←Back to work

MASTER'S THESIS · EXPLAINABLE RECOMMENDER SYSTEMS

The Impact of Explanation Characteristics on User Understanding in Recommender Systems

An empirical study examining how different ways of presenting explanations affect what users understand about recommendations — and how well they can reason about the system behind them.

CORE IDEA

Explain.
Understand.
Reason.

Role

Independent Research

Context

Master's Thesis · DDIS

Method

Controlled User Study

Sample

Pre-study N=26 · Main Study N=33

01 / CONTEXT

Explanations do not automatically create understanding.

Recommender systems are increasingly used to help people navigate large information spaces. Yet users often receive recommendations without fully understanding why they were generated or how their own behaviour influences the result.

Previous research has investigated many different explanation styles, but its findings are difficult to compare. One reason is that user understanding is often treated as a single outcome, even though understanding can involve very different cognitive abilities.

RESEARCH QUESTION

How do explanation presentation characteristics in recommender systems influence users' comprehension and enabledness?

02 / MY ROLE

An independent research project within DDIS.

I conducted the thesis independently as part of the Dynamic and Distributed Information Systems group at the University of Zurich.

The work covered the full research process: reviewing related literature, developing the conceptual framework, designing the experiment, creating and refining the explanation interfaces, running the user studies, analysing the resulting data and deriving implications for explanation design.

Supervisor & Mentor

Kathrin Wardatzky

Research Group

Dynamic and Distributed Information Systems

University

University of Zurich

03 / UNDERSTANDING

Understanding is more than knowing what an explanation says.

01

Comprehension

Knowing what is true and why.

Can users correctly identify the reasons behind a recommendation and understand their relative importance?

02

Enabledness

Knowing how to predict and act.

Can users apply what they learned to predict system behaviour, reason about changes and choose appropriate actions?

04 / STUDY DESIGN

From explanation design to controlled experimentation.

01

Research

I reviewed research on explainable recommender systems, user understanding and methods for objectively evaluating how people understand algorithmic systems.

02

Pre-study

A pre-study with 26 participants was used to compare and refine alternative textual, icon-based and visual explanation designs.

03

Main study

33 participants completed a controlled within-subjects study in which the explanation presentation changed while the underlying information remained constant.

04

Analysis

Task performance was analysed separately for comprehension and enabledness using repeated-measures statistical analyses and additional exploratory tests.

EXPERIMENTAL CONDITIONS

Same information.
Different presentation.

The study deliberately kept the explanatory content constant. Only the way that information was presented changed, allowing the experiment to isolate the effect of presentation.

Text explanation modality showing recommendation reasons as a ranked bullet list.
M1

Text

Recommendation reasons presented as concise textual statements ordered by importance.

Text and icons explanation modality showing recommendation reasons with icons and short descriptions.
M2

Text + Icons

The same textual content complemented by simple icons and a lightweight interactive cue.

Visual explanation modality showing recommendation reasons as circular progress charts.
M3

Visual

The relative strength of recommendation factors communicated through compact visual representations.

BASELINE

No explanation

Participants saw only the recommendation list, without an accompanying explanation.

M1

Text

Recommendation reasons were presented as concise textual statements ordered by importance.

M2

Text + Icons

The same textual content was complemented by simple icons and included an interactive what-if element.

M3

Visual

The relative strength of recommendation factors was communicated through compact visual representations.

EXPLANATION FACTORS

01

Topic fit

02

Similar readers

03

Same venue

05 / MEASUREMENT

Measuring what users can actually do.

Rather than relying only on participants' perceived understanding, the study used objective tasks that required them to demonstrate their understanding.

COMPREHENSION TASKS

Identify and interpret

Participants identified which factors influenced a recommendation and determined their relative importance.

ENABLEDNESS TASKS

Predict and reason

Participants predicted how recommendations would change when factors or system conditions were modified.

MENTAL MODEL

Explain the system

An open-ended task asked participants to describe how they believed the recommendation mechanism worked.

KEY FINDINGS

Presentation mattered more for reasoning than for recognition.

01

COMPREHENSION

No significant effect

Comprehension remained relatively stable across all explanation conditions.

p = .817
02

ENABLEDNESS

Significant effect

Participants' ability to reason about the recommender system differed depending on the explanation condition.

p = .009
03

INTERACTION

Suggestive, but not significant

The pattern differed between comprehension and enabledness, but the interaction did not reach statistical significance.

p = .066
06 / MENTAL MODELS

Users often understood the pieces without understanding the whole.

Participants were generally able to identify individual recommendation factors and predict straightforward outcomes. More difficult tasks emerged when several factors had to be combined or when users had to simulate the system's behaviour.

The written explanations reinforced this pattern. Many participants mentioned relevant factors but only partially explained how those factors worked together to generate a recommendation.

Recognising relevant information is not the same as building a coherent mental model of the system.

ADDITIONAL INSIGHT

Domain knowledge mattered.

Participants who were more familiar with scientific literature performed better on both comprehension and enabledness tasks. General familiarity with recommender systems, in contrast, was not associated with significantly different performance.

07 / IMPLICATIONS

Explanation design should follow the user's goal.

01

Simple can be enough

If users mainly need to identify the reasons behind a recommendation, relatively simple explanations may already provide sufficient support.

02

Reasoning needs structure

Tasks that require prediction or deeper reasoning benefit from explanation designs that make relationships between multiple factors easier to understand.

03

More is not automatically better

Additional visual richness or interactivity does not by itself guarantee better understanding. Complexity can also increase cognitive demand.

08 / LIMITATIONS

A controlled experiment, not a complete real-world system.

The participant sample was recruited through convenience sampling and may not represent the broader population of users of academic recommender systems.

The recommendation interface deliberately simplified the underlying recommendation logic so that the explanation conditions could be compared in a controlled setting.

The structure and number of enabledness tasks were not fully identical across all conditions, which limits some direct comparisons.

09 / OUTCOME

Understanding should not be measured as a single outcome.

The study shows that the effectiveness of an explanation depends on what kind of understanding it is expected to support. The ability to recognise recommendation reasons and the ability to reason about system behaviour represent different challenges.

This distinction provides a more precise way of evaluating explainable recommender systems and highlights the importance of aligning explanation design with the cognitive task users are expected to perform.

PERSONAL TAKEAWAY

The thesis strengthened my experience in designing and running user research, translating abstract concepts into measurable tasks, analysing quantitative and qualitative data, and connecting empirical findings back to the design of human-centered digital systems.

BACK TO SELECTED WORK

View projects→