From Counterfactual Edits to Human Narratives: LLM-Driven Interpretations for Recommender System Explanations

Department of Computer Science, University of Innsbruck, Austria Faculty of Information Technology, Czech Technical University in Prague, Czechia
*Equal contribution
35th ACM International Conference on Information and Knowledge Management (CIKM '26)

Abstract

Recommender systems increasingly influence what users discover and consume online, yet the decision logic behind their outputs often remains opaque. Counterfactual explanation (CFX) methods provide an effective way to show which changes to a user's interaction history would alter a recommendation. In recommendation settings, however, CFXs often require many edits, producing long, low-level lists that are hard for users to interpret. Moreover, most evaluations emphasize edit counts while largely overlooking whether the resulting CFXs are coherent and actionable for users. We argue that this gap between generating valid CFXs and communicating model behavior stems from the lack of user-centered abstraction and limited qualitative evaluation. To bridge this gap, we investigate large language models as mediators that summarize raw CFX interaction edit lists into concise natural language explanations describing higher-level preference patterns. We optimize these mediators for correctness, informativeness, and linguistic quality through preference optimization, and evaluate the resulting explanations with human-calibrated metrics derived from separate human-labeled datasets. On the MovieLens-1M dataset, across multiple counterfactual generators and language models, the resulting summaries achieve high overall linguistic quality, while the strongest semantic results are obtained with high-success CFX methods such as ACCENT and SPINRec. Direct Preference Optimization improves the semantic metrics in almost all evaluated settings.

Introduction

Counterfactual explanations (CFXs) offer “what-if” reasoning: they identify a minimal set of past interactions whose removal would change the recommendation. In practice, however, CFXs for recommender systems often contain dozens of interactions (figure below). Such CFXs may be algorithmically minimal, yet they are long, low-level lists of removed interactions that are hard for users to interpret. Natural language explanations (NLEs) are readable, but existing methods such as FIRE rely on weaker attribution signals rather than causal interventions.

Interactions removed by each CFX generator

(a) Interactions removed by each CFX generator

User history size (25th, 50th, 75th, 99th percentile)

(b) User history size (25th, 50th, 75th, 99th percentile)

On three datasets, CFXs that push the top item out of the top-5 of a variational autoencoder recommender often remove dozens of interactions (a), a substantial share of a typical user's history (b).

Our approach: LLM as a mediator

Placed between the CFX generator and the user, an LLM mediator summarizes the contrast between the CFX interactions and the user's remaining history into a concise NLE that preserves the underlying counterfactual grounding.

Contributions

  • LLM mediator framework turning CFXs into concise, faithful NLEs.
  • Evaluation methodology that decomposes correctness and informativeness metrics into simple interaction-level matching judgments, with human-calibrated final evaluation.
  • End-to-end comparison of 7 CFX generators × 3 LLMs, with and without LLM fine-tuning via Direct Preference Optimization (DPO).

Methodology

Overview of our methodology

A user's interaction history \(H_u\) yields a top-\(K\) recommendation list \(\mathrm{Top}_K(H_u)\) with top item \(i_u^\star\). A CFX \(C_u \subseteq H_u\) satisfies \(i_u^\star \notin \mathrm{Top}_K(H_u \setminus C_u)\); \(\bar C_u = H_u \setminus C_u\) is the rest of the history. The LLM sees the metadata \(\phi\) (e.g. release year & genres) of \(k_{\mathrm{gen}}\) sampled items from each set: \[ T = \mathrm{LLM}_{\mathrm{gen}}\left(\phi(C_{u,k_{\mathrm{gen}}}), \phi(\bar C_{u,k_{\mathrm{gen}}})\right) \]

Evaluation metrics

Correctness
Does the NLE match the user's CFX interactions?
\[ \mathrm{Cor}(T, C_u) = \frac{1}{|C_{u,k_{\mathrm{eval}}}|} \sum_{i \in C_{u,k_{\mathrm{eval}}}} \mathrm{score}_{\mathrm{match}}\bigl(T, \phi(i)\bigr) \in [0,1] \]
Informativeness
Does it distinguish them from the rest of the user's history? Generic NLEs score ≈ 0.
\[ \mathrm{Info}(T, C_u, \bar C_u) = \mathrm{Cor}(T, C_u) - \mathrm{Cor}(T, \bar C_u) \in [-1,1] \]
Linguistic Quality
Is the NLE well written?
\[ \mathrm{LQ}(T) = \mathrm{score}_{\mathrm{ling}}(T) \in [0,1] \]

Two evaluators: prompted LLM judges score NLEs during DPO; human-calibrated ModernBERT-large classifiers, trained on human labels (accuracy 0.91 for LQ and 0.92 for interaction-level matching), give the final scores (examples in the table below).

Optimization: we fine-tune the LLM mediator with DPO on NLE pairs for held-out users ranked by the LLM judges, and select the final model with the human-calibrated evaluators.

Example NLEs (obtained with ACCENT + DPO-tuned Ministral 8B) scored by the LLM judge used during DPO (LLM judge) and the human-calibrated classifier used for the final evaluation (Human-cal.).

Natural Language Explanation (NLE)
(Because you watched …)
Correctness ↑ Informativeness ↑ Ling. Quality ↑
LLM
judge
Human-
cal.
LLM
judge
Human-
cal.
LLM
judge
Human-
cal.
comedy and drama films from 1990s 0.350.53 0.020.08 0.831.00
adventure romances from 1995 0.210.12 0.110.12 0.941.00
drama films from 1960s to 1990s 0.830.50 0.440.40 0.890.00
mystery and action films from 1970s to 1990s 0.600.40 0.280.25 1.001.00
comedy films from 1980s and 1990s 0.460.25 -0.15-0.20 0.891.00

Experiments and Results

Setup

  • Data & recommender: MovieLens-1M, variational autoencoder recommender; CFXs push the top item out of the top-5.
  • Item metadata: Release year & genres. No movie titles, to avoid reliance on LLM knowledge; sample sizes \(k_{\mathrm{gen}}=k_{\mathrm{eval}}=20\).
  • CFX generators: Jaccard, Cosine, LIME-RS, SHAP, ACCENT, LXR, SPINRec.
  • LLMs: Ministral 8B, Gemma 3 12B, Qwen3 8B, each in vanilla and DPO-tuned form.

Results

  • CFX generators: ACCENT and SPINRec have the highest success rates (share of cases with \(|C_u| \le 50\); 0.99 and 1.00) and the most compact CFXs (10.36 and 12.25 interactions on average). Simple CFXs (\(|C_u| < 3\); these need no summary) are rare for all generators (at most 7%), so most CFXs benefit from summarization.
  • Correctness & informativeness: Generally highest for the stronger CFX generators, especially ACCENT and SPINRec. DPO improves correctness in all but one generator–LLM pair and informativeness in most.
  • Linguistic quality: High overall, with many settings above 0.80 and the strongest above 0.85; some spread is expected given the multiple optimization objectives.

Key findings

  • Better CFXs, better NLEs: High-success CFX generators yield the most faithful NLEs.
  • Metric design matters: Optimizing faithfulness must still push towards readability.

BibTeX

@inproceedings{10.1145/3799682.3839968,
  author = {Cahlik, Vojtech and Mohammadi, Amir Reza and Alves, Rodrigo and Kordik, Pavel and Zangerle, Eva},
  title = {From Counterfactual Edits to Human Narratives: LLM-Driven Interpretations for Recommender System Explanations},
  year = {2026},
  isbn = {9798400725395},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3799682.3839968},
  doi = {10.1145/3799682.3839968},
  booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management},
  pages = {6954–6959},
  numpages = {6},
  keywords = {recommender systems, counterfactual explanations, natural language explanations, large language models},
  location = {Rome, Italy},
  series = {CIKM '26}
}