What Happened and Why? Trace-Guided Micro-Episodes with Elicited User Explanations for Product Iteration
A workshop position paper about how interaction traces and short, in-context user explanations can work together to support product iteration in creative AI tools.
Abstract
Teams shipping AI workflows in design tools can measure usage yet often struggle to explain why features fail. In creative work, standard metrics are ambiguous: a long session could imply productive exploration or frustrating struggle with stochastic outputs. We argue for trace-guided micro-episodes, a unit of analysis binding interaction logsāwhat users didāto their intent. Rather than relying on disruptive surveys, we propose a āutility-for-rationaleā paradigm: systems offer optional, context-aware controls at likely friction points, capturing user explanations as a byproduct of real-time error recovery. This approach converts ambiguous telemetry into causal evidence without breaking flow. We posit this methodology serves a dual purpose: equipping teams with diagnostic clarity to iterate on vague failure modes (e.g., controllability vs. quality) while generating the grounded alignment data required to train future agents.
Overview
Creative systems produce detailed logs, but those logs are often hard to interpret on their own. A long interaction sequence might reflect productive exploration, repeated verification, or a user struggling to recover from an unsatisfying result. This paper starts from a simple observation: traces are good at showing what happened, but they are often poor at revealing why.
To make those traces more informative, the paper proposes trace-guided micro-episodes: short windows of interaction paired with the local interface context and a lightweight explanation from the user at a moment of friction. Rather than treating explanation as a separate survey task, the proposal is to gather it through useful recovery-oriented interactions inside the tool itself.
Why it matters
For product teams, this framing offers a more grounded way to interpret analytics from creative workflows. Instead of collapsing many behaviors into a single metric, micro-episodes can help distinguish between broad but practically different issues such as controllability problems, output-quality mismatches, and verification burden. The goal is not to replace telemetry, but to make it easier to reason about.
The paper also matters methodologically. It suggests that user explanations do not need to be collected only through disruptive post-hoc surveys. When explanation is tied to an action that helps the user move forward, it can become part of the workflow rather than an interruption. That opens up a more modest and more realistic path toward gathering situated evidence for both product iteration and future agent training.
Cite this paper
- Venue Herding CATs: Making Sense of Creative Activity Traces (CHI 2026 Workshop)
- Publication Date February 25, 2026
- PDF what-happened-and-why.pdf
Use the BibTeX below if you want to cite the paper or add it to a reference manager.
BibTeX
@inproceedings{tao2026whw,
title = {What Happened and Why? Trace-Guided Micro-Episodes with Elicited User Explanations for Product Iteration},
author = {Tao, Sirui and McCarthy, William P. and Dow, Steven P.},
booktitle = {Herding CATs: Making Sense of Creative Activity Traces (CHI 2026 Workshop)},
year = {2026},
month = feb,
url = {https://dylantao.github.io/projects/what-happened-and-why/},
pdf = {https://dylantao.github.io/projects/what-happened-and-why/what-happened-and-why.pdf}
}