Our customers' feedback lived in eight places, and the loudest complaint set the roadmap. So we built a nightly pipeline that reads all of it and ranks it. Three times, we found our own bias inside the scores.
We build TextYess, an AI-first CRM for eCommerce brands. Our AI agents handle pre-sales and support for those brands, our merchants, over WhatsApp, voice, onsite chat, and email.
Our merchants' feedback lived in eight places: Zendesk tickets, chats with our own AI bot, recorded sales calls, CRM notes, product usage in our data lake, Linear, Slack, Notion. Nobody on a small team can read all of that every day. So we planned from what we remembered, and what we remembered was whoever had complained loudest that week. A problem mentioned quietly, once a week, in three different places, was invisible.
Psychologists call this availability bias: memory ranks by ease of recall, and nothing is easier to recall than the last angry message. While it is happening it does not feel like bias. It feels like being close to the customer.
So we built a pipeline that reads everything every night and ranks what it finds. This is what it does, and the three places our own bias survived inside it.
a ranked list instead of a memory
Our first attempt made things worse. We built a report generator that read everything and summarized it, and for a while it told us the same twelve things every morning, one of which we had fixed two weeks earlier. A report that repeats stale findings trains people to stop opening it, and once nobody opens it you are back to planning from memory.
The version that works is a nightly pipeline of cloud jobs. It reads all eight sources, groups the day's conversations into problems and feedback themes, keeps the quotes behind each item, scores everything, and puts a ranked list in front of us in the morning. We triage what we agree with into Linear, our issue tracker, in one click. The next night the pipeline checks items against reality: each item's Linear issue against the pull requests that actually merged, plus a rotating re-check of open items against the merged changelog, asking "already fixed, or already being fixed?" The stale twelve are exactly the kind of thing that check catches.
No model runs when a page loads. The app only reads a database precomputed overnight, so the data is up to a day old, every page says how fresh it is, and there is a log of every job every night. This is boring plumbing on purpose: the point is to replace recall with a record we can audit.
We decide what to build from the top of that list, so getting the ranking right is most of the job. Any bias that survives into the scoring flows straight back into the roadmap the tool was built to protect. Three of ours survived.
the scores inherited our biases
Each item carries three scores. Priority answers "does this matter". Momentum answers "is it worth this week": priority lifted by how live the item is, mentions in the last seven days plus how recent the last one was. Friction is where it hurts right now, merchant breadth and mention volume amplified by how negative the sentiment is.
Priority is a sum of named contributions, visible next to the total: revenue exposed, its share of all active revenue, fit with our ideal profile, merchant breadth, mention volume, how much came unprompted, whether we caused the problem, sentiment, recency. Revenue and fit dominate; volume and sentiment only modulate, because a hundred mentions from merchants we do not want to serve is not a hundred reasons to build. Feedback nobody asked for counts extra, and so do problems we caused ourselves, so the queue leans toward our own mess.
Priority's first version mapped the score to five levels with fixed cutoffs: eight or more meant urgent. The score is unbounded, so as we grew every number drifted up until most of the queue had crossed eight. Nothing was miscalculated; urgent had just stopped meaning anything. Levels are now cut against the current population, so urgent means the top of what is actually in front of us.
Momentum's first version decayed freshness to zero at thirty days, on the reasoning that month-old feedback is stale. Then we measured. Across 565 items, 258 recurred, giving us 705 gaps between one mention and the next. Half of re-mentions arrive within three days, so a fast decay points the right way. But 17% arrive after day thirty, and those are the slow structural problems. The cliff was deleting exactly the feedback we had built the tool to catch. We had rebuilt our own recency bias as a decay curve.
Recurring items are rarely one merchant repeating themselves: of 182 that kept collecting mentions, 69.8% eventually reached a merchant who was not there at the start. Freshness now halves every ten days and never quite reaches zero, so a dormant item sinks but never disappears, and a trivial one cannot climb on recency alone, because priority is still the base.
Friction's first version put a floor under its sentiment multiplier, so a feature merchants loudly loved still scored about 85% of full friction. Praise was ranking as pain. Both failures scored feedback the way it feels rather than the way it behaves in the record.
two intuitions failed the backtest
By then we had stopped trusting our own reasoning about scores and started backtesting it. The test: given only what the system knew at the time, how well does a signal predict that an item gets mentioned again within 14 days? The base rate was 45.4%.
Breadth was the obvious candidate for a momentum driver. Items already mentioned by three or more distinct merchants re-surfaced within the window 67.5% of the time; single-merchant items, 37.2%. A gap that size looks like a law of nature. Then we held volume constant by comparing only items with four or more prior mentions, and the separation nearly vanished: 64.6% for multi-merchant against 63.2% for single. Most of breadth's apparent signal was volume in disguise, and volume already reaches momentum through priority, so adding it would have counted the same signal twice. It stayed a priority input.
The second intuition failed outright. We expected unfixed problems to get angrier over time, and we wanted sentiment escalation as an early warning. Of 138 items with enough sentiment history, 74 stayed flat, 40 got worse, 24 improved. Merchants do not reliably escalate; a problem mostly comes back at the same temperature, so we shipped nothing. A backtest feels objective whether or not it supports you, and we would have defended a wrong number much longer than a wrong hunch.
no invented quotes
None of this works if a claim in the list cannot be traced. Every quote attached to an item must be a verbatim substring of the raw fetched source, or it is dropped. Agents propose JSON, loaders validate and commit it, and that is the only path into the database. A run can downgrade its own status grade, never upgrade it.
We also score each night's run on nine mechanical dimensions, and keep that score separate from the question of truth: a run can take full marks and still be wrong about what a merchant meant. For those cases, anyone on the team can flag the exact claim. The house rule underneath it all: the tool is allowed to be wrong, it is not allowed to make things up, because one fabricated quote means you can never trust any of them again. That asymmetry is a bias too, and we are keeping it.
Availability bias has not gone anywhere. The inbox still opens with the last angry message on top, and it is still the easiest thing to act on. What changed is that there is a second list in the room, one that remembers, and can show the quote behind every claim. When the loudest complaint tops that list too, it wins on the merits. Reading it before the inbox is a choice we make again every morning.
step one of a product agent
This pipeline is the first step toward an autonomous product agent: a system that does the heavy lifting of listening so our attention goes to judgement. We take this internal work as seriously as anything we ship to merchants, because its value compounds. Every hour it saves comes back every day after, and the quality of what we build is capped by the quality of what we hear.
That is also who we hire for. We look for product people who want to automate the organising part of their own job rather than protect it: less backlog administration, more time with merchants, more owning a problem from the first quote to the shipped fix. If the system above reads like a machine you would want to build, we should talk.