We built LitmusLab to solve a specific problem: how do you know if the AI features you're shipping are actually working for each type of user, across the full range of inputs they'll throw at it in production?
The honest answer is that most teams don't know. They ship, they watch for complaints, they hope.
We wanted to prove LitmusLab could do better. So we built Screened, a real AI application — not a toy — and ran it through three levels of testing. This is what we found, what we'd do differently, and what it means for teams shipping AI features.
What We Built
Screened is a social movie recommendation app. You log in, it looks at what you and your friends have been watching, and it recommends what to watch next, weighted toward titles your friends loved that you haven't seen yet.
The AI layer is the interesting part. Screened doesn't just return popular movies. It reasons about your taste, your friends' recent activity, and tries to surface recommendations with a genuine social connection: "Jordan watched this last week and gave it five stars. Here's why it fits what you've been watching."
That's a harder problem than it sounds. The AI has access to real user data: watch histories, ratings, timestamps. It needs to use that data accurately. It can't just sound plausible. It needs to be right.
We built Screened at demo.litmus-lab.com. Four users, real watch histories, real friend groups. The recommendation engine runs on GPT-4o.
Then we tested it with LitmusLab.
Three Suites, Three Questions
We set up three test suites to demonstrate LitmusLab's capabilities at increasing levels of depth: basic, intermediate, advanced. They're not meant to be run sequentially as a gating strategy. They show what's possible at each layer of evaluation.
Suite 1: Does it work?
Structural checks only. Status codes, response times, valid JSON. One early run hit 50% due to a response time spike that resolved on its own with no code change. Subsequent runs passed 100%.
This is the floor. Most teams have APM tools that surface this signal already. What's different here is that it lives in the same place as everything else, visible to everyone on the team — not just the engineers watching Datadog.
Suite 2: Is it correct?
This is where scenario-driven testing starts. We defined four users as test scenarios: Alex, Jordan, Morgan, and Casey, each with their own watch history and friend group. Then we ran five test cases across all four scenarios: 20 executions total.
The checks here are a mix of deterministic and AI-evaluated:
- Did it avoid recommending titles the user already watched?
- Are friend attributions accurate? Did Jordan actually watch the title the AI credited to Jordan?
- Is the response consistent with the scenario data?
Result: 100% pass rate across all 20 executions.
The AI was using the right data. Friend attributions were accurate. No already-watched titles slipped through. At the scenario level, Screened was behaving correctly.
But passing Suite 2 only tells you the AI is using the right data accurately. It says nothing about whether the recommendations it produces are actually worth making.
Suite 3: Is it actually good?
This is where it gets interesting.
Suite 3 adds contextual evaluation. We give the evaluator not just the scenario data, but Reference Responses: human-authored definitions of what a correct, quality answer looks like for each test case and scenario combination.
One of the five test cases was: What's a movie I haven't seen that a friend really loved?
"Really loved" is doing a lot of work in that sentence. We defined it precisely in the Reference Response: a 5-star rating from a friend. Not 4 stars. Not a well-regarded film the AI associates with a friend's taste. An actual five-star rating in the scenario data.
Results:
- Casey: passed (83%)
- Morgan: failed (75%)
- Jordan: failed (75%)
- Alex: failed (70%)
For Morgan, the evaluator flagged: "Nightcrawler is not a valid recommendation as it is not in the list of 5-star titles. Gone Girl is not in the list of 5-star titles either. The reasoning is well-justified with references to Morgan's watch history and friend's ratings, but the selection of titles is incorrect."
For Alex: "The response recommends Poor Things and Anatomy of a Fall, which are not in the list of 5-star titles friends rated. Poor Things is linked to Jordan without a valid 5-star rating in the scenario data."
The AI wasn't making things up. It was inferring what a friend would probably love based on their taste — which sounds helpful until you realize the user asked for something a friend actually watched and gave five stars. Nightcrawler and Gone Girl don't appear in anyone's watch history at all.
The AI answered a different question than the one that was asked. And without a Reference Response defining exactly what "really loved" means, there's no way to catch that in testing.
Suite 1 wouldn't have caught this. Suite 2 wouldn't have caught this. Suite 3 caught it because we defined what "really loved" means, gave the evaluator the actual watch history to check against, and ran it across four different user scenarios.
What This Means
Suite 2 passing told us the AI is factually accurate. Suite 3 told us accuracy isn't the same as quality.
This is a prompt scoping failure, not a hallucination. The AI didn't invent something false. It answered a reasonable interpretation of the question instead of the intended one. That's harder to catch, because the response looks coherent and helpful. A user asking what their friend loved gets a plausible, well-reasoned answer. They'd have no way to know it wasn't grounded in their friend's actual watch history.
The prompt doesn't enforce what "loved" means, so the AI fills the gap with its best inference. The app works. The AI is coherent. The responses sound right. A manual review might not flag it because the reasoning is good enough that it reads as correct until you check the underlying data.
The fix is straightforward: tighten the prompt to tell the AI that "really loved" means 5-star ratings only, and only recommend titles that appear in the scenario data at that threshold. But you can only write that fix if you know the problem exists.
That's what the third suite is for.
Try It
Screened is live at screened.litmus-lab.com. No account required. Ask it what to watch tonight.
If you want to see how we tested it, let us set up a time to demonstrate.