How to do AI analysis you can actually trust

Original: Lenny Rachitsky · 17/02/2026

Summary

Ever run an AI analysis on customer data, only to discover the numbers were fabricated and the insights completely generic?

Key Insights

“Ever run an AI analysis on customer data, only to discover the numbers were fabricated and the insights completely generic?” — Introduction to the problem of unreliable AI analysis.

“After 2,000+ hours of testing customer discovery workflows with AI, she’s identified the failure modes that break AI analysis and the reliable fixes for each one.” — Highlighting Caitlin Sullivan’s extensive experience and the value of her findings.

“The final verification pass that stress-tests everything before it hits a deck.” — Describing a crucial step in ensuring the reliability of AI-generated insights.

Topics

Full Article

[![](https://substackcdn.com/image/fetch/$s_!OYz9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d4c0b48-1813-4432-894d-5011ef111807_3016x3016.png)](https://substackcdn.com/image/fetch/$s_!OYz9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d4c0b48-1813-4432-894d-5011ef111807_3016x3016.png)

If you’re a premium subscriber Add the private feed to your podcast app at add.lennysreads.com Ever run an AI analysis on customer data, only to discover the numbers were fabricated and the insights completely generic? In this episode, Caitlin Sullivan, a user-research veteran who’s trained hundreds of product and research professionals, shares her four prompting techniques for getting trustworthy, actionable insights out of any LLM. After 2,000+ hours of testing customer discovery workflows with AI, she’s identified the failure modes that break AI analysis and the reliable fixes for each one. Listen now: YouTube | Apple | Spotify In this episode, you’ll learn:

* How to catch the two types of AI quote hallucinations
* Why AI defaults to useless generic themes and insights
* Which LLM is best for analysis work (and which one fabricates the most)
* How to turn vague signal into actual decision clarity
* The final verification pass that stress-tests everything before it hits a deck

Referenced

* [Caitlin Sullivan](https://www.linkedin.com/in/caitlindsullivan/)

[Read more](https://www.lennysnewsletter.com/p/how-to-do-ai-analysis-you-can-actually-db6)

Building AI product sense, part 2

Lenny Rachitsky · explanation · 77% similar