The questions I’ll always ask in a product review

February 9th 2025

I figured some of these might be useful to folks earlier in their career, certainly I wish I had them earlier, so here goes.

Why did we build this? (Bad answer: because our competitor has it. Better: because we are losing deals to xyz who has a great version of it)
Who, specifically, is it for? (Bad answer: Sales. Better: the person in charge of reporting on online revenue every week)
What is the most ambitious version of this product? (here I’m testing for how well they’ve thought through the space)
How does this version compare to it? (here I’m looking for smart scoping decisions, 80/20s etc.)
What is the biggest assumption we’ve made that needs to be true for success? How have we tested that? (if there’s a bet we’re making, we should discuss it and see if we can ascertain things before going live)
What’s the estimated impact here if this works out exactly as we planned? (numerically what will this feature do: drive sales, stop churn, change satisfaction, reduce support queries?)
What does this look like in 2 years and what needs to prove true for us to continue resourcing it? (here I’m thinking about what does ‘done’ look like, or will this be a feature we’ll be building for the rest of our lives? e.g. reporting)

What’s our confidence this feature will actually work for ~everyone in real world scenarios? What is needed to improve that?
How has it performed thus far? What’s the worst outcome? What’s the most amazing thing we’ve seen? What’s the most common?
How will we know if it’s not working live in production?
What couldn’t we get working reliably, if anything?
What guardrails have we built here?
Do we have enough instrumentation to investigate issues in production?

What did customers say when you showed this to them?
What was the energy/vibe like - one of relief (e.g., “finally, guys!”) or excitement (e.g., “holy shit!”)?
What questions did customers ask? What wasn’t obvious? What concerns came up immediately?
How long did it take for it to click with them? Was there an “aha” moment?
Were the words we used familiar to them?
How did this feature compare to their top 3 feature requests?
Did they read the launch notes? Did they understand what we claimed to have built before they saw it?

What do you believe will be the #1 feature request in the days following launch? When is that shipping?
What’s the most important thing we’re not shipping on day 1? (not necessarily same as above)
What’s the least important thing that we are shipping?
Does this juxtaposition make sense?

How long does it take for a user to see value here? Can we shorten it?
What’s the breakdown of tool time vs task time? Have we tried to minimize each? (UI improvements, ‘quick start’ templates)
How has the onboarding performed in testing? Are they engaging with it, or just clicking “next-next-next”?
What’s the biggest barrier to adoption?