An idea can sound compelling and still rest on an uncertain assumption.
People will upload their documents. Teams will trust the recommendation. Customers will switch from their spreadsheet. Someone will pay for the time saved.
Before building the full application, I would work out which assumption matters most. Then I would design a test that could produce an inconvenient answer.
If every possible result can be read as encouragement, the experiment is not helping you decide.
Write down the assumption
Take a hypothetical product that turns supplier quotes into a comparison.
The broad idea is easy to describe. The important uncertainty might be more specific: "Purchasing staff will find a structured comparison useful enough to use it during a real buying decision."
That is something we can investigate without first building a complete procurement platform.
We might need an upload flow, a comparison view and a way to correct extracted information. We probably do not yet need a marketplace, advanced analytics or a complex subscription system.
Make the critical part real
A useful experiment should expose the difficult part rather than hide it.
If the value depends on handling messy documents, I would want representative messy documents, not only carefully chosen examples.
If the value depends on saving effort, I would look at the review and correction work too. An automatically generated comparison is less compelling if someone has to spend longer checking it than they would have spent making it by hand.
For AI features, representative evaluation matters. NIST's Generative AI Profile recommends testing and evaluation across development and deployment rather than relying on isolated demonstrations.
Decide what evidence you will record
I would define the observations before running the test:
- Can participants complete the task?
- Where do they need help?
- What do they correct?
- Which information do they still look for elsewhere?
- Would they use the result in their actual workflow?
- What would stop them from coming back?
I would also separate positive reactions from demonstrated behavior.
"That looks useful" and "I used this to complete a real task" are different kinds of evidence. Both can be informative, but I would not treat them as interchangeable. It is the same rule I follow when I test previews with real users: watch what people do, not just what they say.
Include a comparison
Where practical, I would compare the experiment with the current method.
Not just the generation time. The whole task: preparing inputs, checking the output, fixing mistakes and moving the result into the next step.
The product needs to improve something that matters, not merely produce an impressive screen halfway through.
Let the result change the plan
Before testing, I would list the possible decisions: continue with the concept, narrow the audience, change the workflow, remove the AI component, run another test or stop.
Stopping is a legitimate outcome. So is discovering that the most valuable product is smaller or less technically elaborate than the original idea. If the answer is to keep going, choose the next deliverable for the evidence it can produce.
The purpose of a prototype is not to defend the plan you started with. It is to make the next decision with better evidence.
If you have an idea you want to test before building the whole thing, email me, or take the v1 check to see where it stands.