Build an Ideation Engine: A Repeatable Product Discovery System

Most “AI ideation” advice is a prompt that produces a longer list. That is not a discovery system. It is autocomplete with ambitions.

I built an ideation engine in OpenClaw to turn project evidence, customer research, and competitive threats into a small number of testable product candidates. The engine runs on the main machine; a portable copy of its operating skill also exists locally. Its value is not the number of ideas it generates. Its value is the trail from observation to rejection, test, and eventual outcome.

This article publishes that process, including the parts that failed.

The output is a test, not an idea

An idea is cheap. A decision-ready candidate is a structured record with:

• a specific user and painful situation;
• an observation tied to a source artifact;
• the existing alternative or workaround;
• a falsifiable claim;
• the smallest credible validation test;
• a score with an evidence-confidence modifier;
• a final disposition: test, merge, park, or reject.

If the engine cannot produce that record, it has generated prose—not product discovery.

The complete pipeline

observations + project state + rejected ideas
                    │
                    ▼
         three independent lenses
                    │
                    ▼
          normalized candidate cards
                    │
                    ▼
      hard gates → deduplicate → score
                    │
                    ▼
           smallest credible test
                    │
                    ▼
 test result → decision → outcome tracker
                    │
                    └──────── feedback into next run

The feedback arrow is the important part. Without it, an ideation engine optimizes for ideas that sound plausible to a model. With it, the system can learn which sources, lenses, and scoring judgments predict real progress.

Step 1: assemble evidence before asking for ideas

The evidence-driven mode reads five classes of input:

Input	Question it answers	Typical artifact
Portfolio state	What already exists?	Active-project index and stage
Known ideas	What must not be duplicated?	Active, merged, and rejected idea registry
Customer evidence	Where does pain repeat?	Prospect notes, interviews, support issues
Operating evidence	Where are we repeatedly doing manual work?	Project logs, incidents, automation inventory
Constraints	What can we realistically test?	Time, access, skills, data, and budget

A source path alone is not evidence. Capture the observation or metric from it. “Read prospect database” is weak. “Seven of 14 sampled profiles describe the same manual compliance-document workflow” is inspectable.

Bulk prompt mode is useful for exploring a supplied question, but it is a different product. A user prompt can be a source of direction; it is not automatically market evidence.

Step 2: use independent lenses

The engine runs three roles independently so they do not converge on the first attractive framing:

Lens	Search behavior	Required evidence
Gap analyst	Finds missing capabilities between current workflows, projects, and customer needs	A named workflow, artifact, or repeated gap
AI adversary	Asks what a faster competitor would build to make the portfolio weaker or obsolete	A concrete vulnerability and plausible attack path
Pattern generalizer	Clusters recurring needs across multiple customers or projects	A count, sample, and falsifiable generalization

Each lens returns at most three candidates. “No high-confidence candidate” is a valid result. The engine is a filter, not a quota.

Step 3: normalize every candidate

Different models produce different rhetoric. A fixed candidate contract removes the style advantage:

candidate_id:
title:
lens:
target_user:
painful_situation:
source_artifact:
source_observation:
existing_alternative:
proposed_outcome:
falsifiable_claim:
smallest_test:
time_box:
success_threshold:
failure_threshold:
dependencies:
duplicate_candidates:

Assign the ID before evaluation and use it in every table. In the first real run, nine candidates entered the gate, three were promoted, five were explicitly merged or rejected, and one was left without a recorded disposition. That bookkeeping gap made the run impossible to reconcile cleanly. Stable IDs and a required final status fix it.

Step 4: enforce hard gates

Gate	Pass condition	Failure action
Source tie	Names an artifact and the exact observation inside it	Reject as ungrounded
Novelty	Not the same user + problem + outcome as an existing candidate	Merge evidence or reject duplicate
Reachability	A path exists to observe or contact at least five target users	Park until access exists
Testability	Has success and failure thresholds inside a time box	Reject as unfalsifiable
Feasibility	The smallest test fits the declared budget and capabilities	Reduce scope or park

A gate that never rejects is a heading, not a control. One 25-prompt batch reported that all 25 ideas passed all four original gates. The source for each job was often the prompt itself, so “source tie” did not measure market evidence. The process had become good at documenting acceptance.

Step 5: deduplicate by problem, not title

Embeddings can help retrieve candidates, but the merge decision should compare four explicit fields:

1. target user;
2. painful situation;
3. desired outcome;
4. proposed mechanism.

If the first three match, treat the records as one opportunity even when the proposed mechanisms differ. Preserve both mechanisms as solution options and combine their evidence. This prevents the registry from filling with differently named versions of the same problem.

Step 6: score evidence, not enthusiasm

The original engine averaged four 0–10 judgments: market pull, feasibility, defensibility, and personal fit. That is a useful portfolio view, but a confident score based on weak evidence still looks precise. I now apply an evidence-confidence multiplier:

base =
  0.35 × market_pull +
  0.25 × feasibility +
  0.20 × defensibility +
  0.20 × personal_fit

decision_score = base × evidence_confidence

Confidence	Evidence level
0.25	Model inference or one anecdote
0.50	Repeated internal observation, not externally tested
0.75	Multiple target users confirm the problem or reveal the workaround
1.00	Behavioral evidence: commitment, usage, payment, or successful pilot

Use the number to order tests, not to declare winners. A score above 7 with low confidence is a research priority. It is not a product decision.

A worked example: execution attestation

The first engine run combined two active systems: an agent-discovery network and a skill marketplace. The pattern lens asked what must exist after an agent finds another agent and installs its capability. The missing layer was evidence of what executed.

Source observation	Discovery establishes where a service is; artifact verification establishes what was acquired. Neither proves what ran.
Candidate	An execution-attestation layer with signed, reviewable evidence.
Falsifiable claim	Teams evaluating agent workflows will require execution evidence before allowing higher-impact autonomous actions.
Smallest test	Show a concrete attestation record to five platform or security engineers and measure whether it changes an approval decision.
Current status	Strong architecture hypothesis; market willingness still requires direct validation.

The engine did not “validate” the market. It found a coherent, testable seam between existing systems. That is the correct claim.

Rejected ideas are first-class data

The first run’s rejection log was more useful than its launch list:

Candidate	Disposition	Reason
Prospect Signal Engine	Merge	Natural extension of the existing startup graph
Ghost Registry Protocol	Park	Speculative competitor behavior and weak near-term test
Automated AI Auditor	Merge	Overlapped an existing advisory audit product
Market-Reality Engine	Internal improvement	A meta-idea for improving this engine, not a customer product
Determinism Monitor	Merge	Overlapped the agent reliability/SLA scope

Keep these records. Otherwise the same attractive ideas return every month with new names.

Step 7: close the learning loop

The engine’s hit tracker defined a target conversion rate above 30%, but the tracker was still empty after the runs I reviewed. That means the system could score ideas but could not yet measure whether its judgment correlated with validation or delivery.

Track these outcomes by source and lens:

• candidates generated;
• hard-gate pass rate;
• tests started within 14 days;
• tests that met their success threshold;
• ideas promoted, merged, parked, and rejected;
• time and money spent per validated candidate;
• shipped products and observed usage or revenue.

If a lens produces exciting ideas that never reach a test, reduce its weight. If a source repeatedly leads to real adoption, sample it more often. That is how the engine improves.

Copy the operating template

For a weekly run, use this sequence:

1. Collect five to ten concrete observations from customers, operations, and active projects.
2. Run the gap, adversary, and pattern lenses independently.
3. Normalize every output into the same candidate contract.
4. Reject anything without evidence, access, or a falsifiable test.
5. Merge by user + problem + outcome before scoring.
6. Apply the evidence-adjusted score.
7. Choose at most one or two tests for the next cycle.
8. Record outcomes before generating another batch.

You can download the ideation scorecard as CSV. The adjacent evidence pipeline is documented in the open-source Automation Miner case study, and the discovery dataset is unpacked in the 42-opportunity audit.