Who should a rep call, and what should the brief say?

A synthetic outbound book for a restaurant software company on a flat $499/month subscription: 20,000 independent restaurants a quarter, capacity to work 5,000. Every restaurant has two hidden numbers, the chance it signs with no rep at all and the lift a rep adds. They are not the same restaurants.

The dashboard picks the wrong model

01 · this quarter's 5,000 calls

Measured LTV:CAC ranks firstPropensity 4.98xonly 49% of its deals needed a rep

LTV:CAC on rep-caused deals ranks firstUplift 2.94x79% rep-caused, within 0.16 of the oracle

Policy	Signed	Rep-caused	LTV:CAC measured	On rep-caused deals
Old rule: intent first, then most reviews	1,174	30%	4.19x	1.29x
Propensity: most likely to sign if worked	1,367	49%	4.98x	2.49x
Uplift: most changed by a rep (holdout-trained)	1,020	79%	3.74x	2.94x
Oracle: true lift (unknowable)	1,036	80%	3.85x	3.10x

Same 5,000 calls: the uplift list adds 133 rep-caused deals ($1.34M of lifetime value) and sends reps to none of the restaurants already on the pricing page. The propensity list spends 1,257 calls on them. The model is a T-learner trained on a 3,000-lead randomized holdout, half worked and half not.

Automated account research invents the fact the pitch depends on

02 · 80 Claude-written briefs

Claude wrote a JSON brief for 40 restaurants whose CRM records were missing some fields. 68 missing fields can't be inferred from anything else in the record.

Prompt	Stated an unknowable field	Wrong	Misread a given field
Open: "write a brief"	12 of 68 (18%)	4	0 of 75
Grounded: "use only the record, else unknown"	0 of 68	0	0 of 75

10 of the 12 were has_own_online_ordering = false, the fact a pitch for a restaurant's own ordering is built on. Three of those restaurants already had it.

The parts I got wrong

I first drew the holdout only from leads the old rule skipped. The uplift model had never seen a pricing-page restaurant go unworked, so it still sent reps to 358 of them, and only 67% of its deals were rep-caused. Drawn from the whole book: 0 and 79%. The holdout has to include the leads you'd work anyway, which is exactly what makes it expensive.
My first scoring of the briefs said 34% of missing fields were invented. Half were valid inferences: a record with 450 monthly app orders does use delivery apps. The real number is 18%.

Run it

stdlib only

python3 run.py                  # under a second: propensity vs uplift
python3 research_eval.py        # scores the 80 cached briefs
python3 research_eval.py --ask  # asks Claude again (Claude Code CLI)