# Experiment workbook

Use this reference to produce a complete test plan. Examples are fictional planning material, not PumpGTM customer outcomes or recommended universal benchmarks.

## Copyable planning document

### Decision and rationale

**Working title:** a short description of the proposed change.

**Decision after the test:** the specific choice the owner can make.

**Assumption at risk:** what must be true for the change to be worth pursuing.

**Current evidence:** source, date, population, and limitation for each supporting observation.

**Competing explanation:** another reason the observed behavior might occur.

**Prediction:** for this eligible audience, this intervention will change this observable behavior by a commercially meaningful amount, through this proposed mechanism.

### Design

| Item | Required detail |
|---|---|
| Method | Interview, audit, prototype, controlled comparison, or feasibility check |
| Unit | Account, person, session, task, or another justified unit |
| Eligibility | Conditions an included unit must satisfy |
| Exclusions | Existing exposure, overlapping campaigns, ineligible product needs, or other relevant restrictions |
| Recruitment | Where the sample comes from and likely selection bias |
| Assignment | How units enter groups and how related units remain together |
| Intervention | The precise material or experience being evaluated |
| Comparison | The current approach or other meaningful reference |
| Constant conditions | What is held comparable and how differences are recorded |
| Observation window | When an outcome becomes eligible and when follow-up ends |
| Operating cap | Maximum time, spend, and eligible exposure |

### Measurement contract

| Metric | Event definition | Numerator | Denominator | Time window | Data source | Deduplication |
|---|---|---|---|---|---|---|

Name one primary metric. Secondary measures explain mechanisms; guardrails detect unacceptable effects. A complaint should not be reclassified as positive engagement simply because it is a reply.

### Feasibility and interpretation

Record the baseline source, minimum worthwhile effect, statistical assumptions if applicable, required volume if calculated, available volume, expected duration, and follow-up lag. Mark any unresolved field explicitly.

Write separate rules for adoption, rejection, inconclusive evidence, and compromised execution. An end date is an operating rule; reaching it does not automatically make the sample statistically adequate.

### Execution checklist

1. Freeze eligibility and assignment rules.
2. Review the actual messages, prototype, or interview guide.
3. Confirm the measurement source can capture the defined events.
4. Record the start date, cap, and full observation window.
5. Log deviations without rewriting the original plan.
6. Analyze only after the planned window, unless a defined adverse event requires stopping.

### Results record

| Group | Assigned eligible units | Recorded exposures | Qualified outcomes | Adverse outcomes | Missing observations |
|---|---|---|---|---|---|

**Interpretation:** decision rule applied, supported conclusion, limitations, and next action.

## Worked case: conversation context

**Fictional brief:** a team using PumpGTM wants to know whether a public hiring signal is a more relevant opening than a generic networking invitation. The brief does not establish that PumpGTM automatically collects hiring signals; the team supplies and checks the signals manually.

**Decision:** whether to invest in another, larger test of context-based opening messages.

**Assumption:** an accurate, relevant public event helps eligible buyers recognize why the conversation could be useful.

**Eligibility:** accounts that meet the team's existing fit criteria and have a dated public hiring announcement relevant to the buyer workflow. Exclude existing conversations and accounts already assigned to another message test.

**Assignment:** randomize at account level. Keep contacts from the same account in the same group. Balance known differences in account type where practical.

**Control draft:** "How are you handling new-account research today?"

**Variant draft:** "Your public hiring page mentions a new sales role. How are you handling new-account research while that role is open?"

These are illustrative drafts. Verify that the page really supports the statement, that the role is still open, and that the question is relevant. Keep sender, channel, subsequent offer, and follow-up treatment comparable.

**Primary outcome:** a unique assigned account responds within the agreed window with a substantive answer about its current research process or a request for a relevant next conversation. Define the window with the team before launch.

**Guardrails:** mistaken factual references, requests to stop, complaints, and negative relevance feedback. The owner must set acceptable limits; this example does not supply industry standards.

**Baseline and sample requirement:** unresolved until historical counts, desired effect, and available eligible volume are supplied. If these cannot be established, classify this as a directional learning exercise. Do not claim a statistically validated winner.

**Interpretation:** assess whether replies actually reference the context and whether the approach is operationally feasible. Positive findings justify a properly sized follow-up, not a public performance claim.

## Interview guide for an uncertain buyer problem

- Think about the last time you needed to identify a new account worth approaching. What initiated the task?
- Walk through what you did, including the sources and tools you checked.
- Where did the process slow down or produce an unusable result?
- What happened next, and who else had to get involved?
- What did you try to change? Why did you keep or abandon that alternative?
- What would need to be different for you to change the current workflow?

Record accounts of actual events separately from predictions about future adoption. If a participant says a task took several hours, label that as their report unless independently measured.

## Arithmetic example

Suppose a fictional control group has 8 qualified account replies from 200 assigned eligible accounts, and a variant has 12 from 200. The observed rates are 4% and 6%. The absolute difference is 2 percentage points; the relative difference is 50%.

Those calculations do not establish significance, causation, or a durable 50% improvement. A decision also needs uncertainty analysis appropriate to assignment, the prespecified decision rule, observation completeness, and the practical value of four additional observed outcomes.

## Exceptions

| Situation | Correct treatment |
|---|---|
| No baseline exists | Measure the current process or plan discovery; do not fabricate a baseline |
| Almost no positive events | Show counts and uncertainty; avoid a confident winner claim |
| One group has delivery problems | Diagnose delivery and report contamination before attributing message effects |
| Audience changes halfway through | Record the deviation and reassess comparability |
| Participant requests withdrawal | Respect the request and record its effect on measurement |
| Several things change together | Interpret the package, then isolate mechanisms in a later test |
| Result looks positive before the end | Follow the original analysis rule unless a defined adverse event requires stopping |
| The test misses the required volume | Mark the result inconclusive or directional, not adequately powered |