The trap pack

An open test for any tool that answers questions about survey data. A survey file holds questions that look alike: what retailers did last season and what they say they would do next season; the same grid asked once for each channel. A tool that reads the file as text will sooner or later give the figure of one under the name of the other. Its arithmetic is right. The number is wrong. This pack finds out how often.

How to run it on any tool

1. Load the codebook and the response files into the tool, the way a customer would.
2. Ask each question in questions.csv exactly as written. Do not rephrase, and do not tell the tool about the look-alikes.
3. Write down the percentage it gives, or NONE if it says it cannot say.
4. Mark each against answer_key.csv. A figure within half a point of the answer is right.

Two numbers matter. How many answers were sound: right, or rightly withheld. And how many were a wrong number: a figure given with confidence that is not the answer to the question asked. A tool that says it cannot say costs a minute. A tool that gives the look-alike’s figure costs a decision.