If you've ever set up a MaxDiff exercise in survey software, you've probably encountered the term Balanced Incomplete Block Design, or BIBD. It's the textbook gold standard: an experimental design where every item appears the same number of times across all choice sets, and every pair of items appears together the same number of times. In theory, BIBD gives you maximum statistical efficiency and zero structural bias. It's clean. It's elegant. It's what your methods professor told you to use.
In practice, almost nobody uses it. The reason is combinatorial math: a true BIBD only exists for specific combinations of items, set size, and number of sets. Change any one parameter and the design either doesn't exist or produces so many sets that respondent fatigue destroys whatever statistical advantage you gained. Even when a BIBD does exist for your parameters, it's a single fixed design — meaning every respondent sees the same set compositions in the same structure. That uniformity can introduce its own biases, especially in online fieldwork where context effects and satisficing are real concerns.
So what does your survey software actually do? If you're using Sawtooth Software, Lighthouse Studio, or any modern MaxDiff tool, the answer is some variation of a "shuffled deck" algorithm. At runtime, the software deals items into sets like cards from a shuffled deck, producing a unique near-balanced design for each respondent. Item frequency balance is guaranteed when the math works out (specifically, when the number of sets times items per set is evenly divisible by the total number of items). Pairwise co-occurrence balance — the harder constraint — is approximate. It's close, and it averages out beautifully across a full sample, but within any single respondent's design, some pairs will co-occur more than others. This is a known and accepted tradeoff in the industry. It's also the tradeoff most researchers never think about, because the software handles it silently.
Questra takes the same approach. Our Decipher/Forsta platform adapter generates MaxDiff designs at runtime using the same shuffled-deck method trusted by the leading tools in the space, embedded directly in the survey XML. No external design files, no separate design software, no quota sheets to manage. In this post, we'll walk through how BIBD works in theory, why runtime generation replaced it in practice, where the real gaps are, and what researchers should actually pay attention to when evaluating MaxDiff design quality.