quizbowl team logo

Quizbowl at the University of Washington

Quizbowl at UW

quizbowl team logo

How to Write Questions

Part II: Difficulty

For the most part, pyramidal difficulty is pretty intuitive. The thing that most people get wrong is not ordering question clues by difficulty, but how much difficulty varies within or across questions. A helpful concept for thinking about this is the "shape" of a tossup. Imagine making a plot where each clue corresponds to a line segment. The x-coordinate of each segment corresponds to where it occurs in the tossup, and the y-coordinate corresponds to the clue's difficulty. Now connect the segments! If you're plotting a high-quality quizbowl question, you should get a nice staircase like this:

On the other hand, the shape of a bad tossup might look something like this:

This visual allows you to be very descriptive. For example, a "flat" tossup is one in which the difficulty of most of its clues are the same. A "smooth" tossup is one in which the difficulty declines gradually. In general, you want your tossups to be somewhere in between: enough variation in clues to be noticeably pyramidal, but not so much that your question ceases to be smooth. (Disclaimer: These are not technical terms!)

The undesirable opposite of a smooth question is a question in which the difficulty drops off dramatically. Consider the following question, which more or less maps to the "bad shape" above:

This leader was promoted to major general after his performance at the Battle of Ridgefield. This leader commanded the winning side at the Battle of Valcour Island. This leader fooled an enemy about the size of his relief force at the Battle of Fort Stanwix. For ten points, name this hero of Saratoga who defected to the British during the American Revolution.

ANSWER: Benedict Arnold

A dramatic drop in difficulty like the one at "Saratoga" is called a difficulty cliff. Cliffs should generally be avoided because they create the problem we were trying to avoid all the way back in Part I: the "buzzer race". When we suddenly reveal a very well-known clue, we have regressed to testing speed rather than knowledge. Small "cliffs" are sometimes unavoidable, but make sure that you have exhausted your options for middle-difficulty clues before including one. And if there aren't many middle-difficulty clues, that might be a sign you shouldn't be writing a question on that topic in the first place!

But question shape isn't the only thing that matters; we also need to control overall difficulty. Before we can talk about overall difficulty, we need to talk about target difficulties. A target difficulty is a metric that describes the difficulty of a question set. It's a predetermined benchmark that helps question writers know what to write and tournament hosts & players to know what to expect. In the past, target difficulties were nonstandard: "college regular" could have meant anything from EFT to ACF Regionals. But since then, quizbowl has mostly settled on the dot scale, which establishes four standard target difficulties. These are:

  1. One Dot (novice difficulty, like ACF Fall)
  2. Two Dots (medium difficulty, like ACF Winter)
  3. Three Dots (qualifier difficulty, like ACF Regionals or SCT)
  4. Four Dots (nationals difficulty, like ACF Nationals or ICT)

We can also think of target difficulty in terms of our visual tool. Imagine a standard set of horizontal lines on your graph. You can think of these as your targets: in a perfect world, each segment on your graph would fall on one of these horizontal lines. Now things won't always work out that way: there's always going to be some deviation from the ideal. But using those target lines to guide your clue selection will help you from veering too far away from a good question. As for the exact y-coordinate of those lines? Well, that's something you'll have to get a feel for by reading lots of question sets at your target difficulty.

We can use the concept of "target difficulty lines" to talk about bonuses, too! You should know approximately what an easy, middle, and hard part feel like at your target difficulty from reading past bonuses. Imagine an easy line, a middle line, and a hard line going through your chart. You'll want to select bonus parts that fall as close to those lines as possible. And since bonuses are easier to adjust for difficulty, you'll want to be even more precise than you were with tossups.

So what are some examples of this going wrong? A pretty common misstep for beginners is to write the "one cool clue" tossup: flat until it cliffs to the "one cool clue" that the author thought of and then cliffs again at the end. It's also common for a tossup to have a "flat middle": the lead-in and giveaway are difficulty-appropriate, but the rest of the clues are all kind of the same difficulty. It's easy to get "tunnel vision" when you're excited about a question: make sure to review your clue selection every time you add a new clue to make sure your shape isn't going off the rails!

It's also common to just overshoot difficulty. This is usually because that bottom target difficulty line wasn't hit properly. The giveaway clues in your tossups and the easy parts in your bonuses should be converted by at least 80% of your target audience. And it's almost always better to err on the side of too easy! The easiest parts are often afterthoughts for new writers, but they are foundational for any question set.

A related topic is clue density, which is about how close important clues are to each other in question. Check out the shape that corresponds to the following tossup:

This man fought a guerilla campaign in the Sierra Maestra mountains against his most famous opponent. This man gave his “History Will Absolve Me” speech while on trial for leading an attack on the Moncada barracks. This ally of Che Guevara overthrew Fulgencio Batista and repulsed the Bay of Pigs invasion. Before dying in 2016, this man transferred much of his power to his brother, Raul. For 10 points, name this longtime President of Cuba.

Fidel Castro

In this case, the clues are so close together that it's hard for the listener to parse a clue before the next one gets read. This has a similar effect of a difficulty cliff: even though there is technically a clue in between, it whizzes by so fast that you experience a dramatic drop in difficulty and a subsequent buzzer race. Even worse, a player might ring in with the wrong answer, but a clue is revealed as they are buzzing that sets them on the right track. So make sure that clues have "room to breathe". It makes them more pleasant to listen to as well!

In the "Notability" section of Part I, I talked a bit about using clues that are newsworthy. I'll go into more detail about what makes a clue worth writing about in Part III: Rewarding the Right People.