What It Really Takes to Predict 911 Dispatcher Success

Every 911 center knows the cost of a hire who doesn't work out: weeks of academy time, months of on-the-job training, and a seat that's empty again. On a recent episode of Testing, Testing 1-2-3, we asked Dr. Dan Biddle how agencies can make those decisions with far less guesswork.

Dan is CEO of Biddle Consulting Group and TestGenius and holds a Ph.D. in industrial/organizational psychology. He has spent more than 20 years as a testifying expert in state and federal court cases on selection, testing, validation, and psychometrics. For about 10 years he was also a consultant to the U.S. Department of Labor, reviewing employers' tests that were on the eve of litigation and helping the government judge which ones would hold up as valid under the Uniform Guidelines on Employee Selection Procedures.

In other words, he has seen a lot of hiring tests, good and bad. Here's what he shared about building one that truly predicts dispatcher success.

Why One Test Isn't Enough

"If all you have is a hammer, everything looks like a nail," Dan said. Many employers pick a single tool they've heard of, whether a cognitive ability test or an interview, and hope it captures everything the job requires.

Cognitive ability testing is the most widely used type of test in America, and for good reason: it's often the strongest single predictor of job performance. But it leaves out a lot. As Dan put it, a test that only screens for smart, capable people says nothing about whether they can read a room, communicate clearly, or understand context. That's the gap between IQ and EQ.

The fix is to measure different aspects of a candidate with tests that don't overlap much. Dan calls this not biting on the same side of the apple. Two tests that measure nearly the same thing add little. Tests that each predict performance in their own way add up.

Three Dimensions in One Test

CritiCall 3D applies that principle to 911 dispatcher hiring. It combines three assessments into one seamless testing session:

• Operational: the original CritiCall test, built over 26 years ago with input from dozens of 911 agencies and adopted by more than a thousand. It measures the core skills a call taker needs, including multitasking.

• Situational: a judgment assessment that measures how candidates handle realistic, interactive situations. It overlaps very little with the operational test.

• Behavioral: a personality assessment that captures traits the other two miss.

Each test had already been through years of study on its own. Dan's team revisited all of the earlier validation studies, then combined the strongest parts of each into what he described as the shortest, most compressed, yet most effective battery they could build for the job. Applicants also no longer have to finish one test, close the session, and start another.

The Number That Matters: .634

The combined battery produced an operational validity coefficient of .634, which Dan called unheard of in the field. Square that number and you get about 40%: the share of what drives job performance in the 911 call-taker role that the test explains.

For context, here is how other approaches compare, using the figures Dan cited:

People in his field, Dan said, celebrate any test that clears about .25. Four courts have ruled that a test with a correlation of just .3 is strong enough to rank applicants strictly by score, even though it leaves 91% of job performance unexplained.

The unstructured interview fares worst. One interviewer asks five questions, another asks twelve, and next week the questions change again. Dan's verdict: they're "a crapshoot." Structured interviews, built from a job analysis and scored with a consistent rubric, do much better, but still nothing like the three-part battery.

What this means for a dispatch center

We asked Dan to put it in practical terms for a dispatch captain without a statistics background. His answer: the first goal of any business decision is to remove uncertainty.

Agencies already test patrol car engines and cooling systems before an officer runs code three at 95 miles an hour. Dan argues that hiring the people who answer 911 calls deserves the same rigor. With a highly predictive test, a center can move its top-scoring applicants forward with confidence and avoid losing $40,000 to $60,000 training someone who was never suited for the job.

That matters because turnover is the industry's biggest pain point. In reviewing hundreds of pages of industry reports while building CritiCall 3D, Dan saw the same notes again and again: did not pass training, did not pass probation. Some statewide reports showed losses as high as 70% from skills-related failures. Better screening means shorter training time, less qualification-related turnover, and stronger teams for the coworkers who rely on each new hire.

Setting Cutoff Scores the Right Way

A valid test still needs a sensible pass point. Set it too low and unqualified applicants slip through. Set it too high and capable people never get a chance.

Dan has little patience for the common 70% cutoff. A 70% might be a top score on one test and a bottom score on another, and it says nothing about what the job requires. Mike Callen, president of TestGenius, shared a line he once heard at a conference: a 70% cutoff is "a lawsuit waiting to happen."

Instead, Dan recommends having an agency's own job experts take the test to set a minimum competency cutoff, using the adjusted Angoff method. From there, agencies can rank applicants or, as Dan prefers, use statistical banding. Banding groups similarly qualified applicants into tiers, such as A players and B players, because no test is perfectly reliable and tiny score differences don't mean much. The one rule: never band below the minimum cutoff. CritiCall 3D includes tools that help agencies set that minimum quickly.

Fair to Applicants, Too

Dan describes good testing as a handshake between employer and applicant. He used a firefighter physical ability test as an example. If a candidate struggles to drag a 165-pound dummy 50 feet, squeaking past the test only lands them in a job that will demand that effort again and again. An accurate test helps people find work that truly fits their strengths.

That's the goal behind CritiCall 3D: a fair, well-rounded battery that serves agencies, applicants, coworkers, and ultimately the public who depends on every 911 call being answered well. As Dan put it, it's the result of 26 years of research and hundreds of job experts, and it "didn't have to turn out this well."

Watch, read, or listen to the full episode

• Watch: Episode video on YouTube
• Read: Full transcript | Show Notes
• Listen: Apple Podcasts | Spotify | Amazon Music

Note: The show notes include the slide deck, executive summary, and validation report.

To learn more about CritiCall 3D, visit criticall911.com.