What’s it testing?

I took a dumb assessment the other day, for a job, and decided to look into who made it.

Criteria is a “leading provider of professionally developed pre-employment tests” made to help businesses make better talent decisions. Based on 'stringent scientific standards’ that are ‘backed by experts in I/O psychology,’ Criteria is able to produce assessments that ‘accurately measure job-related competencies in a consistent and reliable way.’

What a bunch of baloney.

Criteria is right to say that ‘an assessment is only as powerful as its validation.’ Except that what it does to validate its assessments isn’t scientific, let alone reliable. Their service isn't testing if the folks who pass their assessments truly are the best possible candidates for the job. It's testing for what it thinks are reliable indicators of people who would make the best candidates (not the same thing.)

In other words: Criteria isn't assessing what it says it measures, or even analyzing if their assessments consistently and reliably produce the best outcomes.

That's because you can’t just take a dataset from the DOL, plug it into an algorithm, look for correlations, and assume that your assessments will work for every job in every industry for every client. That's bad science.

Furthermore, you can't measure content validity unless the questions themselves are specific to the job you’re hiring for. And you certainly can’t know the ‘ability of the test to correlate with (the best possible) business outcome’ unless you look at the opportunity costs of factoring in other variables (such as hiring the candidates you didn’t pass the assessments), and testing how those factors actually perform (which you can’t).

So no, I don’t believe that Criteria is actually testing for what it says it does. And I don’t think it helps anyone to subscribe to their crap.

The only reliable way to reliably test if someone would be great at a job is to give it to them—the same job, the same tasks—and to measure how they perform. Not by giving them an aptitude-test from some high-altitude think-tank. But by actually testing what your testing.