2/3/2010 -- Why is developing a developer-based exam such a challenge? And how serious does Microsoft take exam piracy and cheating? Microsoft Learning Group psychometrician Liberty Munson gives us the lowdown in Part 2 of her Q&A. (Go here to read Part 1.)

Are there any particular challenges in developing developer or business-oriented exam questions as opposed to IT-related ones?
Yes, we struggle with writing real world, relevant multiple choice items for developer exams because in the real world, most developers use Intellisense when writing code and, as a result, don't memorize command lines. It's very hard to replicate that experience in a multiple choice and other "recognition" item types. If we were to remove this aspect of their work from the content domain, we'd no longer have an exam that is a valid measure of the way the technology is used by developers.

The best solution for developer exams is to rely on performance-based testing, but this is complicated. Because there is rarely, if ever, one and only one end state that will solve a developer problem, scoring is proving challenging for us. Although scoring is based on the end state, not the process used to get there, with developer exams the end states could look very different from one candidate to the next and still have solved the problem.

Because we understand the concerns that developers have about the relevance of our developer exams, we are working on solving the scoring problem so we can implement performance-based testing for these exams as quickly as possible, but it won't be in time for the next wave of .NET exams. We are leveraging lessons learned from the .NET 3.5 exams and hope to create better developer exams while we figure out performance based testing for developers.

We have similar problems with some types of business-oriented (PRO) exam questions. For some job roles, case study items offer the best solution because these items require candidates to read and extrapolate information from multiple sources to determine the appropriate course of action or solution. Although these item types require more reading than our typical items, the nature of the case study allows us to mimic some of the complexities of certain job role problems.

"Microsoft investigates every lead that we get from braindump sites to test centers that are engaging in inappropriate behaviors to candidates who provide score reports as evidence of passing an exam." -- Liberty Munson, exam psychometrician, Microsoft Learning Group

With piracy and cheating as a big concern, what can you reveal regarding how you determine if cheating or piracy is taking place?
Unfortunately, I can't tell you much about the specific actions that we're taking to identify and prevent cheating because the more we tell people, the better the cheaters get at thwarting our anti-piracy efforts.

But, I will tell you this: Everyone leaves a "footprint" in the data when they take an exam; using statistics to analyze this footprint, I know if something anomalous occurred during a particular testing session. And, we do look at these footprints.

Microsoft takes cheating very seriously, and we have developed four major pillars in our anti-piracy campaign.

First, we want to educate candidates about what Microsoft considers cheating: colluding or working with others in a collaborative way during the exam, using braindump sites to prepare for the exam, having someone else take the exam for you (proxy testing), and falsifying score reports are all behaviors that Microsoft considers "cheating."

Second, we look for ways to protect our content, such as having candidates sign non-disclosure agreements before taking an exam, geoblocking (not delivering exams in certain countries), and investing in performance-based testing.

Third, Microsoft investigates every lead that we get from braindump sites to test centers that are engaging in inappropriate behaviors to candidates who provide score reports as evidence of passing an exam or cheat in ways that we identify through that footprint that I mentioned above.

Finally, Microsoft enforces our anti-piracy policies: we ban candidates and close test centers, and we have successfully sued multiple braindump sites with satisfying results. We are looking for ways to share more of our enforcement activities publicly and hope to do more in the future.

What does the future of test-taking look like, and what are the scenarios we may see in five, 10, 20 years from now?
I don't think we'll see huge changes in the next five years (although certifying bodies will continue to experiment with innovative item types, delivery methods and approaches to piracy, the changes will likely be small), but over the next 20 years as technology continues to expand, I think more IT certification bodies will leverage:

• Performance-based item types
• Remote proctoring where the proctor is not at the same location as you but is monitoring your computer strokes and watching you via webcam
• "On the fly" anti-piracy analyses that will end the exam when anomalous test taking behavior occurs
• Innovative delivery methods that make stealing content and cheating extremely difficult

In the last item, I don't know what these will look like, but the breakthrough will be spectacular.

If there were a psychometricians club, how big would it be? And what are the typical issues it would address in its newsletter or journal?
The psychometricians club would probably include several thousand people. Typical issues addressed in our newsletter/journal would include innovations in item types and exam development, practical considerations in exam development to ensure validity and reliability while still meeting business constraints related to budget and resources, advancements in statistics and psychometrics (yes, psychometricians are constantly developing new statistics to run or different ways to interpret the data, especially as technology advances), changes in best practices and other standards as technology changes, best approaches for establishing validity (what's acceptable and what's not), speaking "psychometrician" to managers and other business leaders, etc.

Do psychometricians have any advantage in Las Vegas?
Probably, but I don't know many psychometricians who gamble because we understand probabilities too well and as a result, tend to be risk averse. After all, the numbers never lie, and the house never loses.

More articles by Michael Domingo:

• Q&A: Microsoft's Resident Psychometrician (Part 1)