Microsoft abandoned adaptive testing many years ago, along with most vendors. CompTIA was the sole holdout of note, though they never seemed to believe in it enough to apply it to any of their exams outside of the A+ realm. Not surprisingly, it just announced that with the next round of A+ exams, they are bidding farewell to the adaptive format and going back to linear. What is surprising, however, is the reason it gave as to why.

CompTIA not kissing it off because it is outdated, subject to ridicule, or something that it never did figure out how to truly implement (my favorite tidbit is the grading scale it used to rank questions: easy, medium, hard). Instead, it's being deep-sixed it because it makes it too hard to introduce questions that aren't graded. CompTIA wants the exams to routinely include new questions that don't figure into your pass/fail results just so they can beta test them and see how well candidates answer them. The questions will look just like all the others and you won't be able to distinguish them from the graded questions, but someone at CompTIA can use the results of them to see if they are good questions or not.

Imagine this scenario: You spend three months studying the objectives of the exams so you can become A+ certified. You know everything there is to know on the topics that are posted. You go to take the exam, and there are 10 questions in there about things you never knew were going to be tested on. You're furious! The study guide you spent money on, the class you attended, the practice test engine you bought, the set of videos….. none of them prepared you for these questions that are randomly interspersed with the others. You sweat and toil over them. How could you not know these topics, you ask yourself. Your inability to feel good about how you answer these questions bleeds over into your comfort level about the other questions. You start to second-guess. If you don't know the answer to these 10, you ponder, then maybe you don't really know the answers to the other 70. You feel weak; you wonder if maybe you shouldn't have studied just one more week. Time runs out. Your score appears: you passed. CompTIA was just fooling with you on those 10 questions.

To look at it from a different perspective, you just paid CompTIA (who has some of the highest priced exams in the industry) money so you could be their guinea pig. While most other vendors kowtow to beta testers (free exams, hats to the first 500, etc.), CompTIA doesn't mind charging you full price and making you worry about beta tests AT THE SAME TIME you're trying to get certified. You have to love capitalism!

In the article linked to above, Tancy Stanbery, senior certification program manager for CompTIA, says that this testing method reduces headaches for candidates who have to deal with beta exams: "Those can be somewhat painful to our candidates...they have to wade through a lot of questions and then they have to wait for results," she explained. She followed that up by saying the upcoming linear A+ exams will feature 80 questions instead of the current 20 to 30, declining to state how many unscored items and scored items this constitutes. I'm sure glad that only going through 80 questions will keep candidates from having to "wade through a lot of questions."

Before the wrong message comes across, I want to point out that CompTIA is not the first organization to think about beta testing on live candidates. The concept is a valid one. What is so poor, however, is the method in which they are implementing it "seeding" unscored questions in with live ones is the easy way out. Compare this with the way in which the LSAT, used for admission into law school, is administered. Instead of using the word "linear,"they use the more understandable "standardized," but serve the same purpose as any certification exam.

The LSAT is divided into five sections. Only four of those sections count toward your score, and one of them is used to collect beta results. You don't know which of the five sections is the beta, but you do know that it is an entire section -- not a single item sprinkled about among all the live items. Dividing the exam into sections allows you to take mental breaks and breathe a sigh of relief at the end of each. If you have a section where you just can't quite figure out what they are looking for, you sigh when you reach the end of it and hope that that was the beta section. That, or any section, does not carry over into the others or affect how you think or feel in them.

I salute CompTIA's decision to abandon adaptive testing and join the rest of the certification world. I do wish, however, that their reasons for so doing were more virtuous. I also hope they will abandon their plan to increase the stress level of those who only want to demonstrate how they've mastered the published objectives and get back to work.

What's your take on this issue? Post your thoughts below!