4/28/2004 -- There's no hotter topic in the IT certification industry right now than performance testing. Whether it be simulation questions, embedded environments or live, hands-on labs, many programs that currently rely on multiple-choice questions are taking a serious look at making the move to hands-on -- not only for security reasons, but also for the promise of providing a better way to accurately test candidates' skills. Even for programs already providing such testing -- such as Cisco, Novell and Microsoft's Office Specialist program -- validating psychometrics on such exams can be a challenge, as traditional testing standards don't necessarily apply.

One place these companies can turn is the Performance Testing Council. According to Council President Wallace Judd, Ph.D, who has been developing performance tests since 1989, the Council is a non-profit consortium of IT certification providers and other vendors focused on "advancing the technology of performance testing," including current initiatives to create valid reliability standards for performance exams as well as standards for electronic delivery and data exchange formats, among others.

On May 27, the Council will be holding a one-day workshop in Los Angeles covering some of the issues involved with creating and implementing such exams. In advance of this seminar, we took some time to talk with Judd about the current state of performance testing, the challenges it faces in becoming more widespread, and what innovations the future may hold.

CertCities.com: Hands-on testing seems to have taken on a "holy grail"mythos in the industry, in that it is a fool-proof fix to all the security problems that braindumps cause. Would you agree to with that assessment?

Wallace Judd: Is performance testing an improvement in test security? It's a substantial improvement...Is it the ultimate answer? I don't believe there is an ultimate answer, but it can improve things a lot.

For example, if somebody wants to cheat on a performance test the way that they can cheat is by memorizing the answers. The way they cheat and memorize the answer is by actually performing the desired behavior. So if by practicing the desired behavior they're "cheating," well, guess what? They're actually learning the skill.

I heard an industry speaker one time mention that performance-testing items may be more memorable, and therefore they could be more vulnerable to cheating. For example, what if an exam has 40 objective areas, but the test itself only has three hands-on tasks? Would that exam be less secure than a traditional multiple-choice exam?

Judd: That would be a mistake.What you'd want to do there is develop a test with, say, 20 performances and then randomly select three, four, or five -- whatever's an appropriate number to administer -- so that they didn't know which of the 20 they were going to get tested on.

There's another way to resolve the issue. For example, if I'm supposed to create a new user on a specific system as a part of a system administration task, I can randomize that user name from among 200 different user names. And so if I want to cheat and try to memorize a set of tasks to do with Sarah Jones, it may be that with Sarah Jones I was supposed to hook up her e-mail in this situation and not create her as a user account. So I'd get everything wrong.

You can actually randomly generate variance within the item that make it very difficult to cheat because now a specific sequence of keystrokes or events is not going to work to solve that problem.

Why isn't performance testing currently more widespread within the certification industry? Is it technology limits? Cost?

Judd: There are a raft of reasons...[One is] it takes a higher level of technological proficiency to implement it. In order to make this happen you have to program it, and you have to create it in some language, whether that's a simulation language or whether it's an actual embedded language within the application, or whether it's an offering language.

Another barrier is that a lot of people have experience with multiple-choice testing, and they consider that to be the standard of testing. Even though there are issues with its reliability, with security, with its validity...Those are much less of concern [however] than the pure economic cost of developing [a performance] test -- [that's] the major hurdle right now.

Performance testing is perceived as more expensive and more difficult to administer, and more expensive to construct and more difficult to administer. In some cases that's true. In some cases it's not true. But most people recognize that performance testing as a methodology is superior to that of multiple-choice testing.

The question is, "What's the cost benefit of doing this more difficult type of testing?" We've already conducted a...study with 900,000 simulated users that let us and our membership know exactly what the cost benefit is for a company to change their correlation between test score and ability by...10 percentage points. If you have a test with these parameters, your recruitment costs, your hiring costs, your termination costs, and the possibility of a wrongful termination suit, then your cost benefit -- if you hire 200 people a year using this certification -- may be in excess of $500,000 by using a performance test over a conventional test.

What are some of the technology issues in getting a hands-on or simulation exam delivered through the various testing vendors?

Judd: One is you've got to have more CPU power in order to execute either a simulation or a hands-on test because it requires more CPU power to run the test item. In addition, the hardware has to have more available memory to either run the application or to run the simulation. If the test material is downloaded, it's a bigger download.

Another one is that a number of different vendors that are content providers use different application tools in order to create their simulations or hands-on [exams]. Consequently, the delivery vendors have to deal with a variety of different kinds of applications that are integrated into their delivery system.

All these problems are real, but over time all these problems are going to get solved.

One way to get around the hardware requirements would be to use thin-client delivery, correct?

Judd: Absolutely...A solution to [the hardware] problem is to have people dial in to a network that already has the setup executing on it, and to have that setup be configured the way you want it to be. The [test-takers would] then dial in with a modem to that environment and proceed from there to solve whatever solutions you need to in that environment. So, yes, that is a solution. But there are a variety of different kinds of solutions to this problem that are being explored and implemented right now.

Would you say that performance testing right now is based solely on face validity (e.g., that it appears to be most valid based on the customer perception)? Is that the only way to measure it right now, or is there other data out there?

Judd: No, that's not the only way to measure it. I think that that is at this point probably the strongest measure that's widely known. But, for example, when I was vice president of Research and Development for Kelly Services, we did a very powerful validation study. What we did is we actually tested people with the real application in the branch. [Then] we told them that there was a job available tomorrow if they wanted to go down to this bicycle shop down the street and they needed a secretary for a day. We paid them the regular day's wage and they worked for this bicycle shop. Well, little did they know that that was a sting operation for Kelly Services. What we did is we actually measured their output in the shop. We saw what they constructed, we saw how they created it, and we measured the effectiveness of what they did.

We then compared that data with the test data that we had from the branch. And that's a perfect validation study. Not only did we ask the examinees to store the data on the disk --- so the data was actually computer scored -- we also said print it out and had them put it into a letter to be mailed. The letters that were put on the desk never got mailed. They got taken to Kelly Services where they were hand scored. Then we compared the hand scoring with the machine scoring and it was 99.8 percent accurate. So we really knew what was happening with those results.

There's maybe a few, maybe even more of those types of studies, but a lot of them have been internal. What we're trying to do is to put people -- companies, developers of certifications -- into a position where they can publicly either release that kind of data or sponsor an experiment that we can verify is objective and get the information out to the entire industry.

What trends in performance testing have you seen in other industries that you think might cross over into IT certification?

Judd: There are a number of other industries that are doing performance testing. I think you'll see a lot of cross-fertilization…

A big development that I see happening that's going to expand dramatically the automation of scoring is this whole RFID development...RFID gives you a way of electronically assessing the position of a certain device or a certain piece of packaging or whatever. And if, for example, what you want to do is see what somebody did, you can put RFID tags on real objects and then electronically assess the location and the position of those objects. So whereas previously you couldn't electronically assess what a warehouseman was doing, now you can...

Increasingly there are ways that we're automating the processes that used to have to involve human [scorers]. For example, some optometricians in some states are able to give injections. The American Board of Optometry wanted to be able to test that but they didn't want to have to test it on the people that were there as subjects because they would get injected eight, 10, 12 times a day. So the American Board of Optometric Examiners bought synthetic arms which can be injected and which feel and accept the needle just like the real thing and cost $35,000 apiece. But they're instrumented and they can automatically tell whether that injection got in there or not.

As technological advances allow us to create and evaluate simulations through virtual reality, through instrumentation of the real world, through a variety of advances what's going to happen is those developments are going to give us means to automate the scoring of performances.

In your years of working in the performance testing industry, what's surprised you most?

Judd: What has surprised me most is how long it's taken to become mainstream. I implemented the first series of performance tests in 1989. Here it is 2004, and a lot of people are just stumbling on the field. It's amazed me that it's taken this long for it to become widespread. And it's still not widespread.

How do you think performance testing will change the IT certification industry over the next few years?

Judd: I think that what will happen is the tests will get more precise. They'll be able to be given in a shorter amount of time. They'll produce more accurate results with a much larger scope. I think that's what's going to happen as a trend over the next three to five years.

What happens as a result of that is probably more interesting because, as you get a better idea of the skills you're brining into the shop, a number of things happen. One is you can train people much more specifically to fill in the gaps that they have in their background and in their own experience. Because you can identify those much better with performance testing, you'll be able to remedy them better.

Another thing that happens is when you're sure that somebody has a certain set of skills because they've in fact performed them, you can manage better to that skill or skill set. If on the job they're not performing those particular duties well that they demonstrated in the test, then it's very clear that you have a motivational issue; you don't have a skill issue. As a manager it's always been difficult to tell the difference between a motivational problem and a skills problem. You always assumed that training might resolve the issue. But training will not resolve a motivational issue. So it lets you manage in a much more precise fashion.

Do you think the costs of performance testing will start to decrease over the next few years?

Judd: Yes. All the tools and technologies that are currently being developed are all aimed at trying to create a much more standardized approach to performance testing, and these tools ideally are adaptable to a broad range of problems. That means that the cost of developing the tools can be amortized over a much broader number of tests, bringing the cost per test down.

What advice would you give today to a smaller testing program that’s thinking about implementing some form of hands-on testing?

Judd: It would be a good idea for them to get information from the Performance Testing Council about other companies in their area that may be doing performance testing, or to try to benefit from the experience of parallel or similar companies, similar attempts at performance testing to see what the range of tools available is to them. And then there are a number of questions that they have to ask: What's the value of performance testing to them? Do they really care whether or not the test does a better job of personnel selection or personnel description?

Another thing they have to consider is, is there instrumentation already available -- either in the application itself or in the operating system -- that allows them to automate the feedback process? If there is that sort of feedback built into the system somewhere, they're going to have a much easier job of creating a [hands-on] performance test item. On the other hand they may want to look at simulation. If they have a relatively small number of screens -- not 600 but 60 -- it's going to be a lot cheaper to simulate 60 screens than 600.

Then they need to look at how quickly does the application change? If the application itself is changing once every three months, probably either simulation or hands-on isn't going to work because it’s going to be too difficult for the test to keep up with the change in the technology.

Are there certain types of test questions -- for example, knowledge-based questions -- that lend themselves better to the traditional multiple-choice format?

Judd: I think that's a popular misconception. It is a misconception because almost never is the memorization of the knowledge itself your goal. Your goal is usually to be able to use that information to get something done.

For example, if I need to know all the parts for a carburetor, I'm not doing it because I like to memorize carburetor parts. I'm doing it because I want to be able to fix that carburetor and I have to tell the guy at the parts counter what I want.

What is the right thing to do [for testing this skill] is to take that carburetor apart and for you to remove three pieces from the table, then for you to hand me a catalog and say, here, order the missing parts for this carburetor from the catalog. Notice that's a performance problem...

You could turn it into a memorization problem because you could have a bunch of different parts and say what's A, what's B, what's C? But that is almost never the performance that you want. That’s not the behavior you're concerned with. So when you put it in a realistic context it changes the entire problem. Notice how much more complex and robust the carburetor problem becomes when you've got the parts all played out on the table and you're missing three, and you have to order the missing three. That's a real problem.

And, incidentally, somebody is much more motivated to solve that problem because they can see it as a real problem...I think that many times the poor performance observed on multiple-choice tests is due to the fact that people don’t understand how what they’re being asked is relevant to the job.

Is there anything else that you want to share about what’s going in performance testing right now that you think our readers would be interested to know about?

Judd: I really believe that as an industry we're headed toward a very robust future, and that as instrumentation becomes more widely embedded in actual objects, as simulation becomes cheaper to execute, as virtual reality becomes a standard game tool, as graphics tools become more and more capable of creating animations that emulate the real world, what will happen is the scope of performance testing will dramatically broaden.

Right now it's difficult to create a performance test for a salesman because you can't create a rich enough environment to simulate the multiplicity of responses that a salesman is going to encounter. But within five years with a flick of the wrist you'll be able to...create a simulation in an hour that is a very robust simulation of the environment a salesperson faces when they go out and try to sell your product. That would be a good test. And right now it wouldn't be a good test at all.

Do you really see the technology coming that fast?

Judd: Sure. If you look at the game world, if you look at the instant replay world, if you look at virtual reality in electronic arts and whole bunch of other places, it's just cascading down on us. As that technology becomes more widely available and cheaper, it'll be put to good use in the world of performance testing.

Thank you for taking the time to talk with us today.

Judd: Thank you very much.

More articles by Becky Nagel:

• CertCities.com’s 10 Hottest Certifications for 2006
  
• 10 Hottest Certifications for 2005
  
• CertCities.com's 3rd Annual Readers' Choice Awards
  
• Q&A: Novell Certs Go Linux