3/20/2002 -- Last week, Cisco surprised most everyone by announcing that it had immediately replaced the current version of its CCNA exam, 640-507, with a new version (640-607) featuring router simulations. On Friday, CertCities.com interviewed Don Field, Cisco's senior manager of certifications, and Nader Nanjiani, marketing programs manager, Internet Learning Solutions Group, to ask about theses changes, as well as find out how they expect the revised exam to affect candidates and to learn what we might expect in the future. Here's what they told us:

CertCities.com: Why did you add these simulations to the new CCNA exam?

Don Field: We wanted to make sure the certification exams we offered showed continual improvement in aligning with the skills, understanding and knowledge that someone really needs to have in order to be considered Cisco-certified. We're very happy with the exams that we've always offered, and we certainly stand behind them, but we also look to continually improve on our ability to do that. We felt -- and feel -- that particularly because of the nature of the kind of work people do in operating, configuring and troubleshooting networks that include Cisco equipment, demonstration of hands-on capabilities is of growing importance in the success that someone will have in performing their job function in an efficient and effective manner. We believe the exam questions we have added will do a much better job in assuring that people have the kind of skills and understanding they need in order to do the kinds of roles people who become Cisco certified tend to do.

I noticed you didn't mention braindumps in your answer….Did [combating braindumps] play any part?

Field: Everyone who offers certification exams needs to deal with the issue of exposure of exam questions. Any time we or any of the other vendors in the industry make a change to our exams, anyone who was engaged in trying to learn the questions instead of trying to learn the content would have a more difficult time doing it. In that sense, it is a spin-off benefit of making the exams that much more aligned with what someone really needs to know.

I understand that you'll be weighing the hands-on portion of the exam heavier than you will the multiple-choice [and other] questions.

Field: Each of the simulations…involves an individual going through several steps. What we have done is designed these in such a way that if someone gets completely through a simulation…with the wrong result but parts that are correct, they may well get partial credit. Therefore, each scenario presented to a candidate [is] worth more. It is in that way a simulation is weighted more heavily.

What technology are you using for these simulations? How did you go about developing them?

Field: They're based on Flash. We've used Flash for the purposes of instruction in the past, and we do today. For example, a significant amount of Flash simulation is used in our Cisco Academy program… We've used that same technology in building the simulations since that technology was readily available to us. [Editor's note: For a demonstration, click here].

How are the test centers adapting to it?

Field: Nicely. There were things that each of test partners, Prometric and Vue, needed to do in order to prepare their systems to be able to deliver these exams -- so not every Prometric center and not every Vue center can deliver this exam, but many can. Vue, in fact, is providing a list on its Web site listing specifically which vendors are offering the exams.

Does the new exam require hardware upgrades on the part of test centers?

Field: It can. The requirement to deliver [this] exam is slightly greater than each of the two vendors have had [for Cisco exams]… so there could be a certain number of test centers that may not have the equipment to run the simulations. Having said that, the systems of several years ago could run these simulations, so it'd have to be a situation where someone was using quite old equipment by today's standards to not be able to run what it is that we offer.

What's your penetration right now as far as availability?

Field: I don't have a statistic on that.

As far as the announcement is concerned, why wasn't the new exam format pre-announced? Instead of switching over instantly, why didn't you give end users…some kind of warning that this was coming?

Field: The candidate who is properly prepared should have the same experience on this as they would have on the previous exam. We didn't change anything in the objectives for the exam; we didn't change anything as far as the material we were testing. We changed the way in which we were asking the questions. So for the candidate who has been preparing based on their understanding of the material, essentially for them there is no change -- this exam will test their capabilities in exactly the same way as the prior exam. To avoid disruption for those individuals who had already registered for the [640]-507 exam, we do allow those who have already registered to take that exam. They've got six weeks to do it, and that covers just about anybody who's registered. In addition, by the way, those who registered for the [640-507] exam before March 12 but prefer to take the new exam can also switch over.

On our site, people are posting, "This is a great thing…but why didn't they tell me before?" There's some anger…[Candidates may have] bought a 640-507 book two days ago and now there's a [640]-607 book out…Do you see where these candidates may be upset?

Field: There are always folks who will be concerned about change, and I can certainly understand that. The good news is that we haven't changed in any way what someone needs to know in order to become a CCNA. The skills required haven't changed. The understanding required hasn't changed. The knowledge required hasn't changed. So an individual who has been preparing for the previous version of the exam and preparing properly according to all the things that they're supposed to do have also been preparing for the new exam -- they just didn't know the new exam was coming. But their preparation will serve them exactly the same way on the new exam as it would have on the exam that they were working on previously.

So, because the objectives haven't changed, you're saying that any [study] materials aimed at the 507 exam are just as going to be applicable to the 607, correct? You're not telling everyone to go out and buy new books…

Field: That's right…To the extent that any materials were appropriate for the previous exam, they are just as appropriate for this exam.

Can you talk a little about …the development process?

Nader Nanjiani: The development process involved gathering a number of scenarios that a networking individual or field technician would face in the real world. Having reviewed those networking scenarios, an approximation was developed to create a scenario that would best validate entry-level skills for network technician. Having approximated that, there were questions written to that effect. As Don mentioned earlier, these types of tools are currently being used in the Cisco Networking Academy curriculum so not only was a certain amount of familiarity built into the systems, but also a good amount of testing went on. After that, the exam was developed, the exam was field tested -- we had internal Cisco employees go through the field test; we tried to gauge how they would perform on the exam, how easy it was to adapt to these exams. Not only did we include these simulation items, we also went ahead and refreshed the questions (the multiple choice and fill-in-the-blank questions that were on the exam). As a result, we've pretty much come up with a new exam that still covers the content outlined for [640]-507. After the field test, we felt comfortable -- given that we had a large amount of data on what the trends are with CCNA and what appropriate test scores are -- going with the launch.

What was your timeline on that? When did you start this process?

Field: The whole process from deciding that this is the type of thing that we would do to where we are is probably nine to 10 months.

You've said in general that you're working on putting [hands-on scenarios] into other exams? Do you have a timetable for that?

Field: We're not in a position to announce anything at this stage. We certainly expect in the coming months -- without being more specific as to how many -- a number of additional exams in the Cisco certification line-up will undergo changes to include simulations. But, of course, we reserve the right to make other changes to our exam questions as well. Our goal remains the same: continuous improvement in aligning our certification exams with the skills and knowledge and understanding someone needs.

So you don't want to give any hints as to what and when the next exam with scenarios will appear?

Field: That's correct. We announce things when we're ready to announce them.

It could be said you just held off pre-announcing the 640-607 so you could be the first to come out with the prep materials before everyone else does, which would be your prerogative, of course… So again, why not pre-announce that these simulations are coming?

Field: Unfortunately, it's the same answer. This change was not a change to content but the way we're assessing content, so for the candidate who is properly preparing there is essentially no change.

So that's why you didn't feel a need to pre-announce it?

Field: That's correct.

What kind of reception have you gotten so far?

Field: Obviously, it's very early to tell. I don't have numbers to share, but we know people have already taken the exam, and certainly, we're going to be interested in all aspects of it. We're going to be interested in the reaction that we hear from customers and we're going to be interested in candidate performance on the exam -- as we do when we make other changes. So, in that regard this is certainly no different.

Do you see this as a natural progression for your program? You're one of the first programs to implement this…

Field: For our program, I think it is a natural progression to continually improve the exams and, in this particular case, we're improving them in a way that matches what our candidates and those that employ them are looking for in terms of validation of hands-on skills….It is a good fit with the way people are looking at Cisco certification.

Would you characterize these exams as harder?

Field: I wouldn't characterize them as harder or easier. I would characterize them as a different and, we believe, better way to assess the same knowledge, skills and understanding that we were assessing before.

Do you expect a higher failure rate with the new CCNA exam?

Field: I honestly don't know what to expect. For the candidates who are preparing by making sure they have the requisite knowledge and skills, I would expect a similar pass rate to the pass rate of the previous CCNA exam.

Just to clarify, eventually you're looking to adding these types of simulations to all your multiple-choice exams?

Field: We haven't said all of our exams. We've said that we intend to add this to additional exams. [Editor's note: Simulations were demonstrated at a Prometric conference in 2001; at that time Cisco expected to integrate them into the CCIE written exam.]

Is there anything else you'd like to share with our readers?

Nadar: One constituent that we're always focused on is the employers who pay high salaries to those employees who do get certified. We want to make sure as we certify people that employers continue to rely on the certifications as an accurate gauge of a candidate's skills. We believe that going to simulations offers us a more accurate reflection of a candidate's skill level. If employers value a certification highly, then naturally candidates stand to benefit in the long run.

More articles by Becky Nagel:

• 10 Hottest Certifications for 2005
  
• CertCities.com's 3rd Annual Readers' Choice Awards
  
• Q&A: Novell Certs Go Linux
  
• Q&A: Don Field and Rick Stiffler, Cisco Certification