7/27/2004 -- When it comes to certification program success stories, it's hard not to think of Cisco. Over the years, the program has come into worldwide prominence with sizable and continued growth, offering a large stable of offerings whose reputations seem only to improve, thanks in part to innovative upgrades like adding simulation questions to its low- and mid-level exams, as well as the highly regarded (and highly feared) jewel in its crown, the lab-based Cisco Certified Internetwork Engineer (CCIE) title.

We recently had the chance to talk with Cisco's Don Field, senior manager of Core Technologies at Internet Learning Solutions Group (ISLG), and Rick Stiffler, senior manager, Advanced Technology Training at ILSG, to ask them about everything from recent additions to the company's Qualified Specialist program, to whether the latest version of the Cisco Certified Network Associate (CCNA) exam is indeed statistically harder than previous versions. Here's what they had to share:

CertCities.com: Let's start with the new Storage specialties that Cisco recently announced. Can you tell us a little bit about why the company decided to add this area?

Rick Stiffler: Storage is one of our new advanced technologies here at Cisco, and the certification that was launched in support of the storage products and technologies is just that -- it's supporting new areas that Cisco has expanded its market into. We've developed the training and the certifications to support these new technologies.

Cisco currently has around 14 specialties in its Qualified Specialist certification program, correct?

R.S.: I believe we’re now closer to 20 right now.

How is the Specialty program going overall?

R.S.: We’re continuing to see continued growth -- very good [growth] -- in the hot areas, especially Storage right now, but also inVoice, Security and Wireless.

Don Field: Let me add that one of the areas that’s been very important to us in designing and offering our specialty certifications has been their use among our channel partners. They’re available to anyone but they are of particular value to our channel partners. In many cases our channel partners are required to have their employees meet a certain job role requirement in order to move up in the Cisco channel partner program. And the specialty certifications often match exactly those specific requirements. So that’s been one piece that has driven the success of the specialty certifications in our program.

Are there any other Specialty areas that are coming up that we should be looking for, or ones that you’re looking at maybe changing or retiring?

R.S.: Nothing that’s currently available for release ... [Cisco CEO] John Chambers did publicly announce that while [Cisco is currently in] six to eight advanced technologies right now, he wants to get that up to 12 or 15. So over the next year or two I think you’ll see [the company getting] into more diverse areas, and we will follow with that.

And you believe with the channel partners and the general audience there’s enough market to support that range of Qualified Specialist titles?

R.S.: Absolutely.

Let's move on to the new, free CCNA Prep Center. Why have you decided to offer this?

D.F.: Anyone who’s taken a Cisco exam is welcome to use the CCNA Prep Center. We launched it as a pilot and it is [currently] in its pilot phase as a way to help those who are seeking CCNA certification find the resources they need to complete the process and actually earn the certification.

It’s quite honestly as straightforward as that. Those who might have tried the CCNA exam but not passed it, those who are taking our two-exam path such that perhaps they have tried and even passed the intro exam on the way to earning CCNA would find it very useful because it has some of the practice materials and simulations that would help someone to be sure they’re properly prepared for the CCNA exam.

It also serves a purpose, an additional purpose, in that those who are seeking to recertify at CCNA by passing the CCNA exam or our [ITND] exam would also find the materials on the site to be helpful.

What's the payoff for Cisco to offer this for free?

D.F.: We’re very interested in seeing those who are seeking our certification be successful. We’re not going to help them be successful by making the exams any easier. The purpose of our certification remains as it always has been, to separate those who know their stuff from those that still have a way to go. But for those who are serious about acquiring the knowledge and skills and understanding that are needed to achieve the CCNA level, we’re happy to provide resources that will help them do that.

[Beside the free resources], we also provide pointers to ... fee-based resources from our Cisco learning partners and from Cisco Press. So there’s the opportunity for people to get the full range of resources that they need.

For us the payoff is seeing more of those who choose to enter this path and are willing to invest the time and sometimes the financial resources to acquire the skills that are needed to be successful at doing so.

Why are you limiting the Prep Center to only those who have taken an exam? Especially considering that a large majority of people come into the Cisco program through the CCNA.... many people that haven’t taken an exam are going to want to have access to this material.

D.F.: Well, in the pilot phase we wanted to make sure that those who are in our program already have the opportunity to use this site. It’s always possible that at a future stage we’ll find a way to make the site available to a larger number of members of our audience. But for now in this pilot phase we really wanted to keep it to those that have demonstrated that they’re in our program and that, you know, may have a need to find additional resources to achieve CCNA or to recertify.

Are you looking at doing something similar for the CCDA?

D.F.: We’re starting this one as a pilot and we’ll see how this goes. And of course it’s always possible that we would do similar things if it turned out that this met the needs that those among our candidates were seeking.

Let’s talk a little bit about the CCIP. There was a big transition in 2003 when it was restructured to be more like the CCNP in some ways. Just wondering, how is it going? Are you seeing the pick up on that title that you wanted to see?

D.F.: ... CCIP is doing very well. We’re very pleased with the changes we made. What we found when we originally launched CCIP was that there were positives to giving people many options for how they could earn a CCIP... [But] what we heard from candidates and from employers was that it made more sense to be clear about what it is that someone has validated when they earn a CCIP versus the many different options that would make it less clear as to exactly what somebody had passed in order to earn the certification.

So we made the change so that it would provide ... both candidates and employers the ability to really know what it meant when somebody earns a CCIP. We focused therefore on the baseline technologies that we found most prevalent in service provider organizations -- MPLS, QoS ... along with advanced routing ... And again, have gotten nothing but positive comments from our candidates and from employers in the service provider industries for doing that.

It seems that people aren’t sure who the CCIP is really for. What is the audience you’re serving with the CCIP and how can someone know whether they should pursue a CCNP or CCIP?

D.F.: Well, CCNP has not only been available longer but it’s our most popular certification by far at the professional level. It's the best known by far and certainly makes sense to me that most folks, as they consider a professional level certification, are looking at CCNP first.

CCIP is very specifically targeted at the service provider community. Telecommunications companies, cable companies are the types of organizations whose employees look at CCIP as being the logical next step and the right certification for them.

That’s not nearly as large an audience as the general audience of enterprises and governmental organizations, education institutions and so on, so it is a certification that is somewhat more targeted.

Let's talk about testing formats. Could you tell me the percentage of traditional-format Cisco exams that now have hands-on simulation questions in them?

D.F.: I believe all do right now. There may be one exception ... But if not all, just about all of them …

R.S.: [For the Qualified Specialist certifications] in Security, Voice, and a couple of the others we do have simulations. Not all have them [right now] ... but that is the process that we’re in. It’s moving to that.

When are you looking at possibly completing that?

R.S.: I wish I had a date I could tell you. It’s just an ongoing process.

Right now, I believe that for exams that have simulation questions there’s generally two to three per exam. Are you looking at expanding the percentage of questions on the exams that are hands-on?

D.F.: We ... don’t look at it exactly that way. We have one clear simple purpose for our exams and that’s to, as I said earlier, separate those who have the knowledge and skills and understanding that are the subject areas of that exam from those that might still have a ways to go.

We pick the question formats, the exam technologies that we use and so on specifically to achieve that purpose. If that purpose is achieved with one simulation, that’s great. If in another exam that purpose really requires three, four, five simulations or some other number, that’s just as fine from our perspective.

So the choice of format of exam questions is really dependent on what we need to identify those candidates that are properly prepared in that area.

We get a lot of questions about how the scoring is weighting with the hands-on questions. Would you care to share any insight onto how heavily the hands-on questions weigh, for example, in the CCNA or one of the CCNP exams?

D.F.: I can share a little bit with you. The one question that we do get from time to time that we have answered publicly and I'll share that answer here ... [is] we do offer partial credit on our simulation questions.

So a simulation question is often worth the weight of several individual [multiple-choice] questions ... on the exam. And at the same time though, an individual who demonstrates partial awareness of what is being tested on a simulation question will in fact get partial credit.

The simulation questions on our exams do form a significant portion of what’s being tested and I would suggest it would be rather difficult for someone to be successful on our exams to have simulations without having sufficient command of the subject to be able to perform on those simulations. But we don’t get into the specifics of what percentage of the exam is tested by simulation.

Do you worry that the unique nature of the simulation questions may make them more memorable, and thus possibly more vulnerable to cheating than other question types?

D.F.: We do a variety of things -- none of which I’ll talk about on this call -- to make sure that we maintain the integrity of our program. There’s nothing that I can think of that would be more important to someone who holds a Cisco certification than to know that we work very hard to make sure that we continue the tradition that I described a couple of times earlier -- that our exams really do a good job of separating those who know their stuff from those that might still have a way to go.

Exam security is part of that. I have no reason at all to believe that simulation questions are more susceptible from a security perspective than other types of questions in specific answer to your question.

Do you feel that, in general, certification exams are more secure now than they were two years ago, or less, or about the same?

D.F.: Well, I certainly wouldn’t want to comment for the entire industry. I think the industry works hard on that issue. We feel very comfortable with the security of our exams and we felt very comfortable with the security of our exams for a long time. So I don’t know how I could compare it to a year or two ago but we worked hard on it, we’ve got the right people doing the right things on it and we’re very comfortable with the security of our exams.

And at the same time comfort does not mean that we’re not very vigilant regarding that issue. We most certainly are because, as I said, it’s critically important to our candidates and our certification holders.

Is Cisco looking at any new testing technologies, formats, questions types to add to your current exams?

D.F.: We’re always looking at new things, we just can’t discuss them.

R.S.: We have an entire department that’s dedicated just to this, and so it is a completely ... ongoing function that we do. And we’re taking every precaution and taking every action necessary to insure the credibility of our certification programs. So it’s going to be an ever-evolving process.

Are any of your exams adaptive at all?

D.F.: No, none of our exams are adaptive.... Like any other technology, we might always consider using in the future things that we don’t do today. So we don’t use adaptive testing today but that doesn’t say anything about whether we may or may not choose to use it in the future. And that’s true of any new technology or different technology.

In March 2003 Cisco announced that it had granted 500,000 certifications. Can you give an update of that number and talk about the different growth areas, and also maybe some of the regions where you’re seeing that growth?

D.F.: We’re over 600,000.... A couple of years before we made the announcement that we had reached 500,000 certifications issued, we announced that we had reached 250,000. [Now] it’s a year-plus later and we’re up in the 600,000s of certifications issued globally.

That's total certifications issued, correct? It doesn't subtract titles that have expired?

D.F.: It does not do any subtracting....

In what regions are you seeing the most growth?

D.F.: According to my colleagues that manage other programs, the growth in Asia-Pacific has outpaced the growth elsewhere in the world. So we’ve seen growth in the Asian countries ... the growth in Japan and many other Asian countries has definitely been faster than the growth that we’ve seen in other parts of the world.

I know that in chatting with folks from other IT programs ... many have seen very much the same trend so I think that there isn’t something Cisco-specific about that but perhaps something more generally prevalent in the industry.

There are also a few factors of seasonality in our business as there are in probably just about any business. For example, the number of certifications that we issue in the month of March in Japan tends to be higher every year than in any other month because that for most companies in Japan, the end of March is fiscal year-end. And so we do see more people earning certifications at that time in that one particular country.

That may or may not be true for other programs but it wouldn’t surprise me if it were.

I've talked with a number of other programs who have wanted to add simulation questions to their exams but haven't been able to pull it off for one reason or another, yet your implementation of simulation questions seems to have gone fairly smoothly. Would you credit this to the technology you use, the investment you made? How exactly were you able to make it work?

D.F.: I don’t know that we do anything any better than any other program so I definitely would not think that that’s the case. And we’d be happy to talk with folks at other programs about how we do what we did, and I’m sure that we could equally learn from them about things they’re doing that we wish we did better. So ... I certainly wouldn’t say that we’re necessarily any better than anybody else is at doing what we do.

We knew or felt strongly that it was important for us to invest in making sure that our exams, that to the best of our ability, identified those people who truly had the knowledge, skills and understanding and that were behind the content that was covered by that exam. Simulations are one of many things that we’ve done to make sure that our exams accomplish that.

There are other techniques that we’ve used -- and we know there are other techniques that others have used, as well -- to do a good job of doing that in their programs. [It's] one piece that we’ve done and we’re certainly pleased with how that’s worked, which is why we’ve expanded it ...

Was it difficult to get them implemented? Was it a long process?

D.F.: Without getting into the details of how we work with our test delivery partners, adding a new technology to any system takes some time and effort on the part of all parties. And certainly that was the case when we moved to introduce simulation first on our CCNA exam and then in many other parts of the program.

So certainly it took some time and effort for us to figure out how to design simulation questions that would have the fidelity that was needed to make sure that they were really testing what we wanted them to test. And time in working closely with our test delivery partners to make sure that those questions could be properly displayed and scored and so on in the exams ...

Overall, yes, we were very pleased with how [the launch] worked. Of course, behind the scenes we found all the things that we needed to fix and improve and so on but that would be true with the introduction of any new technology in any system.

There seems to be a sentiment out there that you significantly raised the difficulty level with the latest batch of Cisco exams. Let's take the latest CCNA exam, for example -- from a psychometric perspective, have you raised the level where the exam is indeed harder than the previous versions?

D.F.: That’s not our intent....We have no interest and no desire in making the exams artificially more difficult, just as we have no interest or desire in making them artificially less difficult than what really is needed for someone to demonstrate that they have the knowledge, skills and understanding that matches that content area. And that’s true across all of our exams.

How do you balance that psychometrically to make sure? Was it a bigger transition when you added the hands-on questions?

D.F.: ... It was not more difficult necessarily. For the purpose of the analysis that we do, it was another exam type, but we were still looking to test knowledge, skills and understanding and that continues to be what we do.

We do use -- I don’t know if most of your readers know what psychometrics are -- but we do use statistical analysis to assess the success of our exams in making that determination as to whether or not someone is properly prepared. We think we do a very good job of that, and adding new exam types was something that just was added to that process. It wasn’t something that disrupted or in other ways was problematic for completing that process as we do for all of our exams.

R.S.: ...We do have a number of highly skilled [people] -- including a number of PhDs in psychometrics -- that [work on] this issue, so it is really something that’s put through a lot of ringers to make sure we do do it right.

Is there currently a slight resurgence going on in the certification industry as a whole? Or do you feel that certification in general is losing a bit of value in the eyes of hiring managers? Is there a current trend one way or another?

D.F.: I’m not seeing either of those trends, necessarily ...We’re very pleased with how our program is going both in terms of the number of people who are seeking Cisco certification and the quality of candidates who are sitting for Cisco exams. I wouldn’t be able to comment on whether or not there’s an industry resurgence.

As to those who may not see ... significant value in certifications, from our perspective that’s fine. Certifications aren’t for everyone. For those who see significant value in being able to validate their skills and be able to demonstrate that for their own use, for their current employer, for a perspective employer, we’re delighted to offer the program.

Clearly, [many] are interested and that’s wonderful, but we’re not necessarily out to convince those that may not see as much value because there certainly can be different perspectives on [certification], and we respect those.

Cisco has won the title of "Best Certification Program" in CertCities.coms' Readers' Choice Awards for two years in a row. Why do you think that your reputation in the industry is so strong?

D.F.: Thank you for saying it’s strong ... we just kind of do what we do and work to make it valuable for our candidates.What makes our program strong is that Cisco is strong. Individuals are interested in Cisco certifications in significant part because they’re interested in Cisco products and solutions and technologies. So ... certainly one thing that contributes very significantly and, I would say, most significantly to the success of our program is the success of our company.

After that, the second piece that we think is important are those things that ... we do to maintain the integrity of the program. Things that assure that only those that really have the requisite skills are able to earn a Cisco certification helps to solidify in the minds of candidates and their employers or potential employers the value of a Cisco certification when someone has earned it. So we think that efforts related to certification integrity are also important.

The last piece I would add is that we’ve been, during the course of the last year or so, expanding the benefits that are available to someone who earns a Cisco certification. It was just about a year ago that we launched the Cisco Certifications Community and it had outstanding response to the knowledge-sharing materials there, the live broadcast that we do on a monthly basis to individuals who are certified, the games that we’ve added to the Community which are learning games about Cisco advanced technology. So we think that the benefits that we’ve added for those who have worked hard to attain a Cisco certification have perhaps played a role as well.

You seem to invest a lot in your program, and that can't be cheap. Is Cisco as a whole very supportive of the program?

D.F.: I certainly can’t speak for everyone at Cisco. We’ve certainly had excellent support from our own management in being able to expand the program both in terms of certifications that we offer, investment in the certifications that we have and investment through Web portals such as the Cisco Certification Community and the new pilot of the CCNA Prep Center in helping candidates achieve and find value for the certifications they’ve earned from Cisco.

R.S.: Cisco really puts customer satisfaction top on their list, and when you look at training and certification and supporting the customer’s experience with our products, technologies and solutions, then I think it becomes pretty evident into how they support it and what they think of what the certifications and training do for them.

Where do you see the Cisco program in the next one to three to five years?
D.F.: ... For us the certification program will continue to be successful as we continue to align it with Cisco’s success. So we will look to make sure that in those areas where it’s important to people or employers, that they can validate skills of the individuals, as Cisco’s technologies change and grow or as we add additional technologies and solutions to the Cisco set of offerings, we’ll look at where certifications need to grow as well, and be modified as well. So a big part of how we’ll manage our program is based on the directions that our company takes in products and solutions and technologies.

More articles by Becky Nagel:

• CertCities.com’s 10 Hottest Certifications for 2006
  
• 10 Hottest Certifications for 2005
  
• CertCities.com's 3rd Annual Readers' Choice Awards
  
• Q&A: Novell Certs Go Linux