The Certification Development Cycle

There’s a lot that goes into maintaining a certification exam. Here is an overview of each of the major steps in the development cycle.

Job Analysis

In order to develop a certification program, you first need to understand the profession you are certifying. A job analysis can tell you what content will be on the certification exam, the number of items on the exam, how items will be allocated across the content, and what makes a candidate eligible to sit for the certification exam. You will have to perform a job analysis as frequently as the profession changes significantly, which is typically every 5-7 years, but can be as frequent as every 1-2 years in IT professions, for example. The main result of a job analysis for a certification exam is the exam content outline. It goes by many other names such as exam blueprint and detailed test plan, but importantly it outlines the content that is on the certification exam.

Item Writing

Item writing is just as it sounds: writing new items for the certification exam. Ideally a certification program should do this every year, but sometimes funds are tight or your certification program doesn’t have a high candidate volume. There are many ways to incorporate item writing into your certification development cycle; it’s just a matter of figuring out what is best for your organization. Just make sure the subject-matter experts (SMEs) are adequately trained. You can’t write the items yourself!

Item Review

After you have subject-matter experts (SMEs) write new items for your certification exam, you should have a different group of SMEs review, edit, and improve those items. This should also occur each year, but maybe you have a lot of previously-reviewed items in your item bank. Ideally, these meetings should be facilitated by a credentialing professional such as a psychometrician or test developer. They will be able to guide the SMEs in editing the items prior to the items being pretested. It is important to pretest items so you can get data on how they perform statistically before they become scored and help determine whether people pass or fail the exam.

Form Assembly & Equating

The next step is to have a credentialing professional use the exam content outline and statistical software to assemble a new version of the certification exam, consisting of scored items with statistics and unscored/pretest items without statistics, and ensure that the new version of the exam is equally as difficult as the previous version. You should try to generate a new version of each certification exam every year and retire the oldest version of the exam, however, low candidate volume may not require doing this as often. The number of exam forms and how frequently they are developed is determined by a program’s candidate volume and the certifying body’s tolerance for item exposure.

Exam Review

However frequently a certification program assembles a new exam form, they also need to review that exam form with subject-matter experts (SMEs). SMEs need to review the exam content to ensure each item is fair, accurate, meaningful, and current (F.A.M.C.). This means that the items are a fair assessment of candidate knowledge, skills, and abilities; that the content is factually correct; that the content is important to what candidates do on their jobs; and that the item content isn’t outdated and irrelevant. This is an arduous process, but it is necessary to provide validity evidence to support interpretations of candidate results.

Standard Setting

Whenever you perform a job analysis, you have presumably changed the content you are assessing on a certification exam, and subsequently, you should ask yourself if the standard of minimum competence has also changed. This means you need to perform a standard setting study (cut score study). This is how you determine the number of items/questions a candidate must answer correctly in order to pass the exam and receive the certification. One can’t just say “C’s get degrees, so a 70% is the cut score.” That would be arbitrary and unfair to the candidates and the public. The standard setting study should be performed on the final version of an examination form after all item and exam development is completed. Going through a standardized, non-arbitrary process (the most common of which is the modified-Angoff method) is how you set a defensible and fair passing standard.

This post is just intended to be an overview of each of the major steps in the certification exam development process. There are many considerations within each of these steps as to what should be done and how frequently. Make sure to ask a credentialing professional for some guidance.