Did you ever wonder how a national certification examination is developed? If so, take a minute to read this article from Vita Greco, MA, Director of Psychometric Services, Professional Testing Corporation(PTC). In this article Ms. Greco, details the process and important steps to ensure transparency and validity of the examination. Professional Testing Company is the organization the Patient Advocate Certification Board worked with on the development of the Patient Advocate Certification Examination. Take a few minutes to read this article and let us know if you have any questions or comments. 

Understanding the Process of How a Certification Examination is Developed

by Vita Greco, MA, Director of Psychometric Services, Professional Testing Corporation

Most professionals navigating their career paths in today’s challenging and competitive workforce will find themselves considering the pursuit of certification. For many, the decision to seek certification may not be an easy one. The level of commitment, time, and cost of obtaining certification is substantial and many wonder if it is worth the effort. However, once professionals educate themselves on the purpose and process of certification, the true value and return on investment can really be appreciated.

To become certified, candidates must meet education and experience requirements and achieve a passing score on a certification examination. Many find that the examination is more challenging than they originally anticipated. It is important to keep in mind that the examination is based on all aspects of the profession. Most practitioners, especially seasoned professionals, have specialized experience, and to have a full understanding of the knowledge the examination is testing, it is important to adequately prepare by reviewing the published content outline for the examination and any recommended references or reading material.  Some also take practice tests to assess their readiness for the certification examination and others may search for review courses or form study groups with other professionals.

It is also important to take into consideration the rigorous process that goes into developing the examination. Similar to how a strong foundation is critical when building a home, the same is true for a certification program. The foundation that a certification program is built on is a job analysis (also known as a practice analysis or role delineation). During this process, a comprehensive review of the profession is conducted and the major areas of responsibility (domains of practice), specific work-related tasks that are associated with those domains, and the knowledge and skills to perform those tasks are identified and then validated, typically through a survey of practitioners. The results of the job analysis are then used to develop the content outline with test specifications. This serves as the “blueprint” for which items are written and examinations are constructed. Since the items on the examination are linked back to the profession through the job analysis, the job analysis provides necessary evidence that the examination is job-related and content-valid. It is important to repeat this process every few years so that it accurately reflects the scope of practice as the profession evolves over time.

The next step is to write and review items (test questions). This process typically begins with a testing professional, such as a psychometrician, conducting item writing training that prepares subject matter experts (SMEs) to write items effectively. SMEs will write practice-related items that target the knowledge specified in the content outline.  Items then go through several stages of review including psychometric review followed by a content review by another panel of SMEs representative of the candidate population. This panel edits and reviews each item to confirm that the knowledge being tested is accurate, reflective of current, best practice as delineated in the content outline, relevant and important to practice, and is free from bias and stereotyping. Approved items are then reviewed for grammar and style and entered into an item bank. To ensure that the item bank continues to reflect current, best practice of the profession, it is important to continuously refresh the item bank with new items by conducting periodic item writing and review initiatives.

Following the item writing and review process, examination drafts are constructed by selecting items from the item bank in proportion to the weightings of each content area as indicted in the test specifications of the content outline. A panel of SMEs will then critically review each item to confirm that all items reflect current, best practice; accurately represent content as delineated in the content outline; have one and only one correct or best answer with plausible distracters; adhere to item writing guidelines; and are appropriate for the candidate population as described in the eligibility criteria. A final review for grammar is done and the final version of examination is then produced and made ready for the administration.

Once examinations are approved, the passing score (also referred to as a standard, passing point, or a cut-score) must be determined. The passing score represents the lowest score on the examination that represents success. It is the minimum level of knowledge that must be demonstrated by a candidate to ensure competency. The passing score is the basis to which pass and fail decisions are determined, so a defensible passing standard is essential. There are typically two different types of approaches that are used to set the standard for examinations – a relative (norm-referenced) testing standard or an absolute (criterion-referenced) testing standard. Using the norm-referenced approach, a normal distribution of scores is assumed, based on a bell-shaped curve, and the standard is set by holding the passing rate consistent from administration to administration. In other words, the actual number of items required to pass the examination may vary depending on the level of competency of the group testing. One of the biggest misconceptions among candidates is that all certification examinations use this method to make pass/fail decisions, when in fact most certification examinations use a criterion-referenced standard, such as the modified Angoff method. A criterion-referenced standard is set by determining the total number of items that must be answered correctly to pass the examination.  Criterion-referenced standards are based on achieving a specific score set before the examinations are administered, do not depend on the relative ability of other candidates’ scores, and do not have a pre-determined passing rate.

After examinations are administered, item analyses and summary statistics are produced. Item performance statistics, such as item difficulty and discrimination indices, are reviewed.  Items that exhibit problematic statistics are flagged and presented to SMEs for further review to determine if any changes to the scoring key should be made. These statistics are also helpful when reviewing and revising the question for future examination forms. Once the SMEs have completed their review of the flagged items, the examinations are scored and pass and fail score reports are sent to candidates.

Certification does not end with passing the examination. Candidates who successfully meet the requirements and pass the examination are granted certification for a specific period of time and then must maintain their certification by renewing it every few years. This process promotes continued competency and life-long learning. Not only do candidates have to meet education and experience requirements to recertify, but they also must demonstrate that they are keeping current in the profession by either passing the examination again or by achieving a specific number of continuing education credits that they have obtained since initially passing the examination.

Those seeking certification may experience tangible benefits, such as career advancement (i.e., new job opportunities or promotions), increase in salary, and formal recognition from employers and peers. They may also experience intangible benefits, such as feelings of personal accomplishment/satisfaction and professional growth. Not only is achieving certification a way to demonstrate commitment to the profession, it also allows the certificant a way to stand out from other practitioners. Successfully earning certification shows that the practitioner has not only taken the initiative to seek certification but has also met specific standards in the industry as set by the certifying organization. Especially for seasoned professionals, seeking and maintaining certification indicates that the knowledge and skills that they have are current in practice.

The benefits to certification are not solely for the certificant. Benefits to certification are experienced by other stakeholders as well. For example, certification programs not only improve the industry by establishing professional standards that can be utilized to build a more qualified workforce, they also help those hiring professionals to make better informed decisions when filling positions, and they help protect the public by providing a way to benchmark competence among practitioners. By taking that step to seek certification, it not only helps propel the professional into a brighter future, but it also raises the bar for the industry and takes the profession to new levels.

Bio: Vita Greco has over 14 years of experience in the administration and development of certification and licensure programs. Her expertise includes managing certification programs of all sizes and industries, launching new credentials, item and exam development, standard setting, analyzing statistical reports, accreditation compliance, marketing, strategic planning, and advising clients on all aspects of their assessment programs. Vita has an M.A. in Industrial/Organizational Psychology from Hofstra University.

About Professional Testing Corporation

PTC is an organization whose services are focused on the measurement aspects of human resources, including the design, development, and administration of tests and testing programs for professional organizations. PTC’s services to professional organizations include test development, candidate application and fee processing, scheduling, testing center arrangements, test administration and scoring, secure client portals, and data analysis and reporting. Services also include item banking, development and management of certificants databases, survey and questionnaire studies, accreditation assistance, board management, job analyses, self-assessment, and psychometric research in areas of interest to PTC clients. To learn more, visit the PTC website at http://www.ptcny.com.