NRPP – Working Toward ANSI Accreditation

NRPP is working toward ANSI/ISO 17024 Accreditation for Personal Certification Bodies and several steps have already been completed. First, there was the Certification Scheme and Job Task Analysis that was conducted by subject matter experts (SMEs) in Indianapolis late last August. In case you missed it, the SME’s determined the need for four initial certifications, two for residential radon measurement and two for residential mitigation. They are:

• Radon Measurement Field Technician
• Radon Measurement Professional
• Radon Mitigation Installer
• Radon Mitigation Specialist

The Measurement Field Technician and Mitigation Installer are entry-level or employee certifications, specifically designed to assess the competency of persons doing specific onsite tasks and who must work under the general supervision of a Measurement Professional or a Mitigation Specialist. In addition to oversight, the Professional and Specialist would be responsible for implementing and documenting direction, quality assurance procedures, system design and final reports.

The Job Task Analysis also determined the need for developing Master-Level or Endorsement Certifications for advanced projects like Multifamily Measurement or Mitigation, Schools and Large Buildings, Radon in Water and perhaps Vapor Intrusion Prevention.

During the JTA Workshop, a set of Job Task Domains, Sub-domains and Knowledge, Skills and Abilities (KSAs) statements were produced for each of the proposed four certification levels.

After the scheme and JTA were approved by the NRPP Certification Board, our consultants at Dainis and Company conducted an online survey of certified professionals in which participants were asked to weigh the domains and subdomains by their relative importance for a worker to be considered competent to perform those job tasks. Almost 400 certified professionals responded. With those responses, our psychometrician consultants created four exam blueprints.

What is a Psychometrician?

A psychometrician is someone who practices the science of educational and psychological measurement, or in other words, testing. Psychometricians measure the validity, reliability, and fairness of an exam program, and are an integral part in the process of creating valid and reliable language tests. They may be involved in all aspects of the development of the program, such as defining the KSAs that the exam is designed to measure; they are also involved in the development of the test specifications, the item writing process. While practicing or overseeing each of these areas, the psychometrician is highly concerned with ensuring that each part of the process conforms to testing industry standards, such as those in the Standards for Educational and Psychological Testing. These standards are considered the primary source for test publishers, test takers, and test users.

Next came the Question Item-Writing Workshop at the end of May. Our psychometricians hosted a webinar to familiarize the Subject Matter Experts (SMEs) with instructions. Candidates should not be penalized for taking harder versions of an examination, nor rewarded for taking an easier version. Test developers can control for statistical equivalence when constructing the exam by careful selection of the items. By using the beta test results, items can be selected so that pre-equated tests forms are generated. experienced field practitioners on the fundamentals of creating fair and valid test questions.

For example:

• The correct answer must be technically correct and accurate, with no room for debate. Part of the process is to eliminate ANY possible way someone could argue for an answer other than the one you deem correct.
• Questions must be meaningful, job-related and stick to the exam blueprint. Items should match the objectives in the blueprint and not use jargon, slang or cultural references.
• No trick questions! We want to test someone’s skills or knowledge, not send them on a mental roller coaster ride.
• Questions should not contain extra wording. (They should be fair to those candidates for whom English is a second language.)
• Item writers must avoid showing what THEY know instead of trying to find what the Candidate knows.
• The stem or scenario must not give a clue to the answer, but the item must include all information required to answer the question. Candidates should be able to answer based on the information provided and make no assumptions.
• Good exams avoid using compound answers. An example of this would be “red and blue” or “menus and radio buttons.” If this is a “choose two” question for example, a candidate could argue that by choosing “red and blue” they chose two answers.
• Writers should avoid “not” questions or negative phrasing in the scenario or stem. This is like tossing out a mental hand grenade or a speed bump at the candidate. Instead of asking, “Which feature is not supported?” turn it around and ask, “Which three features are supported?”
• No trivia questions! A technical professional knows how to find information in a manual or technical standard or use a help file. We need to ask about something one should know to perform the job.

The ANSI Accreditation process requires our exams to be professionally evaluated. While our existing exams were written by well-intentioned, knowledgeable subject matter experts, to earn ANSI accreditation, psychometricians are required to make sure every question is fair, impartial and well written.

Assemble and Deliver Beta Exams

Once reviewed, edited, and approved, items are placed in the item pool. The item pool or item bank is a depository of all items that are viable candidates for the examination. Beta examinations are currently being conducted during a limited time period. A main goal of the beta exam is to field test the entire item bank. While this may require multiple beta exam forms, it will provide us as many items as possible to use when developing the operational forms.

Analyze Beta Exam Results

Item data are reviewed to see if they performed as intended. As a minimum, statistics are reviewed from a psychometric perspective to determine item difficulty and item discrimination.

Item Difficulty is measured by the proportion of candidates who answer the question correctly. In other words, the correct response option for an item should be chosen more frequently than the incorrect options.

Item Discrimination is measured by distinguishing responses between low scoring candidates and high scoring candidates. If low scoring candidates are getting an item correct, while the high scoring candidates are missing it, there may be a problem. For example, perhaps the wording of the item results in higher scoring candidates misinterpreting the question and selecting the incorrect response, while lower performing candidates answered it correctly. This case is called negative discrimination. The goal is to have positive discrimination where higher performing candidates answer the question correctly while lower performing candidates miss it. As a result, the question has the predictive ability of total exam performance.

Construct Equivalent Exam Forms

While designing the exam to meet the specifications of the test blueprint, our psychometricians strive to maximize the reliability of the examination. This assures that the same results could be replicated if the same candidates were tested again under similar circumstances.

For many reasons like test security and repeat test takers, it is necessary to have multiple forms of an exam, making its critical that the exams are operationally equivalent from a both a content and statistical perspective, as well as being reliable measures. In other words, no matter which form a candidate receives, the candidate will have the same number of questions on a topic as a candidate receiving a different form.

We also must ensure statistical equivalence, so candidates are not be penalized for taking harder versions of an exam, nor rewarded for taking an easier version. Our psychometricians can control for statistical equivalence when by carefully selecting questions using the beta test results.

Establish the Passing Scores

After the exams are constructed, the passing score for each exam must be determined. In accordance with testing guidelines, pass/fail standards must be established in a manner that is generally accepted as being fair and reasonable, rather than arbitrarily set.