Why such studies are important for examination and curriculum development
This article has been adapted from a presentation given by AAMA Chief Executive Officer Balasa in April at the Health Professions Network (HPN) 2015 Spring Conference,
Why Competencies Matter, in Grand Rapids, Michigan.
In the realm of professional and occupational testing and credentialing, it is axiomatic that the content of a licensure or certification examination must be linked to the knowledge and skills required for safe and effective performance in the profession or occupation. Note the following from the 2014 Standards for Educational and Psychological Testing:
When test content is a primary source of validity evidence in support of the interpretation for the use of a test for employment decisions or credentialing, a close link between test content and the job or professional/occupational requirements should be demonstrated.
Comment: … When evidence of validity based on test content is presented for a job or class of jobs, the evidence should include a description of the major job characteristics that a test is meant to sample. It is often valuable to also include information about the relative frequency, importance, or criticality of the elements. For a credentialing examination, the evidence should include a description of the major responsibilities, tasks, and/or activities performed by practitioners that the test is meant to sample, as well as the underlying knowledge and skills required to perform those responsibilities, tasks, and/or activities.1
Another way to state this fundamental principle is that a professional or occupational credentialing examination must be based on a robust and psychometrically defensible occupational analysis (also known as a role delineation study, job task analysis, or practice analysis) in order for it to differentiate between those candidates who are sufficiently knowledgeable in the profession or occupation, and those who are not. The Accreditation Standards of the Accreditation Board for Specialty Nursing Certification (an accreditor of nurse specialty certification programs) articulate this principle as follows:
Standard 7: Validity
The certifying organization has conducted validation studies to assure that inferences made on the basis of test scores are appropriate and justified. …
Several measures can be taken to promote the content validity of a certification examination program. One of the most important of these is conducting a practice analysis (also referred to as job analysis, job task analysis, or Role Delineation Study (RDS)). … The practice analysis should define the tasks of a particular job as well as the knowledge required to perform the tasks competently. … Linking this information to the examination content is of key importance.2 [Emphasis added.]
Another accreditor of certification programs is the National Commission for Certifying Agencies (NCCA). Note the following from the NCCA Standards for the Accreditation of Certification Programs:
Standard 14: Job Analysis
The certification program must have a job analysis that defines and analyzes domains and tasks related to the purpose of the credential, and a summary of the study must be published. …
2. Validation of the delineated domains, tasks, and associated knowledge and/or skills is typically accomplished by surveying current certificants and/or a representative sample of the population that is the intended target audience for the certification.
3. Validation surveys should include rating scales specifically selected and tailored as necessary to assess the critical domains and tasks (and associated knowledge and/or skills if included) to be examined.
4. It is important for surveys to sample broadly within the population as defined by the program to ensure representation by key characteristics, such as major practice area, job title, work setting, geography, ethnic diversity, gender, years of work experience, geographic region (including international, if applicable), and other demographic variables. [Emphasis added.] Stakeholders such as educators, supervisors, and employers may be included, if appropriate. The population from which the sample is drawn should be clearly defined, justified, and related to the purpose of the credential. The sample size and methods by which it is drawn should be psychometrically defensible.3
The importance of basing curriculum requirements on the needs of employers is becoming recognized in the realm of programmatic and specialized academic accreditation.
Note the following from the Commission on Accreditation of Allied Health Education Programs (CAAHEP) Standards and Guidelines for the Accreditation of Educational Programs:
II. Program Goals
A. Program Goals and Outcomes
There must be a written statement of the program’s goals and learning domains consistent with and responsive to the demonstrated needs and expectations of the various communities of interest served by the educational program. The communities of interest that are served by the program must include, but are not limited to, students, graduates, faculty, sponsor administration, employers, physicians, and the public. [Emphasis added.]
Program-specific statements of goals and learning domains provide the basis for program planning, implementation, and evaluation. Such goals and learning domains must be compatible with the mission of the sponsoring institution(s), the expectations of the communities of interest, and nationally accepted standards of roles and functions. Goals and learning domains are based upon the substantiated needs of health care providers and employers, and the educational needs of the students served by the educational program. [Emphasis added.]
B. Appropriateness of Goals and Learning Domains
The program must regularly assess its goals and learning domains. Program personnel must identify and respond to changes in the needs and/or expectations of its communities of interest.4
The Accrediting Bureau of Health Education Schools (ABHES) Accreditation Manual, 17th edition, contains the following language:
V.A.2. Program goals are in keeping with the requirements of the profession.
Program goals are written in a manner to ensure that the curriculum is current with industry standards, meets the demands of the communities of interest (e.g., students, graduates, employers, and the public) and that students obtain related hands-on training, if applicable, that enables them to obtain viable employment in the field.5 [Emphasis added.]
Although an occupational analysis is not mentioned specifically in the above excerpts, the inclusion of employers as a community of interest, and the references in the CAAHEP Standards to “nationally accepted standards of roles and functions,” and “the substantiated needs of health care providers and employers,” point out the need for academic accrediting bodies to incorporate the findings of an occupational analysis into their accreditation standards.4
An occupational analysis that ascertains the knowledge and skills necessary for safe and effective practice in a profession or occupation is essential for both licensing and certifying testing bodies, and also for accreditors of academic programs. Related issues will be presented in future Public Affairs articles.
Questions? Contact Donald A. Balasa, JD, MBA, at email@example.com or 800/228-2262.