How Written Examinations Are Developed
The American Board of Orthopaedic Surgery (ABOS) administers computer-based written examinations as a part of initial Certification and for Recertification at the end of a Diplomate’s Maintenance of Certification (MOC) cycle. Additionally, Subspeciality Certification in Surgery of the Hand and Orthopaedic Sports Medicine are earned by successfully completing a computer-based examination. Producing the final product of a scored examination requires a great deal of expertise, time, effort, and expense.
The first step in creating a valid examination is the development of an examination blueprint. The ABOS constructs blueprints for its examination content based on the opinions of subject matter experts within that subspecialty nominated by partnering associations and societies. This process ensures that the examination content accurately represents what is being done by practicing orthopaedic surgeons. The blueprint is periodically reviewed and modified to accurately reflect changing trends in practice. (To learn more about blueprint development, click here to read a journal article on rapid blueprinting.)
The ABOS contracts with the American Board of Medical Specialties (ABMS) in the test development process, specifically the item-writing process by which examination questions are written, edited, and placed into a final examination. Over many years the ABOS has created a significant bank of questions. New questions are composed by volunteers of our Question Writing Task Force (QWTF). The QWTF is made up of approximately 40 members, all experienced orthopaedic surgeons, who each submit 8 to 12 new questions annually. In addition to images and/or diagrams, each question is accompanied by the appropriate references. While some members of the QWTF have experience of as much as 20 years or more, the ABOS ensures that new volunteers are added each year. Some years, members spend time reviewing old questions to keep the bank up to date.
New questions are reviewed by professional editors prior to a two-day QWTF meeting each spring. At the QWTF meeting, the new questions are reviewed and discussed among orthopaedic surgeons who specialize in the subspecialty corresponding to the questions content area. If a question is approved by this peer review, it will be added to the bank of questions. Each new question accepted, along with questions that have performed well on previous examinations, is designated for use on one or more examinations, including the Part I Certifying Examination or any of the current existing Subspecialty/Practice-Profiled Recertification Examinations.
The Part I Examination is then edited during a one-day meeting by the ABOS Field Test Task Force (FTTF), which is comprised of approximately 20 ABOS Diplomates. Another level of editing is performed by the ABOS Written Examination Committee, followed by a final editing by the Chair of the Written Examination Committee. Prior to being administered at testing centers in North America and Canada, the questions on the examination have gone through five levels of editing. In addition, all questions in the bank are re-edited on a rolling three- to five-year schedule to ensure that each question remains current, applicable, and accurate.
There are 10 Practice-Profiled Examinations available for Recertification in the areas of General Orthopaedics, Orthopaedic Sports Medicine, Orthopaedic Trauma, Foot and Ankle Orthopaedics, Surgery of the Spine, Pediatric Orthopaedics, Muskuloskeletal Oncology, Surgery of the Hand, Adult Reconstruction and Shoulder and Elbow. These examinations consist of questions only within that subspecialty and also go through the same blueprint, question writing, and examination review process. The Part I examination is administered locally at Prometric Testing Centers one day per year. Diplomates choose the location that is best for them. Candidates cannot register for the Part I examination unless they have paid the examination fee. and then receive a scheduling permit. The ABOS highly recommends that Candidates review the examination tutorial on their own computer so they can familiarize themselves with the user interface employed by Prometric.
Beginning in 2023, Subspeciality and Practice-Profiled/Recertification examinations are administered at Pearson Professional Centers. Diplomates will schedule directly with Pearson after paying the examination fee. The ABOS recommends that Diplomates review the examination tutorial on their own computer so they can familiarize themselves with the user interface employed by Pearson.
After the examinations are administered, psychometricians summarize the statistical performance of each examination question. A Key Validation Subcommittee reviews the data and poorly performing questions are deleted before scoring. The remaining questions are reviewed in a Standard Setting Exercise. A group of Board Certified, practicing orthopaedic surgeons volunteer to take the examination, review the statistics of each question, and determine the difficulty of each item (Anghoff method). The psychometricians then analyze the degree of difficulty of each question and the overall examination itself to ensure that the examination is valid, reliable, and produces scoring that is scalable from one year to the next. With this information, the ABOS Written Examination Committee meets for a Standard Setting Call to determine the passing standard. Examinees whose score meets or exceeds the standard will pass the examination. Thus, the examination is developed so that a Candidate or Diplomate would have the same statistical likelihood of passing or failing no matter the year the examination is taken. The ABOS does not set an over-all pass rate, only a standard that can be scaled across numerous years.
As the process outlined above shows, there are many steps to be completed. These steps, in addition to quality control measures taken by the ABOS staff, are the reason it takes over a month, and sometimes up to two months from exam administration to posting of results to an examinee’s ABOS Dashboard.
