How the ABOS Part I Examination is Developed
Producing the American Board of Orthopaedic Surgery (ABOS) Part I Examination requires a great deal of expertise, time, effort, and expense. The ABOS depends on a dedicated group of staff, business partners, and volunteers.
The first step in creating a valid examination is the development of an examination blueprint. The ABOS Part I blueprint is shared with the American Academy of Orthopaedic Surgeons (AAOS) Orthopaedic In-Training Examination (OITE). The ABOS constructs examination blueprints with diverse groups of subject matter experts from across the country. In the case of the ABOS Part I Examination Blueprint, the subject matter experts include many orthopaedic surgery residency program directors and representatives from the AAOS Education Council. This process ensures that the ABOS Part I Examination content accurately represents what orthopaedic surgeons completing residency should know. The blueprint is cyclically reviewed and modified to accurately reflect changing trends in practice.
The ABOS contracts with the American Board of Medical Specialties (ABMS) in the test development process, specifically the item-writing process by which examination questions are written, edited, and placed into the ABOS Examination Item Bank and ultimately placed on the final version of each examination. Over many years, the ABOS has created a large bank of questions. New questions are composed by volunteers of the ABOS Question Writing Task Force (QWTF). The ABOS QWTF is made up of approximately 40 members, all experienced ABOS Board Certified orthopaedic surgeons, who each submit up to 12 new questions annually (in addition to reviewing questions in the Item Bank on a regular basis). In addition to images and/or diagrams, each question is accompanied by the appropriate references. While some members of the ABOS QWTF have experience of as much as 20 years or more, the ABOS ensures that new volunteers are added each year. Some years, members spend time reviewing old questions to keep the bank up to date.
New questions are reviewed by professional editors prior to the ABOS QWTF meeting. At the ABOS QWTF meeting, the new questions are reviewed and discussed among orthopaedic surgeons who specialize in the subspecialty corresponding to the question content area. If a question is approved by this peer review process, it will be added to the Item Bank. Each new question accepted, along with questions that have performed well on previous examinations, is designated for use on one or more examinations, including the ABOS Part I Examination or any of the ABOS Subspecialty Certification or Practice-Profiled Recertification Examinations.
The ABOS Part I Examination is constructed based on the blueprint using items from the Item Bank. It is then administered and reviewed accordingly during a 1-day meeting by the ABOS Field Test Task Force (FTTF), which is comprised of approximately 20 ABOS Diplomates. Another level of review is performed by the ABOS Written Examination Committee, followed by a final review by the Chair of the Written Examination Committee. Prior to being administered at testing centers, the questions on the examination have gone through five levels of review. In addition, all questions in the bank are re-reviewed on a rolling 3- to 5-year schedule to ensure that each question remains current, applicable, and accurate.
Candidates take the ABOS Part I Examination in July, with an alternate date in August available for those with life events.
After the examination is administered, psychometricians summarize the statistical performance of each examination question. A Key Validation Subcommittee reviews the data, and poorly performing questions are deleted before scoring. The psychometricians then analyze the degree of difficulty of each question and the overall examination itself to ensure that the examination is valid, reliable, and produces scoring that is scalable from 1 year to the next. With this information, the ABOS Written Examination Committee meets to review the analysis data and determine the passing standard. Examinees whose score meets or exceeds the standard will pass the examination. Thus, the examination is developed so that a Candidate would have the same statistical likelihood of passing or failing no matter the year the examination is taken. The ABOS does not set an over-all pass rate, only a standard that can be scaled across numerous years.
Once the passing standard score is set, letters are drafted to each Candidate letting them know if he or she is successful. In addition, all Candidates receive a report showing how they performed in the various content areas.
The entire process is monitored by the ABOS Associate Executive Director and the ABOS Written Examination Committee. The goal is a fair, standardized, valid method of measuring orthopaedic knowledge.
