Strengthening the planning and design of Objective Structured Clinical Examinations

Background Although Objective Structured Clinical Examinations (OSCEs) offer innovative, objective, and fair methods of clinical assessment, their quality is compromised by poor planning and design. Aim This study aimed to describe the development and present evidence-based recommendations on strengthening the planning and design of OSCEs for a South African public College of Nursing. Setting A South African public College of Nursing. Methods Recommendations were developed based on synthesising two sets of qualitative data. Set 1 included two main themes with each of the four sub-themes related to barriers and facilitators towards quality in OSCE designs from 14 nurse educator interviews and 15 external moderator reports. Set 2 included 12 quality measures to be adopted in the quality design of OSCEs derived from an integrative literature review. The draft recommendations were reviewed by eight experts to be finalised. Results Seven recommendations were developed for strengthening OSCEs’ planning and design, related to: (1) policy framework, standard operating procedures and stakeholder code of conduct; (2) blueprinting and mapping of the OSCE content; (3) developing a bank of OSCE stations; (4) scoring rubric and standard-setting method selection; (5) examiners and standardised patients’ recruitment and training; (6) venue selection; and (7) station piloting. Conclusion The seven developed recommendations can strengthen the quality of OSCEs in the South African public College of Nursing context. Contribution The developed recommendations can assist nurse educators in planning and designing to conduct quality OSCEs following piloting and implementation.


Introduction
Objective Structured Clinical Examinations (OSCEs) represent a dynamic assessment approach for formative and summative clinical evaluations.They entail a series of time-constrained stations where students undertake structured tasks while undergoing independent assessment by examiners within a secure setting (Castellani et al. 2020).Objective Structured Clinical Examinations promote evaluating affective, cognitive, and psychomotor learning domains on large student numbers within a short time using realistic clinical scenarios in a safe simulation laboratory environment (Castellani et al. 2020).In the affective domain, students' attitudes, values, and professionalism are assessed, allowing educators to gauge their interpersonal skills, empathy, and ethical decision-making.Within the cognitive domain, OSCEs evaluate students' knowledge, understanding, and critical thinking abilities related to clinical concepts, diagnostic reasoning, and treatment planning.Furthermore, in the psychomotor domain, students' physical skills, procedural competency, and clinical techniques are scrutinised, providing insights into their proficiency in executing clinical tasks effectively.By encompassing these diverse domains, OSCEs offer a holistic approach to assessing students' competency and readiness for clinical practice (Castellani et al. 2020).In an OSCE, students perform similar clinical tasks, and their performance is graded against a predetermined scoring rubric, thereby promoting uniformity, validity, and reliability of assessment (Cheema & Ali 2021).Objective Structured Clinical Examinations, as compared to traditional assessment methods such as mini clinical evaluations and case-based discussions, are innovative and provide for objectivity, standardisation and fairness and are, therefore, the preferred method for clinical assessment (Malau-Aduli et al. 2022).
Background: Although Objective Structured Clinical Examinations (OSCEs) offer innovative, objective, and fair methods of clinical assessment, their quality is compromised by poor planning and design.

Strengthening the planning and design of Objective Structured Clinical Examinations
Read online: Scan this QR code with your smart phone or mobile device to read online.
While OSCEs have numerous benefits compared to traditional methods of clinical assessment, they require considerable resources, including time, personnel, and facilities.They are therefore expensive and require substantial preparation and planning (Bdair, Abuzaineh & Burqan 2019;Cheema & Ali 2021).Furthermore, maintaining consistent standardisation across multiple stations and examiners can be challenging, which may result in variations in scoring and evaluation.Despite efforts to simulate clinical scenarios, OSCEs may lack the complexity and realism of actual patient encounters, limiting their ability to assess clinical competence fully.
The timed, high-stakes nature of OSCEs can cause stress and anxiety in students, potentially impacting their performance and distorting assessment outcomes (Bdair et al. 2019).
Poorly planned and designed OSCEs and human errors may compromise accurate implementation of the examinations, leading to questionable credibility, uniformity, validity and reliability of results (John et al. 2021).Therefore, OSCE planning and design must be based on robust research evidence to strengthen their quality, credibility, and legal defensibility.
Aligning OSCE design with international academic standards, and evidence-based recommendations could assist in this regard (John et al. 2021;Krusen & Rollins 2019).
Globally, evidence-based recommendations and guidelines for implementing OSCEs are available (Dewan, Khalil & Gupta 2024).However, these guidelines do not consider resource-constraint settings and are therefore not adaptable to the study context (a South African public College of Nursing).In the research context, the authors observed that the planning and design phase of OSCEs is often compromised, particularly among campuses with limited resources.For example, campuses lacking adequate spacing, trained examiners, and manikins struggled to plan and design real-life scenarios effectively.Additionally, at the time of the study, there were no existing evidence-based recommendations or guidelines to assist nurse educators in strengthening the planning and design of OSCEs, leading to inconsistencies in examinations.Therefore, the study aims to develop and present evidence-based recommendations (hereby referred to as 'recommendations') to strengthen the planning and design of OSCEs for a South African public College of Nursing.

Research design and methods
A qualitative research design was employed, incorporating multiple methods such as individual semi-structured interviews with nurse educators and a document analysis of data extracted from external moderator reports.In addition, an integrative literature review summarising existing literature on best practices for managing the quality of OSCEs in health science education was conducted.The qualitative data from the interviews and document analysis, along with the findings from the integrative literature review, were synthesised to develop comprehensive recommendations for enhancing the design of OSCEs at a South African public College of Nursing.This article that forms part of a larger doctoral study that developed a best practice guideline managing quality OSCEs at a South African public College of Nursing, focusses on the planning and design of an OSCE.

Setting
The

Methods
The methods used in the qualitative interviews, document analysis and integrative literature review are explained as follows.

Qualitative interviews
Fourteen nurse educators (N = 14), each with at least 2 years' expertise in nursing education and who were involved in conducting OSCEs from the five campuses, were purposely selected to participate.Face-to-face, individual interviews were conducted and audio-recorded by the first author -a male nurse educator and master's degree holder with experience in conducting qualitative interviews -to obtain interviewees' experiences regarding quality OSCE management at the college under study.The majority (n = 13) of the 14 participating nurse educators were female, with ages ranging from 33 to 60 years, with between 3 and 30 years of work experience.
The first three interviews conducted with nurse educators from Campus Five served as the pilot study.As the first author was employed at this campus, pilot interviews were conducted by the third author -a female nurse educator with a PhD in Nursing Science -who was neither affiliated with the campus nor had a pre-established relationship with any participants.No changes were needed to the interview schedule and the data were included in the main study.
Interviews took between 60 and 90 min each and were conducted in a venue and at a time that were convenient for the participants.Data were collected during a 2-month period in 2019.Immediately after the interviews, the first author transcribed the recordings verbatim and coded the data with the assistance of an independent coder.The data were categorised into two themes with four sub-themes each, related to facilitators and barriers of quality OSCE designs, according to Tesch's data analysis method.Tesch's data analysis method involved systematically coding qualitative data by organising, categorising, and interpreting themes by the first author, with assistance of the co-coder to derive meaningful insights (Weyant 2022) (see Table 3 for the themes).

Document analysis
Following the completion of the interviews, the first author examined the documents.These documents consisted of reports provided by external moderators regarding the college's OSCEs.The selection process involved picking a deliberate set of 30 extensively detailed reports from external moderators (n = 30).To extract information about comments related to the OSCEs' planning and design, a data extraction tool developed by the first author was utilised (see Table 1 with an example).
The point of data saturation was achieved after scrutinising 15 of the external moderator reports (n = 15).The process of thematic analysis was conducted by the first author in collaboration with an independent coder.This involved thoroughly reviewing the extracted data, performing manual coding, and categorising the content into coherent topics.These topics were further organised into three overarching themes and their respective sub-themes.As a result of the large overlap with the data from qualitative interviews, the themes derived from the document analysis were merged with those of the qualitative interviews, resulting in two main themes with four sub-themes, each related to facilitators and barriers of quality OSCE planning and design (see Table 3).

Integrative literature review
An integrative literature review was conducted to summarise existing literature regarding best practices for managing the quality of OSCEs in health science education.Table 2 outlines the search strategy employed.
The first and third authors independently screened titles and abstracts as well as the full text of obtained literature, according to the following criteria governing literature selection: research and non-research documents, as well as grey literature regarding the quality management of OSCE design in health science education, published in English, between January 2010 and March 2021.
Thirteen eligible full-text articles were critically appraised.
The first author and an independent reviewer assessed each article using one of two Johns Hopkins appraisal tools: one for research evidence and one for non-research evidence.
Each article received a score to determine its inclusion or exclusion in the data synthesis and extraction process.This score was calculated by dividing the number of 'yes' responses on the critical appraisal tool by the total number of items on the tool, and then multiplying by 100.As a result of the limited number of articles available on the topic and to ensure the inclusion of relevant articles with sufficient rigour, a minimum score of 60% was set as the threshold for article selection.The first and third authors independently extracted data from all 13 articles.Extracted data were synthesised using thematic analysis according to Cooper (1998).The extracted data were read, compared, ordered, and coded.The coded data were categorised and grouped under themes and sub-themes.A total of 12 quality measures to be adopted in quality OSCE planning and design were identified (see Table 3).

Synthesising the qualitative data into draft recommendations
Synthesising the qualitative data into draft recommendations was carried out using data source triangulation.Data source triangulation refers to a combination of different sources (such as interviews and literature) wherein researchers synthesise both similarities and disparities to arrive at conclusions aligning with the findings according to Carter et al. (2014), as follows: 1. Identifying the sets of themes (two sets of themes were derived) • Set one: Three main themes with four sub-themes each from the qualitative interviews and document analysis related to measures that facilitate and barriers that hinder quality planning and design of OSCEs.• Set two: Twelve quality measures to be adopted in the quality planning and design of OSCEs from the integrative literature review.

Comparing themes:
The authors read and noticed overlaps, intersections, or complementary aspects between the two sets of themes.For example, a peer-review system (theme 1.1 interviews and document analysis) was  linked to conducting peer review workshops (Quality measure 6 -integrative literature review).

Synthesising themes:
The authors integrated, refined, and categorised the themes from both sets to derive a unified understanding of the phenomenon under study.This involved looking for patterns, relationships, and connections between the themes.For example, while specific quality measures were crucial for the quality planning and design of OSCEs, it was the alignment of these quality measures with facilitating measures in place and addressing the barriers that enhanced the quality of OSCE planning and design particularly for the study context.

Deriving and organising recommendations:
Once the integrated set of themes was established, the authors identified key insights and implications for practice or policy.These insights would form the basis of the recommendations.The authors organised the recommendations into a coherent framework, with a logic sequence enhancing the quality design of OSCEs.In this case, the recommendations were grouped according to overarching themes such as laying the foundation in terms of policy framework, standard operating procedures, and a code of conduct, followed by blueprinting and mapping, whereafter recommendations regarding the details of planning and designing the content of the OSCE were included, such as the OSCE bank, scoring rubrics, recruitment and training requirements, the OSCE venue and piloting.The recommendations largely followed the sequence of the identified quality measures from the integrative literature review as this was deemed logical in the planning and design of OSCEs, resulting in seven draft recommendations.Controlled Trials (RCTs) and evidence-based guidelines), Level II (well-designed RCTs), Level III (controlled trials without randomisation), Level IV (case-control or cohort studies), Level V (systematic reviews of descriptive and qualitative studies), Level VI (single descriptive or qualitative studies), and Level VII (expert opinions or committee reports), aiding in the assessment of the strength and reliability of evidence for clinical decision-making.

Recommendations
The recommendations will now be described.

Recommendation One: Develop a policy framework and standard operating procedures as well as an Objective Structured Clinical Examination stakeholder code of conduct
Rationale: Standardises OSCE designs, assists in preparing for a well-designed OSCE and provides guidance to OSCE stakeholders (students, nurse educators, external moderators) on what is acceptable behaviour and how to prevent and address unprofessional conduct and maintain discipline and OSCE credibility (Castro-Yuste et al. 2020;Khan et al. 2013).
Content experts (who may include internal staff and external moderators), with sufficient clinical, health sciences and nursing education experience and those acquainted with the curriculum, published principles and standards underlying OSCEs, that are appointed as part of the OSCE organising committee, must: Develop an OSCE policy framework and standard operating procedures, including: • The criteria for and establishment of an OSCE design committee • Standards on the minimum resources required to conduct an OSCE (e.g.physical space, type of fidelity manikins, trained examiners)

Recommendation Two: Conduct blueprinting and mapping of the Objective Structured Clinical Examination content
Rationale: Ensures the examination of correct standards and appropriate skill, aligning directly with the students' level of competence required at their level of training, and providing authentic clinical assessment opportunities, directly related to relevant aspects in delivering safe care and curriculum requirements (Hastie et al. 2014;Kelly et al. 2016;Mitchell et al. 2015;Obizoba 2018;Pell et al. 2010).
Content experts must: • Align OSCE content with curriculum learning objectives to ensure that all components of the curriculum are proportionally assessed (Ware et al. 2014) • Obtain consensus on sufficient OSCE stations that promote adequate content coverage aligned with students' required clinical competences (Goh et al. 2016;Ogah et al. 2016) • Use a prescribed method, tool, or instrument, selecting relevant OSCE station matter and ensure adequate inclusion of all learning domains and competences (Goh et al. 2016;Khan et al. 2013).This tool could include elements such as OSCE content per learning outcome, the number and duration of stations, station types for each clinical skill, weighting of OSCE content, instructions for students, examiners, and standardised patients as well as resource requirements such as manikins, standardised patients (Goh et al. 2016).• Verify the OSCE content blueprinting tool through evaluation by three independent content experts to ensure relevance and objectivity, then have it signed off and handed over to the station writers (Goh et al. 2016).

Recommendation Three: Develop a bank of Objective Structured Clinical Examination stations
Rationale: Improves OSCE efficiency, reliability and validity by: finding a suitable expert panel to design and write OSCE stations; providing an understanding regarding OSCE duration, resources required and strategies to be employed in an OSCE; and providing the opportunity for station writers to quality assure stations to confirm the clinical accurateness and suitability of the required clinical tasks by students (Khan et al. 2013).
Content experts should: • Develop and maintain an OSCE station bank and related tools (Khan et al. 2013;Ware et al. 2014) • Prior to incorporating an OSCE tool into the OSCE bank, it is recommended to undertake peer review, piloting, and psychometric analysis, as suggested by Khan et al. (2013) Once OSCE blueprinting and mapping have been finalised, content experts: • Should select station writers to design and develop stations using station writing templates • Should select an educational expert to coordinate the writing of stations and guide station writers regarding the required station types for assessing selected curriculum outcomes (Hastie et al. 2014;Khan et al. 2013).• To attain essential content representativity, a minimum of 10 and up to 20 stations, lasting between 5 and 10 min for each OSCE, should be prepared (Brannick, Erol-Korkmaz and Prewett 2011).• Selecting a station type (e.g., unobserved, observed, linked and technology-enhanced stations) and station numbers should be based on the OSCEs': ß Aim, Capacity to evaluate domains of learning, Ability to maintain uniformity and standardisation in all stations regarding the equipment, stock, stationery, scoring rubrics and scenarios as well as the available resources and conditions of an institution (Agarwal et al. 2010).• Peer review workshops are held for all station writers and nurse educators, enhancing capacity, to evaluate and critique (using a questionnaire to measure appropriateness and accuracy of the station content) and adapt stations written by their peers (Khan et al. 2013).

Recommendation Four: Select a scoring rubric and standard-setting method
Rationale: Enables examiners to allocate scores based on the demonstrated clinical skills and the intended objective of the OSCE, fostering equitable outcomes in pass or fail evaluations (Daniels & Pugh 2018;Kamal et al. 2020;Khan et al. 2013).
Station writers, appointed as part of the OSCE organising committee, must: • Choose a checklist when the aim of the OSCE is to evaluate the actions anticipated from students at each station (Hastie et al. 2014;Ware et al. 2014).
• Choose a holistic or global rating scale when the objective of the OSCE is to assess the execution of a specific action (Khan et al. 2013;Schleicher et al. 2017).• Preferentially employ the criterion-referenced approach as a standard-setting method to ascertain the complexity and significance of each OSCE item and to establish threshold scores aligned with the necessary student proficiency (Hastie et al. 2014;Khan et al. 2013;Yousuf, Violato & Zuberi 2015).• In the selection of the scoring, consider overriding factor as well as the assessment of the various domains including affective, cognitive and psychomotor skills.

Recommendation Five: Train recruited examiners and standardised patients
Rationale: It allows examiners to evaluate students consistently and objectively, while also helping standardised patients understand the significance of providing precise and dependable portrayals of the required clinical scenarios, repeatedly for each student participating in an OSCE (Gormley 2011).
In terms of training examiners -who will be assessors during the OSCEs and are sourced internally within the staff complement as well as externally from staff in nearby clinical facilities -prior to the OSCEs, content experts should: • Develop an OSCE examiner training programme to: ß train examiners according to minimum assessment standards ß provide examiners the chance to practise scoring utilising appropriate scoring rubrics (Gormley 2011).• Document the outcomes of the training workshops (Hastie et al. 2014).
In terms of training standardised patients, the OSCE organising committee: • Should implement a robust training programme, which can include role play, equipment, and devices to simulate authentic medical interactions (Daniels & Pugh 2018).• Following training, it is essential to assess the performance of each standardised patient before permitting them to participate in OSCEs (Khan et al. 2013).

Recommendation Six: Select an appropriate Objective Structured Clinical Examination venue
Rationale: To have a dedicated, custom-built venue to run OSCEs in an appropriate way (Ware et al. 2014).
The OSCE organising committee should: • Ensure that the venue -such as a custom-built skills laboratory -is mapped, considering the station placement and type (e.g. with patients, manned, unmanned) and flow patterns (Ware et al. 2014)

Recommendation Seven: Conduct an Objective Structured Clinical Examination station piloting
Rationale: To recognise and rectify practical discrepancies, ensuring impartiality and feasibility of assigned tasks within the recommended timeframe (Ware et al. 2014).
The OSCE organising committee should: • Prior to finalising the stations into the OSCE, conduct station piloting.This involves familiarising examiners with the OSCE prerequisites and scoring criteria, along with evaluating the adequacy of instructions, the suitability of time allotment for each task, the coherence among tasks, and the arrangement of OSCE station sequence (Ware et al. 2014).

Discussion
This The venue should be considered during the development of the policy framework to ensure consistency in resources, as inconsistencies were identified as a problem in the study context.The venue should be booked in advance to avoid disruptions and to accommodate the number of bookings.Furthermore, the suitability of a venue is important in enhancing privacy and reducing noise, and therefore enhancing the reliability of an OSCE (e.g.students not being able to hear each other if performing a similar skill, affecting student outcomes) (Lim et al. 2023).
Finally, a mock OSCE could be used to pilot the OSCE and OSCE banks and tools, as this method has been reported to assist in students' time management skills, alleviate student anxiety and improve their clinical skills.However, from the literature, a mock OSCE often seems to be used as a way of training students, examiners, and standardised patients (Gilani et al. 2022).A mock OSCE should, therefore, include a debriefing that not only focuses on students' performance but also encourages self-reflection for students, standardised patients, and examiners involved in the process.Generally, little is known regarding piloting an OSCE, which requires further exploration.
In summary, the developed recommendations were wellsupported by existing literature, except for the sixth recommendation related to selecting the appropriate OSCE venue.The developed recommendations can enhance the planning and design of OSCEs, vital in enhancing OSCE quality.

Limitations
The study was limited for several reasons.Firstly, it focused on strengthening the OSCE planning and design as a basis for its implementation.Secondly, the recommendations were developed based on synthesised data from the College in South Africa and some recommendations may not apply to all educational contexts.Thirdly, external moderators' firsthand accounts and opinions could not be obtained.Fourthly, limited evidence-based methodologies relating to nonclinical topics limited our ability to develop non-clinical recommendations such as this one.Lastly, utilising qualitative approaches for sampling and data collection limited the number of participants and thus limited generalisability.

Conclusion
A set of seven evidence-based recommendations was formulated concerning the establishment of a policy framework, standard operating procedures, and a code of conduct for OSCE stakeholders.These recommendations also covered the blueprinting and arrangement of OSCE content and stations in the bank, as well as the selection of scoring rubrics and standard-setting approaches.In addition, it is advised to focus on examiner and standardised patient recruitment and training, venue selection for the OSCE, and the implementation of OSCE station piloting.These recommendations offer valuable guidance for nurse educators seeking to enhance the planning and design of OSCEs.This is pivotal in elevating the quality of OSCEs.
However, the recommendations require further development, piloting, and implementation, considering alignment with national Professional Board and Council requirements, and factors that influence the objectivity of examiners.Finally, OSCE piloting should be further explored.
study was conducted at a South African public College of Nursing consisting of five main and 19 satellite campuses.

TABLE 1 :
Document analysis' data extraction tool.
OSCE, Objective Structured Clinical Examination.
'Objective Structured Clinical Examination' or 'OSCE' or 'Health Science Education' and 'Quality'.OSCE, Objective Structured Clinical Examination; CINAHL, Cumulative Index to Nursing and Allied Health Literature; ERIC, Education Resources Information Center.

Table 3
outlines the two sets of themes, synthesised into seven draft recommendations.

Table 4
outlines the final seven recommendations after review, including the level of evidence as per Lobiondo-Wood and Haber (2021).Lobiondo-Wood and Haber (2021) categorise levels of evidence into a hierarchy from the highest to lowest: Level I (systematic reviews or meta-analyses of Randomised

TABLE 3 :
Two sets of themes synthesised into seven draft recommendations.Adapted from Shah, R., Edgar, D.F. & Evans, B.J., 2018, 'The use of simulated and standardised patients in education, training and assessment', Optometry in Practice 19(1), 1 OSCE, Objective Structured Clinical Examination.*, Standardised patients are individuals trained to simulate real patients' medical conditions, histories, and symptoms consistently and accurately. Source:

TABLE 4 :
Final recommendations after review.
(Ware et al. 2014)eive the bells that signal the station rotation times during the OSCE(Ware et al. 2014)ß In case recording technology is employed, a control room is required, overseeing live video feeds from every station(Ware et al. 2014).