‘Q&A with the Chairs of the FRCA’
Drs Mark Blunt & Richard Telford, Chairs of the Primary & Final FRCA Exams
President, Dr Clare Howard opened the meeting and welcomed members, especially trainees, to the first LSA meeting of the 2017-18 session. She started by remembering Dr Gordon Bush who recently passed away and was a Life Vice President of the society. Members stood in silence to remember and honour him and his contributions to Anaesthesia. Dr Howard then welcomed the speakers, Drs Blunt and Telford.
Dr Blunt started his talk by asking the audience if anyone felt that the FRCA exams were not an important part of the assessment of anaesthetists? He spoke about methods of assessment of medical specialists and Miller’s pyramid of professional authenticity. He said the GMC expects certain qualities in a professional as do other professional bodies but a lot of these are behavioural components which need to be assessed using work place based assessments. So he asked where exams fit in? He thought that their role is in assessing knowledge. However they are expensive: the venue, staff, examiners, equipment for OSCEs and now new computer based testing. They also have to be acceptable: to the profession, educators, public, patients and candidates. He said that their educational impact is important. This relationship is complex and their impact is often negative but they are working to provide more focussed feedback for candidates.
Dr Blunt commented on the fact that the exam assesses knowledge at a point in time and that knowledge retention can be poor. He also spoke about the reliability of the exams; that is, will the test and retest give the same results? The RCoA looks at this across all the exams: written, OSCE and oral assessments. He said that a lot of work is done in training, appraising and reassessing examiners as well as performing examiner scoring evaluation. They evaluate and review all questions and normalise difficult papers. He explained how they measure the reliability of oral exams using item response theory and Rasch modelling. He explained how reliability can be improved by increasing testing duration. He spoke about the validity of the exam (does the assessment measure what you think it’s measuring and is meant to measure?)
Dr Blunt said that the basis for questions is sampling of the content domain as detailed in the syllabus which are created by content experts. He said test procedures ensure administration errors are avoided and the difficulty and discrimination of questions is understood. He explained the threats to validity. Assessment of understanding is more than just answering a series of questions. He explained the OSCE is meant to be objective but it can promote a scattergun approach to history taking, with candidates asking as many questions as they could to achieve those on the mark sheet. Therefore, taking a poor unstructured history could still achieve full marks. To counter this, educationalists have accepted that the opinion of an expert examiner is valuable in post graduate medical exams. The history taking station has consequently changed to allocate marks on the quality of interaction and the way the history is taken.
Dr Blunt thought that the multiple parts of exams were a bit like pixels, with many required to fully assess the quality of a candidate. He ended by saying reliability does not relate to the type of exam, however longer exams are more reliable and validity is not always as simple as it looks. WPBAs are the only way we can successfully assess the behavioural components.
Dr Richard Telford then spoke about the history of the FRCA examinations and how the reduction in mortality in anaesthesia was due to the introduction of safety standards, monitoring and training. He explained how the pass mark for the MCQ and SAQ were set. He said the exam had undergone a review focussing on validity and reliability in 2015, the results of which had led to modifications approved by the GMC in 2016. The original proposal wanted to establish restrictions on applications for the Final FRCA until 4 out of 7 modules had been signed off so that candidates could demonstrate adequate clinical exposure prior to sitting the exam and to extend the time to pass the exam before progression was delayed until the end of ST5. The GMC rejected these because different schools of anaesthesia have different training structures although agreed that the clock won’t stop with exam failure until midway through ST5.
He said that MCQs had improved validity and statistical analysis suggested acceptable reliability. He said the SAQs were going to change to constructive response questions (CRQs) which would be phased in during 2019 and would more closely resemble current A-level science questions. They would measure knowledge and application level cognitive skills and could include artefacts such as X-rays, ECGs and blood test results. There would be a number of subsections which increase in difficulty and complexity as the question progresses. They would also be suitable for computer based testing.
He said the structured oral exam would change so that candidates would be tested by six examiners, with six minutes each. The long case would decrease from three sections to two sections and the number of short cases increase to six. Two short cases would be stand alone and four would be merged with an associated science. He said they were trying to minimise candidate swaps in the exam and he gave examples of an SAQ as a CRQ and example of a short case.
He ended by speaking about differential attainment and the ongoing observational study of curriculum units to see which show the greatest differential attainment. He said there is a survey happening of candidate’s social and cultural background and their exam preparation strategies. He also explained that there is a study ongoing of candidate attainment in the Primary and Final MCQ to see if outcome is influenced by gender, ethnicity or linguistics.
Several questions were taken from the audience and Dr David Gray gave the vote of thanks.
Dr Gemma Roberts October 2017
Liverpool Society of Anaesthetists