Minutes

Committee Meeting

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Minutes from Meeting

Wednesday 10th October 2012

‘FRCA – Where from and where to?’

Dr Simon Bricker, Chester and Dr Liam Brennan, Council Member, Royal College of Anaesthetists

The meeting was opened by the President Dr Christine Bell who then handed over her duties to the incoming President, Dr Janice Fazackerley. Dr Fazackerley reviewed the last two years and paid tribute to Dr Bell.

‘FRCA – Where from?’ – Dr Simon Bricker

Dr Fazackerley introduced the first speaker, Dr Simon Bricker, who started his talk by looking at the origins of Anaesthesia. He commented that anaesthetics were often given by unqualified practitioners and he illustrated this by showing an advert for dental extractions where the anaesthetist was the wife of the dentist.  She had not even attended school, never mind having had any formal training. He spoke about developments in anaesthesia from laryngoscopy and tracheal intubation to the introduction of intravenous anaesthesia. It was in the 1930s when anaesthesia in the UK started to become formerly organised. The Diploma of Anaesthesia was introduced in 1935 and the first Chair of Anaesthesia in Oxford was established in 1937. The Diploma was then developed in 1948 and modelled on the FRCS with an emphasis on anatomy and pathology. This exam had a pass rate of 20-30%. In 1953 the FFARCS was introduced with a significant weight placed on basic science. This also had a similarly low pass rate of 25%. Indeed Dr Bricker demonstrated that the pass rate did not significantly increase over the following 30 years. He then went on to illustrate the sometimes bizarre nature of the exam questions particularly in the viva. This began to change in the 90s when training became far more structured with the exam following suit with the consequence that a far higher pass rate was expected. The College also became influenced by educational theory which was used to justify the many changes in the exam.

Dr Bricker noted that the College had come under significant pressure when PMETB was established in the early 2000s to defend the standards of the FRCA exam. In particular, it had to defend the reliability and validity of the exam. Dr Bricker then looked at what reliability actually meant. This included the defining of the various “question instruments” such as the MCQ, SAQ, OSCE and SOE. Their use then had to be justified and questions mapped against the curriculum. Reliable pass/fail boundaries also had to be established. He also looked at the validity of the exam demonstrating the complex nature of this concept.

Dr Bricker went on to discuss some aspects of the current format. He spoke about multiple choice questions as tests of factual knowledge and their failings. The single best answer had been introduced into the final FRCA in 2010 to better test judgement, problem solving and the application of knowledge to clinical practice. He described the format and then gave an example from the College website and argued that at one extreme they may be too simple and at the other far too complicated and getting a balance was often difficult. He then spoke about the short answer question which he commented had been abandoned by my many other Colleges. Handwriting and legibility could often be a significant issue and illustrated this with some examples from his own experience. He quoted from the RCA Bulletin in 2006 where Peter Nightingale and Chris Dodds had justified the presence of the SAQ paper to demonstrate well-argued solutions to problems rather than just asking candidates to write a list of facts. They argued that it was also there to demonstrate effective written communication. Dr Bricker then demonstrated both the failings of some candidates and also of questions in both of these respects.

In 2011, the RCA had given guidance for the final SAQ stating that 10% of the marks for each question would no longer be given for clarity, judgement and the ability to prioritise. They also said that marks would no longer be deducted for serious errors. Dr Bricker argued that increasingly, the exam was merely a test of factual knowledge and a retreat from professional judgement which is an extremely important aspect of being an anaesthetist as well as a doctor.

Dr Bricker looked at the structured oral examination and gave his opinion that increasingly this aspect of the exam was less of a discussion and more of an interrogation. He spoke about the old closed marking scheme and the problem of the isolated “one”. This had been changed in recent years to a more open system of individual examiner marking. Dr Bricker demonstrated that this also had its problems. He gave some anonymous feedback from examiners who gave their opinions that increasingly the structured oral examination had become a tick box exercise. Dr Bricker finished his talk with some anecdotes from his time as an examiner.

‘FRCA Where To?’ - Dr Liam Brennan, Chair RCOA Examinations Committee.

Dr Brennan began his talk commenting that many people looked to the past with rose tinted spectacles when indeed the reality was often significantly different. He went back to 1990 when he had passed the FRCA and looked at how it then worked. There was a 35% pass rate, no curriculum on which to base the exam and no external regulation or scrutiny about the validity or reliability of the exams. Oral questions were unstructured; there was no examiner training and little transparency in the appointment of examiners or the process of the examination. This contrasted with the position when the FRCA was last reviewed in 2011. This process ensured that the FRCA fulfilled the standards published by the GMC in 2010 for curricular and assessment systems. The College had to show mapping of the exam to the curriculum and the written components of the FRCA had to demonstrate appropriate statistical reliability. The structured oral examination was more difficult to assess but a new marking system had been recently introduced which had helped reduce disparity.

Dr Brennan then looked at the cost of the FRCA compared to other post-graduate exams and confirmed comparability. He also commented that visitors, mainly consultant anaesthetists, were consistently complimentary about the exam process. There was also lay representation from the patient liaison group.

Dr Brennan then looked at new methods of assessment. In particular, he spoke about the newly introduced single best answer MCQ with which examiners were now becoming more familiar. The College had been relatively slow at introducing this method and were still using the written short answer question format which has been discontinued in many other post-graduate examinations. Dr Brennan looked at how the various assessments mapped to Miller’s Triangle. MCQs demonstrated factual knowledge whereas single best answers may better assess application of this knowledge. OSCEs may help to assess competence but overall performance can only be assessed in the work place which is now being done with work placed based assessments.

Dr Brennan then discussed the future. He said that the GMC is tightening up on all aspects of assessment and the exemption of overseas qualifications for the Primary FRCA would soon cease unless the RCA could demonstrate equivalence by robust quality assurance of these exams. He also thought that exams may become more computer based which have a number of advantages such as lower manpower requirements and more rapid delivery of results. He explained the increasing use of statistical analysis in exam performance to estimate reliability. “Generalisability theory” allows assessment of the variability of individual factors such as candidates, examiners, venue and different days of the exam. This is currently being trialled by the College. He voiced his concerns about examiner availability which was increasingly becoming problematic. Less than half of the examiners were allowed professional leave from their Trusts to fulfil this role and more than a quarter expected examiners to either use annual leave or pay this time back to their Trust. The main problem appeared to be the attitude of clinical and medical directors and Dr Brennan expected this to get worse rather than better in the future.

In conclusion Dr Brennan felt that the College was under constant external scrutiny to justify its examination systems which it was endeavouring to do. However he understood the reluctance of examiners and the wider anaesthetic community to constant change in the examination process.

Questions were then taken from the floor to the two speakers who were then joined by Mr Richard Bryant, the RCA Director of Training and Examinations. The first question asked about the competence of the GMC in making judgements about the assessment methods of the anaesthetic curriculum. Dr Brennan answered that there had been very significant improvement in recent years in the relationship between the GMC and the RCA and he now felt that the GMC had very significant credibility in this area. Professor Hunter asked whether basic science was still important in the exam. Dr Brennan answered that knowledge of this area was central to anaesthetic practice and its place had been robustly defended by the College. Dr Peter Bamford asked about the timing of the Final FRCA during the intermediate years of training when some trainees may have not had clinical exposure to areas of practice that are examined. Dr Brennan firstly said that there was little enthusiasm for an exit exam in anaesthesia and the assessment of higher and advanced training was very much more clinically based. Therefore the place of the FRCA in ST4 was ideal providing two or more units of training had been completed. He then went on to talk about a review that is currently under way into the shape of training in the future. It is looking as to whether CCT holders were ready for the consultant role. There were some ideas being put forward about post CCT credentialing and other forms of assessment. There were a number of other questions about perceived lowering of standards and the possibility of candidates passing despite making serious errors during the exam. Dr Brennan tried to reassure the audience that patient safety was at the heart of the exam and the increasing lay scrutiny of the assessment process ensured that to be the case.

The vote of thanks was given by Dr David Gray who thanked both speakers for their contribution and also paid tribute to the new President for her longstanding commitment to education and training in the region. The meeting finished at 20:45.

Ewen Forrest
Hon Secretary
11th October 2012

 

 

 

.
   
Last updated: 22 October, 2012 LSA