The FRCA Examinations - What Now and Where Next?
Dr Andrew Tomlinson (Senior Vice-president RCoA, Chair of Examinations Committee 2007-2009), Dr J-P van Besouw (Chair of Examinations Committee), Dr Liam Brennan (Chair of the Final FRCA Board of Examiners), Dr Hazel Adams (Chair of the SAQ Core Group 2007-2009)
The meeting was opened by the President, Professor Jenny Hunter, who initially informed the meeting of the sad news of the death of a former President of the Society, Dr Tom Bryson. Professor Hunter spoke of both her personal memories and the professional achievements of Dr Bryson which was then followed by a minute’s silence in his memory.
Professor Hunter then introduced the speakers for the meeting, Dr Andrew Tomlinson, who is now the Senior Vice President of the Royal College of Anaesthetists, and former Chair of the Examinations Committee, Dr J P Van Besouw, who is the current Chair of the Examinations Committee, Dr Liam Brennan, current Chair of the Final FRCA Board of Examiners, and Dr Hazel Adams, the recent Chair of the SAQ Core Group.
Dr Tomlinson gave a brief initial overview of the changing exam structure, which had its routes in a PMETB publication from 2005. This document set out the quality standards of curriculum assessments. Firstly, Dr Tomlinson looked at the Primary examination; recent changes included the removal of negative marking from September 2008, the reduction in attempts from 6 to 5, and the validity of a pass in the MCQ being reduced from 3 years to 2 years. The marking system for the OSCE exam has also changed and the closed marking system has been abolished in favour of a numerical scoring system. In addition, the two parts of the exam have now been separated and there is no longer a requirement to re-take a section already passed within the previous two years. Dr Tomlinson then went onto the final exam which similarly had had negative marking removed from September 2008, the separation of the written exam from the structured oral exam and will (2010) have the introduction of single best answer type questions in the MCQ paper. In a similar way to the Primary, closed marking has now been abolished and the pass mark is pre-determined which required a standard setting exercise.
The question and answer section then began. Dr Van Besouw spoke about the importance of adhering to PMETB standards in both the curriculum and assessments, as without recognition by PMETB specialty training would not be recognised. He explained that PMETB was answerable to the GMC with which it will soon merge. Professor Hunter asked if standards were being lowered. Dr Van Besouw answered that standards will be maintained, however, performance by candidates in recent examinations seemed to be decreasing and there had been a significant reduction in pass rates over the last two years, particularly for the Primary exam. He thought that part of the explanation for this was the increasing pressure at CT level to pass the Primary exam within 18 months, whereas in the past many trainees had taken nearly two years to pass the exam. Secondly, training time and clinical exposure have been reduced because of the changes in the EWTD and increasingly time is being spent providing service needs in intensive care. Dr Van Besouw continued with reinforcing how much control PMETB now has on so many aspects of training. He explained the origins of MMC which had been introduced as a way of reducing the long service at SHO level that some junior doctors spent, particularly in surgery and medicine (although not in anaesthesia). This was perceived as a waste of time by Government. MMC was also seen as a way of filling unpopular posts. With the uncoupling of training between CT and ST, the RCA was trying to change the system back to where it had been before MMC. Indeed in Wales, there is now a three year CT programme, with the third year varying depending on the requirements of the trainees. The panel was asked whether the RCA supported a 3 year CT programme, Dr Tomlinson was affirmative with flexibility in the third year. This, however, did not have the support of PMETB. It was pointed out by the panel that in 2010 the Government is pushing for a national recruitment at ST3 level and there is likely to be an excess of CT2’s so a significant number will be pushed out of training into service jobs until further ST3 posts come up. Professor Hunter returned to exam questions and commented that current trainees appear to find the exams far more traumatic than perhaps in the past. Dr Brennan commented that for some candidates it may be their first experience of a high-stake examination because many undergraduates are examined by continual assessment. However, Dr Adams commented that she thought that these exams had always been an intimidating experience. Dr Gray commented that the examples of single best answers published in the BJA seemed far too simple. Dr Brennan answered and explained that following a challenge by PMETB, the College had been forced to move away from traditional MCQ answers. He explained the educational theory behind them and the consultation that the RCA had had with other Colleges, and single best answers seemed the most logical progression. Dr Hunter asked how the exam was testing judgement rather than just pure knowledge. Dr Brennan answered that the structured oral examination was the best part of the exam for this purpose but it was also tested to some degree in the MCQ. Dr Wilkes asked for an example of a single best answer which Dr Brennan gave and explained the amount of work that had gone into each question. Dr Parker asked why all the questions and answers could not be published on the internet? Dr Tomlinson replied that there was inevitably a limited number of questions and if they were all published, candidates would tend to learn the answers rather than understanding the reasons behind them. He also commented that observers in the exam often said how lenient examiners actually were. Dr Brennan said that in Australia all the questions of the equivalent examinations were published on the internet immediately after the exam. However, there was a huge amount of work that went into generating more. Dr Van Besouw commented on whether we wanted the candidates to know the curriculum or just answers to exam questions. Dr Brennan also added that not all areas of the curriculum are examined so some candidates may neglect these areas if all questions are published. Dr Tomlinson said there was a great deal of pressure from PMETB to sample as much of the curriculum as possible, which was really undeliverable because of the pressure on time to be spent on formal assessments by both examiners and candidates.
Dr Gray then asked about candidates who had just missed the pass mark in the short answer questions and the fairness of this to candidates, suggesting that candidates who were one mark short of a pass merited an independent re-scoring of their paper. Dr Adams explained that one of the positive aspects of PMETB was that it had made the College re-evaluate its examination methods. Dr Adams went onto discuss the educational theory behind the short answer questions. She explained that these were designed not just to test knowledge but also for this knowledge to be used for interpretation and then to predict likely outcomes. This would allow the exam to test judgement as well as simple knowledge. The short answer question section was now demonstrably the most reliable part of the Final exam and Dr Adams outlined the amount of work that goes into this section. Dr Robin asked whether exams were being taken too early. Dr Tomlinson thought that exit exams were a good idea. However, people only work when exams appeared on the horizon so that the exam system in North America was being changed to bring exams forward. He also pointed out that to push exams back may make the later years of training more difficult when trainees are tailoring their training to the particular areas of practice that they wish to pursue as consultants. Dr Howard asked whether there was still an examiners’ meeting at the end of every day. Dr Brennan said this was still the routine, and examiners were always pressed to justify their decisions. Dr Dodwell asked if the structured oral examination was now too rigid and no longer allowed a free discussion of a topic between candidate and examiner. Dr Van Besouw said that the College had to demonstrate identical standards for every sitting and every candidate, and therefore the structure of the exam had to be fairly rigid to ensure consistency. Similarly the close marking system went because PMETB did not like two people being able to fail a candidate so the summation of all marks has become more important.
Dr Oscar Freddy asked if the current exam was in line with competency based training, with basic science being confined to the primary exam, and clinical science to the final exam. Dr Van Besouw said that training programme directors had insisted that the Primary exam had some clinical knowledge within it which could be tested prior to entrance into ST3. Dr Brennan also commented that the Final exam may be better taken in ST4 when trainees had more practical knowledge and experience.
Dr Gray made some final comments and asked whether examiners were becoming increasingly hostile and belligerent towards trainees, as this had been the impression by candidates at the May sitting of the Primary. Dr Van Besouw explained that examiners’ performances were audited and they were videoed for their first year. Dr Adams commented that candidates often had little idea of their own performance and if they were unsuccessful often focused on the wrong reason for their failure. Dr Brennan added that examiners may be seen as antagonistic to the candidates because they are constantly trying to move them on because of the relatively rigid structure of the oral exam, and this can be seen as both aggressive and dismissive by candidates.
Professor Hunter closed the meeting by thanking the panel for their time in coming up to Liverpool, and gave the final brief words to Dr Gray who also thanked the panel and said he felt greatly reassured that the standards of the exam remained high. The meeting finished at 20.45.