Validating a Professional Dilemmas Question Bank

Thirteen years ago, we were rounded up into our main medical school lecture theatre and told we were about to hear something important. The dean of our school, a man we rarely saw, came out with a suited man from Health Education England (HEE). They explained that the pilot last year had been a success and that we would be the first cohort to sit a new exam, the Situational Judgement Test (SJT).

The SJT would test our Judgement in handling tricky, but realistic, scenarios we may encounter as an F1 doctor. It would make up 50% of the score that selected which Foundation School we would go to. Lastly, we were told, this is an exam that you can't revise for, so don't bother.

Today this exam has become the most important post-graduate exam in U.K. Medicine. It has been rebranded as the Professional Dilemmas paper and is part of the Multi-Speciality Recruitment Assessment (MSRA). It is used in the majority of ST1/CT1 training programmes for interview shortlisting and for some as the sole means of deciding who gets selected and who does not.

Is an exam this important to our careers really measuring something so intrinsic to us, that it can't be revised for? Or worse, is left up to chance.

Every exam cycle an Educational Psychologist and Statistician from the Work Psychology Group (WPG), the company that oversees the MSRA and the original SJT, sit down and analyse the performance of the exam.

They produce a technical report providing a narrative summary of how the last round of MSRA sittings went and some statistical analysis to show the validity of the exam. This is done for each speciality programme and the reports are then sent to NHS England. In the past, these reports were released for public consumption, now they are kept private and even marked as confidential.

Within the technical reports there is one particularly important statistic, Cronbach's Alpha, a measure of the internal consistency of the exam.

The purpose of an exam is to accurately grade a candidate's performance in some domain. For GCSE Biology, this is your knowledge of the topics within the GCSE Biology syllabus. For the Professional Dilemmas paper, this is your judgement in resolving Professional Dilemmas across the domains of Empathy & Sensitivity, Professional Integrity & Working under Pressure.

An exam that is consistent, all else being equal, will give the same candidate the same grade, no matter what variation of the exam they took. An exam that is not consistent will assign each candidate a random grade and call it a day.

This makes intuitive sense. If an exam has two versions of the same paper and Billy-Bob sits both of them. An excellent exam may rank Billy-Bob in the 12th percentile for both papers. A poor exam may rank Billy-Bob in the 10th percentile for one, and the 60th percentile for the other.

Importantly, an exam that is consistent, is an exam that can be studied for. This is because, in the context of the Professional Dilemmas paper, a consistent exam must have an implied model of what constitutes excellent judgement. By learning what this model is, through exposure and practice, you would know what the examiners consider to be excellent Judgement, you can then use this to improve your performance in the exam. But this only works if the exam is consistent.

For the past several years we have collected publicly available information and made multiple Freedom of Information requests to obtain the technical reports produced for the MSRA and the aforementioned Cronbach's alpha for the Professional Dilemmas paper.

We wanted to know if the Professional Dilemmas paper is a random number generator, as some allege it to be, or if it is an exam that could be revised for.

These reports show that the Professional Dilemmas paper consistently has a Cronbach's alpha of 0.8 to 0.81. 1.0 would be a perfectly consistent exam, 0.0 would be a completely random exam. 0.8 is generally considered to be a good level of consistency. For reference, the Clinical Problem Solving paper has a Cronbach's alpha of 0.88.

The Professional Dilemmas paper has enough consistency that we should be able to learn a implicit model of Judgement they are assessing against. By learning this model we can improve our performance.

From the age of 15 we have had to sit an array of exams, one after another. We start out doe-eyed and idealistic, reading textbooks and other resource materials to learn things from base principles. Eventually, as we become more time poor and our background knowledge has accumulated, we converge to the tried and tested method of "just do the question bank".

We know that by working through thousands of questions we can drill down knowledge, identify gaps that need further reading and pass the exam.

Something we take for granted however, is that for a question bank to be useful, it must be valid.

For a question bank to be valid, it must be consistent with the exam it is modelling.

Usually this is simple. In most medical exams there is a ground truth, e.g. the Vagus nerve is the tenth cranial nerve.

But for Professional Dilemmas there is no ground truth. It is an exam assessing Judgement, a topic that is inherently subjective. As such, for the MSRA, ground truth is determined by whatever the WPG says it is.

Yet this is too harsh. All Professional Dilemma questions are written by senior doctors who, on the whole, have a similar sentiment on what constitutes good vs. poor judgement. Questions are also piloted before they are used, and analysed to make sure they are valid when compared to the existing question set.

This is reflected in the Professional Dilemmas paper having a reasonably high internal consistency of 0.8. It is not perfect, two people can reasonably disagree on what may constitute perfect judgement for a tricky scenario, but it is not completely random either.

Now there are many MSRA Professional Dilemma question banks, but none have been validated.

Without validation, you can only say that the question bank is measuring your judgement against that of whoever wrote the question. Their judgement may differ from that of the exam writers, in which case practicing using their questions may lead to worsening of your performance.

Since its inception, twenty-three official MSRA questions have been released as part of a practice paper produced by the WPG. Before then, 228 Foundation Programme SJTs were released, also in practice papers. This gives a total of 251 available questions, with only 23 at the exact level that the Professional Dilemmas paper is examining.

For Professional Dilemmas there is a paucity of validated questions available. This is what we have set out to change.

We plan to create the first validated Professional Dilemma Question Bank. To achieve this we have created a validation process consisting of 3 major steps: Pre-Validation, Validation and Post-Validation.

We are currently about to start Step #2 - Validation.

For the past 3 years we have been collecting and writing scenarios for Professional Dilemma questions. These have all been written by Subject Matter Experts (SMEs), senior doctors who have completed the Foundation Programme.

These scenarios were developed into stems, the question part of a 'question'. We then validated the writing standard and scenario complexity of these using a 'spot-the-odd-one-out' test on naïve users.

Stems that passed this test consistently were further developed. At least 10 plausible options were developed for each question. These were debated and discussed and a provisional ordering (for ranking questions) or selection (for multiple-choice questions) were chosen.

In parallel we have also developed an A.I. model that can score consistently highly on an unseen set of official Professional Dilemma questions. Over time we have improved this performance from 85% to 90.2% on this question set. This may sound like a small increase, but for an exam where most people score within a tight range of one another, even small gains in your absolute score, make a large difference in your standardised (relative) score. An absolute score of 90.2% would place this model in the top ~1.2% of candidates sitting the PD paper. This is equal to a standardised score of 360, enough to get into any training programme. To learn more about how the MSRA is marked see our piece here.

900 provisional questions have gone through 3 iterations of validation against this model. Only questions that showed high levels of consistency, at the end of this process, have been progressed to the next stage. This has left us with 521 hand-written questions.

We are wary of using an A.I. model in validation. Though we have high levels of confidence with this model from internal testing, we believe that further validation is required to truly say this question set is valid. This brings us to Step #2.

We plan to open up our internal Question Bank of 251 official SJT questions for anyone to practice on for free. This will always remain free for people to practice on, including for future MSRA exam cycles.

We are providing early access to this Question Bank via our Early Access Programme, places for this will be limited to 1500 people. As part of the Early Access Programme we will also be piloting our pre-validated questions. This mirrors the process undertaken by the WPG for the official MSRA. You will be shown either an official question or a pre-validated question. Pre-validated questions will be clearly marked as such, but only after you have answered it. We can then use this data to measure the internal consistency of our pre-validated questions and pick out ones that have very high levels of consistency with the official question set.

Questions that pass this validation process will then be released as a new Question Bank from the 1st September 2026 for the next round of the MSRA. Users who have signed up to the Early Access Programme and answered at least 150 questions will be given free access to this.

We believe that this Question Bank will give people a distinct advantage in improving their performance on the Professional Dilemmas paper. To maintain this advantage and prevent it from being diluted we will be limiting sign ups to a maximum of 25% of all MSRA takers.

We plan on obtaining official MSRA scores from candidates following the exam cycle. We can then use this data to further validate our Question Bank against real-world MSRA performance. Eventually we hope that our Question Bank will be able to not only improve your performance, but be able to accurately predict what score you will get in the official exam.