Validating a MSRA SJT Question Bank

We take for granted when we do a question bank that its questions are valid. This is because for most medical question banks, validation is fairly straightforward. We can easily verify that a question asking us about second-line management of asthma is correct.

But for Situational Judgement Test (SJT) questions, validation is much trickier. How do you verify that the judgement of a third-party question writer is consistent with that of the official exam writers?

This is very important, as if their judgement is not consistent with the official exam, practising on their questions may lead to worsening of your MSRA SJT exam performance.

At examify we take this question very seriously, in fact it is our whole raison d'être.

We have developed a robust, statistical process for validating all our MSRA SJT questions to ensure they match the exam as closely as possible. By doing so we believe we can give you an edge in the MSRA. You can read about our process below.

Ground Truth

To validate a question bank, we need some ground truth to compare our questions and answers against.

For the past 13 years we have been collecting official SJT practice questions, both for the MSRA and for adjacent exams like the now-defunct UK Foundation Programme (UKFPO) SJT. In total we have collected 289 official SJT questions, enough to act as a validation set.

Crucially, every one of these practice questions has been written by the Work Psychology Group (WPG), the same group that writes and oversees the MSRA SJT exam.

Item Writing

Before we begin formal validation, we need to develop a pool of items (questions). To do this, we have mirrored how the WPG themselves develop items.

This is a three step process consisting of:

Ideation

In conjunction with Subject Matter Experts (SME) we have come up with hundreds of scenarios which can form the basis of a MSRA SJT question.

For a scenario to be valid it must present a dilemma, with competing interests and/or demands, that a Foundation Year 2 doctor may reasonably come across.

Ideas that meet this standard then progress to further development.

Item Writing

Ideas are written into a stem (the question part of a question). We have a writing style guide that we have developed to make sure our questions are written in a way that is consistent with the official questions.

We also do English Not First Language (ENFL) checks and generate names for stakeholders, within the scenario, randomly to remove bias.

The stem then undergoes a validation check. We randomly mix the stems with 4 official stems, which we then present to a group of blinded raters. If they are able to pick out our pilot stem from the official ones, at a rate better than chance, then the stem fails this validation.

For each successful stem we develop multiple appropriate responses. Based on the types of responses that are developed the question is either selected to be a multiple-choice or a ranking type question.

The responses that form the best set are then selected and a provisional answer key is developed by a SME.

Concordance

Questions are then presented randomly to 4-5 naïve SMEs. They are asked to answer the question independently. We then check the concordance between their responses. Items that do not pass our threshold for concordance are sent back as needing further development. Items with high concordance move to the next stage of validation.

The item writing process, as you can see, is a laborious process. At each stage there are many questions that don't make the cut. We are very strict with our progression criteria, ensuring only the highest quality questions make it through.

After a multi-year effort we have developed a pool of 480 questions that are ready for the next stage.

Piloting

Piloting is where the validation process really begins. This again, has been set up to mimic how the WPG validate their own new items.

The WPG include 5-6 new questions in every live MSRA SJT paper. These questions are not flagged as being pilots, so applicants sitting the exam do not know if they are sitting a pilot question. Pilot questions do not contribute to a candidate's score.

The performance of pilot questions is analysed using various statistical methods including measures of internal consistency and if applicant performance on the pilot questions matches their performance on the existing bank (discrimination).

We have modelled this same process by opening up our bank of 289 official practice questions for anyone to access for free on our platform. When answering these questions you will occasionally be shown a pilot question. We believe these questions are already of very high quality, due to our rigorous item development process, including initial concordance validation.

Pilot questions are batched, once we have enough statistical data to validate a single batch we then switch to a new batch for validation.

With this process we can ensure our questions are valid along several constructs including that we are pitching at the same difficulty as the official MSRA SJT and that our answer keys are consistent with the official questions.

Questions that pass this validation process will be entered into a new validated MSRA SJT question bank, the first of its kind, that we hope to release closer to Round 1 of the 2027 MSRA. We say hope as we only plan to release this if our questions prove to be valid. Promisingly, initial data has been very encouraging but we are still in the early phase of this process.

A Validated Question Bank

We believe that a validated MSRA SJT question bank will give you an edge in preparing for the exam, an edge that should allow you to perform better and ultimately to rank higher.

To preserve this edge we intend to limit sign-ups to this question bank. We will prioritise people who signed up to our free tier and helped us validate the bank. We will maintain access to our free tier, containing all 289 official practice questions, with no cap on sign ups for this.

In the future we plan to obtain MSRA scores from users of our question banks. We expect that this data will prove useful, not just in helping us further validate and refine our bank, but also to give it a predictive function. What this means is that we should be able to give you a reasonable range of expected MSRA scores based on your performance on our bank.

If you'd like to help us validate the bank, and benefit from it first, you can start practising on the free tier today.