UCAT Scoring and Scaling: How does it work?

Posted by MichealH Alexander on May 20th, 2021

A common question among students preparing to sit the UCAT is how does UCAT scoring work? This blog answers that question. Practical implications for the UCAT test taker are covered in Part 2 of this blog.

Many UCAT companies claim that they can predict UCAT scores and percentiles from the number of UCAT questions that you answer correctly. This is unfortunately completely misleading and inaccurate. Most UCAT preparation companies have little to no understanding of the scoring process and therefore their platforms provide misleading information regarding predicted UCAT scores.

In reality, UCAT scoring is an extremely complicated process. In this blog we will discuss how UCAT is scored and scaled. UCAT Training

How is UCAT scored and scaled?

Pearson VUE uses Item Response Theory (IRT or ‘Rasch modelling’) for UCAT scoring and scaling. This takes into account several factors such as the number of UCAT questions you got right (your raw UCAT score); the difficulty of UCAT questions; the Point Biserial Correlation Coefficient (PBCC); the probability that you obtained the right answer through random guessing; Differential Item Functioning (DIF); the cohort of students sitting the UCAT test, among other factors. To add to this complexity is the fact that you are randomly allocated one of 3 UCAT tests that Pearson VUE uses every year (this is to enhance the integrity of UCAT, reduce cheating and ‘item leakage’).

While it is true that all questions carry equal marks, the final UCAT score takes into account the difficulty of UCAT questions: so a student who answers 20 easy UCAT questions correctly will get a lower score than a student who answers 20 difficult UCAT questions correctly. That does not mean you should skip the ‘easy’ UCAT questions and focus only on the ‘hard’ UCAT questions: this is because the scoring also takes into account the number of UCAT questions you get right.

How will you know what is a hard and easy UCAT question?

Furthermore, what is a difficult UCAT question for you may be easy for another candidate. However, there are UCAT questions which most students find generally easy or difficult. So how do you know which (type of) UCAT questions most students find easy or difficult? On MedEntry’s online platform, you will obtain statistical information for each UCAT question such as: the proportion of students who got the question right, the proportion of students who chose each option, how long other students spent on the UCAT question and so forth, which is extremely useful information. You should examine it carefully in your targeted UCAT preparation.

Because students of MedEntry are truly representative of the cohort who sit the UCAT (from all over Australia and NZ), you have a relevant and representative comparison group. Students of other prep companies are either local (eg. to a city) or state (eg. Victoria) or country (eg. NZ) only.

What is PBCC?

If a student who gets a high overall score in a UCAT subtest gets a question in that subtest wrong, it could mean there is something wrong with that question and its PBCC is low. Similarly, if a student who gets a low overall score in a UCAT subtest gets a question right, again there may be something wrong with the question and its PBCC is also low. UCAT aims for questions with high PBCC (at least 0.3). PBCC may be low because the UCAT question was poorly written, due to item ‘leakage’, or due to other factors. A question which previously had a high PBCC, which was used in the past and led UCAT testers to believe was a good question, may be deleted from scoring this year if its PBCC falls.

UCAT trial questions

UCAT test designers have a test bank of questions from which they randomly draw about 200 questions each year. They obviously want to keep adding to the UCAT test bank because of ‘leakage’ of some questions, the need to change UCAT question types and so forth. They therefore have UCAT question or ‘item’ writers who write questions regularly and consistently. These are then ‘tested’ to see if they are suitable by using them as ‘trial UCAT questions’ every year. While the item writers have some idea of the suitability of the UCAT questions they created, they are not certain of their suitability unless they are tested on the ‘live UCAT cohort’ (for example, they may not know whether the PBCC is high enough). Approximately 10% of the total questions you will have in UCAT are these ‘trial questions’. Obviously, you will not know what these trial UCAT questions are (but you can guess which ones are likely to be the trial questions: this will be discussed in the MedEntry workshop). UCAT Question Bank

UCAT scoring software

Pearson uses a software package for IRT calculations, which costs about ,000 to buy. There are only a handful of such software providers in the world. To make things even more complex, there are certain assumptions the user of the package must input before the package can be used. Only experienced Psychometricians can do this. MedEntry purchases and uses these complex software packages for our UCAT scoring/scaling in order to accurately simulate UCAT scoring.

More details on how the UCAT scaling process works will be provided during the MedEntry UCAT workshops. What is provided here is a very brief and simplified summary. Psychometricians do the scaling work which is very complicated.