These are the main elements of the review flow that will be very easy to follow once you get started:
- Getting started with an invitation email
- At the beginning of your reviewer link, getting a quick summary about the review you're about to make.
- Checking the question index
- Going through the scoring cycle for each question
- Checking your team's review stats (including yours and the applicant's leaderboard!)
Email with invitation to review applications
This email is the first way to know that you'll be a reviewer of short-answers submitted by candidates. This email will:
- let you know to which role you've been invited as a reviewer.
- contain a unique reviewer link. No painful log in or password details needed. Just this link, which you can always access in case you can't review all answers in one go.
The intro page of your reviewer link
After you click on your reviewer link, you'll first see an overview of what to expect from your job as a reviewer. You'll see:
A. The number of applicants from whom you'll review answers.
B. The total number of answers that you'll rate and score.
C. A reminder of the job description and the main skills that your team are looking for.
You just need to click on the 'start reviewing' button after having this overall framework in mind.
You'll be taken to the questions index after you decide to start reviewing answers. Questions will be displayed in the same order displayed to candidates.
For each question in the index you'll see:
- number of answers reviewed by you out of the total allocated to you; and,
- the text of each question and the skills you are looking for so you have full information before you start rating responses.
You'll see that the platform will take you to question no. 1 by default, but you can click on any other question if you'd prefer rating answers in a different order.
|Note: The total number of answers to be reviewed might not be the same for all questions because that depends on the number of reviews per answer that were set and the number reviewers assigned to review applications (click here to learn more about the review allocation process).|
Behavioural science in the reviewer flow
Chunking is one of the behavioural design principles that comes to action at this point of the reviewer flow. Evidence shows that it's hard to simultaneously compare candidates in multiple areas at once, and that leads to cognitive load. We can also fall prey to the halo effect, if a candidate starts of strong (or weak), that affects everything we read thereafter. So that's why instead of reviewing a candidate's application in full vertically, we make it easy for you to do direct, horizontal comparisons of candidates by reviewing a batch of answers to question 1, then to question 2, and so on.
Scoring all the answers to a same question
Sorry if this is a long section, but it's where you spend most of your time. We want to give you all the possible details so you can use your time wisely.
The typical scoring cycle
Answers to questions will appear as soon as you decide to score them. This is how the scoring cycle works:
1. You'll be able to see one answer at a time and then score it on a 1-to-5 scale.
2. Below each rating scale you'll see the relevant skills that are being assessed and your Review Guide. With this Review Guide as framework, click on the relevant star from the scale to depict how well a candidate answered the question.
3. Once you score an answer, you'll be automatically cycled on to the next answer waiting to be reviewed. Every score you give will be automatically saved (no need to click any "Save" buttons).
4. Points 1-3 repeat until you reach the last answer to a given question. Once you score the last answer to a given question, you'll be taken again to the 'questions index' to start a new rating cycle. When you score the last answer to the last question (i.e. your final, final review), you'll be taken to the review stats.
Behavioural science in the reviewer flow
Rules for Scoring, Anonymisation and Randomisation are behavioural design principles that come into action at this point of the reviewer flow:
Rules of Scoring. Your Review Guide will help you to be objective and consistent when identifying good, not so good and bad answers (click here to learn more about a typical structure of a Review Guide).
Anonymisation. Names and other socio-demographic details can be distracting and result in inadvertent bias. That's why the review flow removes names and all candidate personal details so you just focus on who's given good answers.
Randomisation. Our brains are heavily affected by ordering effects and small contextual factors around us. If we're hungry or tired, our scores are less reliable. All of this leads to lots of noise in the reviews, and a lack of objectivity about who's really good. That's why the review flow randomises the order of all candidate responses everytime you start a scoring cycle. So answer 1 from question 1 and answer 1 question 2 are not necessarily from the same candidate. That way, no candidate is disproportionately advantaged or disadvantaged by where they show up in the pile.
Scoring the last answer to the last question of the review flow
In this case, you'll be taken to the review stats page rather than to the 'questions index'. So we recommend you go SLOW at this point of the review flow because it won't be possible to go back and re-rate responses.
Rechecking the scores you've given until a certain point
If it's possible to recheck a specific scores (click here to learn about possible scenarios), you just need to:
a. click on the coloured grid above the rating scale. The number of cells indicate the total number of answers per question to be reviewed. The colour coding of each cell helps you easily identify the distribution of scores. The redder the score, the lower; the greener the score, the higher.
b. click on the link to the question index if you need to change the scores given to a different set of answers, and then click on the corresponding coloured grid to change ratings.
Getting review stats
We love to give feedback. We are also aware of the effort you've put into reviewing applications so you deserve a cup of tea and a cake, along with some cool insights that will let you know more about the review process. At this point of the review flow, you'll be able to check:
- the distribution of scores given by each member of the reviewing team.
- The level of disagreement between you and the rest of reviewers and their review progress.
- The candidate leader board.
- An opportunity to give us feedback. We love to know more about your experience using Applied.
What happens after finishing a review?
You and your team can shortlist candidates based on the scores you've given (click here to learn how).