Be careful when interpreting references

As a hiring manager, it's important to figure out how well references know the person that they're vouching for. After all, applicants will do their best to provide references who are going to speak positively about them. In fact, applicants may provide references who hardly know them.

Mutual friends (e.g., professors) are good at providing calibrated assessments; however, they're hard to find for every applicant. So without being able to use the same assessment methodology for all applicants, one can't help but question of the practical value of references (especially optional ones) that are provided by the applicant. The main thing it seems to assess is whether the applicant has self-awareness of how much those asked to serve as references want to help them.

Just food for thought - processes intended to provide more signal may not actually do that.