How do you handle Quality on Quality? If you do these scores, do you go back and change the original evaluation score if there are mistakes?

Calibration sessions with your team. This ensures you and the team that the evaluation is based on same/similar baselines. 

Here's a guide in Klaus' blog  226+ pinging @Valentina Thörner36 here, she has loads of experience on hosting and participating in calibration sessions.