Familiarize yourself with terms used for classification in AI and its role in scorecard building.
“Classification” in the AI field is defined as systematically grouping data into categories. The AI model in PeopleQX is trained to identify and classify comparison results between AutoQA and benchmark scores into pre-defined labels:
- True Positive (TP)
- True Negative (TN)
- False Positive (FP)
- False Negative (FN)
The classification formula behind it is known as the F1 score, which combines two elements — precision and recall to assess the model’s performance.
Precision: measures if the number of conversations that the model labeled as correct is true. High precision helps reduce the number of false alarms, flagging only valid issues for attention.
Recall: measures the model’s ability to correctly identify actual issues in conversations. A higher recall rate reduces the number of missed problems so that valid issues aren’t overlooked.
The role of classification in scorecard building
Classification is an outcome of measuring the match rate between AutoQA and Manual QA evaluations. The match rate is derived from the percentage of True Positives/True Negatives (correct answers) across the scorecard. A high match rate means that AutoQA was able to match Manual QA’s assessment.
If answers are mostly classified as False Positive/False Negative, the match rate will be lowered. Using this, you can determine if questions need refining so that AI can score them accurately.
Viewing classifications in scorecard building
When does classification show up in your scorecard? If you’ve added benchmark scores, they appear in the ‘Comparison’ tab of the scorecard quality phase.
A handy guide is provided to explain what the classifications mean. Simply click on the bottom-arrow icon in the ‘AutoQA scoring guide’ panel to display an illustrative example.
With these classifications, you can review audio samples to confirm the match rate between AutoQA and Manual QA. Or if you think AI was incorrect, you can change the answer in ‘Manual QA evaluation’, and the match rate changes accordingly.
At the end of it, your objective would be to achieve high match rates with more True Positives or True Negatives. To get there, you’ll want to write more effective questions for AI scoring so it levels up to the benchmark scores.