EcoSensors builds a defect-detection classifier for manufacturing with defects ~0.5% prevalence. Missing a defect (false negative) has very high operational cost, but inspecting false positives also consumes resources. Select TWO evaluation metrics the team should prioritize during validation and hyperparameter tuning to ensure the model performs well in this imbalanced, high-cost-of-miss setting.

Precision-Recall AUC (PR-AUC) is crucial for highly imbalanced datasets where the positive class is rare. It focuses on the positive class, which is the defect in this scenario. A high PR-AUC indicates good performance in identifying defects without generating too many false alarms. AUC-ROC can be misleading in imbalanced datasets because it considers both true positive rate and false positive rate, where the large number of true negatives can inflate the score. F1-score, or an F-beta score with beta 1, is also vital. F1-score balances precision and recall. Since missing a defect (false negative) is very costly, recall (the ability to find all positive samples) is highly important. An F-beta score with beta 1 specifically weights recall more heavily than precision, aligning with the requirement to minimize false negatives even if it means tolerating some false positives. Accuracy is unsuitable for imbalanced datasets as a model predicting only the majority class can still achieve high accuracy while missing all defects. Specificity is less relevant here as it measures the model's ability to correctly identify non-defects, which is secondary to finding defects and minimizing false negatives.