Amazon MLA-C01: Model Evaluation and Selection — Study Guide

By Laurynas Brazys · 3 August 2026 · #amazon #mla-c01 #certification #exam-prep #study-guide

Part of the AWS Machine Learning Engineer Associate MLA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

Accurate model evaluation begins with selecting metrics that match the business objective and the class / label characteristics. For binary classification, the confusion matrix — true positives, false positives, false negatives, true negatives — is the canonical primitive from which precision (TP / (TP+FP)), recall or sensitivity (TP / (TP+FN)), specificity (TN / (TN+FP)) and accuracy ((TP+TN) / total) are derived. Precision and recall form a trade-off that is captured in the F1 score (2 * precision * recall / (precision + recall)), which is the harmonic mean and emphasizes balance between false positives and false negatives. When class imbalance is severe, accuracy can be misleading; area under the ROC curve (AUC) measures separability across thresholds and is robust to imbalance in many cases, while the precision–recall AUC (AUPRC) is more informative when positive class prevalence is low. For regression, root mean squared error (RMSE) penalizes larger errors quadratically and is sensitive to outliers; use mean absolute error (MAE) when robustness to outliers is preferred.

Threshold selection is an operational decision: many algorithms produce a probability score and require a threshold to convert to class labels. Use ROC and precision–recall curves to locate thresholds that maximize a chosen objective such as F1 or a business-weighted cost function. Practical methodology is to compute per-threshold precision, recall, and F1 from the confusion matrix and then pick the threshold that meets a target precision or recall constraint. Cross-validation complements this by reducing variance in metric estimates: use stratified K-fold for imbalanced classification to preserve class ratios across folds. In code, scikit-learn’s StratifiedKFold with n_splits and shuffle=True plus a fixed random_state gives reproducible folds; log per-fold metrics and aggregate means and standard deviations to quantify expected variance. When using SageMaker, you can orchestrate CV runs as separate training jobs and track them with SageMaker Experiments APIs: CreateExperiment, CreateTrial, CreateTrialComponent, and then LogMetric and LogHyperParameter for each fold.

The bias–variance trade-off guides model complexity and regularization choices. High-bias models underfit and produce high error on training and validation sets, while high-variance models overfit and have low training error but high validation error. Remediate bias by increasing model capacity, adding informative features, or reducing regularization strength; remediate variance by adding regularization (L1/L2), reducing complexity, using dropout, or increasing training data. Use cross-validation estimates to detect variance: large spread across fold metrics implies high variance. For iterative training on SageMaker, incorporate early stopping (for XGBoost set hyperparameter early_stopping_rounds and eval_metric=‘auc’ or ’error’) and use SageMaker Automatic Model Tuning with a Bayesian strategy to search hyperparameter_space; configure HyperparameterTuner with objective_metric_name, objective_type=‘Maximize’ or ‘Minimize’, hyperparameter_ranges, max_jobs, and max_parallel_jobs.

Key services and configuration

AWS services form a tightly integrated toolchain for tracking experiments, registering models, detecting drift, and enforcing deployment guardrails. Use Amazon SageMaker Experiments to organize runs; call sagemaker.create_experiment, sagemaker.create_trial, and sagemaker.log_metric to persist hyperparameters and metrics. For centralized model lifecycle management use the SageMaker Model Registry (ModelPackageGroup and ModelPackage) and control deployment flow through the model package attribute ModelPackageApprovalStatus which supports values such as “PendingManualApproval”, “Approved”, and “Rejected”. Integrate manual approvals into a CI/CD flow by invoking UpdateModelPackage to change approval_status from an AWS Lambda or a Step Functions human-approval action.

For post-deployment monitoring and bias/drift detection use SageMaker Model Monitor and SageMaker Clarify. Enable real-time data capture on an endpoint by setting DataCaptureConfig in the CreateEndpointConfig or UpdateEndpointConfig call; specify CaptureOptions: [{“CaptureMode”:“Input”}, {“CaptureMode”:“Output”}], DestinationS3Uri, and SamplingPercentage to collect representative payloads. Use Model Monitor DefaultModelMonitor to GenerateBaseline (baseline_statistics and baseline_constraints JSON) from a reference dataset and then create a monitoring schedule with create_monitoring_schedule, supplying a MonitoringScheduleConfig, schedule_expression (cron or rate), and MonitoringInputs such as EndpointInput with local_path and S3InputMode. For fairness and bias, use SageMaker Clarify as a ProcessingJob with the clarify_config parameters compute_bias and compute_data_drift, or use the SageMaker Clarify builtin in a Processing job and persist results to S3 for dashboards.

Several AWS capabilities are commonly used together for data aggregation, feature preparation, and anomaly detection:

Amazon S3 for storage and secure isolation (bucket policies, S3 Block Public Access, and KMS encryption via SSE-KMS).
AWS Glue and AWS Glue Data Catalog to discover, unify, and catalog datasets from S3 and JDBC sources.
Amazon SageMaker Data Wrangler for drag-and-drop transforms and automatic visualization of distributions, missing values, and skew.
Amazon SageMaker Feature Store for online and offline feature serving with APIs PutRecord and GetRecord and for consistent training/serving features.
SageMaker Model Monitor and SageMaker Clarify for drift, bias, and monitoring.
Amazon Lookout for Metrics for automatic time-series anomaly detection where appropriate.

Design patterns and trade-offs

When operational overhead must be minimized, prefer managed services and built-in algorithm features rather than building custom ETL or custom monitoring pipelines. For example, if training a binary classifier for fraud detection, XGBoost is an efficient built-in choice in SageMaker that offers low-latency training and hyperparameters tailored for imbalance, such as scale_pos_weight which you should set to (num_negative / num_positive). Configure XGBoost hyperparameters in the Estimator or training container: objective=‘binary:logistic’, eval_metric=‘aucpr’ or ‘auc’, and early_stopping_rounds to terminate noisy runs. This avoids building custom oversampling pipelines (SMOTE) unless domain-specific synthetic sampling is required.

For feature engineering, use SageMaker Data Wrangler to apply encoding transforms (one-hot for low cardinality categorical features, ordinal/label encoding or target encoding for high-cardinality features) and then materialize clean features to SageMaker Feature Store or S3. Data Wrangler removes much of the operational burden and produces scripts or processing jobs you can reuse. If you need automated model selection with minimal effort, SageMaker Autopilot can automatically preprocess mixed categorical and numerical features and generate candidate models; however, Autopilot abstracts away transformations and may not be optimal for custom feature regimes.

Model comparison and promotion is most robust when metrics, artifacts, and lineage are recorded. Use SageMaker Experiments to compare runs, then create a ModelPackage in a ModelPackageGroup. Promote by setting ModelPackageApprovalStatus to “Approved” and produce an EndpointConfig referring to the model package ARN. Where human control is required, leave the model in “PendingManualApproval” and wire a human approval workflow using Step Functions or AWS CodePipeline with an approval action that calls UpdateModelPackage to move it to “Approved.”

Common pitfalls and decision criteria

A common pitfall is confusing a drop in production F1 with model fault when the underlying issue is data drift. The most likely cause of a sustained F1 drop months after deployment is input distribution drift or label drift (concept drift) rather than a sudden bug. To diagnose, enable DataCaptureConfig on the endpoint to capture request and response payloads to S3, run Model Monitor monitoring schedules against the baseline constraints, and compute feature distribution statistics and PSI (population stability index). Also run Clarify data drift and bias checks to detect shifts that affect fairness metrics.

Another frequent error is mishandling class imbalance by naively resampling without accounting for temporal leakage or business costs. Prefer algorithm-level controls first — for XGBoost set scale_pos_weight and tune eval_metric to aucpr or custom objective — before moving to sampling strategies that can introduce duplicated examples. For reproducibility and experiment tracking, always use SageMaker Experiments APIs to log hyperparameters and metrics and register model packages with artifact URIs and provenance metadata so that promotions are auditable.

Practical Problem: Use-Case Scenario

Company: FinSight Inc. — fraud detection for on-line transactions with data in Amazon S3 and on-premises MySQL, requirement for secure data isolation, automated anomaly detection and visualization, manual approval before production deployment, low startup latency for iterative training, and centralized model versioning with minimal operational overhead.

Data aggregation and security: Use AWS Glue to crawl the S3 transaction logs and create Glue Catalog tables, set bucket encryption with SSE-KMS and restrict access through IAM policies and VPC endpoints. For on-prem MySQL, use AWS Glue JDBC connection inside a VPC with a secure VPN or AWS Direct Connect, or use AWS DMS to replicate required tables into an S3 landing zone. Register datasets in the Glue Data Catalog and grant SageMaker execution roles least-privilege access.
Feature preparation and anomaly detection: Use Amazon SageMaker Data Wrangler to connect to the Glue Catalog and the S3 landing zone, apply transforms (missing-value imputation, label encoding for categorical variables, scaling for numerics), and run Data Wrangler’s built-in profiling to visualize distributions and flag anomalies. Export processed features into SageMaker Feature Store offline store for training and the online store for low-latency lookups at inference.
Model training and minimizing startup latency: Use SageMaker Estimator for XGBoost with objective=‘binary:logistic’, eval_metric=‘aucpr’, and set scale_pos_weight=(neg_count/pos_count). To reduce startup latency across iterative training jobs, cache Data Wrangler outputs in S3 and reuse the same training container image and instance type rather than reconstructing data ingestion each time; use SageMaker Pipelines with cache_config enabled so repeated steps with unchanged inputs/hyperparameters skip infrastructure startup.
Experiment tracking and model selection: Instrument each training run with SageMaker Experiments (CreateExperiment, CreateTrial, LogMetric). Use HyperparameterTuner configured with objective_metric_name=‘validation:aucpr’, objective_type=‘Maximize’, strategy=‘Bayesian’, max_jobs and max_parallel_jobs to find best hyperparameters. Compare models in Experiments and register chosen candidates in the SageMaker Model Registry by creating a ModelPackage in a ModelPackageGroup.
Manual approval and deployment gating: Leave newly created model packages in ModelPackageApprovalStatus=‘PendingManualApproval’. Use an AWS Step Functions workflow with a human approval task or an AWS CodePipeline approval action; once approved, run UpdateModelPackage to set approval to ‘Approved’ and trigger automatic creation of an EndpointConfig and UpdateEndpoint to perform deployment.
Monitoring, bias, and drift: Enable DataCaptureConfig on the real-time endpoint with CaptureOptions for Input and Output, DestinationS3Uri, and SamplingPercentage. Generate baseline statistics with Model Monitor’s GenerateBaseline from the training dataset and schedule continuous monitoring with create_monitoring_schedule. Run SageMaker Clarify as a processing job regularly to compute bias and data drift metrics. If F1 falls below baseline constraints, use Model Monitor alerts to trigger an automated retrain pipeline (SageMaker Pipeline) that logs new runs in Experiments and produces new model packages for human review.

AWS rationale: This approach uses managed SageMaker capabilities—Data Wrangler and Feature Store to minimize preprocessing overhead, XGBoost with scale_pos_weight to handle imbalance without complex resampling, SageMaker Experiments and Model Registry for auditable experiment and model lifecycle management, DataCaptureConfig plus Model Monitor and Clarify for automated drift and fairness detection, and Step Functions/CodePipeline integrated with Model Registry to provide the required manual approval gate. These services together minimize operational burden while preserving security, traceability, and the ability to respond to model degradation.

← Model Training and Hyperparameter Optimization · All domains · Model Deployment and Inference →

Practice these questions → · Timed practice on ExamRoll.io →