In our MLA-C01 question bank, about 30 questions ask how to scale endpoints and container-based inference, which makes it one of the largest single-topic groups. The exam guide is explicit too: Task 3.2 lists "SageMaker endpoint auto scaling policies based on demand and time" and "choosing specific metrics for auto scaling (model latency, CPU utilization, invocations per instance)," and Task 3.1 lists "choosing the deployment target among SageMaker endpoints, Kubernetes, ECS, EKS, and Lambda." Knowing the policy names is not enough; you need to combine the right policy and metric for the situation.
This post starts with how auto scaling works and then covers the three policy types and how to combine them, choosing a metric, cooldowns and capacity limits, endpoints that can scale to zero, choosing an instance type, and deployment targets outside SageMaker (EKS, ECS, Lambda). The basic differences among the four inference options (real-time, serverless, asynchronous, batch) were covered first in Deployment Infrastructure and Scaling Strategies.
---
How Auto Scaling Works
A restaurant manager calls in more staff when customers pour in and sends people home early when it gets quiet. Calling people in and sending them home too often only causes confusion, though, so after each adjustment the manager waits and watches for a while. Auto scaling for SageMaker endpoints has the same structure.
| Component | Role | |------|------| | CloudWatch metric | Shows the endpoint's load as a number (for example, invocations per instance) | | CloudWatch alarm | Decides whether the metric has crossed the threshold | | Application Auto Scaling | Calculates and adjusts the instance count according to the policy | | Scalable target | The instance count of a production variant, adjusted within the minimum and maximum capacity |
The setup order is: register the scalable target (with minimum and maximum capacity), define the policy, then apply the policy. You must specify both the endpoint name and the variant name, and the required permissions are handled by the service-linked role for Application Auto Scaling.
!Loop in which CloudWatch metrics and alarms drive an Application Auto Scaling policy that adjusts the endpoint instance count within its minimum and maximum
---
The Three Policy Types and How to Combine Them
| Policy | How it works | Best fit | |------|------|------| | Target tracking (recommended) | Keeps the metric near a target value; alarms are created automatically | Most cases, simplest to set up | | Step | Adds a different number of instances depending on how far the alarm is breached | Fine-grained control; required to scale out from zero | | Scheduled | Changes minimum and maximum capacity at set times, once or on a recurring schedule | Predictable peaks such as 9 a.m. every day |
AWS recommends target tracking in most cases. For example, with a target of 70 invocations per instance, it adds instances when the value goes above 70 and removes them when it drops. Scheduled scaling cannot be set up in the console; you create it only with the AWS CLI or the Application Auto Scaling API.
A frequent correct answer on the exam is a combination of policies. Use scheduled scaling to add instances ahead of predictable peaks such as the morning rush, and let target tracking follow the unexpected fluctuations. Target tracking keeps working after a scheduled action runs. If several policies on the same target conflict, the one that provides more capacity wins.
Exam tip: "Prepare in advance for the traffic spike every Monday morning and respond automatically to other changes" is scheduled scaling plus target tracking. "Automatic scaling with the least operational effort" is a single target tracking policy.
---
Which Metric Should Drive Scaling?
A metric used for target tracking must go down in proportion as you add instances. Only then does the logic "adding instances brings the metric back to the target" hold.
| Metric | Type | When to use it | |------|------|------| | InvocationsPerInstance | Predefined metric, 1-minute resolution | The default for typical real-time endpoints | | ConcurrentRequestsPerModel and ConcurrentRequestsPerCopy | Predefined high-resolution metrics, 10-second resolution | Sudden traffic bursts, LLMs that stream responses as tokens | | CPUUtilization, GPUUtilization, MemoryUtilization | Custom metrics | Compute-heavy models where the CPU or GPU hits its limit first | | ModelLatency | Custom metric | When you manage a response-time target (SLA) directly | | ApproximateBacklogSizePerInstance | Custom metric | Queue length of an asynchronous endpoint |
High-resolution metrics are emitted every 10 seconds, so they scale out much faster when traffic grows. Scale-in, however, happens at the same speed as with standard metrics. CPUUtilization can exceed 100% up to the number of vCPUs on the instance (up to 400% with 4 vCPUs), so be careful when you set the target value.
Exam tip: "Scaling is too slow for a sudden jump in concurrent requests, causing latency" is the high-resolution concurrent requests metric. "The model uses up the GPU and slows down" is a custom metric based on GPUUtilization.
---
Cooldowns, Capacity Limits, and Load Testing
Cooldown: Keeping Capacity from Swinging Too Often
A cooldown is a waiting period that delays the next adjustment after one has been made. If you do not specify one, the default of 300 seconds applies to both scale-out and scale-in.
Instances are added and removed too often: increase the cooldown Instances arrive too slowly for growing traffic: decrease the scale-out cooldown Scale in slowly, scale out quickly: make the scale-in cooldown longer than the scale-out cooldown (for example, 600 and 300 seconds)
Minimum and Maximum Capacity
The minimum capacity of a regular real-time endpoint must be at least 1, and when traffic drops to zero the endpoint scales in to that minimum. SageMaker does not enforce a separate limit on the maximum, but it cannot exceed your account's instance quota. To scale out manually in a hurry, raise the minimum; to scale in, lower the maximum.
Set the Target Value with a Load Test
The method AWS recommends is to find, with a load test, the maximum requests per second (MAX_RPS) one instance can handle within your latency target, and then multiply it by a safety factor to set the target.
Target InvocationsPerInstance = MAX_RPS × safety factor × 60 Start with a safety factor of 0.5 and adjust it as you test
Exam tip: "Scaling happens so often that instances keep being added and removed" is solved by increasing the cooldown.
---
Endpoints That Can Scale to Zero
Keeping instances running all night for a model that gets no requests wastes money. Whether an endpoint can go down to zero depends on its type.
| Type | Scale to zero | How, and what to watch for | |------|------|------| | Real-time endpoint (regular) | No | Minimum capacity of at least 1 | | Real-time with inference components | Yes | Set the minimum instance count to 0; scaling back up from zero needs a step policy and an alarm, and requests fail during the few minutes of provisioning | | Asynchronous endpoint | Yes | Minimum capacity of 0, requests wait in the queue; adding a HasBacklogWithoutCapacity step policy scales out on the first request | | Serverless endpoint | Automatic | Goes to zero automatically when idle; the first request causes a cold start |
Cold starts in serverless inference are reduced with provisioned concurrency. The concurrency you specify stays warm and ready, and that value can be adjusted with Application Auto Scaling based on a target metric or a schedule. Serverless endpoints support 1–6 GB of memory and a maximum concurrency of 200 per endpoint, and they do not support GPUs, VPC configuration, multi-model endpoints, data capture, or Model Monitor.
Exam tip: "Infrequent requests, waiting a few minutes is acceptable, and payloads are large" is an asynchronous endpoint that scales to zero. "Intermittent short requests, but the latency of the first response must be reduced" is serverless with provisioned concurrency.
---
Choosing an Instance Type: Inference Recommender
Auto scaling decides "how many," but before that you have to decide "which instance." SageMaker Inference Recommender actually deploys the model to several instance types, runs load tests, and gives recommendations that compare performance and cost.
| Job type | What it does | Duration | |------|------|------| | Default (Inference recommendations) | Automatic load tests on recommended candidate instances | Usually within 45 minutes | | Advanced (Endpoint recommendations) | Custom load tests where you choose the instances, traffic pattern, and latency target | Around 2 hours on average |
It works for both real-time and serverless endpoints, and you pay only for the instances used while the job runs. For inference resources running outside SageMaker, such as EC2, Lambda, and ECS on Fargate, AWS Compute Optimizer recommends rightsizing based on recent utilization history.
Also keep the basic sense of hardware choice in mind. Light inference, such as models for tabular data, is fine on CPU instances, while deep learning inference fits GPU instances or AWS Inferentia-based instances (ml.inf2). Inferentia is a chip built for inference, an option that delivers the same throughput at lower cost.
---
Choosing a Deployment Target: SageMaker, EKS, ECS, or Lambda
Not every model has to live on a SageMaker endpoint. If the company already has a platform it uses, deploying onto that platform is often the right answer.
| Deployment target | Strengths | How it scales | Clue in the question | |------|------|------|------| | SageMaker AI endpoint | Fully managed, ML-specific features such as Model Monitor, deployment guardrails, and multi-model | Application Auto Scaling | "minimal operational overhead," "together with mod