RayService Autoscaling Example
Ray’s built-in autoscaler that grows or shrinks your cluster automatically whenever its current set of allocated nodes cannot satisfy pending resource requests. The below example shows basic configuration options that demonstrate how this can be configured, and how to test it. We’ll use the load-generation tool Blazemeter, which generates client calls to the LLM for long periods of time in order to trigger autoscaling.
See the official Ray documentation for more information on basic autoscaling here:
https://docs.ray.io/en/latest/serve/autoscaling-guide.html#resnet-autoscaling-example
Those docs define the important autoscaling parameters that we use in this example as:
target_ongoing_requests is the average number of ongoing requests per replica that the Serve autoscaler tries to ensure. You can adjust it based on your request processing length (the longer the requests, the smaller this number should be) as well as your latency objective (the shorter you want your latency to be, the smaller this number should be).
max_replicas is the maximum number of replicas for the deployment. Set this to ~20% higher than what you think you need for peak traffic.
Here is a RayService YAML which includes an autoscaling_config section, and deploys the Llama 3.2-3b LLM from CakeFS (shared EFS drive in your Cake cluster):
apiVersion: ray.io/v1
kind: RayService
metadata:
name: llama32-3b-autoscale
namespace: ray-service
spec:
serveConfigV2: |
applications:
- name: llama32-3b-autoscale
route_prefix: "/"
import_path: "ray.serve.llm:build_openai_app"
runtime_env:
env_vars:
VLLM_USE_V1: "0"
ENGINE_START_TIMEOUT_S: "1600"
SERVE_LOG_LEVEL: "DEBUG"
VLLM_LOGGING_LEVEL: "DEBUG"
HF_TOKEN: "YOUR_HF_TOKEN"
args:
llm_configs:
- model_loading_config:
model_id: "Llama-3.2-3B-Instruct"
model_source: "/home/ray/shared/models/models--meta-llama--Llama-3.2-3B-Instruct/snapshots/0cb88a4f764b7a12671c53f0838cd831a0843b95"
deployment_config:
health_check_timeout_s: 600
autoscaling_config:
min_replicas: 1
max_replicas: 4
target_ongoing_requests: 1
engine_kwargs:
tensor_parallel_size: 1
pipeline_parallel_size: 1
tokenizer_pool_size: 2
tokenizer_pool_extra_config: "{\"runtime_env\": {}}"
max_model_len: 8192
trust_remote_code: true
rayClusterConfig:
rayVersion: '2.47.1' # should match the Ray version in the image of the containers
enableInTreeAutoscaling: true
headGroupSpec:
rayStartParams:
dashboard-host: '0.0.0.0'
template:
metadata:
annotations:
karpenter.sh/do-not-disrupt: "true"
spec:
containers:
- env:
- name: RAY_GRAFANA_IFRAME_HOST
value: "https://<CLUSTER_NAME>/grafana"
- name: RAY_GRAFANA_HOST
value: "http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80"
- name: RAY_PROMETHEUS_HOST
value: "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
name: ray-head
image: rayproject/ray-llm:2.47.1-py311-cu124
imagePullPolicy: Always
resources:
requests:
cpu: 500m
memory: 1200Mi
# Allows us to use ptrace for debugging (CPU Flame Chart)
securityContext:
capabilities:
add:
- SYS_PTRACE
volumeMounts:
- name: log-volume
mountPath: /tmp/ray
- name: ray-service-shared-volume
mountPath: /home/ray/shared
ports:
- containerPort: 6379
name: gcs-server
protocol: TCP
- containerPort: 8265 # Ray dashboard
name: dashboard
protocol: TCP
- containerPort: 10001
name: client
protocol: TCP
- containerPort: 8000
name: serve
protocol: TCP
volumes:
- name: log-volume
emptyDir: {}
- name: ray-service-shared-volume
persistentVolumeClaim:
claimName: shared-volume
workerGroupSpecs:
- groupName: gpu
minReplicas: 1
maxReplicas: 4
numOfHosts: 1
rayStartParams:
num-gpus: "1"
template:
metadata:
annotations:
karpenter.sh/do-not-disrupt: "true"
spec:
nodeSelector:
karpenter.sh/nodepool: ray-serve-gpu
karpenter.k8s.aws/instance-family: g6e
containers:
- env:
name: ray-worker
image: rayproject/ray-llm:2.47.1-py311-cu124
imagePullPolicy: Always
resources:
limits:
nvidia.com/gpu: 1
requests:
cpu: 6000m
memory: 20Gi
# Allows us to use ptrace for debugging (CPU Flame Chart)
securityContext:
capabilities:
add:
- SYS_PTRACE
volumeMounts:
- mountPath: /tmp/ray
name: log-volume
- name: ray-service-shared-volume
mountPath: /home/ray/shared
ports:
- containerPort: 6379
name: gcs-server
protocol: TCP
- containerPort: 8265 # Ray dashboard
name: dashboard
protocol: TCP
- containerPort: 10001
name: client
protocol: TCP
- containerPort: 8000
name: serve
protocol: TCP
volumes:
- name: log-volume
emptyDir: {}
- name: ray-service-shared-volume
persistentVolumeClaim:
claimName: shared-volumeThe important section for autoscaling is below:
autoscaling_config:
min_replicas: 1
max_replicas: 4
target_ongoing_requests: 1
The max_replicas: 4 tells the autoscaler to launch up to 4 GPU workers, and we’ve set the target_ongoing_requests parameter to a low value to encourage the autoscaler to eagerly launch workers to keep the number of queued client requests to a minimum.
Note: Alternative syntax for autoscaling using the num_replicas parameter also exists. See below. However, we recommend the previous syntax for more control
num_replicas: autoAfter applying this yaml, we can start Blazemeter, using a YAML file such as the following, to start 10 virtual users that ramp up over 10 seconds, and hold that load constant for 30 minutes. The think-time is set to 5s to allow for a slight delay between virtual users’ requests.
load-test.yaml
version: 1
execution:
- scenario: llama_chat # reference to the scenario below
concurrency: 10 # peak VUs
ramp-up: 10s # time to reach peak (1 → 10)
steps: 10 # linear ramp (start at 1, add 1 each step)
hold-for: 30m # keep peak load; adjust or remove as needed
scenarios:
llama_chat:
default-address: http://localhost:8000
requests:
- label: chat_completion
method: POST
url: /v1/chat/completions
headers:
Content-Type: application/json
body: |
{
"model": "Llama-3.2-3B-Instruct",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user",
"content": "${__RandomFromList(Describe the moon.,Name three prime numbers.,Who wrote Faust?,Give me a haiku on rain.,Translate 'good morning' to Spanish.,)}"
}
],
"temperature": 0.9,
"max_tokens": 2000
}
think-time: 5s # per-VU pause before the next iterationTo start the Blazemeter test, port-forward to the ray service port 8000, and run:
uv run --with bzt bzt -report load-test.yamlAs it runs, you’ll see statistics and response codes (HTTP 200):
For more about Blazemeter usage, see docs at https://github.com/Blazemeter/taurus/blob/master/site/dat/docs/CommandLine.md and https://gettaurus.org/kb/Basic1/#Scaling-With-Cloud-Provisioning?utm_source=BM&utm_medium=kb&utm_campaign=creating-a-new-taurus-test
You can use k9s or Lens to open a shell in one of the GPU workers to see the low-level details of GPU, CPU, and RAM usage on the worker nodes.
You can use the following commands to install and run the excellent nvitop command:
python3 -m pip install --user pipx
python3 -m pipx ensurepath
. ~/.bashrc
pipx run nvitop -m full -c --colorfulNvitop shows GPU memory and utilization, driver version, CUDA version, as well as GPU type, CPU and RAM, and much more:
Once the autoscaler launches pods, Karpenter will go to work launching Kubernetes nodes to accommodate them.
They will appear in k9s:
On the Ray Dashboard, the Replica count will show 4:
You can also run Ray CLI commands to find out more about what’s running. For example, running ray-status will show 4 GPUs active:
(base) ray@llama32-3b-autoscale-raycluster-gxq4h-gpu-worker-c2wsl:~$ ray status
======== Autoscaler status: 2025-07-07 13:43:31.532417 ========
Node status
---------------------------------------------------------------
Active:
4 gpu
1 headgroup
Pending:
(no pending nodes)
Recent failures:
(no failures)
Resources
---------------------------------------------------------------
Total Usage:
6.0/25.0 CPU (4.0 used of 4.0 reserved in placement groups)
4.0/4.0 GPU (4.0 used of 4.0 reserved in placement groups)
0B/1.09TiB memory
118.42KiB/79.85GiB object_store_memory
Total Constraints:
(no request_resources() constraints)
Total Demands:
(no resource demands)The Ray Dashboard Cluster page shows details on each node, including GRAM and GPU usage:
Clicking on the View Config, you can see the full set of parameters that Ray used, including the default ones that we did not specify, including smoothing_factor, downscale_delay_s and many more:
Once the load test is complete, Ray will wait 10 minutes (by default) to shut down the worker pods, and Kubernetes will shut down the GPU nodes.
In the ray Cluster Dashboard, the workers that were scaled-down will show as “Dead”, with a reason “Expected termination” stating that they were intentionally terminated by the autoscaler.
Finally, the Ray Serve Dashboard shows that the replicas is now back to 1:
Advanced Ray Serve Autoscaling
See the Ray documentation for coverage of more advanced configuration settings:
https://docs.ray.io/en/latest/serve/advanced-guides/advanced-autoscaling.html
See the official docs for the AutoscalingConfig settings:
https://docs.ray.io/en/latest/serve/api/doc/ray.serve.config.AutoscalingConfig.html
Conclusion
This example demonstrates how to configure and test Ray's autoscaling capabilities for a RayService deployment. By leveraging tools like Blazemeter for load generation and observing the behavior through the Ray Dashboard and CLI commands, you can effectively confirm that your Ray cluster scales out to handle increased demand and scales back down when the load subsides, optimizing resource utilization and maintaining service availability.