Amazon SageMaker HyperPod adds data capture, Hugging Face, NVMe

Amazon SageMaker HyperPod has added a set of inference features aimed at enterprise production workloads: multi-tier data capture, direct deployments from the Hugging Face Hub, node-local NVMe weight loading with cloud fallback, and automatic Route 53 DNS record management. These capabilities include declarative CRD configuration, pod-level IAM controls, and runtime compatibility with vLLM, TGI, and SGLang.

What are the new HyperPod inference features?

HyperPod now supports three explicit capabilities: configurable data capture at the SageMaker endpoint, the Application Load Balancer, and at the model pod; direct model sourcing from the Hugging Face Hub with token isolation and revision pinning; and automatic DNS record creation and management via Route 53 tied to ACM certificates. The release also highlights node-local NVMe for loading weights to reduce cold-start latency, automatic fallback to cloud storage, and granular pod-level AWS IAM permissions for security boundaries.

Beyond those items, the feature set exposes declarative CRD fields for dataCapture, tlsConfig, and dnsConfig, and lists built-in support for inference runtimes vLLM, TGI, and SGLang. The Hugging Face source emits Kubernetes events to report deployment success or failure.

How does data capture work?

Data capture captures inference traffic at three tiers: Tier 1 at the SageMaker AI endpoint, Tier 2 at the Application Load Balancer, and Tier 3 at the model pod, and you can enable any combination. Tier 1 writes to {s3Uri}/{hash}/sme/, requires endpoint registration, and is compatible with SageMaker AI Model Monitor; Tier 2 enables ALB access logs for client IPs, request paths, and latencies; Tier 3 captures full input and output payloads at the inference container and defaults to 100 percent sampling.

Configuration requires an S3 bucket (for example s3://amzn-s3-demo-bucket/captures/) and appropriate IAM permissions for the operator to write to it; if you do not provide one, HyperPod uses the TLS certificate bucket. The dataCapture CRD can set initialSamplingPercentage (Tier 1 and Tier 3 examples use 100), bufferConfig.batchSize (example value 100), bufferConfig.flushIntervalSeconds (example value 60), and payloadConfig.maxPayloadSizeKB (example value 1024). The documentation recommends adding a kmsKeyId to encrypt captured data and controlling sampling to manage cost.

The Inference Operator requires an S3 PutObject permission scoped to ARN patterns such as "arn:aws:s3:::hyperpod-tls*/data-capture/*" and, if using a customer-managed KMS key, kms:Decrypt and kms:GenerateDataKey permissions with conditions tied to S3 and the operator account.

How do Hugging Face, NVMe, and Route 53 integration work?

You can deploy models directly from the Hugging Face Hub without pre-staging weights in S3 or FSx, using tokenSecretRef for gated models, commitSHA for revision pinning, and token isolation; the flow supports vLLM, TGI, and SGLang runtimes. The sample YAML shows a modelId of "mistralai/Mistral-7B-Instruct-v0.3", a token secret created via kubectl create secret generic hf-token-secret, and an instanceType example of "ml.g5.24xlarge".

For performance, HyperPod can load weights from node-local NVMe storage to reduce cold-start latency and will automatically fall back to cloud storage when needed. For custom domains, add tlsConfig and dnsConfig to your deployment YAML with an ACM certificate ARN (example acmArn arn:aws:acm:us-west-2:123456789012:certificate/abc12345-1234-1234-1234-abc123456789) and a Route 53 hostedZoneId (example Z1234567890ABC). The operator requires IAM permissions for acm:DescribeCertificate and acm:GetCertificate, S3 PutObject on the TLS bucket, and Route 53 actions such as route53:ChangeResourceRecordSets against the hosted zone.

Why it matters

Enterprises get two practical wins: deeper, configurable observability into inference traffic for monitoring and auditing, and simpler deployment paths for community models. Pod-level capture and the ability to pin Hugging Face revisions reduce operational friction for teams that must balance governance and speed. Node-local NVMe addresses a common production complaint, cold-start latency, while Route 53 automation reduces manual DNS operations.

What to watch

Adoption of Tier 3 pod-level capture with controlled sampling and kmsKeyId settings will show whether teams prioritize deep observability over storage cost. Also watch deployments pulling modelId values such as "mistralai/Mistral-7B-Instruct-v0.3" from the Hugging Face Hub to see how often teams use direct Hub sourcing versus pre-staging weights.