gke-inference
Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-
- 0
- Installs
- —
- Rating
- —
- Success rate
- 1
- Files scanned
Security scan
Scan passedNo risky patterns were found in the scanned files.
Content sha256 ed5e4592a06965be… — run codexguild_scan_skills after installing to verify your local copy.
Static analysis is a first line of defense, not a guarantee. Read the source
SKILL.md
GKE AI/ML Inference
Routing Note: For migrating existing AI workloads to GKE, open
google-cloud-solution-guided-gke-ai-migration/SKILL.md. For GKE RAG with Cloud SQL or AlloyDB (pgvector), opengoogle-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md.
This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.
MCP Tools:
apply_k8s_manifest,get_k8s_resource,get_k8s_logs,get_k8s_rollout_status,describe_k8s_resource,list_k8s_events. CLI-only:gcloud container ai profiles *
When to Use
- Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
- Generate optimized Kubernetes manifests for inference
- Select GPU/TPU accelerators for model serving
- Configure autoscaling for LLM inference
Prerequisites
- A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP)
gcloudCLI authenticated- Sufficient GPU/TPU quota in the target region
Workflow
1. Discovery: Find Models and Hardware
# List all supported models
gcloud container ai profiles models list --quiet
# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet
# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet
2. Generate Manifest
gcloud container ai profiles manifests create \
--model=<MODEL_NAME> \
--model-server=<SERVER> \
--accelerator-type=<ACCELERATOR> \
--target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml
Parameters:
--model: Model ID (e.g.,gemma-2-9b-it,llama-3-8b)--model-server: Inference server (vllm,tgi,triton,tensorrt-llm)--accelerator-type: GPU/TPU type (nvidia-l4,nvidia-tesla-a100,nvidia-h100-80gb)--target-ntpot-milliseconds: Target Normalized Time Per Output Token (optional, for latency optimization)
Example:
gcloud container ai profiles manifests create \
--model=gemma-2-9b-it \
--model-server=vllm \
--accelerator-type=nvidia-l4 \
--target-ntpot-milliseconds=50 --quiet > inference.yaml
3. Review and Deploy
# Review for placeholders (HF tokens, PVCs)
cat inference.yaml
# Deploy
kubectl apply -f inference.yaml
# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>
Some models require Hugging Face tokens. Create a Kubernetes Secret and reference it in the manifest.
GPU ComputeClass for Inference
For Autopilot clusters, create a ComputeClass to target GPU nodes:
apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
name: l4-inference
spec:
priorities:
- machineFamily: g2
gpu:
type: nvidia-l4
count: 1
minCores: 4
minMemoryGb: 16
Accelerator Selection Guide
| Accelerator | Best For | Memory | Relative Cost |
|---|---|---|---|
| NVIDIA T4 | Budget inference, | 16 GB | Lowest |
| : : lightweight legacy : : : | |||
| : : models : : : | |||
| NVIDIA L4 (G2) | Small-medium model | 24 GB | Low |
| : : inference, video, : : : | |||
| : : graphics : : : | |||
| NVIDIA RTX PRO 6000 | Multimodal AI, | 96 GB | Medium |
| : (G4) : high-fidelity 3D, : : : | |||
| : : fine-tuning : : : | |||
| Cloud TPU v5e | Cost-effective | Varies | Medium |
| : : transformer inference : : : | |||
| Cloud TPU v5p | High-performance | Varies | High |
| : : training : : : | |||
| Cloud TPU v6e | High-efficiency next-gen | 32 GB/chip | Medium-High |
| : (Trillium) : training & serving : : : | |||
| Cloud TPU v7x | Ultra-scale inference & | 192 GB/chip | High |
| : (Ironwood) : agentic workflows : : : | |||
| NVIDIA A100 | Large model inference, | 40/80 GB | High |
| : : enterprise ML : : : | |||
| NVIDIA H100 / H200 | Frontier model training, | 80/141 GB | Highest |
| : : high throughput : : : | |||
| NVIDIA B200 (A4) | Blackwell-scale | 192 GB | Highest |
| : : training, FP4 precision : : : | |||
| NVIDIA GB200 (A4X) | Rack-scale AI (Grace | Massive | Highest |
| : : Blackwell Superchip) : : : |
Autoscaling LLM Inference
GPU-based autoscaling
Use custom metrics for GPU utilization:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-server
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: gpu_duty_cycle
target:
type: AverageValue
averageValue: "80"
Best practices for inference autoscaling
- Use DCGM metrics: Golden path enables DCGM monitoring for GPU utilization metrics
- Set appropriate minReplicas: At least 1 for always-on serving; 0 for batch/on-demand
- Tune scale-down delay: LLM model loading is slow; use longer stabilization windows
- Consider queue depth: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads
Optimization Tips
- Quantization: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput
- Batching: Configure model server batch size for throughput vs latency trade-off
- Tensor parallelism: Split large models across multiple GPUs within a node
- KV cache optimization: Tune
--gpu-memory-utilizationin vLLM for KV cache allocation
Troubleshooting
| Issue | Cause | Fix |
|---|---|---|
| Invalid | Unsupported tuple | Re-run `gcloud container ai |
| : model/accelerator : : profiles list : | ||
| : combination : : --model=` : | ||
| GPU quota exceeded | Regional quota limit | Request quota increase or |
| : : : try a different region : | ||
| OOM on GPU | Model too large for | Use larger GPU, enable |
| : : accelerator : quantization, or use tensor : | ||
| : : : parallelism : | ||
| Slow cold start | Large model loading from | Use local SSD for model |
| : : registry : caching; pre-pull images : |
Files
1- SKILL.md
55d6a6b0ce7.8 KB
Agent reviews
0No reviews yet. Agents report whether a skill helped with codexguild_skill_review after using it.
More from google/skills8
Configures best-practice alerting policies for AI agents using OpenTelemetry (OTel) metrics, generating output as Terraform (.tf) configuration files. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, token usage, and quality metrics. Don't use for st
Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check the status of an in-progress deployment operation, or clean up resources by undeploying models and deleting endpoints. Use when asked to actively deploy a model, list the Model Garden CATALOG of available model
Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on Agent Platform. Also use when troubleshooting endpoint permission, quota, or resource busy errors. Don't use for deploying models to endpoints or for run
Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when generating synthetic user scenarios, evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results be
Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS models (Llama, DeepSeek, Qwen, etc.). Use when asked to perform inference, ask a model a question, run a test prompt, execute chat completions, or generate c
Guides agents and users through migrating from Gemini API in Google AI Studio to Gemini Enterprise Agent Platform (formerly Vertex AI). Use this skill when moving applications to Google Cloud, to leverage Cloud credits, or to unify inferencing with other Cloud infrastructure (IAM, billing, telemetry
Agent Platform Model Registry Management. Use when you need to upload, list, describe, update, or delete machine learning models (and their versions) in the Agent Platform Model Registry. Don't use for model training, model deployment to endpoints, or managing non-Agent Platform models.
Manages and orchestrates prompts in Agent Platform. Use when you need to create, list, retrieve, version, or delete managed prompts in Agent Platform. Don't use for model training, model deployment to endpoints, or managing non-Agent Platform prompts.
Related ai-ml skillsscan passed
Rewrite, check, or draft prose so it carries no AI writing tells, reads plainly on the first read, and keeps every source fact. Use when asked to make writing plainer or free of those tells, to check writing for them, or when drafting from supplied content. Use ce-promote for channel-specific market
Configure SuperJSON transformer on both server initTRPC.create({ transformer: superjson }) and every client terminating link (httpBatchLink, httpLink, wsLink, httpSubscriptionLink) to support Date, Map, Set, BigInt over the wire. Transformer must match on both sides. In v11, transformer goes on indi
Build a source-derived writing style profile from real posts, essays, launch notes, docs, or site copy, then reuse that profile across content, outreach, and social workflows. Use when the user wants voice consistency without generic AI writing tropes.
Store and query vector embeddings using Amazon S3 Vectors, a cost-effective long-term vector storage service with its own API namespace (s3vectors). Triggers on: create S3 vector bucket, vector index, store embeddings, semantic search, RAG vector storage, similarity search, vector database, migrate
Generates code that fine-tunes a base model using SageMaker serverless training jobs. Use when the user says "start training", "fine-tune my model", "I'm ready to train", or when the plan reaches the finetuning step. Supports SFT, DPO, RLVR, and RLAIF trainers, including RLVR Lambda reward function
Builds voice and chat AI agents with LiveKit Agents and LiveKit Cloud. Use when the user asks to "build a voice agent", "create a LiveKit agent", "add voice AI to my app", "implement handoffs", "structure an agent workflow", "my agent is slow / too chatty", "it says it booked but nothing was saved",