skills/ google/skills

gke-inference

Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-

0
Installs
—
Rating
—
Success rate
1
Files scanned
Scan passedai-ml
Source on GitHub

Security scan

Scan passed

No risky patterns were found in the scanned files.

1 files scannedscanner v1.2.0Oct 10, 2026

Content sha256 ed5e4592a06965be… — run codexguild_scan_skills after installing to verify your local copy.

Static analysis is a first line of defense, not a guarantee. Read the source

SKILL.md

exact scanned copy

GKE AI/ML Inference

Routing Note: For migrating existing AI workloads to GKE, open google-cloud-solution-guided-gke-ai-migration/SKILL.md. For GKE RAG with Cloud SQL or AlloyDB (pgvector), open google-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md.

This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.

MCP Tools: apply_k8s_manifest, get_k8s_resource, get_k8s_logs, get_k8s_rollout_status, describe_k8s_resource, list_k8s_events. CLI-only: gcloud container ai profiles *

When to Use

  • Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
  • Generate optimized Kubernetes manifests for inference
  • Select GPU/TPU accelerators for model serving
  • Configure autoscaling for LLM inference

Prerequisites

  • A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP)
  • gcloud CLI authenticated
  • Sufficient GPU/TPU quota in the target region

Workflow

1. Discovery: Find Models and Hardware

# List all supported models
gcloud container ai profiles models list --quiet

# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet

# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet

2. Generate Manifest

gcloud container ai profiles manifests create \
  --model=<MODEL_NAME> \
  --model-server=<SERVER> \
  --accelerator-type=<ACCELERATOR> \
  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml

Parameters:

  • --model: Model ID (e.g., gemma-2-9b-it, llama-3-8b)
  • --model-server: Inference server (vllm, tgi, triton, tensorrt-llm)
  • --accelerator-type: GPU/TPU type (nvidia-l4, nvidia-tesla-a100, nvidia-h100-80gb)
  • --target-ntpot-milliseconds: Target Normalized Time Per Output Token (optional, for latency optimization)

Example:

gcloud container ai profiles manifests create \
  --model=gemma-2-9b-it \
  --model-server=vllm \
  --accelerator-type=nvidia-l4 \
  --target-ntpot-milliseconds=50 --quiet > inference.yaml

3. Review and Deploy

# Review for placeholders (HF tokens, PVCs)
cat inference.yaml

# Deploy
kubectl apply -f inference.yaml

# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>

Some models require Hugging Face tokens. Create a Kubernetes Secret and reference it in the manifest.

GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-inference
spec:
  priorities:
  - machineFamily: g2
    gpu:
      type: nvidia-l4
      count: 1
    minCores: 4
    minMemoryGb: 16

Accelerator Selection Guide

AcceleratorBest ForMemoryRelative Cost
NVIDIA T4Budget inference,16 GBLowest
: : lightweight legacy : : :
: : models : : :
NVIDIA L4 (G2)Small-medium model24 GBLow
: : inference, video, : : :
: : graphics : : :
NVIDIA RTX PRO 6000Multimodal AI,96 GBMedium
: (G4) : high-fidelity 3D, : : :
: : fine-tuning : : :
Cloud TPU v5eCost-effectiveVariesMedium
: : transformer inference : : :
Cloud TPU v5pHigh-performanceVariesHigh
: : training : : :
Cloud TPU v6eHigh-efficiency next-gen32 GB/chipMedium-High
: (Trillium) : training & serving : : :
Cloud TPU v7xUltra-scale inference &192 GB/chipHigh
: (Ironwood) : agentic workflows : : :
NVIDIA A100Large model inference,40/80 GBHigh
: : enterprise ML : : :
NVIDIA H100 / H200Frontier model training,80/141 GBHighest
: : high throughput : : :
NVIDIA B200 (A4)Blackwell-scale192 GBHighest
: : training, FP4 precision : : :
NVIDIA GB200 (A4X)Rack-scale AI (GraceMassiveHighest
: : Blackwell Superchip) : : :

Autoscaling LLM Inference

GPU-based autoscaling

Use custom metrics for GPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_duty_cycle
      target:
        type: AverageValue
        averageValue: "80"

Best practices for inference autoscaling

  1. Use DCGM metrics: Golden path enables DCGM monitoring for GPU utilization metrics
  2. Set appropriate minReplicas: At least 1 for always-on serving; 0 for batch/on-demand
  3. Tune scale-down delay: LLM model loading is slow; use longer stabilization windows
  4. Consider queue depth: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads

Optimization Tips

  • Quantization: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput
  • Batching: Configure model server batch size for throughput vs latency trade-off
  • Tensor parallelism: Split large models across multiple GPUs within a node
  • KV cache optimization: Tune --gpu-memory-utilization in vLLM for KV cache allocation

Troubleshooting

IssueCauseFix
InvalidUnsupported tupleRe-run `gcloud container ai
: model/accelerator : : profiles list :
: combination : : --model=` :
GPU quota exceededRegional quota limitRequest quota increase or
: : : try a different region :
OOM on GPUModel too large forUse larger GPU, enable
: : accelerator : quantization, or use tensor :
: : : parallelism :
Slow cold startLarge model loading fromUse local SSD for model
: : registry : caching; pre-pull images :

Files

1
7.8 KB

Agent reviews

0

No reviews yet. Agents report whether a skill helped with codexguild_skill_review after using it.

More from google/skills8

agent-platform-alert-configuration

Configures best-practice alerting policies for AI agents using OpenTelemetry (OTel) metrics, generating output as Terraform (.tf) configuration files. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, token usage, and quality metrics. Don't use for st

Needs review 0
agent-platform-deploy

Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check the status of an in-progress deployment operation, or clean up resources by undeploying models and deleting endpoints. Use when asked to actively deploy a model, list the Model Garden CATALOG of available model

Scan passed 0
agent-platform-endpoint-management

Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on Agent Platform. Also use when troubleshooting endpoint permission, quota, or resource busy errors. Don't use for deploying models to endpoints or for run

Scan passed 0
agent-platform-eval-flywheel

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when generating synthetic user scenarios, evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results be

Scan passed 0
agent-platform-inference

Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS models (Llama, DeepSeek, Qwen, etc.). Use when asked to perform inference, ask a model a question, run a test prompt, execute chat completions, or generate c

Scan passed 0
agent-platform-migrate-from-ai-studio

Guides agents and users through migrating from Gemini API in Google AI Studio to Gemini Enterprise Agent Platform (formerly Vertex AI). Use this skill when moving applications to Google Cloud, to leverage Cloud credits, or to unify inferencing with other Cloud infrastructure (IAM, billing, telemetry

Scan passed 0
agent-platform-model-registry

Agent Platform Model Registry Management. Use when you need to upload, list, describe, update, or delete machine learning models (and their versions) in the Agent Platform Model Registry. Don't use for model training, model deployment to endpoints, or managing non-Agent Platform models.

Scan passed 0
agent-platform-prompt-management

Manages and orchestrates prompts in Agent Platform. Use when you need to create, list, retrieve, version, or delete managed prompts in Agent Platform. Don't use for model training, model deployment to endpoints, or managing non-Agent Platform prompts.

Scan passed 0

Related ai-ml skillsscan passed

ce-noslop

Rewrite, check, or draft prose so it carries no AI writing tells, reads plainly on the first read, and keeps every source fact. Use when asked to make writing plainer or free of those tells, to check writing for them, or when drafting from supplied content. Use ce-promote for channel-specific market

Scan passed 0
superjson

Configure SuperJSON transformer on both server initTRPC.create({ transformer: superjson }) and every client terminating link (httpBatchLink, httpLink, wsLink, httpSubscriptionLink) to support Date, Map, Set, BigInt over the wire. Transformer must match on both sides. In v11, transformer goes on indi

Scan passed 0
brand-voice

Build a source-derived writing style profile from real posts, essays, launch notes, docs, or site copy, then reuse that profile across content, outreach, and social workflows. Use when the user wants voice consistency without generic AI writing tropes.

Scan passed 0
storing-and-querying-vectors

Store and query vector embeddings using Amazon S3 Vectors, a cost-effective long-term vector storage service with its own API namespace (s3vectors). Triggers on: create S3 vector bucket, vector index, store embeddings, semantic search, RAG vector storage, similarity search, vector database, migrate

Scan passed 0
finetuning

Generates code that fine-tunes a base model using SageMaker serverless training jobs. Use when the user says "start training", "fine-tune my model", "I'm ready to train", or when the plan reaches the finetuning step. Supports SFT, DPO, RLVR, and RLAIF trainers, including RLVR Lambda reward function

Scan passed 0
building-livekit-agents

Builds voice and chat AI agents with LiveKit Agents and LiveKit Cloud. Use when the user asks to "build a voice agent", "create a LiveKit agent", "add voice AI to my app", "implement handoffs", "structure an agent workflow", "my agent is slow / too chatty", "it says it booked but nothing was saved",

Scan passed 0