Topic · Automation and Agents
Inference and Model Serving skills for Claude Code and Codex
Skills for running models fast and cheaply: serving with vLLM, SGLang and TensorRT-LLM, local inference with llama.cpp and GGUF, quantization with AWQ, GPTQ, HQQ and bitsandbytes, speculative decoding, Jetson edge serving, GKE inference and managed endpoints on Google's Agent Platform and Replicate. They help an agent pick a serving stack and tune it to the hardware.
Highest-scoring inference skills in the October 22, 2026 snapshot: agent-platform-deploy from google/skills (90/100), huggingface-community-evals from huggingface/skills (86/100) and speculative-decoding from Orchestra-Research/AI-Research-SKILLs (77/100). Scores come from the Suede Skill Check.
- 39 listed skills
- 6 repositories
- 7 with full pages
26 from official publishers. 8 bundle scripts. 5 connect to an outside service. Listed in AI Agents and LLM Engineering (35), DevOps and Cloud (3), Developer Tools and Frameworks (1).
All 39 inference skills
No matching skills. Try fewer words or clear your search.
| Skill | Repository | What it does |
|---|---|---|
| agent-platform-deploy 90/100 · A | google/skills | Deploys open models or custom weights from Model Garden to Agent Platform endpoints, checks deployment status and cost, troubleshoots errors, and undeploys models. |
| build-models 71/100 · C | replicate/skills | Packages and builds custom AI models with Cog for deployment on Replicate, including cog.yaml, predict.py, model inputs and outputs, and Docker images. |
| hf-cloud-serving-image-selection 84/100 · B | huggingface/skills | Picks the right serving container for a SageMaker model deployment and finds its current image URI, preferring HuggingFace Deep Learning Containers. |
| huggingface-community-evals 86/100 · B | huggingface/skills | Runs evaluations for Hugging Face Hub models on local hardware using inspect-ai and lighteval, including backend selection and choosing between vLLM, Transformers, and accelerate. |
| huggingface-local-models 80/100 · B | huggingface/skills | Selects models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm, covering quant selection, conversion, and OpenAI-compatible serving. |
| huggingface-zerogpu 85/100 · B | huggingface/skills | Guides writing and reviewing Gradio code for Hugging Face ZeroGPU Spaces, covering @spaces.GPU usage, requirements configuration, and ZeroGPU-specific constraints like pickle isolation. |
| speculative-decoding 77/100 · C | Orchestra-Research/AI-Research-SKILLs | Accelerates LLM inference with speculative decoding, Medusa multiple heads, and lookahead decoding, covering draft models, tree-based attention, and parallel token generation. |
| agent-platform-endpoint-management 84/100 · B | google/skills | Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on… |
| agent-platform-inference 70/100 · C | google/skills | Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS… |
| aks-gpu-inference 84/100 · B | microsoft/azure-skills | Diagnose Day-2 AKS GPU and KAITO incidents using profile-aware, read-only evidence. WHEN: 'Insufficient nvidia.com/gpu', GPU pod Pending, model-load… |
| awq-quantization 69/100 · D | Orchestra-Research/AI-Research-SKILLs | Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B)… |
| deepstream-sop 77/100 · C | NVIDIA/skills | Use this skill when building, deploying, evaluating, debugging, or measuring latency for the DeepStream SOP Inference Microservice — a… |
| fine-tuning-serving-openpi 69/100 · D | Orchestra-Research/AI-Research-SKILLs | Fine-tune and serve Physical Intelligence OpenPI models (pi0, pi0-fast, pi0.5) using JAX or PyTorch backends for robot policy inference across… |
| foundationpose-setup 81/100 · B | NVIDIA/skills | Install or repair the FoundationPose perception pipeline and build its FoundationStereo TensorRT engines. Use for SAM3/TAO dependency conflicts… |
| gguf-quantization 69/100 · D | Orchestra-Research/AI-Research-SKILLs | GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when… |
| gke-inference 84/100 · B | google/skills | Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring… |
| gptq 77/100 · C | Orchestra-Research/AI-Research-SKILLs | Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4×… |
| hf-mem 67/100 · D | huggingface/skills | Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub |
| hqq-quantization 69/100 · D | Orchestra-Research/AI-Research-SKILLs | Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4/3/2-bit precision without needing calibration… |
| jetson-inference-mem-tune 70/100 · C | NVIDIA/skills | Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM/VLM workload on any NVIDIA Jetson. |
| jetson-llm-benchmark 70/100 · C | NVIDIA/skills | Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output. |
| jetson-llm-serve 68/100 · D | NVIDIA/skills | Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin. |
| jetson-package 75/100 · C | NVIDIA/skills | Pick Jetson-compatible containers, vLLM runtime images, and Jetson AI Lab PyPI indexes; maps Orin SM 8.7 vs Thor SM 11.0 and JetPack-specific… |
| jetson-speculative-decoding 68/100 · D | NVIDIA/skills | Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck. |
| llama-cpp 77/100 · C | Orchestra-Research/AI-Research-SKILLs | Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when… |
| openrlhf-training 69/100 · D | Orchestra-Research/AI-Research-SKILLs | High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM… |
| optimizing-attention-flash 69/100 · D | Orchestra-Research/AI-Research-SKILLs | Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training/running transformers with long… |
| physical-ai-infrastructure-setup-and-resilient-scaling 78/100 · C | NVIDIA/skills | Use when the user wants to set up, scale, validate, or harden NVIDIA physical AI infrastructure for synthetic data generation workflows across local… |
| quantizing-models-bitsandbytes 69/100 · D | Orchestra-Research/AI-Research-SKILLs | Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models… |
| run-models 60/100 · D | replicate/skills | Run AI models on Replicate via predictions, webhooks, and streaming. |
| serving-llms-vllm 69/100 · D | Orchestra-Research/AI-Research-SKILLs | Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference… |
| sglang 77/100 · C | Orchestra-Research/AI-Research-SKILLs | Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic… |
| tao-finetune-clip 89/100 · B | NVIDIA/skills | CLIP vision-language model for image-text retrieval, zero-shot classification, embedding extraction, ONNX export, and TensorRT deployment. Use when… |
| tao-finetune-video-clip 88/100 · B | NVIDIA/skills | InternVideo2-CLIP L14 (TAO video_clip) for video-text retrieval, zero-shot classification, embedding extraction, LoRA fine-tuning, ONNX export, and… |
| tao-port-huggingface-model 84/100 · B | NVIDIA/skills | Integrate a HuggingFace Computer Vision model into the NVIDIA TAO Toolkit ecosystem (tao-core config, tao-pytorch trainer, tao-deploy TensorRT… |
| tao-run-inference-service 85/100 · B | NVIDIA/skills | Start, query, and stop a network-specific TAO inference microservice ({network_arch}-inference-microservice) by delegating container execution to… |
| tao-train-sparse4d 87/100 · B | NVIDIA/skills | Prepare calibrated multi-camera data and train, evaluate, run inference, export, or quantize TAO Sparse4D 3D detection and tracking models. Includes… |
| tensorrt-llm 77/100 · C | Orchestra-Research/AI-Research-SKILLs | Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100)… |
| tilegym-monkey-patch-kernels-to-transformers 83/100 · B | NVIDIA/skills | Integrate TileGym kernels into Hugging Face `transformers` models by replacing the library's submodule(s) and certain class(es)' implementations… |
How to choose inference skills
Start from the hardware. huggingface-local-models and llama-cpp cover CPUs and Apple Silicon, the Jetson skills cover NVIDIA edge devices, and vLLM, SGLang and TensorRT-LLM cover data-center GPUs.
Quantization skills change model accuracy. Run an eval before and after, and keep the unquantized weights so you can compare outputs on your own prompts.
Hosted options (Replicate, Agent Platform endpoints, GKE) bill by usage and read cloud credentials, shown under Before you install. Set a budget alert before you leave an endpoint running.
Ranked inference shortlists
Related topics
Fine-Tuning and Model Training 127Google Cloud 146Kubernetes 72Evals and Benchmarks 27LLM APIs and SDKs 28
Find your next useful skill
Explore Rising Skills with quality scores and source reports, or search from inside your agent.
Get Rising Skills by email · Read Rising Skills · Install the finder · State of Agent Skills