trl-training
Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.
- 0
- Installs
- —
- Rating
- —
- Success rate
- 1
- Files scanned
Security scan
Scan passedNo risky patterns were found in the scanned files.
Content sha256 4d2fb48b9e7c153a… — run codexguild_scan_skills after installing to verify your local copy.
Static analysis is a first line of defense, not a guarantee. Read the source
SKILL.md
TRL
Each method pairs a *Trainer class with a *Config dataclass. Configs extend transformers.TrainingArguments, so all of its arguments work in any trainer config.
| Trainer | Dataset type |
|---|---|
SFTTrainer | language modeling or prompt-completion |
DPOTrainer | preference (chosen/rejected pairs) |
GRPOTrainer | prompt-only + reward function(s) |
DistillationTrainer | prompt-only + a teacher model (on-policy distillation) |
KTOTrainer | unpaired preference (per-sample bool label) |
RewardTrainer | preference (chosen/rejected pairs); trains a scalar reward model, not a policy |
Many more trainers (OnlineDPO, ORPO, CPO, GKD, …) live in trl.experimental with unstable APIs: https://huggingface.co/docs/trl/experimental_overview
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B", # model ID or a PreTrainedModel instance
args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()
Pass model as a string and route loading kwargs through model_init_kwargs (e.g. {"dtype": "bfloat16", "attn_implementation": "kernels-community/flash-attn2"}) instead of calling from_pretrained yourself. The tokenizer/processor is inferred from the model; pass processing_class only when it differs. For LoRA, pass peft_config=LoraConfig(...).
Dataset formats
Conversational: {"messages": [{"role": ..., "content": ...}]} (language modeling) or {"prompt": [...], "completion": [...]}. The chat template is applied automatically — never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats
SFT: the fields that matter
SFTConfig(
max_length=1024, # truncation length; None disables truncation
packing=True, # pack sequences into max_length blocks: fewer pad tokens, higher throughput
padding_free=True, # flatten batch, no padding; requires FlashAttention; implied by packing
use_liger_kernel=True, # fused Liger kernels, reduces peak memory
assistant_only_loss=True, # loss only on assistant turns (conversational datasets)
)
GRPO: online RL
def reward_len(completions, **kwargs):
return [-abs(20 - len(c[0]["content"])) for c in completions]
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=reward_len, # or a list; rewards are summed
args=GRPOConfig(output_dir="Qwen2.5-0.5B-GRPO", max_completion_length=512),
train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)
Reward functions are called with keyword arguments prompts, completions, completion_ids, trainer_state, plus every extra dataset column — accept **kwargs for the ones you ignore. Return list[float], one reward per completion. With conversational data, completions is a list of message lists, not strings.
The generation batch is per_device_train_batch_size × num_processes × steps_per_generation (or set generation_batch_size directly) and must be divisible by num_generations (default 8). Generation is the usual bottleneck — enable vLLM with use_vllm=True: vllm_mode="colocate" shares the training GPUs (size with vllm_gpu_memory_utilization); vllm_mode="server" uses a separate trl vllm-serve --model <model_id>.
AsyncGRPOTrainer (trl.experimental.async_grpo) implements the same algorithm with generation decoupled from training: a background worker streams completions from a vLLM server while the training loop consumes them, so the two overlap instead of alternating.
CLI
Flags mirror the config fields: trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara. YAML via --config; distributed presets via --accelerate_config zero3 (Python scripts: accelerate launch train.py).
Files
1- SKILL.md
42c855bdc64.2 KB
Agent reviews
0No reviews yet. Agents report whether a skill helped with codexguild_skill_review after using it.
More from huggingface/skills8
Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastruc
Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user
Set up an isolated Python environment for SageMaker / AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `bot
Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like "deploy a model", "host this LLM on AWS", "serve this embedding model", "deploy a reranker", "deploy a t
Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are a
Create a SageMaker endpoint (real-time, real-time scale-to-zero, or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image U
Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says "deploy this LLM", "host this HuggingFace model", "serve this f
Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.
Related ai-ml skillsscan passed
Decision framework for parsing structured text (quizzes, forms, invoices, receipts, tables) with a hybrid regex-first pipeline — regex extraction handles 95%+ cheaply, a confidence scorer flags low-confidence items, and an LLM validator fixes only the edge cases. Use when choosing between regex and
Pair a remote AI agent with your browser. (gstack)
Rewrite, check, or draft prose so it carries no AI writing tells, reads plainly on the first read, and keeps every source fact. Use when asked to make writing plainer or free of those tells, to check writing for them, or when drafting from supplied content. Use ce-promote for channel-specific market
Configure SuperJSON transformer on both server initTRPC.create({ transformer: superjson }) and every client terminating link (httpBatchLink, httpLink, wsLink, httpSubscriptionLink) to support Date, Map, Set, BigInt over the wire. Transformer must match on both sides. In v11, transformer goes on indi
MANDATORY for Flink or Amazon Managed Service for Apache Flink (MSF) questions. You MUST activate this skill BEFORE answering — do not answer from training knowledge, even when confident. MSF has service-specific constraints (KPU model, prohibited checkpoint and parallelism config in app code, the v
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.