Gemma 4 Fine-tuning Guide | Unsloth Documentation

Gemma 4 Fine-tuning Guide | Unsloth Documentation

Fine-tuning an open-weight large language model is no longer a research-only activity; it is a standard step in many production pipelines. A base model knows how to generate fluent text, but it does not know your data format, your domain vocabulary, or your product's tone. That is where fine-tuning comes in. With Google's Gemma 4 family and Unsloth's optimized training stack, the gap between "pretrained model" and "deployable assistant" shrinks to a single, fast training run.

This guide is a practical walkthrough of the Gemma 4 fine-tuning workflow as described in the Unsloth documentation. It covers the environment you need, the installation steps, dataset preparation, LoRA configuration, training, and finally how to run and export your fine-tuned model. The documentation page this guide is based on was last updated on 2026-07-18T14:46:20.453Z, so the instructions below reflect the current state of the Unsloth toolchain.

What This Guide Assumes

Unsloth is built around a simple proposition: fine-tuning should be fast, use as little VRAM as possible, and not require you to rewrite your training loop. It accomplishes this through a set of optimized kernels for attention and linear layers, automatic quantization handling, and tight integration with Hugging Face's Transformers and TRL training stacks.

Gemma 4, from Google, is the open-weight model family this workflow is designed for. Depending on the size of the variant you choose, you will need more or less GPU memory, but the training code itself stays nearly identical.

You should be comfortable with Python, have a basic understanding of what LoRA and QLoRA are, and know your way around a terminal. You do not need to be a machine learning engineer — you need a GPU and the willingness to follow a few commands.

Requirements

Before you run anything, make sure your environment meets the minimum set of requirements.

Hardware

• An NVIDIA GPU with support for CUDA. Unsloth's speedups come from custom kernels that target modern GPU architectures, so a GPU from the Turing generation (RTX 20 series) or newer is recommended. Ampere, Ada Lovelace, and Hopper architectures will give you the best results.
• Sufficient VRAM. The exact amount depends on the Gemma 4 variant you want to fine-tune and the quantization level you use. With 4-bit QLoRA, you can fine-tune a smaller Gemma 4 model on consumer hardware with 8–16 GB of VRAM. Larger variants will require a workstation or cloud GPU with 24 GB or more.
• Enough system RAM and disk space to hold the model weights, the training dataset, and the checkpoints you save.

Software

• A 64-bit Linux distribution or Windows Subsystem for Linux (WSL2). Most instructions assume a Unix-like shell.
• Python 3.9 or newer.
• CUDA toolkit and NVIDIA drivers. Precisely which version you need depends on the PyTorch release you install; the official PyTorch website and the Unsloth documentation list compatible combinations.
• A recent version of PyTorch. Unsloth tracks PyTorch releases closely, and installing a current build will save you from version mismatch issues.

If you are using Google Colab, you are mostly covered: Colab ships with recent PyTorch and CUDA versions, and Unsloth provides a specific install path for Colab environments.

Step-by-step Installation

The installation process for Unsloth is deliberately short. The library is distributed as a Python package, and the easiest way to get it is through pip.

Open a terminal and install the core package:

pip install unsloth

This pulls in the main Unsloth package along with its dependencies, including transformers, datasets, trl, peft, and accelerate. If you find that your environment already has a specific version of PyTorch and you want to avoid Unsloth overwriting it, you can skip the dependency resolution and install only the library itself:

pip install --no-deps unsloth

On Google Colab, the recommended path is to install the colab-new extra, which resolves the environment-specific dependency versions automatically:

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

For a bleeding-edge installation directly from the repository, you can use the same command with the colab-new extra on any Linux machine:

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

Once the installation finishes, verify that the library imports cleanly and that the version looks sane:

python -c "import unsloth; print(unsloth.__version__)"

You should see a version string printed with no warnings about missing CUDA extensions. If you receive an error mentioning a missing CUDA runtime, double-check your PyTorch installation and your CUDA driver version before proceeding.

Some users also prefer to install a specific version of PyTorch first, then add Unsloth on top. This is a good approach if your project is pinned to a particular transformers release:

pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install unsloth

Just remember: install PyTorch first, then Unsloth, and let Unsloth align the rest of the training stack.

Loading Gemma 4 with Unsloth

The entire Unsloth API is centered on two functions: FastLanguageModel.from_pretrained and FastLanguageModel.get_peft_model. The first loads a model and applies optimizations and quantization; the second turns the loaded model into a parameter-efficient, trainable LoRA model.

The standard loading block looks like this:

import torch
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-4-...",  # pick the Gemma 4 variant you need
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

Three arguments deserve special attention:

• model_name — this is the Hugging Face identifier of the Gemma 4 checkpoint you want to fine-tune. The exact identifier depends on the size of the model and whether you want the base model or an instruction-tuned version. Unsloth hosts pre-quantized and optimized variants of these models on its Hugging Face hub, and those are the ones you should use to benefit from the speed improvements.
• max_seq_length — sets the context window for training. The Gemma 4 family supports long contexts, but you should set this to a value that matches your actual data. A value of 2048 is fine for most chat datasets, while longer-document tasks may need 4096 or more. Keep in mind that memory usage grows with sequence length.
• load_in_4bit — enables QLoRA-style quantization. This cuts the VRAM footprint of the model dramatically, allowing you to fine-tune on consumer GPUs that would otherwise be too small. If you have a high-end GPU with plenty of memory, you can set this to False and fine-tune in full or 8-bit precision.

The variable dtype can be left as None, which lets Unsloth choose the optimal float type based on your hardware. On Ampere and newer GPUs, that will typically be bfloat16, which is more stable than float16 during training.

Preparing Your Dataset

Unsloth does not force you into a single dataset format. If you have ever used Hugging Face's datasets library or the TRL SFTTrainer, you already know how to feed data in. The most common format for chat fine-tuning is a conversation-style layout, where each conversation is a list of messages with role and content fields.

For example, the popular ShareGPT-style format looks like this:

{
    "conversations": [
        {"from": "human", "value": "Explain what a vector database is."},
        {"from": "gpt", "value": "A vector database stores embeddings..."}
    ]
}

You can load such a dataset with the Hugging Face datasets library:

from datasets import load_dataset

dataset = load_dataset("json", data_files="my_data.json", split="train")

If your data is in plain text, you can define a simple chat template and reformat your dataset directly:

def format_chat(example):
    text = ""
    for turn in example["conversations"]:
        if turn["from"] == "human":
            text += f"<|user|>\n{turn['value']}\n"
        elif turn["from"] == "gpt":
            text += f"<|assistant|>\n{turn['value']}\n"
    return {"text": text}

dataset = dataset.map(format_chat, remove_columns=dataset.column_names)

The SFTTrainer that we will use later accepts a dataset_text_field argument and handles the rest. The important thing is that your dataset contains a text field that represents the fully formatted conversation, including the special tokens that separate roles.

Configuring LoRA Adapters

With the model loaded and the dataset prepared, the next step is to configure the parameter-efficient fine-tuning setup. Unsloth exposes a thin wrapper around PEFT that makes this straightforward:

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

The key hyperparameters are:

• r — the rank of the LoRA matrices. A rank of 16 is a good default for most tasks. Larger values increase expressiveness but also use more VRAM during training.
• lora_alpha — the scaling factor for the LoRA weights. Keeping it equal to the rank is a common starting point.
• lora_dropout — set to 0 because the Unsloth kernels and fused optimizers already add their own regularization during training; dropout on the adapter layers is rarely needed.
• target_modules — the projection layers where LoRA adapters are attached. Covering all four attention projections plus the MLP projections is standard for Gemma-class models.
• use_gradient_checkpointing="unsloth" — activates the memory-saving gradient checkpointing implementation from Unsloth. This is one of the reasons you can train larger models on smaller GPUs.

Training with the SFT Trainer

Now that the dataset and the model are ready, we train. Unsloth works directly with trl.SFTTrainer, so you get the full power of the Hugging Face training ecosystem without learning a new API.

from trl import SFTTrainer
from transformers import TrainingArguments

training_args = TrainingArguments(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=5,
    max_steps=60,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=1,
    output_dir="gemma4-finetuned",
    optim="adamw_8bit",
    seed=3407,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=training_args,
)

Note the choice of optim="adamw_8bit", which reduces optimizer state memory. Combined with 4-bit quantization and gradient checkpointing, this is what allows a modest consumer GPU to fine-tune a Gemma 4 model.

Start the training run:

trainer.train()

During training you will see the loss decreasing in the logs. If you encounter a CUDA out-of-memory error, lower the per_device_train_batch_size to 1 or 2 and increase gradient_accumulation_steps to compensate. A low batch size with accumulation produces nearly the same result as a large batch while using much less memory.

For most small and medium-sized datasets, a few hundred steps are enough. You do not need to train for many epochs; LoRA converges quickly, and overtraining a small adapter on a small dataset is easy to do.

Saving and Reloading Your Model

Once training finishes, you have two options: save the adapter weights only, or merge the adapter back into the full model and export it to a standard format.

The simplest option is to save just the LoRA adapter:

model.save_pretrained("gemma4-lora")
tokenizer.save_pretrained("gemma4-lora")

This gives you a small set of files, typically a few tens of megabytes, that contain only the trained adapters. To use the fine-tuned model again, reload the base Gemma 4 model with Unsloth and then load the adapter on top:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-4-...",
    max_seq_length=2048,
    load_in_4bit=True,
)

from peft import PeftModel

model = PeftModel.from_pretrained(model, "gemma4-lora")

If you want a fully merged model that can run without Unsloth or PEFT, you can merge the adapter into the base weights:

model = model.merge_and_unload()
model.save_pretrained("gemma4-merged")
tokenizer.save_pretrained("gemma4-merged")

The resulting directory contains a standard Transformers checkpoint that can be loaded like any other model.

Usage Examples

Running Inference

After fine-tuning, switch the model to inference mode with a simple Unsloth helper:

model = FastLanguageModel.for_inference(model)

Then pass your prompt with the same chat format you used during training:

messages = [
    {"role": "user", "content": "Write a support response for a user whose order is delayed."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

The output should reflect the style, format, and knowledge you trained into the model — which is the entire point of the exercise.

Exporting to GGUF for Local Deployment

One of the most common post-training tasks is exporting the fine-tuned model to GGUF so it can be used with tools like llama.cpp or Ollama. Unsloth provides a single command for this:

model.save_pretrained_gguf(
    "gemma4-gguf",
    tokenizer,
    quantization_method="q8_0",
)

You can choose other quantization methods such as f16, q4_k_m, or q5_k_m. The q8_0 method preserves most of the model's quality while producing a reasonably sized file that runs well on CPU hosts.

Best Practices and Troubleshooting

The two most common problems during a fine-tuning run are out-of-memory errors and slow training. Here is how Unsloth addresses both:

• Use 4-bit quantization. If load_in_4bit=True is not set, set it. This alone can cut VRAM usage by up to three-quarters.
• Enable `use_gradient_checkpointing="unsloth"`. This trades a small amount of training time for a large reduction in memory.
• Use the AdamW 8-bit optimizer. The optimizer state is often the hidden memory consumer; 8-bit state reduces it substantially.
• Shrink the context window. If your task does not require long documents, keep max_seq_length modest.
• Increase gradient accumulation instead of batch size. A batch size of 1 with a high accumulation count is safer than a large batch that may not fit.

On the speed side, make sure you are loading the Unsloth-hosted model checkpoints rather than the stock Google checkpoints. The Unsloth versions have precomputed optimizations and quantized weights designed to work with the library's fast kernels. Loading a vanilla checkpoint from the Hub will work, but you will miss most of the performance benefit.

Conclusion

The Unsloth documentation for Gemma 4 describes a workflow that is short, repeatable, and easy to adapt to different model sizes and datasets. Install the library, load a quantized Gemma 4 model, attach a LoRA adapter, train with the SFT trainer, and save the result — that is the entire cycle. What matters more than the code is the discipline around it: prepare a clean dataset with consistent formatting, choose a LoRA rank that matches your task complexity, keep sequence lengths honest, and resist the urge to overtrain.

The fine-tuned model you end up with will not just be a copy of Gemma 4 with new text. It will be a version of the model that knows your specific format, your specific domain, and your specific constraints. And, with Unsloth's memory-efficient training loop, you can get there on hardware that most teams already own. Whether you are building a customer support assistant, a code-generation helper, or an internal document summarizer, this guide gives you the documented path from base model to deployable fine-tune.