# Detect What You Can Name: PaliGemma 2 Object Detection with Transformers # Detect What You Can Name: PaliGemma 2 Object Detection with Transformers ## TL;DR - PaliGemma 2 can do prompt-driven object detection by generating structured `` tokens instead of detector head tensors. - The largest current PaliGemma 2 mix checkpoint for this workflow is `google/paligemma2-28b-mix-448`; use `10b` or `3b` variants when hardware is tighter. - With current `transformers`, the core loop is compact: load a processor and model, prompt with `detect {label} ; {label}`, parse the generated location tokens, and draw boxes. ## Abstract Classic object detectors return boxes and class scores from a dedicated detection head. PaliGemma 2 takes a different route: it treats detection as conditional text generation, so the model emits bounding boxes as location tokens followed by labels. This post shows how to use that behavior with the current Transformers API, turn generated text into pixel boxes, and wrap the result in a reusable function. By the end, you will have a clean baseline for prompt-driven detection that you can extend into a notebook, script, or Gradio demo. ## Requirements - Python 3.10+ - A Hugging Face account with access accepted for the PaliGemma/Gemma license - A CUDA GPU for the larger checkpoints, or a smaller PaliGemma 2 variant for constrained hardware - `transformers>=5.12.0` - `torch` - `accelerate` - `pillow` - `gradio` if you want the demo UI - `torchao` if you want int4 weight-only loading for the 28B checkpoint ```bash pip install -U "transformers>=5.12.0" torch accelerate pillow gradio torchao hf auth login ``` ## Why PaliGemma 2 Mix 448? The original notebook used `google/paligemma2-3b-mix-224`. That is still useful, but the `448` checkpoints give the vision encoder more input resolution, which matters for localization. The current PaliGemma 2 mix family on Hugging Face includes: | Model | Parameters | Why use it | |---|---:|---| | `google/paligemma2-28b-mix-448` | 27.7B | Strongest option when you can use quantization or large GPU memory. | | `google/paligemma2-10b-mix-448` | 9.7B | Good balance when 28B is too large but you still want the higher-resolution `448` variant. | | `google/paligemma2-3b-mix-448` | 3.0B | The most practical local/dev option. | This tutorial uses `google/paligemma2-28b-mix-448` as the main model because it is the largest current PaliGemma 2 mix checkpoint with detection prompt support. If you are experimenting on a smaller GPU, change only `MODEL_ID`; the rest of the code stays the same. There is not a single official Hugging Face object-detection leaderboard that directly ranks these PaliGemma 2 mix checkpoints against one another. The choice here is based on current checkpoint availability, input resolution, model size, and explicit detection prompt support in the model cards. ## How PaliGemma 2 Represents Detections PaliGemma detection prompts look like this: ```text detect person ; bicycle ; dog ``` The model responds with text shaped like this: ```text dog ; person ``` Each detection contains four location tokens in this order: ```text label ``` The location values live on a 0-1024 grid. To draw them on the original image, divide by 1024 and scale `y` by image height and `x` by image width. ## Load the Model and Processor For the 28B checkpoint, int4 weight-only loading is the most practical starting point. If you use the `10b` or `3b` checkpoint and have enough memory, set `USE_INT4 = False`. ```python import torch from transformers import AutoProcessor, PaliGemmaForConditionalGeneration, TorchAoConfig MODEL_ID = "google/paligemma2-28b-mix-448" USE_INT4 = True processor = AutoProcessor.from_pretrained(MODEL_ID) model_kwargs = {"device_map": "auto"} if USE_INT4: model_kwargs["quantization_config"] = TorchAoConfig("int4_weight_only", group_size=128) else: model_kwargs["torch_dtype"] = torch.bfloat16 model = PaliGemmaForConditionalGeneration.from_pretrained( MODEL_ID, **model_kwargs, ).eval() ``` If you want the smaller development model: ```python MODEL_ID = "google/paligemma2-3b-mix-448" USE_INT4 = False ``` ## Parse Generated Location Tokens Keep the parser separate from model inference. That makes it easy to unit test and easy to replace later if you add segmentation tokens. ```python from dataclasses import dataclass import re @dataclass(frozen=True) class Detection: label: str box_xyxy: tuple[int, int, int, int] LOC_PATTERN = re.compile( r"\d{4})>\d{4})>" r"\d{4})>\d{4})>" r"\s*(?P