It started with a Reddit post

Part of my job as a developer advocate at Voxel51 is to keep an eye out for real, messy, unglamorous computer vision problems — the kind that never show up in benchmarks but are exactly what practitioners actually face. So when this post crossed my feed, I couldn't resist:

Hey guys. I recently lost a turbine RC plane over the desert and ran an aerial mapping grid over the search area. I now have about 3,000 high-res images to comb through and was thinking that maybe someone knew how to cook up a quick cv script that could flag stuff for manual review. I built the plane myself and it would be sick to find it.

Note: one of the images is renamed "0000" — it's a false positive that has a shadow on the top left that looks EXACTLY like the plane... (I went and checked it out irl).

Three thousand near-identical desert tiles. A handmade carbon-fiber airframe somewhere in the frame. And a known decoy — a shadow that fooled the pilot in person. This is a search problem, a data problem, and a tooling problem all at once. It's also a perfect showcase for the workflow I evangelize every day: stop writing one-off scripts, and start treating your dataset as the thing you build, query, and share.

Encapsulating the Workflow with FiftyOne

The instinct most people have here is to write a standalone OpenCV script that dumps a CSV file and a contact sheet JPEG. That gets you exactly one answer, once, with no way to iterate. FiftyOne flips the model: the dataset is the source of truth, every computation writes back onto it, and the App becomes your review surface. Tooling becomes reusable, results become inspectable, and the whole thing becomes shareable.

Concretely, three pieces of the platform carried this project: the FiftyOne Brain for embeddings and uniqueness, the Hugging Face integration for distribution, and — the part I'm most excited about — the plugin system for packaging the actual detector.

From a folder of images to a queryable dataset

Loading the survey is one call, and the FiftyOne Brain turns raw pixels into something you can search and visualize — CLIP embeddings, a similarity index, and a 2D UMAP map you can lasso in the App:

import fiftyone as fo
import fiftyone.zoo as foz
import fiftyone.brain as fob

dataset = fo.Dataset.from_dir(
    dataset_dir="scans",
    dataset_type=fo.types.ImageDirectory,
    name="Ariel_Scans",
    persistent=True,
)
dataset.compute_metadata()

clip_model = foz.load_zoo_model("open-clip-torch")
dataset.compute_embeddings(clip_model, embeddings_field="clip_embeddings")
fob.compute_similarity(dataset, embeddings="clip_embeddings", brain_key="clip_sim")
fob.compute_visualization(
    dataset, embeddings="clip_embeddings", brain_key="clip_viz", method="umap"
)

And because every artifact — images, metadata, embeddings, indexes — lives on the dataset, I could publish the whole thing to the Hugging Face Hub and let anyone pull it back in a single line. No setup, no recompute:

from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub("harpreetsahota/ariel_scans", persistent=True)

The detector, packaged as a plugin

Here's the heart of it. The desert background is almost uniform mid-gray, and the only dark features are scrubby vegetation — so tone alone can't separate the plane from a bush. The detector instead scores each dark blob on shape and surface character: a carbon-fiber airframe is elongated, fills its bounding box, has a smooth low-variance interior, and is solid and convex. Vegetation is none of those.

I could have left that as a script. Instead, I wrapped it as a FiftyOne plugin — and that decision is the whole point of this post. A plugin exposes the logic as operators that run identically from the Python SDK and from the App's UI, can be delegated to run in the background, and can be installed by anyone with one command:

fiftyone plugins download https://github.com/harpreetsahota204/plane_finder

The plugin ships two operators:

• preview_dark_mask — a calibration helper that writes the thresholded dark mask back as a heatmap overlay, so you can tune the threshold by eye in the App before committing to a full run.
• find_plane — the detector, which writes a plane_candidates detections field (score in confidence, plus per-detection attributes like rect_aspect and interior_std) and a per-image max_plane_candidates_score for ranking.

Driving them from the SDK is just get_operator and a call:

import fiftyone.operators as foo

find_plane = foo.get_operator("@harpreetsahota/plane_finder/find_plane")

# Keep only confident candidates, cap boxes per image so the field stays clean
await find_plane(
    dataset,
    label_field="plane_candidates",
    dark_threshold=105,
    min_score=0.5,
    max_per_image=5,
)

The exact same operator, with the exact same form inputs, is one backtick-keystroke away in the App — every parameter rendered as a labeled, described field, with a checkbox to delegate execution for the full 6,500-image run. I didn't write that UI. The plugin system generated it from the operator's input schema. That parity between code and UI is, to me, the single most underrated thing about building on FiftyOne.

Turning 3,000 images into ranked shortlists

Once the scores are on the dataset, review is just querying. Saved views persist on the dataset and show up in the App's Saved Views menu, so the community member opens the data and immediately has a set of curated entry points — each attacking the search from a different angle:

from fiftyone import ViewField as F

# Everything with a candidate, ranked; then the top 100 for a fast first pass
candidates_by_score = dataset.match(
    F("max_plane_candidates_score") > 0
).sort_by("max_plane_candidates_score", reverse=True)
dataset.save_view("candidates_by_score", candidates_by_score, overwrite=True)
dataset.save_view("top_shortlist", candidates_by_score.limit(100), overwrite=True)

# The most airframe-like crops: elongated AND smooth-surfaced
carbon_like = dataset.filter_labels(
    "plane_candidates",
    (F("rect_aspect") > 3.0) & (F("interior_std") < 13.0),
).sort_by("max_plane_candidates_score", reverse=True)
dataset.save_view("carbon_like", carbon_like, overwrite=True)

Being able to filter on the detector's own per-detection attributes (rect_aspect, interior_std, solidity, bbox_diag) is what makes this expressive. I saved eight views in total:

• candidates_by_score — every flagged sample, ranked by its best score.
• top_shortlist — the top 100 candidates — the fastest first pass.
• high_confidence — only the strong detections, weak boxes dropped.
• carbon_like — the most airframe-like: elongated and smooth-surfaced.
• best_geometry — strongest pure shape, independent of surface smoothness.
• review_band — the mid-scoring "maybe" pile, so nothing borderline is missed.
• thermal_frames — the small thermal/IR frames, reviewed apart from the RGB.
• unusual_candidates — candidates ranked by Brain uniqueness — the plane is an anomaly in a sea of identical desert.

Zooming in: patch embeddings

The sharpest tool is to explode every candidate into its own crop with to_patches, then run the Brain on the crops instead of the whole frames. Each embedding now describes just the blob, so look-alikes cluster tightly: find one true patch and a similarity search pulls the rest, and ranking by uniqueness floats the real airframe above the repetitive bush and shadow patches.

patches = dataset.to_patches("plane_candidates")
dataset.save_view("plane_patches", patches, overwrite=True)

dataset.compute_patch_embeddings(
    clip_model, "plane_candidates", embeddings_field="patch_embeddings"
)
fob.compute_similarity(
    dataset, patches_field="plane_candidates",
    embeddings="patch_embeddings", brain_key="patch_sim",
)
fob.compute_visualization(
    dataset, patches_field="plane_candidates",
    embeddings="patch_embeddings", brain_key="patch_viz", method="umap",
)

fob.compute_uniqueness(
    patches, embeddings="patch_embeddings", uniqueness_field="patch_uniqueness"
)

I bundled the view-building and patch-embedding step into a single publish_views.py script that recomputes everything and pushes the enriched dataset back to the Hub. One note worth flagging for anyone adapting it: because compute_embeddings spins up a multiprocessing DataLoader, the script body lives inside a main() guarded by if name == "main" — otherwise the spawned workers re-import and re-run the whole pipeline.

def main():
    dataset = fo.load_dataset("Ariel_Scans")
    # ... build the eight views, compute patch embeddings ...
    dataset.save()
    push_to_hub(dataset, "ariel_scans", exist_ok=True)

if __name__ == "__main__":
    main()

Why the plugin system is the real story

Strip away the desert and the drone, and what this project really demonstrates is what the FiftyOne plugin system unlocks for developers:

• Write once, run anywhere — one operator definition runs from the SDK in a notebook, from a script, and as a point-and-click panel in the App — with the UI generated from the input schema.
• Scale without rewrites — the same operator runs inline on a small view or as a delegated background job over thousands of full-res images, just by toggling a flag.
• Shareable by design — fiftyone plugins download means the community member runs my exact detector without copying code or matching my environment by hand.
• Composable with the rest of the platform — the plugin's output is ordinary FiftyOne labels, so it immediately works with views, the Brain, similarity search, and the Hub.

That's the pitch I make to developers all the time, and this is it in miniature: the gap between "I hacked together a CV script" and "I shipped a reusable, UI-enabled, shareable tool" is a thin plugin wrapper around code you've already written.

Did it find the plane?

The tooling did its job: it collapsed a 3,000-image haystack into a handful of short, ranked lists a human can review in minutes, kept the notorious "0000" shadow decoy honest, and shipped as a dataset and a plugin anyone can pull down and rerun. Whether the airframe turns up is now a question of eyes on a shortlist rather than endurance — which is exactly where you want a search-and-rescue problem to land.

If you want to try it yourself: pull the dataset from the Hub, install the plugin from GitHub, and start at the top of top_shortlist. And if you build something like this, come share it with the Voxel51 community — the next messy real-world problem is always around the corner.

And if the plane does turn up, somewhere a Redditor owes the internet fifty bucks.

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.

Document understanding has long been one of the most challenging problems in computer vision. Traditional OCR vision systems excel at character recognition but struggle with complex layouts, tables, formulas, and structured data extraction. Enter GLM-OCR—a lightweight yet powerful multimodal model that's changing how we approach document parsing.

In this post, we'll explore what document understanding is and how to build efficient, production-ready AI document understanding pipelines using GLM-OCR integrated with FiftyOne, a powerful dataset management and visualization platform. We'll cover everything from basic text extraction to advanced structured data parsing, with practical document understanding integration examples you can run today.

What is document understanding?

Document understanding is an AI-powered approach that goes beyond traditional OCR, enabling machines to read, interpret, and extract structured data from documents. By combining vision and language understanding, AI document understanding systems can handle tables, formulas, and layouts, making OCR vision systems smarter and streamlining workflows with seamless document understanding integration.

How GLM-OCR differs from traditional OCR vision systems

GLM-OCR represents a fundamental shift in how we approach document understanding. To appreciate its value, we need to understand how it differs from traditional OCR vision systems.

Traditional OCR vision systems: Character recognition only

Classic OCR vision systems like Tesseract, PaddleOCR, or commercial solutions follow a character-by-character recognition approach:

• Image preprocessing: binarization, noise reduction
• Text detection: finding text regions
• Character recognition: identifying individual characters
• Post-processing: spell checking, language models

Limitations:

• Outputs flat text strings—no understanding of document structure.
• Struggles with complex layouts (tables become garbled text).
• Cannot handle formulas (mathematical expressions become meaningless text).
• Requires extensive post-processing to extract structured data.
• No semantic understanding—just character patterns.

GLM-OCR: Document understanding through multimodal AI

GLM-OCR takes a completely different approach. Instead of treating documents as collections of characters, it uses vision-language models to understand documents semantically:

1. Visual-Language Fusion: The CogViT encoder captures both pixel-level details AND layout semantics simultaneously.
2. Contextual Reasoning: Uses language understanding to interpret relationships between text blocks, tables, and figures.
3. Structure-Aware Generation: Generates structured outputs (Markdown, JSON, LaTeX) that preserve document semantics.
4. Multi-Token Prediction: Predicts multiple tokens per step and uses context to correct errors—like semantic proofreading.

Key differences:

Why GLM-OCR vision systems matters

Traditional OCR vision systems give you text, but you still need to parse tables manually or with complex regex, reconstruct formulas from text descriptions, build custom parsers for each document type, and handle edge cases and formatting variations.

GLM-OCR gives you semantic structure directly:

• Tables are already in HTML/Markdown format.
• Formulas are ready-to-render LaTeX.
• Structured data matches your exact schema.
• Document hierarchy is preserved.

This fundamentally changes how you build document processing pipelines—from "extract text then parse" to "extract structure directly".

Key Advantages:

• Lightweight: At 0.9B parameters, it runs efficiently on consumer hardware.
• Multimodal: Handles PDFs and images with equal proficiency.
• Structure-First: Outputs semantic Markdown, JSON, or LaTeX—not just text.
• Open Source: Apache-2.0 licensed, deployable anywhere.
• Context-Aware: Uses language understanding to improve accuracy.

The FiftyOne document understanding integration

While GLM-OCR is powerful on its own, integrating document understanding with FiftyOne unlocks several key benefits:

1. Efficient Batching: Native support for batched inference, dramatically improving throughput.
2. Dataset Management: Seamless integration with FiftyOne's dataset workflows.
3. Visualization: Built-in tools for exploring OCR results and debugging.
4. Flexibility: Easy switching between operation modes without reloading models.

Get started now →

Getting started with your document understanding integration

Installation

First, install the required dependencies. Since GLM-OCR is relatively new, you'll need the latest transformers from source:

pip install git+https://github.com/huggingface/transformers.git
pip install -U timm fiftyone

For the best text viewing experience in FiftyOne, install the Caption Viewer plugin:

fiftyone plugins download https://github.com/harpreetsahota204/caption_viewer

This plugin provides intelligent formatting for OCR outputs with proper line breaks, table rendering, and JSON pretty-printing—essential for reviewing complex document extractions.

Loading your dataset

FiftyOne makes it easy to work with document datasets. Let's start with a sample dataset from Hugging Face Hub:

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub(
  "Voxel51/scanned_receipts",
  overwrite=True,
  persistent=True,
  name="scanned_receipts",
  max_samples=50
)

This loads 50 scanned receipt images—perfect for demonstrating GLM-OCR's capabilities.

Registering and loading GLM-OCR

The FiftyOne integration makes GLM-OCR available as a zoo model:

import fiftyone.zoo as foz

# Register the model source
foz.register_zoo_model_source(
  "https://github.com/harpreetsahota204/glm_ocr",
  overwrite=True
)

# Load the model
model = foz.load_zoo_model("zai-org/GLM-OCR")

The model automatically detects and uses the best available device (CUDA, MPS, or CPU), making it easy to get started regardless of your hardware setup.

Four operation modes for document understanding

GLM-OCR supports four distinct operation modes, each optimized for different use cases:

1. Text recognition

The most straightforward mode—extract plain text while preserving formatting:

model.operation = "text"
dataset.apply_model(
  model,
  label_field="text_extraction",
  batch_size=8,
  num_workers=2
)

Output: Clean text with preserved layout and structure. Perfect for general OCR tasks where you need readable text output.

2. Formula recognition

Extract mathematical formulas in LaTeX format—essential for scientific and technical documents:

model.operation = "formula"
dataset.apply_model(model, label_field="formulas", batch_size=8)

Output: LaTeX-formatted mathematical expressions ready for rendering in scientific documents or notebooks.

3. Table recognition

Parse complex table structures into HTML or Markdown:

model.operation = "table"
dataset.apply_model(model, label_field="tables", batch_size=8)

Output: Properly structured tables in HTML or Markdown format, preserving cell relationships and hierarchies.

4. Custom structured extraction

The most powerful mode—extract structured data using custom JSON schema prompts:

model.operation = "custom"
model.custom_prompt = """请按下列JSON格式输出图中信息:
{
  "company": "",
  "address": {
    "street": "",
    "city": "",
    "state": "",
    "zip_code": ""
  },
  "invoice_number": "",
  "dates": {
    "purchase_date": "",
    "purchase_time": ""
  },
  "items": [
    {"name": "", "quantity": "", "price": ""}
  ],
  "totals": {
    "purchase_amount": "",
    "tax_amount": "",
    "total": ""
  },
  "payment": {
    "payment": "",
    "change": ""
  }
}
"""

dataset.apply_model(
  model,
  label_field="structured_extraction",
  batch_size=8,
  num_workers=2
)

Output: JSON-formatted structured data matching your exact schema. This is incredibly powerful for building ETL pipelines, data warehouses, or API integrations.

Get started now →

Real-world document understanding example

Let's see GLM-OCR's document understanding in action with a real receipt. After running text extraction, here's what we get:

Sarang Hae Yo
TRENDYMAX (M) SDN. BHD.
Company Reg. No.: (583246-A).
P6, Block C, GM Klang Wholesale City,
Jalan Kasuarina 1,
41200 Klang, Selangor D.E.
Whatsapp : +6012-5060668
www.saranghaeyo.com
GST Reg : 000384098304

TAX INVOICE
CB#: GM3-46792 15/01/2018 1:48:36 PM
Cashier: W001 - erah @ C6
Remarks:

Qty Description Price Disc Total
1 MS LOYALTY PACKAGE
15.00 0.00 15.00 SR
10 IB-RDM IRON BASKET ROUND 27 * 25
28.90 -28.90 260.10 SR
...

The text extraction preserves the layout beautifully. But with structured extraction, we get machine-readable JSON:

{
  "company": "Sarang Hae Yo",
  "address": {
    "street": "P6, Block C, GM Klang Wholesale City, Jalan Kasuarina 1,",
    "city": "Klang",
    "state": "Selangor D.E.",
    "zip_code": "41200"
  },
  "invoice_number": "GM3-46792",
  "dates": {
    "purchase_date": "2018-01-15",
    "purchase_time": "14:36 PM"
  },
  "items": [
    {
      "name": "MS LOYALTY PACKAGE",
      "quantity": "1",
      "price": "15.00"
    },
    {
      "name": "IB-RDM IRON BASKET ROUND 27 * 25",
      "quantity": "10",
      "price": "28.90"
    }
  ],
  "totals": {
    "purchase_amount": "329.10",
    "tax_amount": "329.10",
    "total": "329.10"
  }
}

This structured output can be directly ingested into databases, analytics systems, or financial software—no manual parsing required.

Visualizing results

One of FiftyOne's greatest strengths is its visualization capabilities. Launch the app to explore your results:

session = fo.launch_app(dataset)

Using the caption viewer plugin

The caption viewer plugin transforms how you review OCR vision system outputs:

1. Click on any sample to open the modal view
2. Click the + button to add a panel
3. Select "Caption Viewer" from the panel list
4. In the panel menu (☰), select the field you want to view:
   1. text_extraction for plain OCR text
   2. structured_extraction for JSON outputs
   3. Any other text field

The plugin automatically:

• Renders line breaks properly
• Converts HTML tables to markdown
• Pretty-prints JSON content
• Shows character counts

This makes it easy to spot issues, verify accuracy, and understand what the model extracted.

Performance optimization

GLM-OCR vision system delivers impressive throughput: approximately 1.86 pages/second for PDFs and 0.67 images/second. The FiftyOne integration's batching support significantly improves efficiency.

Batch size recommendations

Choose your batch size based on available hardware:

• CPU: batch_size=2-4 (slower but works anywhere)
• GPU (8GB): batch_size=8-16 (good balance)
• GPU (16GB+): batch_size=16-32 (maximum throughput)

Runtime configuration

One of the integration's best features is the ability to change operation modes without reloading the model:

# Start with text recognition
model.operation = "text"
dataset.apply_model(model, label_field="text")

# Switch to table recognition
model.operation = "table"
dataset.apply_model(model, label_field="tables")

# Use custom prompt for structured extraction
model.operation = "custom"
model.custom_prompt = "Extract invoice data as JSON..."
dataset.apply_model(model, label_field="invoice_data")

You can also adjust generation parameters such as increasing max tokens for longer outputs:

# Increase max tokens for longer outputs
model.max_new_tokens = 16384

# Adjust batch size for memory constraints
model.batch_size = 4

Real-world structured output use cases

The true power of GLM-OCR vision systems emerges when you need structured data, not just text. Here are practical use cases where structured extraction is essential, along with how to use the output downstream.

Use case 1: Financial document processing

Challenge: Extract structured financial data from invoices, receipts, and statements for accounting automation.

Why structured output is essential: Financial systems require specific fields (amounts, dates, line items) in exact formats. Flat text requires complex parsing that breaks with format variations.

Implementation:

model.operation = "custom"
model.custom_prompt = """Extract invoice information as JSON:
{
  "vendor": "",
  "invoice_number": "",
  "date": "",
  "due_date": "",
  "line_items": [
    {"description": "", "quantity": "", "unit_price": "", "total": ""}
  ],
  "subtotal": "",
  "tax": "",
  "total": "",
  "payment_terms": ""
}"""

dataset.apply_model(model, label_field="invoice_data")

Downstream Usage:

import json
from accounting_system import InvoiceProcessor

processor = InvoiceProcessor()

for sample in dataset:
  invoice_json = json.loads(sample.invoice_data)

  # Direct database insertion
  invoice_id = processor.create_invoice(
    vendor=invoice_json["vendor"],
    invoice_number=invoice_json["invoice_number"],
    date=invoice_json["date"],
    total=float(invoice_json["total"])
  )
  
  # Add line items
    for item in invoice_json["line_items"]:
      processor.add_line_item(
        invoice_id=invoice_id,
        description=item["description"],
        quantity=float(item["quantity"]),
        unit_price=float(item["unit_price"])
  )
  
  # Trigger approval workflow if amount exceeds threshold
  if float(invoice_json["total"]) > 10000:
    processor.send_for_approval(invoice_id)

Benefits: No manual data entry, automatic categorization, immediate integration with accounting software.

Use case 2: Medical record digitization

Challenge: Extract structured patient information from scanned medical forms, lab reports, and prescriptions.

Why structured output is essential: Healthcare systems require structured data for EHR integration, insurance claims, and clinical decision support. Unstructured text cannot be reliably parsed.

Implementation:

model.operation = "custom"
model.custom_prompt = """Extract patient information from medical document as JSON:
{
  "patient_name": "",
  "date_of_birth": "",
  "patient_id": "",
  "visit_date": "",
  "vital_signs": {
    "blood_pressure": "",
    "heart_rate": "",
    "temperature": "",
    "weight": ""
  },
  "diagnosis": [],
  "medications": [
    {"name": "", "dosage": "", "frequency": ""}
  ],
  "lab_results": [
    {"test_name": "", "value": "", "unit": "", "reference_range": ""}
  ],
  "physician": ""
}"""

dataset.apply_model(model, label_field="medical_data")

Downstream usage:

from hl7 import create_patient_message
from ehr_system import EHRClient

ehr = EHRClient()

for sample in dataset:
  medical_json = json.loads(sample.medical_data)

  # Create HL7 message for EHR integration
  hl7_message = create_patient_message(
    patient_id=medical_json["patient_id"],
    name=medical_json["patient_name"],
    dob=medical_json["date_of_birth"]
  )
  
  # Send to EHR system
  ehr.send_message(hl7_message)

  # Extract and validate lab results
  for lab in medical_json["lab_results"]:
    ehr.add_lab_result(
      patient_id=medical_json["patient_id"],
      test_name=lab["test_name"],
      value=float(lab["value"]),
      unit=lab["unit"],
      reference_range=lab["reference_range"]
    )
    
    # Flag abnormal values
    if not ehr.is_normal_range(lab["value"], lab["reference_range"]):
      ehr.alert_physician(medical_json["physician"], lab)

  # Check for drug interactions
  medications = [m["name"] for m in medical_json["medications"]]
  interactions = ehr.check_drug_interactions(medications)
  if interactions:
    ehr.alert_pharmacist(interactions)

Benefits: Automated EHR population, real-time alerts for abnormal values, drug interaction checking, reduced manual data entry errors.

Use case 3: Legal document analysis

Challenge: Extract structured information from contracts, NDAs, and legal agreements for compliance monitoring and analysis.

Why structured output is essential: Legal analysis requires identifying specific clauses, dates, parties, and obligations. These must be structured for automated compliance checking and risk assessment.

Implementation:

model.operation = "custom"
model.custom_prompt = """Extract contract information as JSON:
{
  "parties": [
  {"name": "", "role": ""}
  ],
  "effective_date": "",
  "expiration_date": "",
  "key_clauses": {
    "termination": "",
    "liability": "",
    "confidentiality": "",
    "payment_terms": "",
    "governing_law": ""
  },
  "obligations": [
    {"party": "", "obligation": "", "deadline": ""}
  ],
  "financial_terms": {
    "amount": "",
    "currency": "",
    "payment_schedule": ""
  },
  "renewal_terms": ""
}"""

dataset.apply_model(model, label_field="contract_data")

Downstream usage:

from contract_analyzer import ContractAnalyzer
from compliance_checker import ComplianceChecker

analyzer = ContractAnalyzer()
compliance = ComplianceChecker()

for sample in dataset:
  contract_json = json.loads(sample.contract_data)
  
  # Store in contract management system
  contract_id = analyzer.store_contract(contract_json)

  # Check compliance against company policies
  violations = compliance.check_policy(
    liability_clause=contract_json["key_clauses"]["liability"],
    termination_clause=contract_json["key_clauses"]["termination"],
    payment_terms=contract_json["key_clauses"]["payment_terms"]
  )
  
  if violations:
    analyzer.flag_for_review(contract_id, violations)

  # Extract obligations for tracking
  for obligation in contract_json["obligations"]:
    analyzer.create_task(
      contract_id=contract_id,
      assigned_to=obligation["party"],
      description=obligation["obligation"],
      deadline=obligation["deadline"]
    )

  # Set expiration reminders
  analyzer.set_expiration_alert(
    contract_id=contract_id,
    expiration_date=contract_json["expiration_date"],
    days_before=90
  )

  # Generate summary for legal team
  summary = analyzer.generate_summary(contract_json)
  analyzer.notify_legal_team(contract_id, summary)

Benefits: Automated compliance checking, obligation tracking, expiration monitoring, risk flagging, reduced manual review time.

Use case 4: Academic paper processing

Challenge: Extract structured metadata, formulas, and references from research papers for knowledge base construction.

Why structured output is essential: Academic databases require structured metadata (authors, affiliations, citations). Formulas must be in LaTeX for rendering. References need to be parseable for citation networks.

Implementation:

# First, extract structured metadata
model.operation = "custom"
model.custom_prompt = """Extract paper metadata as JSON:
{
  "title": "",
  "authors": [
    {"name": "", "affiliation": "", "email": ""}
  ],
  "abstract": "",
  "keywords": [],
  "publication_date": "",
  "journal": "",
  "doi": "",
  "references": [
    {"title": "", "authors": "", "year": "", "doi": ""}
  ],
  "sections": [
    {"heading": "", "content": ""}
  ]
}"""

dataset.apply_model(model, label_field="paper_metadata")

# Then extract formulas separately
model.operation = "formula"
dataset.apply_model(model, label_field="formulas")

Downstream usage:

from knowledge_graph import KnowledgeGraph
from citation_network import CitationNetwork
from latex_renderer import render_formula

kg = KnowledgeGraph()
citation_net = CitationNetwork()

for sample in dataset:
  metadata = json.loads(sample.paper_metadata)

  # Create paper node in knowledge graph
  paper_id = kg.create_paper(
    title=metadata["title"],
    abstract=metadata["abstract"],
    doi=metadata["doi"],
    publication_date=metadata["publication_date"]
  )
  
  # Add authors and affiliations
  for author in metadata["authors"]:
    author_id = kg.get_or_create_author(author["name"])
    kg.link_author_to_paper(author_id, paper_id)

    if author["affiliation"]:
      org_id = kg.get_or_create_organization(author["affiliation"])
      kg.link_author_to_org(author_id, org_id)

  # Build citation network
  for ref in metadata["references"]:
    cited_paper_id = citation_net.get_or_create_paper(
      title=ref["title"],
      authors=ref["authors"],
      year=ref["year"]
    )
    citation_net.add_citation(paper_id, cited_paper_id)

  # Extract and render formulas
  if sample.formulas:
    formulas = sample.formulas.split("\n\n") # Assuming formulas separated by double newline
    for i, formula in enumerate(formulas):
      rendered_image = render_formula(formula)
      kg.add_formula(paper_id, formula, rendered_image, position=i)

  # Extract key concepts from sections
  for section in metadata["sections"]:
    concepts = kg.extract_concepts(section["content"])
    for concept in concepts:
      kg.link_paper_to_concept(paper_id, concept)

  # Generate recommendations based on citation network
  similar_papers = citation_net.find_similar(paper_id, limit=10)
  kg.add_recommendations(paper_id, similar_papers)

Benefits: Automated knowledge graph construction, citation network analysis, formula rendering, concept extraction, paper recommendations.

Use case 5: Supply chain document processing

Challenge: Extract structured data from shipping manifests, customs forms, and delivery receipts for logistics automation.

Why structured output is essential: Logistics systems require structured data for tracking, customs clearance, and inventory management. Each document type has specific fields that must be extracted accurately.

Implementation:

model.operation = "custom"
model.custom_prompt = """Extract shipping manifest information as JSON:
{
  "shipper": {
    "name": "",
    "address": "",
    "contact": ""
  },
  "consignee": {
    "name": "",
    "address": "",
    "contact": ""
  },
  "shipment_id": "",
  "tracking_number": "",
  "ship_date": "",
  "expected_delivery": "",
  "items": [
    {
      "description": "",
      "quantity": "",
      "weight": "",
      "value": "",
      "hs_code": ""
    }
  ],
  "total_weight": "",
  "total_value": "",
  "carrier": "",
  "service_type": "",
  "special_instructions": ""
}"""

dataset.apply_model(model, label_field="manifest_data")

Downstream usage:

from logistics_system import LogisticsSystem
from customs_api import CustomsAPI
from inventory_system import InventorySystem

logistics = LogisticsSystem()
customs = CustomsAPI()
inventory = InventorySystem()

for sample in dataset:
  manifest = json.loads(sample.manifest_data)

  # Create shipment in logistics system
  shipment_id = logistics.create_shipment(
    tracking_number=manifest["tracking_number"],
    shipper=manifest["shipper"],
    consignee=manifest["consignee"],
    carrier=manifest["carrier"],
    service_type=manifest["service_type"]
  )
  
  # Submit customs declaration
  customs_declaration = customs.create_declaration(
    shipment_id=shipment_id,
    items=manifest["items"],
    total_value=float(manifest["total_value"]),
    total_weight=float(manifest["total_weight"])
  )
  
  # Check for restricted items
  for item in manifest["items"]:
    if customs.is_restricted(item["hs_code"]):
      logistics.flag_for_review(shipment_id, item)

    # Update inventory system
    inventory.update_in_transit(
      item_description=item["description"],
      quantity=int(item["quantity"]),
      expected_delivery=manifest["expected_delivery"]
    )
    
  # Calculate shipping costs
  cost = logistics.calculate_cost(
    weight=float(manifest["total_weight"]),
    service_type=manifest["service_type"],
    destination=manifest["consignee"]["address"]
  )
  logistics.set_cost(shipment_id, cost)
  
  # Set up tracking alerts
  logistics.set_delivery_alerts(
    shipment_id=shipment_id,
    consignee_email=manifest["consignee"]["contact"],
    expected_delivery=manifest["expected_delivery"]
  )
  
  # Generate shipping labels
  label = logistics.generate_label(shipment_id)
  logistics.print_label(shipment_id, label)

Benefits: Automated customs processing, real-time tracking, inventory updates, cost calculation, delivery notifications.

Multi-mode processing for complex document understanding

To understand some documents, they require multiple extraction modes. Here's how to combine them:

# Extract text for full-text search
model.operation = "text"
dataset.apply_model(model, label_field="full_text")

# Extract tables for data analysis
model.operation = "table"
dataset.apply_model(model, label_field="tables")

# Extract formulas for rendering
model.operation = "formula"
dataset.apply_model(model, label_field="formulas")

# Extract structured metadata
model.operation = "custom"
model.custom_prompt = "Extract key information as JSON..."
dataset.apply_model(model, label_field="metadata")

Now each document has multiple representations, enabling different downstream uses:

• Full text: Search and retrieval
• Tables: Data analysis and visualization
• Formulas: Rendering in documentation
• Metadata: Database storage and filtering

Quality assurance workflows

Use FiftyOne's filtering and evaluation capabilities to build QA workflows:

# Filter samples with empty extractions
empty_data = dataset.match(F("structured_extraction") == None)
print(f"Found {len(empty_data)} samples with no structured extraction")

# Review problematic samples
session = fo.launch_app(empty_data)

# Validate JSON structure
import json
invalid_json = []
for sample in dataset:
  if sample.structured_extraction:
    try:
      json.loads(sample.structured_extraction)
    except json.JSONDecodeError:
      invalid_json.append(sample.id)

if invalid_json:
  print(f"Found {len(invalid_json)} samples with invalid JSON")
  review_view = dataset.select(invalid_json)
  session = fo.launch_app(review_view)

Architecture deep dive

Understanding GLM-OCR's vision system architecture helps optimize your usage:

Three-stage pipeline

1. Visual ingestion: CogViT visual encoder captures pixel-level and layout information.
2. Multimodal reasoning: GLM-V-based vision-language fusion aligns visual features with language understanding.
3. Structured generation: Decoder with Multi-Token Prediction (MTP) generates structured output while correcting errors.

The MTP mechanism is particularly clever—it predicts multiple tokens per step and uses context to fix errors on the fly, acting like semantic proofreading rather than naive character recognition.

Batching implementation

The FiftyOne integration implements efficient batching through:

• SupportsGetItem: Custom GetItem transform loads images as PIL Images.
• TorchModelMixin: Enables DataLoader-based batching.
• Left padding: Correctly handles decoder-only model generation.
• Variable-size images: Handles different image sizes in the same batch.

This architecture allows processing multiple documents simultaneously while maintaining accuracy.

OCR vision system best practices

1. Choose the right operation mode

• Use text for general OCR and human-readable output
• Use formula for scientific documents with mathematical expressions
• Use table when you need structured table data
• Use custom for domain-specific structured extraction

2. Optimize batch sizes

Start with smaller batches and increase until you hit memory limits. Monitor GPU/CPU usage to find the sweet spot.

3. Store raw outputs

Always store the raw GLM-OCR output (Markdown/JSON) alongside processed data. This allows reprocessing as your downstream logic evolves.

4. Combine with LLMs

GLM-OCR excels at structure extraction. Feed its output into LLMs for:

• Summarization
• Question answering
• Risk analysis
• Report generation

5. Handle edge cases

For low-quality scans, consider preprocessing:

• Binarization
• De-skewing
• Noise reduction

This helps the visual encoder and improves structure detection.

The future of OCR workflows with AI-powered document understanding

GLM-OCR vision systems combined with FiftyOne provides a powerful, flexible solution for AI-powered document understanding. The integration's batching support, visualization capabilities, and multiple operation modes make it suitable for everything from quick prototypes to production pipelines.

Key takeaways:

• Fundamentally different: GLM-OCR understands document semantics, not just characters.
• Structure-first output: Get Markdown, JSON, and LaTeX directly—no parsing needed.
• Lightweight yet powerful: 0.9B parameters with SOTA accuracy (94.62 on OmniDocBench V1.5).
• Structured extraction: Extract domain-specific data via custom prompts.
• Open source: Apache-2.0 licensed, deployable anywhere.
• Developer-friendly: Easy integration with existing workflows via FiftyOne.

Whether you're building ETL pipelines, processing receipts, analyzing contracts, extracting data from scientific papers, or digitizing medical records, GLM-OCR with FiftyOne gives you the tools to build robust document understanding systems that output structured data ready for downstream processing.

GLM-OCR next steps

1. Try it yourself: Run the example notebook with your own documents.
2. Experiment with modes: Test different operation types on your use case.
3. Build a pipeline: Integrate structured extraction into your workflow.
4. Optimize: Tune batch sizes and parameters for your hardware.
5. Extend: Combine with LLMs for end-to-end document understanding.

The future of AI document understanding is here—and it's more accessible than ever.

Get started now →

Resources

• GLM-OCR FiftyOne Implementation
• GLM-OCR Model Card
• Example Notebook
• FiftyOne Documentation
• Caption Viewer Plugin

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.

In document digitization, traditional OCR pipelines have long been the bottleneck—fragile, multi-stage systems that struggle with complex layouts, multilingual content, and scientific notation. Enter LightOnOCR-2-1B, a revolutionary 1-billion-parameter vision-language model that challenges the status quo by delivering state-of-the-art document OCR performance in a remarkably compact package.

What is document OCR?

Document OCR is a technology that allows machines to read and extract text from scanned documents, images, and PDFs. Traditional OCR systems often struggle with complex layouts, tables, forms, and multilingual content, making accurate text extraction challenging.

Modern AI OCR models like LightOnOCR-2 go beyond basic character recognition by leveraging vision-language understanding to preserve document structure, layout, and even scientific notation. This makes document OCR ideal for document management system OCR applications, enabling faster, more accurate digitization and streamlined workflows.

Breaking the document OCR size-performance tradeoff

What makes LightOnOCR-2 remarkable isn't just its accuracy—it's how efficiently it achieves it. The model achieves the highest score on OlmOCR-Bench, a comprehensive document OCR evaluation benchmark, while being 9x smaller than prior best-performing models.

This isn't just a marginal improvement; it's a fundamental shift in what's possible with compact vision-language models. The AI OCR model processes documents at an impressive 5.71 pages per second on H100 GPUs, substantially outperforming larger alternatives. This speed advantage, combined with its smaller footprint, makes LightOnOCR-2 practical for real-world deployment scenarios where computational resources matter.

End-to-end document OCR architecture: Simplicity meets performance

Unlike traditional document OCR systems that rely on brittle multi-stage pipelines (text detection → text recognition → layout analysis → post-processing), LightOnOCR-2 uses a single, unified end-to-end architecture. This approach offers several key advantages:

• Fully differentiable: The entire pipeline can be optimized end-to-end, eliminating error propagation between stages
• Naturally ordered output: The model produces clean, naturally ordered text without requiring complex post-processing heuristics
• Robust to layout complexity: Excels at handling tables, forms, receipts, scientific notation, and multi-column layouts

The AI OCR model is trained at 1540px maximum longest edge resolution, enabling it to capture fine details in dense mathematical notation and small text—critical for scientific and technical documents.

Multilingual document OCR excellence with a French focus

LightOnOCR-2 demonstrates strong multilingual capabilities, with particular emphasis on European languages and French documents. The AI OCR model was trained on a large-scale, high-quality distillation mix that includes extensive coverage of:

• Scanned documents with varying quality and degradation
• French-language content
• Scientific PDFs with dense typography and mathematical notation

This training approach enables the AI OCR model to handle the nuances of different languages and document types without requiring language-specific preprocessing or post-processing steps.

Advanced document OCR training techniques

The development of LightOnOCR-2 incorporates several cutting-edge techniques:

• RLVR optimization: The model uses Reinforcement Learning from Visual Representations (RLVR) with IoU-based rewards to refine its performance, particularly for tasks involving bounding box prediction for embedded images.
• Checkpoint averaging: Multiple model checkpoints are averaged to improve robustness and generalization, reducing variance in predictions across different document types.
• Task-Arithmetic merging: The model leverages task-arithmetic merging techniques to combine capabilities from different training stages, resulting in a more capable and robust final model.

Practical AI OCR model usage with FiftyOne

The AI OCR model integrates seamlessly with FiftyOne, making it easy to apply OCR to document datasets:

import fiftyone as fo
import fiftyone.zoo as foz
# Register and load the model
foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/LightOnOCR-2",
    overwrite=True
)
model = foz.load_zoo_model("lightonai/LightOnOCR-2-1B")
# Apply to your dataset
dataset = fo.Dataset.from_images_dir("/path/to/documents")
dataset.apply_model(
    model,
    label_field="ocr_text",
    batch_size=8,
    num_workers=2
)

The integration supports efficient batch processing, allowing you to process large document collections quickly. With configurable parameters for maximum tokens, custom prompts, and batch sizes, the model adapts to various use cases and computational constraints.

Get started now →

Document OCR use cases

LightOnOCR-2 excels in several key scenarios:

• Scientific PDFs: Handles dense typography and accurately transcribes LaTeX mathematical notation
• Scanned documents: Robust to moderate degradation, noise, and rotation
• European languages: Strong Latin script support, especially for French
• Complex layouts: Multi-column documents, tables, forms, and structured content

Looking forward to a more accessible document OCR future

LightOnOCR-2 represents a significant step forward in making high-quality document OCR accessible and practical. By combining state-of-the-art performance with compact size and fast inference, it opens up new possibilities for document processing applications. The AI OCR model is released under Apache 2.0 license, making it freely available for both research and commercial use. The team has also released the training dataset and LightOnOCR-bbox-bench evaluation suite, contributing to the broader OCR research community. As document digitization remains a critical need across industries—from legal and healthcare to research and education—models like LightOnOCR-2 show we don't need to choose between accuracy and efficiency. Sometimes, the best solution does more with less.

Get started now →

Document OCR resources:

• Model
• Paper
• FiftyOne Integration

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.

Introduction

The field of computer vision has seen remarkable progress with vision foundation models that excel at specific tasks—CLIP for vision-language alignment, DINOv3 for dense perception, SAM3 for segmentation. But what if you could distill the best of these foundation models into a single, unified architecture? That's exactly what C-RADIOv4 achieves.

C-RADIOv4 is the latest release in NVIDIA Labs' agglomerative vision foundation model family. Through innovative multi-teacher distillation, it creates a unified student model that retains and improves upon the distinct capabilities of multiple state-of-the-art teachers: SigLIP2, DINOv3, and SAM3.

In this blog post, we'll explore what makes C-RADIOv4 special, dive into its technical innovations, and show you how to leverage its powerful capabilities in FiftyOne for computer vision workflows.

What are foundation models?

Foundation models are large, pretrained models that learn broad, reusable representations from massive datasets and can be adapted to many downstream tasks with little or no task-specific training.

What is C-RADIOv4?

C-RADIOv4 is an agglomerative foundation model that learns from multiple teacher models simultaneously. Instead of training on labeled data, it distills foundation model knowledge from three cutting-edge vision models:

• SigLIP2: The frontier text-image foundation encoder, providing enhanced zero-shot classification capabilities
• DINOv3: A self-supervised learning powerhouse that delivers strong dense perception and semantic segmentation
• SAM3: The latest iteration of Segment Anything Model, enabling powerful segmentation capabilities

The result? A single vision language model that combines text-image alignment, dense perception, and segmentation—all while maintaining competitive performance at a fraction of the parameters of its teachers.

Get started with C-RADIOv4 →

C-RADIOv4: Vision foundation model variants

C-RADIOv4 comes in two variants:

Both foundation models achieve impressive results:

• Zero-shot accuracy: 83.09% (H) and 82.01% (SO400M) on ImageNet-1K
• kNN accuracy: 86.59% (H) and 85.76% (SO400M) on ImageNet-1K
• Semantic segmentation: 55.20 mIoU (H) and 55.14 mIoU (SO400M) on ADE20k

Remarkably, C-RADIOv4-H achieves performance competitive with DINOv3-7B (which has 6.7B parameters) on dense tasks, despite having an order of magnitude fewer parameters.

C-RADIOv4: Key technical innovations for vision foundation models

1. Stochastic resolution training

Previous RADIO vision foundation models trained at just two resolutions, which could lead to inconsistent behavior at inference time—a phenomenon called "mode switching." C-RADIOv4 solves this by training across a wide range of resolutions:

• Low-resolution partition: {128, 192, 224, 256, 384, 432} pixels
• High-resolution partition: {512, 768, 1024, 1152} pixels

This enables smooth resolution scaling and substantial quality improvements at low resolutions. The model demonstrates strong robustness even at resolutions higher than those it was trained on, achieving 57.72 mIoU at 1536px on ADE20k.

2. Shift equivariance

One challenge in foundation model distillation is that students can learn not just useful features, but also fixed-pattern noise from teachers. C-RADIOv4 addresses this with two forms of shift equivariance:

• Shift Equivariant Loss: Randomly shifts crops for both student and teachers independently, preventing the model from learning position-dependent artifacts
• Shift Equivariant MESA: Applies exponential moving average matching with different crops, further reducing fixed-pattern noise

These techniques ensure the model learns robust, semantically meaningful features rather than spurious patterns.

3. ViTDet mode for high-resolution efficiency

For high-resolution inference, C-RADIOv4 supports "ViTDet mode," which uses windowed attention instead of full global attention. This dramatically reduces inference time:

• The SO400M variant with ViTDet window size ≤ 12 is faster than SAM3's encoder
• ViT-H with window size 8 is nearly as fast as SAM3
• Latency scales much more favorably with resolution compared to full attention

This makes C-RADIOv4 practical for real-world applications requiring high-resolution processing.

4. Balanced summary loss

Different teachers produce summary features with different angular dispersions. Without normalization, teachers with larger dispersion would dominate the loss. C-RADIOv4 uses an angular loss formulation that normalizes for each teacher's cone radius, ensuring balanced learning from all teachers.

Getting started with C-RADIOv4 in FiftyOne

FiftyOne provides seamless integration with C-RADIOv4, making it easy to leverage these powerful capabilities in your computer vision workflows.

Installation

First, install FiftyOne and register the C-RADIOv4 model source:

# Install FiftyOne
pip install fiftyone

# Register the RADIO model source
import fiftyone.zoo as foz
foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/CRADIOv4",
)

Quick start: Computing embeddings

The most common use case is extracting image embeddings for similarity search, clustering, and visualization:

import fiftyone as fo
import fiftyone.zoo as foz

# Load your dataset
dataset = foz.load_zoo_dataset("quickstart", shuffle=True)

# Load the C-RADIOv4 model
model = foz.load_zoo_model("nv_labs/c-radio_v4-h")

# Compute embeddings with efficient batching
dataset.compute_embeddings(
    model=model,
    embeddings_field="radio_embeddings",
    batch_size=16,      # Adjust based on GPU memory
    num_workers=4       # Parallel data loading
)

# Launch the FiftyOne App to explore
session = fo.launch_app(dataset)

Model configuration options

C-RADIOv4 offers flexible configuration for different use cases:

# Global image embeddings (default)
model = foz.load_zoo_model(
    "nv_labs/c-radio_v4-h",
    output_type="summary",      # Global semantic features
    feature_format="NCHW"        # or "NLC"
)

# Spatial attention features
spatial_model = foz.load_zoo_model(
    "nv_labs/c-radio_v4-h", 
    output_type="spatial",       # Patch-level spatial features
    feature_format="NCHW",       # Required for spatial
    apply_smoothing=True,         # Gaussian smoothing for cleaner heatmaps
    smoothing_sigma=1.51         # Smoothing strength
)

C-RADIOv4: Practical use cases for vision foundation models

1. Embedding visualization with UMAP

Create beautiful 2D visualizations of your image embeddings to understand dataset structure:

import fiftyone.brain as fob

# Compute embeddings first
dataset.compute_embeddings(
    model=model,
    embeddings_field="radio_embeddings"
)

# Create UMAP visualization
results = fob.compute_visualization(
    dataset,
    method="umap",  # Also supports "tsne", "pca"
    brain_key="radio_viz",
    embeddings="radio_embeddings"
)

# Explore in the App
session = fo.launch_app(dataset)

The visualization reveals clusters of similar images, making it easy to identify patterns and outliers in your dataset.

2. Similarity search

Build powerful similarity search capabilities:

import fiftyone.brain as fob

# Build similarity index
results = fob.compute_similarity(
    dataset,
    backend="sklearn",
    brain_key="radio_sim", 
    embeddings="radio_embeddings"
)

# Find similar images
sample_id = dataset.first().id
similar_samples = dataset.sort_by_similarity(
    sample_id,
    brain_key="radio_sim",
    k=10  # Top 10 most similar
)

# View results
session = fo.launch_app(similar_samples)

This is incredibly useful for:

• Finding duplicate or near-duplicate images
• Discovering similar content for data augmentation
• Building recommendation systems
• Quality control and anomaly detection

3. Spatial attention heatmaps

Visualize what regions of an image the model focuses on using spatial attention features:

# Load spatial model with smoothing
spatial_model = foz.load_zoo_model(
    "nv_labs/c-radio_v4-h",
    output_type="spatial",
    apply_smoothing=True,
    smoothing_sigma=1.51,
    feature_format="NCHW"
)

# Generate attention heatmaps
dataset.apply_model(
    spatial_model, 
    "radio_heatmap",
    batch_size=16,
    num_workers=4
)

# View heatmaps in FiftyOne App
session = fo.launch_app(dataset)

The heatmaps use PCA visualization (as described in the C-RADIOv4 paper) to show object boundaries and regions of interest. This is particularly useful for:

• Understanding model behavior
• Debugging and interpretability
• Identifying important image regions
• Quality assessment

4. Dataset curation: Uniqueness and representativeness

Use embeddings to score how unique or representative each sample is:

import fiftyone.brain as fob

# Compute uniqueness scores
fob.compute_uniqueness(
    dataset,
    embeddings="radio_embeddings"
)

# Compute representativeness scores
fob.compute_representativeness(
    dataset,
    representativeness_field="radio_represent",
    method="cluster-center",
    embeddings="radio_embeddings"
)

# Find most representative samples
representative_view = dataset.sort_by("radio_represent", reverse=True)

These scores help with:

• Active learning: Select diverse samples for labeling
• Dataset balancing: Identify underrepresented regions
• Data cleaning: Find outliers and edge cases
• Efficient annotation: Prioritize representative samples

5. Duplicate detection

Automatically find and remove near-duplicate images:

import fiftyone.brain as fob

# Detect duplicates using embeddings
results = fob.compute_uniqueness(
    dataset,
    embeddings="radio_embeddings"
)

# Filter to most unique samples
unique_view = dataset.sort_by("uniqueness", reverse=True)

This is essential for:

• Cleaning datasets before training
• Reducing storage costs
• Improving dataset quality
• Preventing data leakage in train/test splits

Advanced workflows for vision foundation models

Comprehensive analysis pipeline

Combine multiple C-RADIOv4 outputs for thorough dataset analysis:

# Step 1: Global embeddings for similarity
embedding_model = foz.load_zoo_model("nv_labs/c-radio_v4-h")
dataset.compute_embeddings(embedding_model, "radio_embeddings")

# Step 2: Spatial heatmaps for attention analysis
spatial_model = foz.load_zoo_model(
    "nv_labs/c-radio_v4-h",
    output_type="spatial",
    apply_smoothing=True,
    smoothing_sigma=0.8
)
dataset.apply_model(spatial_model, "radio_heatmap")

# Step 3: Build similarity index
import fiftyone.brain as fob
fob.compute_similarity(dataset, embeddings="radio_embeddings", brain_key="radio_sim")

# Step 4: Compute additional metrics
fob.compute_uniqueness(dataset, embeddings="radio_embeddings")
fob.compute_representativeness(dataset, embeddings="radio_embeddings")

# Step 5: Comprehensive exploration
session = fo.launch_app(dataset)

This pipeline gives you:

• Similarity search for finding related images
• Attention visualization for understanding model focus
• Uniqueness scores for identifying outliers
• Representativeness scores for dataset balancing
• Interactive exploration in the FiftyOne App

Performance optimization for vision foundation models

Batch processing

For optimal performance on large datasets, use batching with parallel data loading:

dataset.compute_embeddings(
    model=model,
    embeddings_field="radio_embeddings",
    batch_size=16,      # Adjust based on GPU memory
    num_workers=4       # Parallel data loading workers
)

Recommended batch sizes by GPU:

• RTX 3090/4090 (24GB): batch_size=16-32
• A100 (40GB/80GB): batch_size=32-64
• Smaller GPUs: batch_size=4-8

GPU acceleration

C-RADIOv4 automatically uses GPU acceleration with mixed precision (bfloat16) on compatible hardware (Ampere+). The vision foundation model automatically handles:

• Device placement (GPU/CPU)
• Mixed precision inference
• Efficient memory management

High-resolution processing

For high-resolution images, consider using the SO400M variant with ViTDet mode enabled (when available through the API) for faster inference:

# Use smaller model for faster inference
model = foz.load_zoo_model("nv_labs/c-radio_v4-so400m")

Real-world applications for vision foundation models

C-RADIOv4 has been successfully deployed in various domains:

• Autonomous Vehicles: Dense perception for scene understanding
• Robotics: Visual navigation and manipulation
• Document Processing: OCR and document parsing (Nemotron Parse)
• Open Vocabulary Segmentation: RADSeg uses RADIO for state-of-the-art segmentation
• Vision-Language Models: Used as vision encoder in Nemotron Nano V2 VL

The vision foundation model's permissive license (NVIDIA Open Model License Agreement) makes it suitable for both research and commercial applications.

Why C-RADIOv4?

Unified capabilities

Instead of maintaining separate models for different tasks, C-RADIOv4 provides:

• Zero-shot classification (from SigLIP2)
• Dense perception (from DINOv3)
• Segmentation capabilities (from SAM3)

All in a single, efficient model.

Efficiency

With 631M parameters (H variant) or 412M parameters (SO400M variant), C-RADIOv4 achieves performance competitive with models having 10x more parameters, making it practical for deployment.

Flexibility

• Works across a wide range of resolutions (128px to 1152px+)
• Supports both global embeddings and spatial features
• Can replace SAM3's vision encoder for segmentation tasks
• ViTDet mode enables efficient high-resolution processing

Quality

The vision foundation model demonstrates:

• Cleaner object boundaries (as shown in PCA visualizations)
• Strong resolution scaling properties
• Robust performance across diverse benchmarks
• Reduced fixed-pattern noise compared to individual teachers

Foundation model distillation and the road ahead for vision models

C-RADIOv4 represents a significant advancement in agglomerative vision foundation models. By combining the strengths of SigLIP2, DINOv3, and SAM3 through innovative foundation model distillation techniques, it delivers a unified model that's both powerful and efficient.

The integration with FiftyOne makes it easy to leverage these capabilities for:

• Dataset exploration and visualization
• Similarity search and duplicate detection
• Attention visualization and interpretability
• Dataset curation and active learning
• Quality control and anomaly detection

Whether you're working on autonomous vehicles, robotics, document processing, or any computer vision application, C-RADIOv4 provides a solid foundation for your workflows.

Resources

• GitHub Repository: CRADIOv4 FiftyOne Integration
• NVIDIA RADIO GitHub: Original Implementation
• Hugging Face Models: C-RADIOv4 Models
• Technical Report: C-RADIOv4 Tech Report
• FiftyOne Documentation: docs.voxel51.com

Citation

If you use C-RADIOv4 in your research, please cite:

@misc{ranzinger2026cradiov4,
      title={C-RADIOv4 (Tech Report)},
      author={Mike Ranzinger and Greg Heinrich and Collin McCarthy and Jan Kautz and Andrew Tao and Bryan Catanzaro and Pavlo Molchanov},
      year={2026},
      eprint={2601.17237},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.17237},
}

Ready to get started? Install FiftyOne, register the C-RADIOv4 model source, and begin exploring your datasets with this powerful unified vision foundation model!

Get started →

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.