How to: LLM-Automated Labeling for Legal Docs

In the world of natural language processing (NLP), labeled data is the foundation of every great model. But if you've ever done legal NLP labeling manually, you know just how time-consuming and exhausting it can be. At Datasaur, a leading data labeling platform, we’ve integrated LLM-automated labeling into the workflow to automate much of the heavy lifting: especially in complex domains like legal contracts.

This post walks you through how we used Data Studio, our NLP labeling tool, to build a legal span-based project, where the labeling is automated by large language models (LLMs) and supported by a human-in-the-loop (HITL) workflow to ensure high-quality results.

At Datasaur, we are committed to making automatic labeling accessible to any type of user. If you are a technical user like a data scientist or a non-technical lawyer – each should have an equally intuitive experience to automatically label data. In Data Studio’s labeling interface, it only takes a few clicks to automatically label all of your data. Let's find out how to set this up!

For a visual guide watch this 3:00 minute video.

Step 1: Upload Your Dataset

Whether you're working with legal contracts, customer support transcripts, financial disclosures, or clinical notes, Datasaur makes it easy to upload virtually any kind of dataset. You can import data in CSV, PDF, JSON, TXT, and many other formats. Once uploaded, you can choose between span-based and/or classification labeling projects depending on your use case.

In this example, we used a synthetic legal contract to demonstrate span-based labeling, where users identify specific text snippets (or “spans”) like names, dates, and clauses. But remember: you can bring in your own real-world legal dataset.

Step 2: Define or Upload Your Labelset

Datasaur supports flexible and customizable label management (aka taxonomy). You can upload your own labelset directly into the platform using CSV files, or create them manually through the interface. If you have multiple labelsets for different users or workflows, you can even assign multiple labelsets to the same project—allowing teams to segment work or perform multi-layered analysis.

For our legal project example, we used a labelset that included:

• Effective_Date
  
• Party_Name
  
• Party_Type
  
• Address
  
• Governing_Law
  
• Agreement_Type
  
• Obligation
  
• Term_Duration
  
• Termination_Clause
  
• Confidential_Information
  
• Payment_Term
  
• Signature_Block
  

These labels served as the instruction set for the model, enabling it to recognize key parts of a contract.

Step 3: Enable ML-Assisted Labeling with LLMs

Here’s where the automation kicks in. Datasaur's extension “ML-assisted labeling” allows you to connect your project to industry-leading LLMs and receive suggested labels in seconds.

You can choose from a variety of LLM providers (including your own), such as:

• OpenAI (ChatGPT)
  
• Azure OpenAI
  
• Anthropic (Claude)
  
• Google Gemini
  
• Cohere
  

Once connected, you can configure the LLM using:

• System prompt (what the LLM should know about the task)
  
• User prompt (the specific instruction for labeling)
  
• Model parameters like temperature, top-p, and token limits
  

After defining your prompt, you simply click "Predict Labels." That’s it, it is really that simple! The model analyzes the passage and returns a JSON of labeled entities, which are immediately rendered as labeled spans in the Datasaur interface. You’ll see exactly which text was tagged under each category—no manual highlighting required.

Step 4: Human-in-the-Loop Validation (HITL)

While LLMs are powerful, there’s no substitute for human oversight—especially in domains like law or medicine. That’s why we support a human-in-the-loop approach.

After the model returns its predictions, a human reviewer steps in to:

• Accept spans that are correct
  
• Adjust boundaries or refine any inaccurate labels
  
• Remove incorrect suggestions
  
• Add any missing spans the LLM didn’t catch
  

This turns labeling from a slow, manual process into a faster, higher-quality QA review.

‍

Why It Works: The Benefits of Combining LLMs with HITL

1. LLMs Are Highly Effective at Legal Labeling

Large language models trained on public and proprietary legal corpora can quickly identify common legal structures and extract relevant entities with high accuracy.

2. It Saves the Labeling Team Time

By integrating LLMs into your data labeling, you cut down hours of manual tagging. Teams can focus their energy where it matters most.

3. Labelers Can Focus on QA, Not Tedious Work

Rather than manually combing through every sentence, labelers become reviewers—ensuring consistency and catching any errors before the dataset goes live for model training and requiring fewer people to review than the typical team sizes required to label.

Final Thoughts

Legal NLP doesn’t have to be tedious. By combining LLM automated labeling with Data Studio, your team can scale faster, label smarter, and produce higher-quality data with less effort.

Whether you're working in the legal, medical, financial, or other enterprise domain, a modern data labeling platform like Datasaur helps you unlock the full potential of AI. 

‍