Introduction

Unstructured.io is an open-source framework for the structured preparation of unstructured documents such as PDFs, Word files, HTML pages, or emails. Its goal is to extract semantically usable content from these heterogeneous formats—such as headings, paragraphs, tables, or lists—and convert it into a unified, machine-readable format. The main use case lies in preparing text data for downstream AI processing, particularly for systems with retrieval-augmented generation (RAG).

The typical application is document analysis, knowledge management, or preparing inputs for embedding models. Multiple processing steps are employed. These four steps form the core of the Unstructured.io pipeline and are executed in every regular use of the library.

• Partitioning
  Splitting the document into logical content elements such as paragraphs, headings, or tables.

• Cleaning
  Removing irrelevant components such as headers, footers, or watermarks.

• Extracting
  Extracting the raw text content along with accompanying metadata.

• Staging
  Formatting this information into a consistent structured format (e.g. JSON or Markdown).

In addition, there are optional extensions:

• Chunking: Longer content is split into smaller text units that are better suited for embedding models. This step is not strictly part of the core library but is necessary for many AI applications.

• Embedding: The resulting text chunks are sent to external models (e.g. OpenAI, HuggingFace, SentenceTransformers) to generate vector representations. This step occurs outside of Unstructured and must be implemented by the user or supplemented via existing example pipelines.

Thus, Unstructured.io handles document preprocessing up to structured text output. For a complete RAG pipeline—including embedding, vector database, retrieval logic, and language model—additional tools are required. Unstructured therefore represents the initial, preparatory stage of such an architecture.

Example PDF Documents for Experimenting

https://github.com/Unstructured-IO/unstructured/tree/main/example-docs/pdf

Supported Data Types

Required External Libraries

Data Pipeline

The unstructured.io workflow is a multi-stage data pipeline that systematically transforms raw documents into AI-ready information. The process begins with Partitioning, where a document is split into its logical elements such as titles, paragraphs, and tables. The subsequent Cleaning step removes disruptive content like headers or footers from these elements. Next comes Extracting, where the pure text content and key metadata are extracted from the elements. The Staging stage then structures these extracted data for the following phases. In Chunking, longer, contiguous texts are divided into smaller, model-optimized sections. The final step, Embedding, converts these text chunks into numerical vectors that can be processed and searched by AI systems such as RAG applications.

[ Partitioning ] > [ Cleaning ] > [ Extracting ] > [ Staging ] > [ Chunking ] > [ Embedding

Core Pipeline Steps (Mandatory Steps)

[ Partitioning ] > [ Cleaning ] > [ Extracting ] > [ Staging ]

Partitioning

The unstructured library offers partitioning functions to break raw documents into structured building blocks such as titles, body text, or list items. This allows users to specifically select only the content they need for their task, for example only the body text for training a summarization model.

Example *.PDF file for testing the unstructured.io library:

pm-partnerschaft-stackitDownload

The partition function has analyzed the above PDF document. Instead of viewing the document only as a long, unformatted text sequence, it interpreted the visual structure and layout of the page to identify individual logical building blocks (Document Elements), e.g. “Title” and “NarrativeText”.

Explanation of each element type from the PDF, e.g. Title:

This is the list of the main document elements such as Text, Header, Image, etc. The complete list of elements is available here: Link

Cleaning

The cleaning function is used to remove unwanted content from documents to prepare the data for downstream tasks such as processing by language models (LLMs). The goal is to obtain “clean” and more relevant data.

The following table provides an overview of the different cleaning functions. Source: Link

Extraction

The Extracting step isolates specific information from already cleaned text elements.

For example, if a text contains the sentence: Please contact support@example.com for further information. Instead of processing the entire sentence, an extraction function like extract_email_address would pull out only the desired data point, namely support@example.com.

Staging

Staging functions in the unstructured package prepare extracted document elements for downstream processing steps. They take a list of structured elements as input, such as Title or NarrativeText. The output is a format-specific dictionary—a structured collection of data in the form of key-value pairs. Each piece of information, such as a text fragment or a metadata entry, is assigned a unique identifier. These identifiers, called keys, can be “text,” “metadata,” or “type.” The goal is to prepare the data so that it can be directly processed by the intended target system.

Originally, specialized conversion functions were available for different use cases:

• A basic conversion like convert_to_csv for translating into tabular formats
• A conversion for machine learning and NLP platforms like stage_for_transformers for preparing training data
• A conversion for vector databases like stage_for_weaviate for semantic indexing

These functions are now deprecated. Further development focuses on the concept of destination connectors, e.g. for Kafka, Weaviate, or MongoDB. This enables automated transfer of data to external platforms after extraction.

The selection of supported platforms is continuously being expanded. If a desired target environment is not included, it can be suggested in the community Slack.

Advanced Pipeline Steps (Optional)

[ Chunking ] > [ Embedding

Chunking

Unstructured uses metadata and document elements recognized by partitioning functions to transform elements into more useful “chunks.”

Chunking Strategy

Currently, Unstructured offers two chunking strategies—“basic” and “by_title.” The main components of the basic strategy are:

basic

The “basic” strategy is the simplest and one of the most commonly used methods in the unstructured library. Its main task is to combine sequential document elements to create chunks that are as large as possible without exceeding a set maximum character count:

• max_characters: a hard limit that must not be exceeded
• new_after_n_chars: a soft limit. When exceeded, a new chunk is started even if the hard limit has not been reached. A new chunk is preferentially started when it makes semantic or technical sense, but this threshold can be ignored as long as the hard limit is respected.
• overlap: if a single element is too large and must be split, this parameter specifies how many characters overlap between the end of one chunk and the beginning of the next. This helps maintain context across chunk boundaries.
• overlap_all: when set to True, overlap is applied not only to oversized, split elements but between all consecutive chunks.

Special case tables: Tables (Table elements) are always treated as standalone chunks and never combined with other elements. If a table itself is too large, it is also split.

With “basic,” we reach the limits of this strategy on highly structured documents such as reports, scientific papers, or manuals.

by_title

The by_title strategy inherits all basic strategy behaviors (such as respecting max_characters) but adds three crucial new rules:

• The strategy identifies Title elements (headings) as the start of a new section. When the algorithm encounters a heading, it performs the following actions:

  • The current chunk is immediately closed.
  • A new chunk is started beginning with this Title element.

  This occurs even if the heading text would have fit perfectly into the previous chunk.

• By default, the strategy does not treat page breaks as hard boundaries. This means a section can span multiple pages without being split into a new chunk. This behavior is controlled by the parameter multipage_sections=True (default).

• Sometimes very short texts, such as individual list entries, are mistakenly identified as Title elements. This can lead to a flood of tiny, unwanted chunks. To address this, there is the combine_text_under_n_chars parameter:

  • This parameter allows multiple consecutive small sections to be combined into a single chunk to best fill the chunking window (max_characters).
  • By default, combine_text_under_n_chars is the same as max_characters. This ensures that small sections are efficiently grouped.
  • A value of 0 means every tiny, title-identified section is defined as a new chunk.

The by_title strategy is the better choice for documents with a clear hierarchical structure such as reports, research articles, manuals, or contracts.

Embedding

After extracting content from documents using the open-source library unstructured, a common downstream step is converting the extracted text elements into vector embeddings. These vectors form the basis for semantic search functions and retrieval-augmented generation (RAG).

The core unstructured library itself does not include native functions for calling embedding providers. Instead, there are two primary methods for generating embeddings for unstructured outputs.

Method 1: The Unstructured ecosystem provides extended functionality in the form of the Unstructured Ingest command-line interface (CLI) and the associated Python library. These components are designed for creating complete end-to-end processing pipelines and offer built-in support for connecting to embedding providers.

In this approach, embedding generation is seamlessly integrated into the data ingestion process (“Ingest Pipeline”). This enables full automation of the workflow from raw document to final vector embedding. Configuration details are available in the official documentation.

Method 2: An alternative method is manually enriching the JSON files produced by unstructured. This approach offers high flexibility, especially in choosing the embedding model, and is suitable for scenarios without a complete ingest pipeline.

The process typically follows a fixed schema:

• Input: A JSON file generated by the unstructured library serves as the input source.
• Reading: The content of the JSON file is loaded into memory as a structured object.
• Embedding Generation: A third-party library, such as sentence-transformers, is used to generate an embedding for the value of the text field of each element in the JSON file. The sentence-transformers/all-MiniLM-L6-v2 model from Hugging Face is a common example.
• Data Enrichment: The generated embedding is added as a new field alongside the corresponding text field in the JSON object.
• Saving: The modified JSON object with the added embeddings is written back to the original file or a new file.

This approach provides full control over the process and can be integrated into existing workflows based on exchanging JSON files.