Skip to main content

agrag.chunking.docling.DoclingChunker

Bases: Chunker

Splits a parsed docling document with docling's hybrid chunker.

The chunker reads the parsed document that the docling loader keeps in Document.metadata["_docling_document"]. Each chunk has page provenance and the headings above it in heading_path. The chunk text is the body without headings, but headings count against the token budget. Chunk ids include the fingerprint, so a re-chunk with new settings does not overwrite the old chunks.

Attributes:

  • tokenizer (str) – The tokenizer that counts size. A name with a slash is a Hugging Face model id, which needs the network the first time.
  • max_tokens (int) – The most tokens in a chunk, headings included.
  • merge_peers (bool) – Whether to merge small neighbours under the same headings.
  • repeat_table_header (bool) – Whether each chunk of a split table repeats its header.
  • omit_header_on_overflow (bool) – Whether to drop headings from a chunk when they will not fit the budget.
  • table_format (Literal['triplet', 'markdown']) – "triplet" writes row, column = value text and "markdown" writes a pipe table.

Functions:

  • chunk – Split a document into chunks.
  • fingerprint – Return the hash of settings(), 16 hex characters.
  • model_copy – Copy the chunker, validating any changed setting.
  • model_post_init – Compute the fingerprint once, after the settings are validated.
  • settings – Return the strategy name and every setting as JSON-safe data.

chunk​

chunk(document:Document) -> list[Chunk]

Split a document into chunks.

Every chunk has non-empty text, indexes run from 0 without gaps, and a chunk with text provenance has text equal to document.text at its offsets.

Parameters:

  • document (Document) – The document to split.

Returns:

  • list[Chunk] – The chunks, in document order, each with chunker and chunker_hash set. A strategy that sets chunker itself keeps its value.

Raises:

  • ChunkingError – The strategy returned chunks that break the contract.

fingerprint​

fingerprint() -> str

Return the hash of settings(), 16 hex characters.

max_tokens​

max_tokens: int = Field(default=1024, gt=0)

merge_peers​

merge_peers: bool = True

model_config​

model_config = ConfigDict(frozen=True, extra='forbid')

model_copy​

model_copy(*, update:Mapping[str, Any] | None = None, deep:bool = False) -> Self

Copy the chunker, validating any changed setting.

A plain copy keeps the fingerprint and the splitter of the original. A copy with changes is built again from its configuration.

model_post_init​

model_post_init(context:Any) -> None

Compute the fingerprint once, after the settings are validated.

omit_header_on_overflow​

omit_header_on_overflow: bool = False

repeat_table_header​

repeat_table_header: bool = True

settings​

settings() -> dict[str, Any]

Return the strategy name and every setting as JSON-safe data.

A setting that is itself a chunker appears as that chunker's settings.

strategy​

strategy: str

The strategy name, "docling".

table_format​

table_format: Literal['triplet', 'markdown'] = 'triplet'

tokenizer​

tokenizer: str = DEFAULT_TOKENIZER