Skip to main content

agrag.chunking.heading.HeadingChunker

Bases: Chunker

Cuts a document into sections at its headings and packs them to a budget.

The chunker reads Document.heading_outline. A heading of split_level or higher (a level number at or below split_level) starts a new section, and no chunk holds such a heading except at its start. A section within chunk_size tokens is one chunk. A larger section is cut at its deeper headings and the parts are packed to the budget. A part that is still too large goes to fallback, and its chunks have the chunker name heading:<fallback strategy>. A document with no headings goes to fallback as a whole and its chunks have the same name. Only the plain text loaders for Markdown and AsciiDoc fill the outline.

Attributes:

  • chunk_size (int) – The most tokens in a chunk, counted with tokenizer. Tokens are counted for each part alone, so a packed chunk can be a few tokens over.
  • split_level (int) – The deepest heading level that starts a new section, from 1 to 6.
  • tokenizer (str) – The tokenizer that counts size. "character" counts characters.
  • fallback (SerializeAsAny[SpanChunker]) – The chunker for a part above the budget and for a document without headings.

Functions:

  • chunk – Split a document into chunks.
  • fingerprint – Return the hash of settings(), 16 hex characters.
  • model_copy – Copy the chunker, validating any changed setting.
  • model_post_init – Load the tokenizer once, so a bad name fails at construction.
  • settings – Return the strategy name and every setting as JSON-safe data.

chunk​

chunk(document:Document) -> list[Chunk]

Split a document into chunks.

Every chunk has non-empty text, indexes run from 0 without gaps, and a chunk with text provenance has text equal to document.text at its offsets.

Parameters:

  • document (Document) – The document to split.

Returns:

  • list[Chunk] – The chunks, in document order, each with chunker and chunker_hash set. A strategy that sets chunker itself keeps its value.

Raises:

  • ChunkingError – The strategy returned chunks that break the contract.

chunk_size​

chunk_size: int = Field(default=256, gt=0)

fallback​

fallback: SerializeAsAny[SpanChunker] = Field(default_factory=RecursiveChunker)

fingerprint​

fingerprint() -> str

Return the hash of settings(), 16 hex characters.

model_config​

model_config = ConfigDict(frozen=True, extra='forbid')

model_copy​

model_copy(*, update:Mapping[str, Any] | None = None, deep:bool = False) -> Self

Copy the chunker, validating any changed setting.

A plain copy keeps the fingerprint and the splitter of the original. A copy with changes is built again from its configuration.

model_post_init​

model_post_init(context:Any) -> None

Load the tokenizer once, so a bad name fails at construction.

settings​

settings() -> dict[str, Any]

Return the strategy name and every setting as JSON-safe data.

A setting that is itself a chunker appears as that chunker's settings.

split_level​

split_level: int = Field(default=2, ge=1, le=6)

strategy​

strategy: str

The strategy name, "heading".

tokenizer​

tokenizer: str = DEFAULT_TOKENIZER