Skip to main content

agrag.chunking

Chunking: how a Document becomes Chunks.

A Chunker splits one document. A Chunking holds the rules that pick a chunker for each document, and DEFAULT_CHUNKING is the preset that Graph uses.

Modules:

  • base – The Chunker contract: how a Document becomes Chunks, and how that is recorded.
  • docling – Docling-native chunking.
  • extras – Opt-in chunkers that need a package extra: semantic, neural and code.
  • heading – The heading-aware strategy: sections packed to a token budget.
  • parent_child – The parent-child strategy: large parents to extract, small children to search.
  • recursive – The recursive strategy: split on the coarsest delimiter that fits the budget.
  • rules – Chunking rules: which chunker a document gets, as data.
  • sentence – The sentence strategy: whole sentences packed up to a token budget.
  • token – The token strategy: fixed-size windows of tokens, with optional overlap.
  • turns – The turn-window strategy: whole chat turns packed to a token budget.

Classes:

  • Chunker – Splits one Document into Chunks and builds their provenance.
  • ChunkerMissingExtraError – A chunker needs a package extra that is not installed.
  • Chunking – An ordered list of chunking rules and a fallback chunker.
  • ChunkingError – A chunker broke the chunk contract or failed to chunk a document.
  • ChunkingRule – A match and the chunker for the documents it matches.
  • CodeChunker – Cuts source code along its syntax tree.
  • DoclingChunker – Splits a parsed docling document with docling's hybrid chunker.
  • HeadingChunker – Cuts a document into sections at its headings and packs them to a budget.
  • NeuralChunker – Cuts where a token classification model predicts a topic break.
  • ParentChildChunker – Cuts a document into parent chunks and cuts each parent into child chunks.
  • RecursiveChunker – Splits on paragraph, sentence and word boundaries, coarsest first.
  • RuleMatch – The documents a rule applies to.
  • SemanticChunker – Cuts where the meaning of neighbouring sentences changes.
  • SentenceChunker – Packs whole sentences into chunks of at most chunk_size tokens.
  • SplitLevel – One level of recursive split rules.
  • TokenChunker – Cuts the text into windows of chunk_size tokens.
  • TurnWindowChunker – Packs whole chat turns into windows of at most chunk_size tokens.

Attributes:

  • DEFAULT_CHUNKING – The preset that Graph uses: docling loads go to DoclingChunker.