agrag.chunking.extras

Opt-in chunkers that need a package extra: semantic, neural and code.

Classes:

• CodeChunker – Cuts source code along its syntax tree.
• NeuralChunker – Cuts where a token classification model predicts a topic break.
• SemanticChunker – Cuts where the meaning of neighbouring sentences changes.

Functions:

• byte_spans_to_char_spans – Convert UTF-8 byte spans of text to character spans.