Skip to main content

agrag.chunking.extras.SemanticChunker

Bases: _ExtraSpanChunker

Cuts where the meaning of neighbouring sentences changes.

Needs the chunk-semantic extra. The chunker embeds sentences with a small static model and cuts where similarity drops below threshold.

Attributes:

  • embedding_model (str) – The model that embeds sentences. The first use downloads it.
  • threshold (float) – The similarity below which a new chunk starts, from 0 to 1.
  • chunk_size (int) – The largest chunk size, as chonkie's semantic chunker counts it.
  • similarity_window (int) – The number of sentences that a similarity looks across.

Functions:

  • chunk – Split a document into chunks.
  • fingerprint – Return the hash of settings(), 16 hex characters.
  • model_copy – Copy the chunker, validating any changed setting.
  • model_post_init – Compute the fingerprint only, so the extra is not needed to build.
  • settings – Return the strategy name and every setting as JSON-safe data.
  • spans – Return the character spans this strategy cuts text into.

chunk​

chunk(document:Document) -> list[Chunk]

Split a document into chunks.

Every chunk has non-empty text, indexes run from 0 without gaps, and a chunk with text provenance has text equal to document.text at its offsets.

Parameters:

  • document (Document) – The document to split.

Returns:

  • list[Chunk] – The chunks, in document order, each with chunker and chunker_hash set. A strategy that sets chunker itself keeps its value.

Raises:

  • ChunkingError – The strategy returned chunks that break the contract.

chunk_size​

chunk_size: int = Field(default=256, gt=0)

embedding_model​

embedding_model: str = 'minishlab/potion-base-32M'

fingerprint​

fingerprint() -> str

Return the hash of settings(), 16 hex characters.

model_config​

model_config = ConfigDict(frozen=True, extra='forbid')

model_copy​

model_copy(*, update:Mapping[str, Any] | None = None, deep:bool = False) -> Self

Copy the chunker, validating any changed setting.

A plain copy keeps the fingerprint and the splitter of the original. A copy with changes is built again from its configuration.

model_post_init​

model_post_init(context:Any) -> None

Compute the fingerprint only, so the extra is not needed to build.

settings​

settings() -> dict[str, Any]

Return the strategy name and every setting as JSON-safe data.

A setting that is itself a chunker appears as that chunker's settings.

similarity_window​

similarity_window: int = Field(default=3, gt=0)

spans​

spans(text:str) -> list[tuple[int, int]]

Return the character spans this strategy cuts text into.

Raises:

strategy​

strategy: str

The strategy name, "semantic".

threshold​

threshold: float = Field(default=0.8, gt=0, le=1)