Skip to main content

agrag.loaders.corpus.base.ProseLoader

Bases: Loader

A loader that makes one Document per source.

Concrete readers reject a source larger than the configured byte limit when the source's byte size is known upfront (SourceRef.byte_size is not None).

Functions:

  • load – Yield documents read from one source.

Attributes:

extensions​

extensions: frozenset[str]

extra​

extra: str | None = None

family​

family = DocumentFamily.PROSE

load​

load(source:SourceRef, stream:BinaryIO, opts:ReadOptions, *, start_at:int = 0) -> Iterator[Document]

Yield documents read from one source.

Parameters:

  • source (SourceRef) – The source to read.
  • stream (BinaryIO) – The open binary stream for the source, positioned at the start.
  • opts (ReadOptions) – The read options for this call.
  • start_at (int) – The record index to resume from. Prose loaders ignore this argument.

Yields:

  • Document – One Document per unit the source contains, in a fixed order.

mime_types​

mime_types: frozenset[str] = frozenset()