Skip to main content

agrag.loaders.corpus.base.Loader

Bases: ABC

Reads one source and yields Document objects.

A Loader keeps no state between calls, so a worker process can reuse one instance across many sources.

Attributes:

  • extensions (frozenset[str]) – The file extensions this loader claims, each with a leading dot.
  • mime_types (frozenset[str]) – The MIME types this loader claims. Empty when the loader relies on the extension alone.
  • family (DocumentFamily) – The document family this loader produces.
  • extra (str | None) – The optional package extra required to use this loader. None for core loaders. The registry raises MissingExtraError when this extra is not installed.

Functions:

  • load – Yield documents read from one source.

extensions​

extensions: frozenset[str]

extra​

extra: str | None = None

family​

family: DocumentFamily

load​

load(source:SourceRef, stream:BinaryIO, opts:ReadOptions, *, start_at:int = 0) -> Iterator[Document]

Yield documents read from one source.

Parameters:

  • source (SourceRef) – The source to read.
  • stream (BinaryIO) – The open binary stream for the source, positioned at the start.
  • opts (ReadOptions) – The read options for this call.
  • start_at (int) – The record index to resume from. Prose loaders ignore this argument.

Yields:

  • Document – One Document per unit the source contains, in a fixed order.

mime_types​

mime_types: frozenset[str] = frozenset()