Skip to main content

agrag.loaders.corpus.base.RecordLoader

Bases: Loader

A loader that makes one Document per record in a source.

Concrete readers read and decode the whole source into memory up front, up to opts.max_document_bytes; that byte limit is what bounds memory use, not incremental reads from disk. Whether records are parsed incrementally from there is format-dependent: the CSV and JSONL readers parse and yield one record at a time, so a malformed record later in the source surfaces only after earlier records have already been yielded. The JSON reader parses the whole source up front, so a malformed source fails before any record is yielded.

Functions:

  • load – Yield documents read from one source.

Attributes:

extensions​

extensions: frozenset[str]

extra​

extra: str | None = None

family​

family = DocumentFamily.RECORD

load​

load(source:SourceRef, stream:BinaryIO, opts:ReadOptions, *, start_at:int = 0) -> Iterator[Document]

Yield documents read from one source.

Parameters:

  • source (SourceRef) – The source to read.
  • stream (BinaryIO) – The open binary stream for the source, positioned at the start.
  • opts (ReadOptions) – The read options for this call.
  • start_at (int) – The record index to resume from. Prose loaders ignore this argument.

Yields:

  • Document – One Document per unit the source contains, in a fixed order.

mime_types​

mime_types: frozenset[str] = frozenset()