Skip to main content

agrag.loaders.corpus.types.ReadOptions

ReadOptions(encoding:str | None = None, max_document_bytes:int = 32 * 1024 * 1024, store_text:bool = True, store_raw_record:bool = False, on_error:ErrorPolicy = ErrorPolicy.RAISE, text_column:str | None = None, id_column:str | None = None, title_column:str | None = None, json_mode:JsonMode = JsonMode.AUTO, csv_mode:CsvMode = CsvMode.ROWS, csv_delimiter:str | None = None, html_selector:str | None = None, normalization:Normalization = Normalization()) -> None

Per-source reader configuration.

Frozen so it is safe to share across worker processes.

Attributes:

  • encoding (str | None) – The text encoding to use. None lets the decoder detect it.
  • max_document_bytes (int) – The largest prose source the loader will read.
  • store_text (bool) – When false, the document text is an empty string.
  • store_raw_record (bool) – When true, a record document keeps its raw row data.
  • on_error (ErrorPolicy) – The error policy to apply inside the reader.
  • text_column (str | None) – The column that holds document text. Required for record sources.
  • id_column (str | None) – The column whose value becomes the document id.
  • title_column (str | None) – The column whose value becomes the document title.
  • json_mode (JsonMode) – The JSON reading mode.
  • csv_mode (CsvMode) – The CSV reading mode.
  • csv_delimiter (str | None) – The column separator. None infers it from the extension.
  • html_selector (str | None) – The CSS selector for the main content of an HTML source.
  • normalization (Normalization) – How to normalize decoded text: byte-order mark, newline form and Unicode form. The default removes the mark, uses LF and applies NFKC. Chunk offsets index the normalized text.

csv_delimiter​

csv_delimiter: str | None = None

csv_mode​

csv_mode: CsvMode = CsvMode.ROWS

encoding​

encoding: str | None = None

html_selector​

html_selector: str | None = None

id_column​

id_column: str | None = None

json_mode​

json_mode: JsonMode = JsonMode.AUTO

max_document_bytes​

max_document_bytes: int = 32 * 1024 * 1024

normalization​

normalization: Normalization = field(default_factory=Normalization)

on_error​

on_error: ErrorPolicy = ErrorPolicy.RAISE

store_raw_record​

store_raw_record: bool = False

store_text​

store_text: bool = True

text_column​

text_column: str | None = None

title_column​

title_column: str | None = None