Skip to main content

agrag.loaders.corpus.decode

The four-step decode pipeline for source bytes.

Every text loader shares this pipeline. It runs, in order: encoding detection via charset-normalizer, byte-order-mark handling, newline normalization, and Unicode normalization. ReadOptions.normalization sets the last three steps. It raises DecodeError on failure rather than silently emitting mojibake.

Functions:

  • decode_text – Decode raw source bytes into normalized text.