Skip to main content

agrag.loaders.corpus.decode.decode_text

decode_text(raw:bytes, opts:ReadOptions) -> DecodedText

Decode raw source bytes into normalized text.

This function detects the encoding, applies opts.normalization and hashes the result. It raises DecodeError instead of returning garbled text.

Parameters:

  • raw (bytes) – The raw source bytes.
  • opts (ReadOptions) – The read options. opts.encoding forces a specific codec when set.

Returns:

  • DecodedText – The decoded text with its encoding and hash.

Raises:

  • DecodeError – The bytes do not decode under the forced encoding, or detection fails and the latin-1 fallback is unavailable.