Skip to main content

agrag.common.data_models.normalization.Normalization

Bases: BaseModel

How a loader turned source bytes into Document.text.

Provenance offsets index the normalized text. A caller who needs offsets into the raw source chooses Normalization(bom="keep", newline="keep", unicode_form="none").

Attributes:

  • bom (Literal['strip', 'keep']) – "strip" removes a leading byte-order mark. "keep" leaves it.
  • newline (Literal['lf', 'keep']) – "lf" turns CRLF and CR into LF. "keep" leaves them.
  • unicode_form (Literal['NFKC', 'NFC', 'NFD', 'NFKD', 'none']) – The Unicode normalization form to apply, or "none".

bom​

bom: Literal['strip', 'keep'] = 'strip'

model_config​

model_config = ConfigDict(frozen=True, extra='forbid')

newline​

newline: Literal['lf', 'keep'] = 'lf'

unicode_form​

unicode_form: Literal['NFKC', 'NFC', 'NFD', 'NFKD', 'none'] = 'NFKC'