Skip to main content

agrag.common.data_models.document.Document

Bases: DataPoint

One unit of source text, before chunking.

A prose source, such as a Markdown file, makes one Document. A record source, such as a CSV file, makes one Document per row.

The way the system computes content_hash depends on the loader. A text loader hashes the decoded text. A docling loader hashes the raw source bytes instead of the parsed output, because docling's parsed output can change between docling versions and between runs on different hardware.

The system computes id from content_hash and record_id unless the caller passes id directly. A record-family document without record_id also mixes in record_index plus source_hash, or uri when source_hash is not set. Pass id only when rebuilding a document from stored data.

Attributes:

  • text (str) – The document text. For a docling source, this holds docling Markdown export. The chunker never reads this field for a docling source. See the Chunk model for docling chunk content instead.
  • title (str) – The document title.
  • uri (str) – The location of the source. This value is not part of the document id.
  • source_format (SourceFormat) – The format the loader used to read this document.
  • family (DocumentFamily) – The shape of the source: one document per file, or one document per record.
  • content_hash (str) – The hash that forms the document id.
  • loader_name (str) – The name of the loader that produced this document, for example "text" or "docling".
  • loader_version (str | None) – The version of the loader package. Does not affect the document id.
  • encoding (str | None) – The text encoding. Text loaders set this field. Other loaders leave it empty.
  • source_hash (str | None) – The hash of the whole source file. Record-family documents set this field.
  • char_count (int) – The number of characters in text.
  • line_count (int | None) – The number of lines in text. Some loaders do not set this field.
  • record_index (int | None) – The 0-based row number in the source. Record-family documents set this field.
  • record_id (str | None) – The value from the configured id column. Record-family documents set this field only when the caller configures an id column.
  • raw_record (dict[str, Any] | None) – The original record data. A loader sets this field only when the caller asks for it.
  • heading_outline (list[HeadingRef]) – The headings in the document, with their offsets. A text loader sets this field for a prose document.
  • document_key (str | None) – The stable identifier for this document's persisted graph node. Independent of id, which changes with every content edit. Defaults to uri when not supplied.
  • turns (list[TurnRef]) – The speaker turns of a chat document, in order. A chat loader sets this field. Turn spans index text and do not overlap.
  • normalization (Normalization | None) – How the loader normalized text. None for a document that no text loader made, such as a docling document or one built by hand.

Functions:

  • id_for – Compute the document id.
  • node_id_for – Compute the persisted Document graph node id.
  • to_node_record – Return this document as a GraphStore write record for its graph node.

char_count​

char_count: int

content_hash​

content_hash: str

created_at​

created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))

document_key​

document_key: str | None = None

encoding​

encoding: str | None = None

family​

family: DocumentFamily

heading_outline​

heading_outline: list[HeadingRef] = Field(default_factory=list)

id​

id: UUID | None = None

id_for​

id_for(*, content_hash:str, record_id:str | None = None, record_index:int | None = None, source_hash:str | None = None, uri:str | None = None) -> UUID

Compute the document id.

A record id, when given, wins over the content hash. Without a record id, a record-family document (record_index is not None) mixes in its source hash and row index, so two rows with identical text but no configured id column still get distinct ids. When the source hash is not available, this falls back to uri so that two different sources still do not collide.

Parameters:

  • content_hash (str) – The document's content hash.
  • record_id (str | None) – The value from the configured id column, when the source has one.
  • record_index (int | None) – The 0-based row number, for a record-family document.
  • source_hash (str | None) – The hash of the whole source file, for a record-family document.
  • uri (str | None) – The document's source location, used in place of source_hash when the caller does not supply one.

Returns:

  • UUID – The document id.

line_count​

line_count: int | None = None

loader_name​

loader_name: str

loader_version​

loader_version: str | None = None

metadata​

metadata: dict[str, Any] = Field(default_factory=dict)

node_id_for​

node_id_for(*, document_key:str) -> UUID

Compute the persisted Document graph node id.

Distinct from id_for(). This id is keyed on document_key, not the content hash, so it stays the same across content changes to the same logical document. Conflating the two ids gives every content version of a document its own graph node instead of one node with a changing content hash.

Parameters:

  • document_key (str) – The document's stable key.

Returns:

  • UUID – The Document graph node id.

normalization​

normalization: Normalization | None = None

raw_record​

raw_record: dict[str, Any] | None = None

record_id​

record_id: str | None = None

record_index​

record_index: int | None = None

resolved_document_key​

resolved_document_key: str

Return the document key. It is never None after construction succeeds.

document_key is typed as optional because callers can omit it and let _resolve_document_key default it to uri. Every constructed Document has a non-None document key by the time callers see it. Use this property instead of document_key where a non-optional value is required, such as computing the persisted Document node id.

Raises:

  • RuntimeError – document_key is still None, which means a validator was bypassed, for example via model_construct.

resolved_id​

resolved_id: UUID

Return the document id. It is never None after construction succeeds.

id is typed as optional because callers can omit it and let _resolve_id derive it. Every constructed Document has a non-None id by the time callers see it. Use this property instead of id where a non-optional value is required, such as building a Chunk.

Raises:

  • RuntimeError – id is still None, which means a validator was bypassed, for example via model_construct.

source_format​

source_format: SourceFormat

source_hash​

source_hash: str | None = None

text​

text: str

title​

title: str

to_node_record​

to_node_record() -> NodeRecord

Return this document as a GraphStore write record for its graph node.

The record excludes text: the persisted node exists for traversal and the update no-op check, not to duplicate the document body already held per-chunk.

turns​

turns: list[TurnRef] = Field(default_factory=list)

uri​

uri: str