Skip to main content

agrag.loaders.corpus.types.LoadStats

LoadStats(documents:int = 0, sources:int = 0, bytes_read:int = 0, skipped:int = 0, quarantined:int = 0, quarantined_items:list[StageFailure] = list()) -> None

Running tally of a corpus walk.

Attributes:

  • documents (int) – The number of documents read so far.
  • sources (int) – The number of sources read so far.
  • bytes_read (int) – The number of bytes read so far.
  • skipped (int) – The number of sources skipped so far.
  • quarantined (int) – The number of sources quarantined so far.
  • quarantined_items (list[StageFailure]) – One StageFailure per quarantined source so far.

bytes_read​

bytes_read: int = 0

documents​

documents: int = 0

quarantined​

quarantined: int = 0

quarantined_items​

quarantined_items: list[StageFailure] = field(default_factory=list)

skipped​

skipped: int = 0

sources​

sources: int = 0