Skip to main content

agrag.chunking.extras.byte_spans_to_char_spans

byte_spans_to_char_spans(text:str, spans:list[tuple[int, int]]) -> list[tuple[int, int]]

Convert UTF-8 byte spans of text to character spans.

Parameters:

  • text (str) – The text the byte offsets index once encoded as UTF-8.
  • spans (list[tuple[int, int]]) – Half-open byte spans.

Returns:

  • list[tuple[int, int]] – The same spans as character offsets. ASCII text returns the spans as given.