A text splitter that splits text based on a token count rather than character count. This is a wrapper around Langchain's TokenTextSplitter.
TokenTextSplitter
Creates an instance of TokenTextSplitter.
Parameters for the text splitter.
The maximum size of each text chunk (in tokens).
The amount of overlap between consecutive text chunks (in tokens).
Splits the given text into chunks based on token count.
The input text to be split.
A promise that resolves to an array of string chunks.
A text splitter that splits text based on a token count rather than character count. This is a wrapper around Langchain's
TokenTextSplitter
.