/var/log

Using tokenizers for LLM requests

There are different tokenizers for different LLM models. For example OpenAI uses tiktoken, while Mistral uses sentencepiece (or you can use AutoTokenizer from the transformers package).

When use tokenizers:

  • When chunkings large documents, make sure each chunk fits in the context window.
  • When sending prompts (optionally augmented with chunks), make sure it fits in the context window.
  • Compute the costs

Tokenizer for Mistral and OpenAI

See the difference between them.

pip install transformers
from transformers import AutoTokenizer
import tiktoken

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
tokenizerOpenAi = tiktoken.get_encoding("o200k_base")

# Dit geeft beiden 7!
tokens = tokenizer.tokenize("Hello my name is Michiel!")
tokensOpenAI = tokenizerOpenAi.encode("Hello my name is Michiel!")

print(len(tokens))
print(len(tokensOpenAI))

Use it with Docling when chunking

tokenizer: BaseTokenizer = HuggingFaceTokenizer(
    tokenizer=AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3"),
    max_tokens=max_tokens
)

chunker = HybridChunker(
    tokenizer=tokenizer,
    max_tokens=max_tokens
)
Tag: | Category: