EmbeddingsDivision documentation
Contents:
About
Python script for manipulating the existing tokenizer.
The solution was tested on meta-llama/Llama-3.2-1B.
Installation
pip install embdiv
Requirements
Python 3.9+
cachetools>=5.5.0,
torch>=2.5.1,
transformers>=4.45.2,