tiktoken 计数切分
tiktoken 计数切分
import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter
enc = tiktoken.get_encoding("cl100k_base")
print("token 数:", len(enc.encode("示例文本…")))
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base", chunk_size=512, chunk_overlap=64)
chunks = splitter.split_text(text)cl100k_base 对应 OpenAI 系模型；换用其它模型或本地 embedding 时，按实际分词器重新校准切片长度。