预算换算示例
预算换算示例
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
budget = 8192 - len(tok("系统提示…").input_ids) - 1024 # 预留输出
used = 0
selected = []
for c in ranked:
n = len(tok(c.text).input_ids)
if used + n > budget:
break
used += n
selected.append(c)中文按 tokenizer 实测而非按字数折算；不同模型的 token/字比例差异明显。