滑窗切分
滑窗切分
def fixed_chunks(text, size=500, overlap=50):
step = size - overlap
return [text[i:i+size] for i in range(0, len(text), step)]
chunks = fixed_chunks(open("doc.txt", encoding="utf-8").read())
print(len(chunks), chunks[0][:60])overlap 必须小于 size；做对照实验时，与其它策略保持同一 embedding 模型与评测集。