不同重叠的切片数量对比
不同重叠的切片数量对比
def chunk(text, size, overlap):
step = size - overlap
return [text[i:i + size] for i in range(0, len(text), step)]
text = open("doc.txt", encoding="utf-8").read()
print("无重叠:", len(chunk(text, 500, 0)))
print("10% 重叠:", len(chunk(text, 500, 50)))
print("20% 重叠:", len(chunk(text, 500, 100)))边界问题更彻底的解法是让切片尽量对齐句子/段落边界，overlap 只作兜底。