SemanticChunker
SemanticChunker
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
splitter = SemanticChunker(
OpenAIEmbeddings(model="text-embedding-3-small"),
breakpoint_threshold_type="percentile",
)
docs = splitter.create_documents([text])
print(len(docs), docs[0].page_content[:80])语义切分要对句子批量向量化，索引成本上升；先用小样本与递归切分做 A/B 再决定是否全面采用。