n8n Token Splitter nodeConfigure Token Splitter in n8n.
The n8n Token Splitter node cuts text into chunks counted in tokens, not characters. It turns the raw string into BPE tokens, splits those tokens into chunks, then converts each chunk back into text. Two parameters drive it, and it runs as a sub-node wired to a root node.
Verified Trustpilot reviews · AI, automation & growth agency
What does the n8n Token Splitter node do?
The Token Splitter takes a raw text string, converts it into BPE tokens (the units a language model actually reads, where a word can count as one token or several), splits those tokens into chunks, and converts the tokens of each chunk back into text. It is a sub-node: it has no main input, it never runs on its own, and it only works once wired to a root node that asks for a text splitter.
Say you index a folder of product manuals for a support assistant. Each PDF arrives as one long string, and a model only accepts a limited number of tokens at a time. Set Chunk Size to 2000 tokens and the splitter hands the loader pieces that stay inside that budget, whatever the language of the document. Counting in tokens instead of characters matters here, because the same paragraph in Spanish or in Japanese does not produce the same token count as in English.
Second scenario, a knowledge base that feeds an AI Agent through a vector store. Chunks that cut a sentence in half give the retriever half an idea, so the answer comes back vague. Chunk Overlap repeats a slice of tokens at the start of the next chunk, which keeps the end of one passage attached to the beginning of the next. The model behind the agent, whether it comes from OpenAI, Anthropic or Google Gemini, bills its own API calls under its own terms, and shorter chunks mean more of them.
Third case, a summarization flow over a long transcript. The Summarization Chain reads the text in pieces, so the size of the piece decides how much context each pass gets. A bigger Chunk Size means fewer passes and a coarser summary, a smaller one means more passes and more detail.
When to reach for something else: this node counts tokens, so it does the extra work of encoding and decoding. If the input is plain prose and an approximate cut is fine, a splitter that counts characters is lighter and easier to reason about. If the input has a structure worth respecting, paragraphs then lines then words, a recursive splitter keeps that structure where the token splitter only sees a flat stream of tokens. Pick the Token Splitter when the limit you are fighting is a model limit expressed in tokens.
Known limits: the node exposes two parameters and nothing else. No separator to choose, no token encoder to pick, no per-document override. It also produces no output of its own in the workflow view, since a sub-node returns its result to the root node that called it. If splitting is the core of your pipeline and you want to learn the surrounding nodes properly, the n8n training covers them, and the n8n review looks at the platform as a whole.
Where do you plug the Token Splitter ?
An n8n AI flow has a root node that receives the workflow items, and sub-nodes clipped onto its ports, one type of sub-node per port. The Token Splitter has no input port of its own: it offers a single output that a root node picks up.
Output (what it plugs into)
- Text Splitter
ai_textSplitter
01Text Splitter
This is the port the node connects through. Drag the connector to a root node that exposes a text splitter slot, and it will call the splitter whenever a long string needs cutting.
Key parameters
- Optional: a root node accepts a text splitter on this port, and the splitter only comes into play when the text you feed it is long enough to need cutting.
- Default Data Loader: the loader that prepares documents before they are embedded, which is where the splitter usually sits in a retrieval flow.
- Summarization Chain: the chain that summarizes a long text in several passes, one chunk at a time.
Which parameters does the Token Splitter expose?
The Token Splitter node has 2 parameters. For each one: the node as you configure it in n8n, what the parameter changes, and our field notes.
Chunk Size
chunkSizeWhat you see in n8n
Notes & use cases
Sets the maximum number of tokens per chunk. The node encodes the text as BPE tokens, counts them, and closes a chunk once the ceiling is reached, so the value is a hard budget rather than an average.
Key parameters
- Chunk Size: a number of tokens, typed in the field or read from the incoming item with an expression such as
{{ $json.chunkSize }}when the size depends on the document.
Chunk Overlap
chunkOverlapWhat you see in n8n
Notes & use cases
Sets how many tokens two consecutive chunks share, so the context at the edge of a chunk is not lost. The tail of one chunk reappears at the head of the next.
Key parameters
- Chunk Overlap: a number of tokens shared between consecutive chunks, kept well under the chunk ceiling since every shared token is stored twice.
0 when the source is already a list of short, independent records.Need help automating Token Splitter with n8n?
A person reads every message.
Token Splitter questions, answered
01Is the Token Splitter included in n8n at no extra cost?
02What do you need to set up before the node works?
03What are the limits of this node?
04When should you pick the n8n Token Splitter node over another splitter?
05n8n or Make for a chunking flow like this one?
Get our weekly integration tips.
No spam. Unsubscribe anytime.
