Three batch processing patterns that keep n8n workflows from breaking at scale
A workflow that handles 200 records without complaint can quietly fall apart at 20,000. Memory limits, API quotas, and the question of what actually finished all become real problems once volume climbs. This guide from n8n breaks down three patterns that keep large batch jobs from derailing.
Batch vs. stream: pick the right model first
Batch processing collects data and processes it in groups. Stream processing handles data continuously as it arrives. Use batch when results don’t need to be immediate, like scheduled data syncs, bulk updates, or reports. Use stream when timing is essential and you can absorb the always-on infrastructure that comes with it.
The three patterns
Chunking
Chunking breaks a large dataset into smaller groups that move through the workflow one at a time. Instead of pushing thousands of records through a single operation, you send 100 or 500 at a time. Smaller batches use less memory and make failures easier to isolate. Larger batches reduce per-group overhead. The goal is finding a size that keeps the workflow moving without overwhelming the next step.
Rate limiting
Smaller batches can still cause trouble if you send them too quickly. Most APIs cap requests within a given period, so a fast workflow can trigger rate-limit errors even when each individual request is valid. The fix is adding pauses between requests or batches and configuring retries when an API asks you to slow down. In n8n, the Loop Over Items node and the Wait node control the pace. The HTTP Request node also has a built-in Batching option that handles this without extra nodes.
Per-item error handling and retry
One bad record shouldn’t stall a healthy batch. Per-item error handling separates failed items from successful ones, then retries or routes those failures without repeating work that’s already complete. Deduplication adds another safeguard when a rerun might include records you’ve already processed. This is the practical application of idempotency: designing the workflow so retrying the same work doesn’t create duplicate results.
n8n nodes that handle this natively
- Split Out node: Turns a list inside a single item into individual items for separate processing
- Aggregate node: Combines separate items back into a single list when the next step needs them together
- Loop Over Items (Split in Batches) node: Sends a set number of items through at a time, giving direct control over batch size
- Wait node: Pauses execution between batches to stay within API rate limits
- Remove Duplicates node: Filters repeated items before they reach the next step
- HTTP Request node: Sends batched data to APIs with built-in batching and request control
n8n has been tested at up to 220 workflow executions per second on a single c5a.large instance running a two-node workflow. For larger self-hosted workloads, queue mode distributes executions across workers without changing the batching logic inside the workflow. Sub-workflows can also manage memory limits by returning a restricted amount of data back to the parent workflow.
The practical starting point: set a batch size your systems can handle comfortably, watch behavior under real load, and tune from there.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
