Docs
README
18.5 Async Iterators & Streams
Overview
Async iterators and streams allow processing of data that arrives over time, perfect for handling large datasets, real-time data, or paginated APIs.
Async Iteration
┌─────────────────────────────────────────────────────────────────────────┐
│ Sync vs Async Iteration │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ SYNC ITERATION: │
│ Data is immediately available │
│ │
│ for (const item of array) { │
│ console.log(item); │
│ } │
│ │
│ ASYNC ITERATION: │
│ Data arrives over time │
│ │
│ for await (const item of asyncSource) { │
│ console.log(item); │
│ } │
│ │
│ Timeline: │
│ ───┬──────┬──────┬──────┬──────┬───▶ time │
│ │ │ │ │ │ │
│ item 1 item 2 item 3 item 4 done │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Async Iterator Protocol
| Method | Returns | Description |
|---|---|---|
[Symbol.asyncIterator]() | Async Iterator | Returns the async iterator |
next() | Promise<{value, done}> | Returns next value |
return() (optional) | Promise<{done: true}> | Cleanup on early exit |
throw() (optional) | Promise | Handle thrown errors |
Creating Async Iterables
1. Async Generator Function
async function* fetchPages(url) {
let page = 1;
while (true) {
const data = await fetch(`${url}?page=${page}`);
const items = await data.json();
if (items.length === 0) break;
yield* items;
page++;
}
}
2. Custom Async Iterable
const asyncIterable = {
async *[Symbol.asyncIterator]() {
yield await fetchItem(1);
yield await fetchItem(2);
yield await fetchItem(3);
},
};
Stream Concepts
┌─────────────────────────────────────────────────────────────────────────┐
│ Data Stream Flow │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Source ──▶ Transform ──▶ Transform ──▶ Sink │
│ │ │ │ │ │
│ produce process process consume │
│ │
│ Examples: │
│ ┌──────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐ │
│ │ API Data │──▶│ Parse │──▶│ Filter │──▶│ Display│ │
│ └──────────┘ └─────────┘ └─────────┘ └────────┘ │
│ │
│ ┌──────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐ │
│ │ File │──▶│Decompress──▶│Transform│──▶│ Save │ │
│ └──────────┘ └─────────┘ └─────────┘ └────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Web Streams API
| Stream Type | Purpose | Method |
|---|---|---|
| ReadableStream | Source of data | getReader() |
| WritableStream | Destination | getWriter() |
| TransformStream | Process data | readable, writable |
Using for await...of
// With async generator
for await (const item of asyncGenerator()) {
console.log(item);
}
// With readable stream
const reader = stream.getReader();
for await (const chunk of reader) {
processChunk(chunk);
}
Backpressure
┌─────────────────────────────────────────────────────────────────────────┐
│ Backpressure Control │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ WITHOUT BACKPRESSURE: │
│ Producer: ████████████████████████▶ Fast │
│ Consumer: ▓▓▓▓▓▓▓▓▓▶ Slow │
│ Buffer: █████████████████████ ← OVERFLOW! │
│ │
│ WITH BACKPRESSURE: │
│ Producer: ▓▓▓▓▓▓▓▓▓▓▓▓▶ Slowed down │
│ Consumer: ▓▓▓▓▓▓▓▓▓▶ At its pace │
│ Buffer: ███████ ← Controlled │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Summary
- •Use async iterators for data over time
- •
for await...ofsimplifies consumption - •Web Streams for large data processing
- •Handle backpressure to prevent memory issues
- •Combine with generators for powerful patterns