Lesson overview | Lesson overview | Next part

Full Dataset Assembly: Part 1: Intuition to 3. Corpus Manifests

1. Intuition

Intuition gives the conceptual and mathematical layer for full dataset assembly. The local variables in this section should be read as pipeline objects: documents, records, tokens, filters, weights, shards, and manifests.

1.1 Assembly as turning accepted records into a trainable corpus

Assembly as turning accepted records into a trainable corpus is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For source set, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

1.2 Source manifests

Source manifests is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For mixture weight, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

1.3 Token accounting

Token accounting is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For token budget, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

1.4 Data order as curriculum

Data order as curriculum is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For manifest, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

1.5 Reproducibility at trillion-token scale

Reproducibility at trillion-token scale is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For shard, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

2. Formal Definitions

Formal Definitions gives the conceptual and mathematical layer for full dataset assembly. The local variables in this section should be read as pipeline objects: documents, records, tokens, filters, weights, shards, and manifests.

2.1 Source set $\mathcal{D}_k$

Source set $\mathcal{D}_k$ is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For source set, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

2.2 Mixture weights $\alpha_k$

Mixture weights $\alpha_k$ is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For mixture weight, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

2.3 Token budget $T$

Token budget $T$ is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For token budget, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

2.4 Shard manifest

Shard manifest is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For manifest, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

2.5 Split assignment

Split assignment is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For shard, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

3. Corpus Manifests

Corpus Manifests gives the conceptual and mathematical layer for full dataset assembly. The local variables in this section should be read as pipeline objects: documents, records, tokens, filters, weights, shards, and manifests.

3.1 Source inventory

Source inventory is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For source set, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

3.2 Version pins

Version pins is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For mixture weight, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

3.3 Hashes/checksums

Hashes/checksums is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For token budget, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

3.4 License fields

License fields is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For manifest, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.

3.5 Build recipe

Build recipe is part of the canonical scope of full dataset assembly. We model the relevant object as a finite collection $\mathcal{D} = \{r_i\}_{i=1}^n$ with record-level metadata $m_i$ and text or token content $x_i$. The practical question is whether the transformation preserves the intended empirical distribution.

A useful local invariant is:

$$\text{valid}(r_i, \mathcal{S}) = 1 \quad \Longrightarrow \quad r_i \text{ can be consumed by the next pipeline stage.}$$

For shard, the invariant should be explicit enough that a checker can fail fast. If the invariant is only written in a notebook comment or an engineer's memory, it will not protect a long-running data build.

Examples:

• A small local experiment can store this object in memory; a frontier-scale run must store it as sharded, versioned, validated records.
• The mathematical object is simple, but the operational contract must survive restarts, parallel workers, schema changes, and audits.
• The notebook for this section uses synthetic data so the same ideas can be executed without external files.

Non-examples:

• A path on disk without a manifest is not a reproducible dataset.
• A metric dashboard without record-level lineage is not a provenance system.
• A filter threshold without an audit sample is not evidence of quality.

Implementation consequence: every transformation should report both a count and a rate. If $n_{\mathrm{in}}$ records enter the stage and $n_{\mathrm{out}}$ records leave, the acceptance rate is

$$a = \frac{n_{\mathrm{out}}}{n_{\mathrm{in}}}.$$

A sudden change in $a$ is a data-drift signal even when the code still runs. This is why pipeline math is inseparable from logging, manifests, and audit slices.

For LLM work, the token-weighted view is often more important than the document-weighted view. A filter that removes 5 percent of documents may remove 30 percent of tokens if it targets long documents. The corresponding token acceptance rate is

$$a_{\mathrm{tok}} = \frac{\sum_i f(r_i)\,T_i}{\sum_i T_i},$$

where $T_i$ is the token count or a deterministic token-count estimate. The distinction matters for compute budgets, mixture proportions, and scaling-law interpretation.