Machine learning systems change whenever their training data changes. A corrected label, a new source, a filtering rule or an updated feature can alter the model even when the code remains untouched. What is data versioning in machine learning? It is the practice of assigning a stable identity to each dataset state and linking that state to the code, configuration and experiment that used it.
The goal is not to save every file forever. It is to make important training runs explainable, reproducible and reversible. A useful version shows what was present, what changed and whether the underlying data can still be recovered.
That discipline matters once several people prepare data, models are retrained regularly or decisions must be audited. CodeCones helps teams connect data, models and delivery controls within reliable production AI foundations.

What Is Data Versioning in Machine Learning?
Data versioning in machine learning is a controlled way to capture, identify and retrieve the exact state of data used by an ML workflow. Each state receives an immutable hash, snapshot, commit or manifest. It travels with the experiment so others can recover the inputs.
"Data" means more than the final training file. A version may cover source extracts, cleansing output, train and validation splits, annotations, features and exclusions. It should preserve source, transformation and ownership lineage.
A timestamp is weak evidence. Simultaneous files can contain different records, while tomorrow's source query may return changed values. Hashes and snapshots identify what was present. A tag such as `fraud-training-2026-11-approved` can describe purpose while the immutable ID remains authoritative.
The objective is trustworthy reconstruction, not an archive that nobody can understand or restore.
Why Data Versioning Matters for ML Teams
The practical value appears when a result must be explained or repeated.
Reproducibility. A model file does not reveal its training data. Reproduction requires the same snapshot, code revision, parameters, seed and environment. An input described only as "current" or "latest" is not fully recoverable.
Rollback and debugging. After quality drops, teams can compare the new dataset with the last accepted one. A reliable diff exposes missing partitions, schema changes, label corrections or population shifts.
Collaboration. Data scientists can test separate versions without overwriting shared files. Reviewers approve a candidate, and pipelines promote its immutable identifier.
Auditability. Version records connect a prediction system to the inputs and transformations that shaped it. They support investigations and documentation, but versioning alone does not guarantee regulatory compliance. Access, lawful use, retention and deletion still require governance.
For teams moving from notebooks to repeatable pipelines, CodeCones' Data Engineering and MLOps services can help connect dataset identity, pipeline orchestration, model registries, deployment and monitoring into one operating design.
What Should Be Versioned?
Start with the layers that can independently change the result:
- Raw inputs: source files, extracts, sensor batches or immutable source references.
- Processed datasets: cleaned, joined, filtered and normalized training tables.
- Labels and annotations: class labels, bounding boxes, reviewer decisions and correction history.
- Splits: exact membership of training, validation and test sets, including leakage controls.
- Features: feature definitions and materialized values when they cannot be recreated exactly.
- Exclusions: consent withdrawals, quality removals and blocked records.
Data versioning, model versioning and a model registry solve related but different problems.
| Control | What it identifies | Typical question answered |
|---|---|---|
| Data version | Exact inputs, labels, features and splits | What data trained this model? |
| Model version | Serialized model and its training result | Which model artifact produced this output? |
| Model registry | Lifecycle status, approval and deployment alias | Which approved model is in production? |
The three records should reference one another. A model entry should name the immutable dataset version, code commit, configuration and environment. A registry alias such as `champion` may move, but the evidence underneath it should not.
How Data Versioning Works

There is no single architecture for ML data versioning. Teams commonly combine four approaches.
1. Immutable Copies
Each release is stored separately and never edited. This works for small batch datasets, but full copies become expensive and names can drift. Add a manifest with paths, checksums, source date, schema and owner. Without it, a dated folder is only a folder.
2. Content-Addressable Files
DVC calculates hashes, stores content outside Git and commits small pointers beside code. Unchanged files can be reused. Storage still depends on layout: changing one value in a monolithic file may store the whole file again.
3. Object-Store Branches
lakeFS adds commits, branches and merges over object storage. Branch creation is a zero-copy metadata operation, which suits shared data lakes. Changed objects still consume storage, and teams must govern cleanup, permissions and garbage collection. Train from a committed version, not uncommitted branch state; a branch name can move.
4. Table Snapshots and Time Travel
Delta Lake and Apache Iceberg record table snapshots through transaction metadata. Pipelines can query earlier states, but time travel works only while required metadata and data files survive retention. Pin the table version or snapshot ID used by the run rather than a moving table name.
The mechanism matters less than the contract: every training run receives an immutable dataset ID, the referenced data can be retrieved for the required period, and changes between accepted versions can be inspected.
How to Choose a Data Versioning Approach

Use this decision framework instead of starting with a vendor list.
| Situation | Practical starting point | Main trade-off |
|---|---|---|
| Small batch dataset, one team | Immutable dated folders plus a hash manifest | Simple, but manual discipline is required |
| Git-centered ML project | DVC with remote object storage | Familiar workflow, but file layout affects storage |
| Large shared object store | lakeFS branches and commits | Strong isolation, but adds platform operations |
| Lakehouse tables already in use | Delta Lake or Apache Iceberg snapshots | Efficient table history, but retention needs care |
| Large binaries already tied to Git | Git LFS | Useful for files, but limited ML lineage |
| Data recreated deterministically | Version source IDs, code and configuration | Saves storage, but reproducibility must be proven |
Ask four questions. How often does the data change? Is it file-based or table-based? How many teams need isolated work? How long must a training version remain recoverable? A small team should not operate lake infrastructure merely to version one CSV. A platform team should not coordinate a shared petabyte-scale lake through copied folders.
The machine-learning data pipeline readiness checklist provides a useful companion review for source ownership, schema, labels, leakage and operational handoffs before a versioning mechanism is selected.
Data Versioning Tools Compared
The best tool depends on where data already lives.
| Tool | Best fit | Important limitation |
|---|---|---|
| DVC | Git-based projects and file datasets | Large changed files may be stored again |
| lakeFS | Shared data lakes on object storage | Requires service operation and garbage collection |
| Delta Lake | Transactional lakehouse tables | Time travel depends on retention and VACUUM policy |
| Apache Iceberg | Multi-engine analytical tables | Snapshot expiration must be governed explicitly |
| Git LFS | Large files closely coupled to Git | Not a complete ML lineage system |
| MLflow | Recording dataset metadata with experiments | A digest is not a substitute for retained data |
| Pachyderm | Versioned, containerized data pipelines | Adds a broader pipeline platform |
| Hugging Face Datasets | Published or shared dataset revisions | Access and hosting model may not fit private data |
Nessie adds Git-like catalog semantics to lakehouse tables. Dolt brings branching to SQL. dbt snapshots preserve changing rows, and Weights and Biases connects artifacts to experiments. Monte Carlo focuses on observability. Each may complement versioning, but none removes the need for an immutable dataset record.
Git alone rarely suits training data. GitHub blocks ordinary Git files larger than 100 MiB, and changing binaries slow repositories. Git LFS uses separate storage and has plan-dependent limits. Keep code and metadata in Git, and substantial data in suitable storage.
When this choice affects production, CodeCones' AI Product Development services can align data versions, evaluation evidence, model controls and release workflows with real operating risks.
A Practical DVC Example
The following workflow shows how to version data in machine learning with DVC. Assume Git is initialized, `data/training.parquet` is prepared, and DVC with S3 support is installed. The example S3 bucket must exist and be writable using your approved credentials. Replace its placeholder URI with your own; never commit credentials.
dvc init
dvc remote add -d storage s3://example-ml-data/dvc
dvc add data/training.parquet
git add .dvc .dvcignore data/.gitignore data/training.parquet.dvc
git commit -m "Version approved training dataset"
dvc push
`dvc add` creates or updates the lightweight pointer, updates `data/.gitignore`, and places content in the DVC cache. The Git commit records the pointer next to the code. `dvc push` uploads data to the configured remote, not your Git repository. Push the Git commit through your normal review process too. Record its immutable commit ID in the experiment tracker and model entry.
After the dataset changes, run `dvc add` again, review the difference at the source or manifest level, commit the updated pointer and push the new content. To recover an accepted version, clone the Git repository into a clean environment, install the same DVC and storage dependencies, and authorize read access to the remote. Replace both quoted placeholders below with the repository URL and approved commit ID before running:
git clone "YOUR_REPOSITORY_URL" ml-restore
cd ml-restore
git checkout "APPROVED_COMMIT_ID"
dvc fetch data/training.parquet.dvc
dvc checkout data/training.parquet.dvc
`dvc fetch` downloads the referenced objects into the cache. `dvc checkout` materializes them in the workspace; it does not download missing remote objects. `dvc pull` combines fetch and checkout and can replace those two DVC commands. A pointer alone cannot recover data removed from both cache and remote.
Production automation should also verify that the remote object exists, access is authorized, the schema matches expectations and the version is approved. Compare checksums and validation results with the approved manifest after restoration. Commands create history; controls make that history trustworthy.
Storage Cost, Retention and Erasure

Storage is where versioning meets operational reality. Version count alone does not determine cost. File size, partitioning and change pattern do.
Consider this hypothetical storage comparison, not a benchmark or a CodeCones client result: a 12 GB table is stored as one file. Correcting one record may create another 12 GB object. If stable partitions are used and only a 300 MB partition changes, the new version may add roughly that partition plus metadata. Using decimal units, 12 GB is 12,000 MB, so 300 MB is 2.5% of the original size and the additional payload is about 40 times smaller than a full copy. This is not a storage bill or a guaranteed saving: compression, metadata, deduplication, rewritten partitions, requests and transfer charges can change the result. Version boundaries should follow change boundaries.
Retention must follow evidence needs. Delta Lake defaults `delta.logRetentionDuration` to 30 days and `delta.deletedFileRetentionDuration` to seven days. The latter is the minimum age of logically removed files before they become candidates for `VACUUM`, not automatic data deletion. Data files are removed when `VACUUM` runs; transaction logs are cleaned separately after checkpoints. Time travel requires both usable log history and its data files, so retaining a log record does not guarantee restoration. Set both retention policies for the required evidence window before cleanup.
Apache Iceberg snapshot expiration removes snapshots from table metadata and can delete files no longer needed by retained snapshots. Expired snapshots cannot be used for time travel. Set snapshot age, minimum snapshot count and branch/tag retention deliberately. Coordinate expiration and orphan-file cleanup with long-running jobs and required model evidence; metadata-only history is not recoverable data.
Deletion requires equal care. GDPR Article 17 provides a right to erasure in defined circumstances, with grounds and exceptions described in the official regulation. Teams need a process to locate a subject across retained versions, assess valid requests and applicable duties, preserve only required audit evidence, and handle downstream copies, caches, replicas and backups. "Immutable" cannot mean "undeletable regardless of obligation." Versioning alone does not establish legal compliance; retention, legal holds, lawful use and erasure need appropriate governance and legal review.
Common Data Versioning Mistakes
Avoid these common failure modes:
- Training from `latest`. The alias may move before the run is investigated.
- Using dates as identity. "Rows before Friday" can change after corrections or late arrivals.
- Versioning only the final table. Sources, labels or split membership may change invisibly.
- Keeping pointers after deleting content. Metadata is useless when referenced files are gone.
- Ignoring quality evidence. Recovering corrupt data reproduces the problem.
- Treating experiment tracking as storage. MLflow dataset tracking logs source, schema and digest metadata, but a digest is not a durable snapshot. Digest computation varies by dataset type; metadata-only datasets do not hash the underlying data. The snapshot still needs a durable home and a tested retrieval path.
- Never testing restoration. Another environment must retrieve and validate the version.
- Retaining everything forever. Unbounded copies increase cost and erasure complexity.
Versioning also supports, but does not replace, production ML monitoring. Monitoring identifies a regression or drift event; version records help explain which data change may have contributed.
Data Versioning Best-Practices Checklist
Use this ten-point checklist to operationalize data versioning:
- Define which raw, processed, label, split and feature layers require versions.
- Give every training run an immutable dataset identifier.
- Link data, code, configuration, environment and model records.
- Capture lineage, schema, quality results, owner and approval status.
- Separate immutable identifiers from movable aliases such as `latest`.
- Choose file or table boundaries that match common changes.
- Test retrieval and rollback from a clean environment.
- Set retention, garbage collection and legal-hold rules explicitly.
- Design erasure and access revocation across historical copies.
- Monitor storage growth and remove unneeded versions through an approved process.
Conclusion
The useful answer to what is data versioning in machine learning is not a tool name. It is an evidence contract: each important model can be connected to the exact data, code, configuration and environment that created it, and that evidence remains retrievable for the period the organization actually needs.
Start with the smallest approach that can honor that contract. Use manifests for simple datasets, DVC for Git-centered file workflows, lakeFS for shared object stores, or table snapshots where a lakehouse already exists. Then prove restoration in practice, define retention and prepare for deletion before scaling the system.
If your team needs a practical design for dataset lineage, reproducible training and production controls, contact CodeCones to review the current data platform and identify the smallest reliable next step.
Frequently Asked Questions
What is data versioning in machine learning?
It assigns an immutable identity to a dataset state and links it to the ML work that used it. The record connects the snapshot, lineage and schema with code, configuration, environment and model output.
Is data versioning part of MLOps?
Yes. It supports reproducible training, controlled promotion, rollback and auditability. MLOps also covers pipelines, testing, registries, deployment and monitoring, so versioning is one control within a larger system.
What is the difference between data versioning and model versioning?
Data versioning identifies training and evaluation inputs. Model versioning identifies the resulting model artifact. A reliable model record links both with the code, parameters and environment.
What is a dataset snapshot?
A snapshot is a stable dataset view at a logical point. Copied files, a content hash, table version or commit may represent it, provided the referenced content remains retrievable.
Which data versioning tool is best for machine learning?
There is no universal winner. DVC fits Git-centered files, lakeFS fits shared object stores, and Delta Lake or Iceberg fit analytical tables. Infrastructure, scale, change pattern and retention should drive the choice.
Can Git version a machine-learning dataset?
Git can version small text datasets, manifests and pointers. It performs poorly with changing binaries, and GitHub blocks files above 100 MiB. DVC, Git LFS or object storage better suits substantial data.
How does DVC version data?
DVC tracks files through metadata and content hashes. Teams commit the metadata to Git while storing content in a DVC cache and remote. Check out the approved Git revision, then use `dvc pull`, or `dvc fetch` followed by `dvc checkout`, to restore data in a clean environment. `dvc checkout` alone needs content already in the cache.
Does MLflow replace a data-versioning tool?
Not usually. MLflow records dataset metadata and connects inputs to experiment runs. Teams still need an immutable source or snapshot in a system designed to retain the underlying data.
How long should dataset versions be retained?
Retain versions long enough to reproduce accepted models, investigate incidents and meet contractual or legal duties. Define rules by version class, and test that cleanup does not break required models.
Do small ML teams need data versioning?
They need reproducibility, not necessarily a large platform. Immutable folders, a checksummed manifest and experiment record may be enough until collaboration, scale or audit needs justify a dedicated tool.
About CodeCones Editorial Team
The CodeCones Editorial Team develops practical engineering guidance for enterprise product and AI teams.
Key Takeaways
- A data version is an immutable reference to the exact dataset state used for a purpose.
- Version raw inputs, processed data, labels and reusable features when each layer can change independently.
- Link the dataset version to the code commit, configuration, environment and resulting model.
- Choose an approach from data shape, change frequency, team size and existing infrastructure.
- Plan retention, deletion and access controls before historical copies accumulate.
- Treat `latest` as a convenience for exploration, never as the permanent identity of a training run.
References
- [1]DVC: Get Started
- [2]DVC: User Guide
- [3]lakeFS: Version Data
- [4]Delta Lake: Time Travel and Retention
- [5]Apache Iceberg: Specification
- [6]MLflow: Dataset Tracking
- [7]GitHub: Large Files
- [8]GDPR Article 17
- [9]DVC: fetch
- [10]DVC: checkout
- [11]DVC: pull
- [12]Apache Iceberg: Maintenance
- [13]European Union: Regulation (EU) 2016/679, Article 17












































