Immutable dataset layers branch into linked experiment and model records on a dark CodeCones background
    CodeCones Insight

    What Is Data Versioning in Machine Learning? A Practical Guide

    Understand ML data versioning, compare DVC, lakeFS and table snapshots, restore an approved dataset, and plan retention and erasure.

    By CodeCones Editorial Team· November 20, 2026· 16 min read
    undefined viewsundefined likes

    Machine learning systems change whenever their training data changes. A corrected label, a new source, a filtering rule or an updated feature can alter the model even when the code remains untouched. What is data versioning in machine learning? It is the practice of assigning a stable identity to each dataset state and linking that state to the code, configuration and experiment that used it.

    The goal is not to save every file forever. It is to make important training runs explainable, reproducible and reversible. A useful version shows what was present, what changed and whether the underlying data can still be recovered.

    That discipline matters once several people prepare data, models are retrained regularly or decisions must be audited. CodeCones helps teams connect data, models and delivery controls within reliable production AI foundations.

    A reproducible ML version record links the data snapshot, code and configuration, environment, and model artifact to one immutable run record. All references must remain linked, retrievable and approved.
    Figure 1: An immutable run record connects data, code, environment and model evidence. Scroll horizontally to read the full diagram.

    What Is Data Versioning in Machine Learning?

    Data versioning in machine learning is a controlled way to capture, identify and retrieve the exact state of data used by an ML workflow. Each state receives an immutable hash, snapshot, commit or manifest. It travels with the experiment so others can recover the inputs.

    "Data" means more than the final training file. A version may cover source extracts, cleansing output, train and validation splits, annotations, features and exclusions. It should preserve source, transformation and ownership lineage.

    A timestamp is weak evidence. Simultaneous files can contain different records, while tomorrow's source query may return changed values. Hashes and snapshots identify what was present. A tag such as `fraud-training-2026-11-approved` can describe purpose while the immutable ID remains authoritative.

    The objective is trustworthy reconstruction, not an archive that nobody can understand or restore.

    Why Data Versioning Matters for ML Teams

    The practical value appears when a result must be explained or repeated.

    Reproducibility. A model file does not reveal its training data. Reproduction requires the same snapshot, code revision, parameters, seed and environment. An input described only as "current" or "latest" is not fully recoverable.

    Rollback and debugging. After quality drops, teams can compare the new dataset with the last accepted one. A reliable diff exposes missing partitions, schema changes, label corrections or population shifts.

    Collaboration. Data scientists can test separate versions without overwriting shared files. Reviewers approve a candidate, and pipelines promote its immutable identifier.

    Auditability. Version records connect a prediction system to the inputs and transformations that shaped it. They support investigations and documentation, but versioning alone does not guarantee regulatory compliance. Access, lawful use, retention and deletion still require governance.

    For teams moving from notebooks to repeatable pipelines, CodeCones' Data Engineering and MLOps services can help connect dataset identity, pipeline orchestration, model registries, deployment and monitoring into one operating design.

    What Should Be Versioned?

    Start with the layers that can independently change the result:

    • Raw inputs: source files, extracts, sensor batches or immutable source references.
    • Processed datasets: cleaned, joined, filtered and normalized training tables.
    • Labels and annotations: class labels, bounding boxes, reviewer decisions and correction history.
    • Splits: exact membership of training, validation and test sets, including leakage controls.
    • Features: feature definitions and materialized values when they cannot be recreated exactly.
    • Exclusions: consent withdrawals, quality removals and blocked records.

    Data versioning, model versioning and a model registry solve related but different problems.

    ControlWhat it identifiesTypical question answered
    Data versionExact inputs, labels, features and splitsWhat data trained this model?
    Model versionSerialized model and its training resultWhich model artifact produced this output?
    Model registryLifecycle status, approval and deployment aliasWhich approved model is in production?

    The three records should reference one another. A model entry should name the immutable dataset version, code commit, configuration and environment. A registry alias such as `champion` may move, but the evidence underneath it should not.

    How Data Versioning Works

    Four versioning patterns: immutable copies with checksummed manifests; content-addressable files with lightweight Git pointers; zero-copy object-store branches; and table snapshots whose history depends on retained metadata and files.
    Figure 2: Four patterns provide different storage and operating trade-offs. Scroll horizontally to read the full diagram.

    There is no single architecture for ML data versioning. Teams commonly combine four approaches.

    1. Immutable Copies

    Each release is stored separately and never edited. This works for small batch datasets, but full copies become expensive and names can drift. Add a manifest with paths, checksums, source date, schema and owner. Without it, a dated folder is only a folder.

    2. Content-Addressable Files

    DVC calculates hashes, stores content outside Git and commits small pointers beside code. Unchanged files can be reused. Storage still depends on layout: changing one value in a monolithic file may store the whole file again.

    3. Object-Store Branches

    lakeFS adds commits, branches and merges over object storage. Branch creation is a zero-copy metadata operation, which suits shared data lakes. Changed objects still consume storage, and teams must govern cleanup, permissions and garbage collection. Train from a committed version, not uncommitted branch state; a branch name can move.

    4. Table Snapshots and Time Travel

    Delta Lake and Apache Iceberg record table snapshots through transaction metadata. Pipelines can query earlier states, but time travel works only while required metadata and data files survive retention. Pin the table version or snapshot ID used by the run rather than a moving table name.

    The mechanism matters less than the contract: every training run receives an immutable dataset ID, the referenced data can be retrieved for the required period, and changes between accepted versions can be inspected.

    How to Choose a Data Versioning Approach

    Choose folders and a manifest for small batch files, DVC and remote storage for Git-centered files, lakeFS branches for shared object stores, or Delta Lake or Iceberg for lakehouse tables. Validate retrieval, retention, access and rollback before scaling.
    Figure 3: Choose from data location and collaboration needs, not tool popularity. Scroll horizontally to read the full diagram.

    Use this decision framework instead of starting with a vendor list.

    SituationPractical starting pointMain trade-off
    Small batch dataset, one teamImmutable dated folders plus a hash manifestSimple, but manual discipline is required
    Git-centered ML projectDVC with remote object storageFamiliar workflow, but file layout affects storage
    Large shared object storelakeFS branches and commitsStrong isolation, but adds platform operations
    Lakehouse tables already in useDelta Lake or Apache Iceberg snapshotsEfficient table history, but retention needs care
    Large binaries already tied to GitGit LFSUseful for files, but limited ML lineage
    Data recreated deterministicallyVersion source IDs, code and configurationSaves storage, but reproducibility must be proven

    Ask four questions. How often does the data change? Is it file-based or table-based? How many teams need isolated work? How long must a training version remain recoverable? A small team should not operate lake infrastructure merely to version one CSV. A platform team should not coordinate a shared petabyte-scale lake through copied folders.

    The machine-learning data pipeline readiness checklist provides a useful companion review for source ownership, schema, labels, leakage and operational handoffs before a versioning mechanism is selected.

    Data Versioning Tools Compared

    The best tool depends on where data already lives.

    ToolBest fitImportant limitation
    DVCGit-based projects and file datasetsLarge changed files may be stored again
    lakeFSShared data lakes on object storageRequires service operation and garbage collection
    Delta LakeTransactional lakehouse tablesTime travel depends on retention and VACUUM policy
    Apache IcebergMulti-engine analytical tablesSnapshot expiration must be governed explicitly
    Git LFSLarge files closely coupled to GitNot a complete ML lineage system
    MLflowRecording dataset metadata with experimentsA digest is not a substitute for retained data
    PachydermVersioned, containerized data pipelinesAdds a broader pipeline platform
    Hugging Face DatasetsPublished or shared dataset revisionsAccess and hosting model may not fit private data

    Nessie adds Git-like catalog semantics to lakehouse tables. Dolt brings branching to SQL. dbt snapshots preserve changing rows, and Weights and Biases connects artifacts to experiments. Monte Carlo focuses on observability. Each may complement versioning, but none removes the need for an immutable dataset record.

    Git alone rarely suits training data. GitHub blocks ordinary Git files larger than 100 MiB, and changing binaries slow repositories. Git LFS uses separate storage and has plan-dependent limits. Keep code and metadata in Git, and substantial data in suitable storage.

    When this choice affects production, CodeCones' AI Product Development services can align data versions, evaluation evidence, model controls and release workflows with real operating risks.

    A Practical DVC Example

    The following workflow shows how to version data in machine learning with DVC. Assume Git is initialized, `data/training.parquet` is prepared, and DVC with S3 support is installed. The example S3 bucket must exist and be writable using your approved credentials. Replace its placeholder URI with your own; never commit credentials.

    
    dvc init
    dvc remote add -d storage s3://example-ml-data/dvc
    dvc add data/training.parquet
    git add .dvc .dvcignore data/.gitignore data/training.parquet.dvc
    git commit -m "Version approved training dataset"
    dvc push
    

    `dvc add` creates or updates the lightweight pointer, updates `data/.gitignore`, and places content in the DVC cache. The Git commit records the pointer next to the code. `dvc push` uploads data to the configured remote, not your Git repository. Push the Git commit through your normal review process too. Record its immutable commit ID in the experiment tracker and model entry.

    After the dataset changes, run `dvc add` again, review the difference at the source or manifest level, commit the updated pointer and push the new content. To recover an accepted version, clone the Git repository into a clean environment, install the same DVC and storage dependencies, and authorize read access to the remote. Replace both quoted placeholders below with the repository URL and approved commit ID before running:

    
    git clone "YOUR_REPOSITORY_URL" ml-restore
    cd ml-restore
    git checkout "APPROVED_COMMIT_ID"
    dvc fetch data/training.parquet.dvc
    dvc checkout data/training.parquet.dvc
    

    `dvc fetch` downloads the referenced objects into the cache. `dvc checkout` materializes them in the workspace; it does not download missing remote objects. `dvc pull` combines fetch and checkout and can replace those two DVC commands. A pointer alone cannot recover data removed from both cache and remote.

    Production automation should also verify that the remote object exists, access is authorized, the schema matches expectations and the version is approved. Compare checksums and validation results with the approved manifest after restoration. Commands create history; controls make that history trustworthy.

    Storage Cost, Retention and Erasure

    Create a version with snapshot identity, lineage, schema and quality; retain it for evidence and legal holds; restore it in a clean environment and verify integrity and approval; then expire or erase through policy review, subject deletion and copy cleanup.
    Figure 4: Retention and erasure require lifecycle controls, not permanent accumulation. Scroll horizontally to read the full diagram.

    Storage is where versioning meets operational reality. Version count alone does not determine cost. File size, partitioning and change pattern do.

    Consider this hypothetical storage comparison, not a benchmark or a CodeCones client result: a 12 GB table is stored as one file. Correcting one record may create another 12 GB object. If stable partitions are used and only a 300 MB partition changes, the new version may add roughly that partition plus metadata. Using decimal units, 12 GB is 12,000 MB, so 300 MB is 2.5% of the original size and the additional payload is about 40 times smaller than a full copy. This is not a storage bill or a guaranteed saving: compression, metadata, deduplication, rewritten partitions, requests and transfer charges can change the result. Version boundaries should follow change boundaries.

    Retention must follow evidence needs. Delta Lake defaults `delta.logRetentionDuration` to 30 days and `delta.deletedFileRetentionDuration` to seven days. The latter is the minimum age of logically removed files before they become candidates for `VACUUM`, not automatic data deletion. Data files are removed when `VACUUM` runs; transaction logs are cleaned separately after checkpoints. Time travel requires both usable log history and its data files, so retaining a log record does not guarantee restoration. Set both retention policies for the required evidence window before cleanup.

    Apache Iceberg snapshot expiration removes snapshots from table metadata and can delete files no longer needed by retained snapshots. Expired snapshots cannot be used for time travel. Set snapshot age, minimum snapshot count and branch/tag retention deliberately. Coordinate expiration and orphan-file cleanup with long-running jobs and required model evidence; metadata-only history is not recoverable data.

    Deletion requires equal care. GDPR Article 17 provides a right to erasure in defined circumstances, with grounds and exceptions described in the official regulation. Teams need a process to locate a subject across retained versions, assess valid requests and applicable duties, preserve only required audit evidence, and handle downstream copies, caches, replicas and backups. "Immutable" cannot mean "undeletable regardless of obligation." Versioning alone does not establish legal compliance; retention, legal holds, lawful use and erasure need appropriate governance and legal review.

    Common Data Versioning Mistakes

    Avoid these common failure modes:

    • Training from `latest`. The alias may move before the run is investigated.
    • Using dates as identity. "Rows before Friday" can change after corrections or late arrivals.
    • Versioning only the final table. Sources, labels or split membership may change invisibly.
    • Keeping pointers after deleting content. Metadata is useless when referenced files are gone.
    • Ignoring quality evidence. Recovering corrupt data reproduces the problem.
    • Treating experiment tracking as storage. MLflow dataset tracking logs source, schema and digest metadata, but a digest is not a durable snapshot. Digest computation varies by dataset type; metadata-only datasets do not hash the underlying data. The snapshot still needs a durable home and a tested retrieval path.
    • Never testing restoration. Another environment must retrieve and validate the version.
    • Retaining everything forever. Unbounded copies increase cost and erasure complexity.

    Versioning also supports, but does not replace, production ML monitoring. Monitoring identifies a regression or drift event; version records help explain which data change may have contributed.

    Data Versioning Best-Practices Checklist

    Use this ten-point checklist to operationalize data versioning:

    1. Define which raw, processed, label, split and feature layers require versions.
    2. Give every training run an immutable dataset identifier.
    3. Link data, code, configuration, environment and model records.
    4. Capture lineage, schema, quality results, owner and approval status.
    5. Separate immutable identifiers from movable aliases such as `latest`.
    6. Choose file or table boundaries that match common changes.
    7. Test retrieval and rollback from a clean environment.
    8. Set retention, garbage collection and legal-hold rules explicitly.
    9. Design erasure and access revocation across historical copies.
    10. Monitor storage growth and remove unneeded versions through an approved process.

    Conclusion

    The useful answer to what is data versioning in machine learning is not a tool name. It is an evidence contract: each important model can be connected to the exact data, code, configuration and environment that created it, and that evidence remains retrievable for the period the organization actually needs.

    Start with the smallest approach that can honor that contract. Use manifests for simple datasets, DVC for Git-centered file workflows, lakeFS for shared object stores, or table snapshots where a lakehouse already exists. Then prove restoration in practice, define retention and prepare for deletion before scaling the system.

    If your team needs a practical design for dataset lineage, reproducible training and production controls, contact CodeCones to review the current data platform and identify the smallest reliable next step.

    Frequently Asked Questions

    What is data versioning in machine learning?

    It assigns an immutable identity to a dataset state and links it to the ML work that used it. The record connects the snapshot, lineage and schema with code, configuration, environment and model output.

    Is data versioning part of MLOps?

    Yes. It supports reproducible training, controlled promotion, rollback and auditability. MLOps also covers pipelines, testing, registries, deployment and monitoring, so versioning is one control within a larger system.

    What is the difference between data versioning and model versioning?

    Data versioning identifies training and evaluation inputs. Model versioning identifies the resulting model artifact. A reliable model record links both with the code, parameters and environment.

    What is a dataset snapshot?

    A snapshot is a stable dataset view at a logical point. Copied files, a content hash, table version or commit may represent it, provided the referenced content remains retrievable.

    Which data versioning tool is best for machine learning?

    There is no universal winner. DVC fits Git-centered files, lakeFS fits shared object stores, and Delta Lake or Iceberg fit analytical tables. Infrastructure, scale, change pattern and retention should drive the choice.

    Can Git version a machine-learning dataset?

    Git can version small text datasets, manifests and pointers. It performs poorly with changing binaries, and GitHub blocks files above 100 MiB. DVC, Git LFS or object storage better suits substantial data.

    How does DVC version data?

    DVC tracks files through metadata and content hashes. Teams commit the metadata to Git while storing content in a DVC cache and remote. Check out the approved Git revision, then use `dvc pull`, or `dvc fetch` followed by `dvc checkout`, to restore data in a clean environment. `dvc checkout` alone needs content already in the cache.

    Does MLflow replace a data-versioning tool?

    Not usually. MLflow records dataset metadata and connects inputs to experiment runs. Teams still need an immutable source or snapshot in a system designed to retain the underlying data.

    How long should dataset versions be retained?

    Retain versions long enough to reproduce accepted models, investigate incidents and meet contractual or legal duties. Define rules by version class, and test that cleanup does not break required models.

    Do small ML teams need data versioning?

    They need reproducibility, not necessarily a large platform. Immutable folders, a checksummed manifest and experiment record may be enough until collaboration, scale or audit needs justify a dedicated tool.

    About CodeCones Editorial Team

    The CodeCones Editorial Team develops practical engineering guidance for enterprise product and AI teams.

    Key Takeaways

    • A data version is an immutable reference to the exact dataset state used for a purpose.
    • Version raw inputs, processed data, labels and reusable features when each layer can change independently.
    • Link the dataset version to the code commit, configuration, environment and resulting model.
    • Choose an approach from data shape, change frequency, team size and existing infrastructure.
    • Plan retention, deletion and access controls before historical copies accumulate.
    • Treat `latest` as a convenience for exploration, never as the permanent identity of a training run.

    Related insights

    Stay Ahead with AI Insights

    Get expert insights on enterprise AI, MLOps, and scalable architecture. Join thousands of professionals building the future of AI.

    By subscribing, you agree to receive updates about AI Assistants. Unsubscribe anytime.