Clinical Data Lifecycle Map: Capture → Lock → Submission → Archive

Why this matters

Locks, submissions, and inspections succeed when you can show your work end‑to‑end. That means:

• Predictable gates (clearly defined decision points).
• Traceable handoffs (each change and movement is recorded).
• Reproducibility (you can re‑create any dataset or listing years later).

If a regulator, auditor, or downstream team asks “Where did this number come from?”, your lifecycle map lets you answer with confidence—and evidence.

Plain language promise: We expand terms at first use.

• EDC = Electronic Data Capture
• DM = Data Management
• PV = Pharmacovigilance (safety)
• IT/CSV = Information Technology / Computer System Validation
• CRF = Case Report Form
• QC = Quality Control
• RACI = Responsible, Accountable, Consulted, Informed (decision matrix)

The lifecycle map (swimlanes)

Swimlanes: Sites • EDC/DM • Vendors • Biostatistics • PV (Safety) • IT/CSV • Archive

Think of each lane as a team with responsibilities, artifacts they produce, and a gate they must pass before the next team can rely on their output.

1) Capture & Ingest

• What happens:

  • Sites enter data into CRFs in the EDC.
  • External vendors (e.g., labs, ePRO, imaging, randomization) send files per an agreed transfer specification (format, columns, codes, timing).

• How it’s controlled:

  • Run structural conformance checks (schema, required fields, datatypes). If a file fails, it does not load (fail‑closed).
  • Land the file as immutable as‑received (exact copy of what arrived), with a checksum.

• Minimal lineage tags to add on load:

  • source_file_tag (e.g., central_lab_2025-08-12_run03.csv)
  • load_id (unique ETL/ingestion run identifier)
  • supplier_version or data_extract_version (if provided)

2) Staging & Curation

• What happens:

  • Standardize units (e.g., mg/dL → mmol/L when needed), timestamps and time zones, and key/ID formats.
  • Enforce keys and relationships (e.g., subject, visit, specimen).
  • Create curated tables used by cleaning, coding, and analysis teams.

• Lineage tags to carry forward:

  • algorithm_version (e.g., unit_norm_v1.2.0)
  • promotion_manifest_id (records what moved from staging → curated)
  • row_hash or lineage_hash for row‑level traceability

• Quality outputs:

  • QC listings of presence (record exists when expected) and value (values within allowed ranges, formats, and units).

3) Cleaning & Coding

• What happens:

  • Run batch edit checks (automated rules).
  • Open and resolve queries with sites.
  • Code medical terms using MedDRA (adverse events) and WHODrug (concomitant medications).
  • Route events for medical review or adjudication as needed.

• Lineage to preserve:

  • Dictionary name + version (e.g., MedDRA 26.1, WHODrug B3 2025 Q2).
  • Audit trail of query lifecycle (opened → answered → closed) and coding decisions.

4) Interim & Lock

• What happens:

  • Use a readiness dashboard to confirm all critical issues are cleared.
  • Perform a lock rehearsal (mock cut) to check timing, extracts, and approvals.
  • Execute the lock ceremony: freeze, obtain sign‑offs, extract datasets, seal the package, start archive steps.

• Evidence to capture:

  • Freeze timestamp, sign‑off records (DM, Biostatistics, PV, Clinical Operations), extract checksums, and an audit trail snapshot.

5) Submission & Reporting

• What happens:

  • Build analysis datasets, tables, figures, and listings (e.g., ADaM, TLFs).
  • Prepare narratives and submission packages.
  • Confirm that the data used for analyses is the same as the data locked (checksum verification).

6) Archive & Retention

• What happens:

  • Archive the final datasets, specifications, audit trails, manifests (indexes of files and checksums), derivation catalogs, and approvals.
  • Run readability/restore tests, especially after system upgrades or migrations.

• Outcome:

  • Years later, you can restore the exact state and reproduce outputs.

Gates, owners & artifacts

Tip: A “gate” is not a meeting. It is a binary decision with named owners and specific artifacts that prove the decision was ready to be made.

Handoffs & lineage: what must travel forward

These items must always move with the data. Without them, reproducibility and compliance suffer.

1. Immutable originals

   • Keep as‑received files exactly as they arrived, with checksums. Never overwrite them.

2. Promotion manifests

   • A machine‑readable list of files/rows promoted from staging → curated, including versions and checksums.

3. Derivation catalog

   • For each transformation or derivation, record the algorithm name, version, and parameters (e.g., “Impute missing onset date: impute_onset_v0.9, method=nearest_visit”).

4. Dictionary versions

   • Record the exact dictionary and version used for coding (e.g., MedDRA 26.1). Store coder conventions.

5. Access & audit trails around key events

   • UAT (User Acceptance Testing), releases, and lock all need time‑bounded audit trail extracts.

6. RACI references

   • Link each gate to the SOP/WI (standard operating procedure/work instruction) and the decision rights (who is Responsible/Accountable/Consulted/Informed).

Pitfalls & misconceptions

• Relying on vendor systems as your archive. Their audit trail does not replace your as‑received store and fixity checks.
• Promoting to curated without algorithm versions. If you cannot name the algorithm and its version, you cannot reproduce the row.
• “Data dump” archives without an index. A pile of files is not an archive. You need an index (what’s inside) and fixity (proof nothing changed).
• Unit and timezone drift. Small inconsistencies earlier cause major reconciliation headaches later. Normalize early, record the method.
• Late discovery of dictionary changes. Switching MedDRA or WHODrug versions mid‑stream requires impact assessment and controlled implementation.

Worked examples

Example A — Central lab feed from capture to curated (self‑contained)

Scenario: Daily hematology results arrive as CSV files from a central lab.

As‑received landing (immutable):

/as_received/central_lab/2025-08-12/
  CLAB_HEM_20250812_03.csv
  CLAB_HEM_20250812_03.csv.sha256
  conformance_log_20250812_run03.txt

Minimum conformance checks (fail‑closed):

• Required columns present (SUBJID, VISIT, TESTCD, RESULT, UNIT, COLLDTTM).
• Datatypes correct (numeric vs character).
• Code lists valid (e.g., TESTCD ∈ allowed list).

Staging transformation (record lineage):

• Convert date‑time to ISO 8601; harmonize time zone to UTC.
• Normalize units (e.g., g/dL → g/L) using unit_norm_v1.2.0.

Row‑level lineage tags added:

• source_file_tag = "CLAB_HEM_20250812_03.csv"
• load_id = "clab_heme_ingest_2025-08-12T19:05Z"
• algorithm_version = "unit_norm_v1.2.0"

Promotion manifest (excerpt):

{
  "promotion_manifest_id": "curate_lab_2025-08-12T20:10Z",
  "source": "/as_received/central_lab/2025-08-12/CLAB_HEM_20250812_03.csv",
  "checksum_sha256": "6d2f...a19",
  "rows_input": 1240,
  "rows_promoted": 1238,
  "exclusions": [
    {"reason": "missing_SUBJID", "count": 1},
    {"reason": "invalid_TESTCD", "count": 1}
  ],
  "algorithms": [
    {"name": "unit_norm", "version": "1.2.0"},
    {"name": "tz_harmonize", "version": "0.7.1"}
  ]
}

QC outputs generated:

• Presence listing (expected vs received by subject/visit).
• Value listing (range checks by TESTCD and unit).
• Differences since prior run (new rows, changed values).

What this enables: If a table or listing looks odd, you can trace any cell back to the exact lab file and the precise transformation version that touched it.

Example B — Lock ceremony evidence and archive package (self‑contained)

Scenario: Database lock at Primary Completion.

Pre‑lock readiness dashboard includes:

• Open critical queries = 0; overdue queries cleared.
• Coding completeness = 100%; adjudications closed.
• Reconciliation gates passed (SAE vs AE, concomitant meds vs exposure, randomization list vs treatment).
• Last data change timestamp ≤ freeze threshold.

Lock ceremony steps & artifacts:

1. Freeze at 2025‑09‑01T18:00Z. System prevents further edits.
2. Sign‑offs collected (DM, Biostatistics, PV, ClinOps) with timestamps.
3. Extract final datasets (SDTM/curated) with checksums.
4. Audit trail snapshot (bounded: last 90 days + lock event).
5. Seal the lock package.

Archive index (excerpt):

{
  "archive_id": "studyXYZ_lock_2025-09-01",
  "components": [
    {"path": "curated/subject.csv", "sha256": "ab91...e3"},
    {"path": "sdtm/ae.xpt", "sha256": "91bd...c5"},
    {"path": "manifests/promotion_manifest_2025-08-30.json", "sha256": "0fd2...19"},
    {"path": "audit/audit_snapshot_2025-09-01.jsonl", "sha256": "70b3...aa"},
    {"path": "admin/lock_signoffs.pdf", "sha256": "7e21...9f"}
  ],
  "created_at": "2025-09-01T20:35Z",
  "created_by": "DM Lead"
}

Fixity & readability tests recorded:

• Verify all checksums immediately after package creation.
• Restore test in a clean environment and confirm extracts open and row counts match.
• Record results in restore_test_log_2025-09-02.txt.

What this enables: Anyone can independently verify the locked state and re‑create analyses using only the archive package.

Example C — Dictionary version change mid‑study (self‑contained)

Scenario: MedDRA upgraded from 25.1 → 26.1 before database lock.

Controlled steps:

1. Impact assessment documented (differences in SOC/PT affecting AE roll‑ups).
2. Change control approved with effective date and environments affected.
3. Re‑coding performed in a sandbox first; QC diffs reviewed (counts by SOC/PT before vs after).
4. Promotion manifest records meddra_version = "26.1".
5. Listings and TLFs regenerated using updated codes; checksums updated.

Key artifact: coder_conventions_v3.0.pdf references the new dictionary and any study‑specific rules.

Practice checklist

• Publish the swimlane map in your Data Management Plan appendix.
• Define gates, owners, and artifacts; link each to the SOP/WI and RACI.
• Require lineage tags on every promoted row (source_file_tag, load_id, algorithm_version).
• Include an archive index and fixity report after lock, plus a documented restore test.
• Record dictionary/version and derivation catalog entries in a central location.
• Run readability tests after any system upgrade or migration.

Key points to remember

• The lifecycle is a sequence of gated handoffs with named owners and versioned artifacts that you can replay years later.
• Ingest is fail‑closed: files that fail structural conformance do not load; always retain immutable as‑received copies with checksums.
• Every promoted row carries lineage tags such as source_file_tag, load_id, and algorithm_version (and supplier_version when provided).
• Normalize units, time zones, and keys/IDs early, and record the exact normalization algorithm and version used.
• Use promotion manifests to describe exactly what moved staging → curated, including counts, exclusions, algorithms, and checksums.
• Maintain row‑level traceability (e.g., row_hash/lineage_hash) so any analysis cell can be traced back to source and transforms.
• Quality control covers presence (expected vs received) and value (ranges, formats, code lists) with clear QC listings.
• Coding requires the exact dictionary name and version (e.g., MedDRA, WHODrug) and documented coder conventions; keep full query and coding audit trails.
• Any dictionary or algorithm change follows change control: impact assessment, sandbox re‑runs, QC diffs, approvals, and updated manifests.
• Lock is a ceremony: freeze timestamp, cross‑function sign‑offs, final extracts with checksums, and a time‑bounded audit trail snapshot.
• Confirm that analyses (datasets, TLFs) are derived from the locked data—verify via checksum/hash alignment.
• An archive is an indexed package, not a folder: datasets, specs, audit trails, manifests, derivation catalog, approvals, plus a fixity report.
• Prove readability with documented restore tests after lock and after any upgrade or migration.
• Do not rely on vendor systems as your archive; maintain your own as‑received store and fixity verification.
• Link each gate to SOP/WI and RACI; a gate is a binary decision with evidence, not a meeting.
• Retain access and audit trails around UAT, releases, and lock; extract time‑bounded snapshots for evidence.
• Clear key reconciliation gates (e.g., SAE vs AE, meds vs exposure, randomization vs treatment) before lock.
• Design handoffs so lineage travels forward; never drop versions or parameters that would break reproducibility.