Archive Integrity & Readability

Why this matters

Years after a study ends, regulators, partners, and your own statisticians may ask you to reproduce a table, listing, or figure—or to prove exactly what data and logic produced a submission. That is only possible if your archive is:

• Complete (everything needed is included),
• Immutable (originals are never changed), and
• Readable (you can still open and parse files after systems are upgraded or retired).

Good Clinical Practice (ICH E6) and 21 CFR Part 11 expect this: electronic records and their audit trails stay accurate, retrievable and readable for the whole retention period.

Plain language: An archive (the lock archive) is the package you keep after database lock that contains the final data, the “as‑received” source files, the specifications that explain the transformations, and the evidence that nothing was altered later.

Archive components & index

Include these building blocks and describe each one in an index so anyone can find and verify it.

1. Final datasets

   • Electronic Data Capture (EDC, e.g., Medidata Rave/REDCap) extracts, curated datasets used for analysis, and any deliverables to statisticians or regulators.
   • Each file gets a checksum (a digital fingerprint, e.g., SHA‑256).
   • At lock, also export open, non‑proprietary formats (e.g., CSV, CDISC ODM XML, SAS transport (XPT), PDF/A) beside any native exports, so the data stay readable without the original system. Before the EDC is retired, each site also receives a readable copy of its own data and audit trail.

2. As‑received source files (immutable)

   • Lab results, external vendor feeds, adjudication outputs, etc., exactly as received.
   • Include promotion manifests that show when and how each file moved from staging to curated areas.

3. Specifications & logic

   • Data transfer specs, mapping documents, derivation rules (algorithm name + version), coding dictionaries and versions (e.g., MedDRA 26.1, WHODrug Global March 2024 in B3 format).
   • The Statistical Analysis Plan (SAP), the extractions it used, and any code lists or controlled terminology used.
   • The annotated case report form (CRF) and the programs (with versions) that built the datasets and outputs; if biostatistics archives the programs elsewhere, an index entry giving their location and checksums.

4. Approvals & completion evidence

   • Lock ceremony sign‑offs, coding/adjudication completion records, protocol amendments that affected data.

5. Audit trails (complete)

   • The full audit trail of every record (who changed what, when, old and new value, reason), exported in a format that can be read without the original system. Focused reports around lock and other key events (e.g., late data changes, dictionary up‑versioning) can be added, never instead.

6. Archive index (the “table of contents”)

   • A single CSV/JSON that lists: path, description, study_id, data_cut_date, format, size_bytes, checksum, system, version (and, optionally, created_by).
   • This index is your map: it ties content to context and makes later verification practical.

Example index snippet (CSV):

path,description,study_id,data_cut_date,format,size_bytes,checksum,system,version
/lock/final/edc_export_2024-10-31.zip,"EDC final export (lock)",ST-101,2024-10-31,zip,1843210,0e7f...a1cb,Rave,2024.3
/as_received/labs/2024-10-29/LAB_ACME_000345.csv,"Central lab feed batch 345",ST-101,2024-10-31,csv,93210,9b2c...772e,SFTP-ACME,v1
/specs/derivations/dv_rules_v2.1.pdf,"Derivation rules v2.1",ST-101,2024-10-31,pdf,1203345,77de...bb19,DM-Git,v2.1

Integrity: fixity & immutability

Fixity means being able to prove that a file’s bits haven’t changed. We do that with checksums (e.g., SHA‑256) created when we first save a file and re‑checked later.

• On write: Generate a checksum each time you add a file to the archive.

• On read / periodically: Re‑calculate and compare the checksum to catch silent corruption (disk faults, accidental edits). Keep at least two copies in separate locations so a file that fails can be restored from a good copy.

• Immutability controls:

  • Store archives on write‑once or retention‑locked storage (e.g., WORM (write once, read many) media or immutable cloud buckets).
  • Restrict write access; enable storage audit logging.

• No in‑place edits: If a correction is needed, create a new file and mark the prior one as superseded (see below). The original stays in the archive with its original checksum.

Tip: Keep the checksum with the file (e.g., <filename>.sha256) and in the archive index.

Readability tests (no‑change principle)

Upgrades happen. Your goal is to prove you can still open and use the archive without altering the contents.

• Media refresh vs. content migration

  • Media refresh: Copying files to new storage hardware/cloud with checksums before/after. Content does not change.
  • Content migration: Converting file formats (e.g., XLS to CSV) or re‑exporting from a new system version. Bits do change. Avoid unless necessary.

• No‑change principle: Prefer readability tests over mass conversions. Show that archived files still open, parse, and produce expected basic results (e.g., row counts, key variable types).

• When to test:

  • Major OS/database/runtime upgrades, EDC vendor upgrades, decommissioning legacy systems, or moving to new storage classes.

• How to test (lightweight and auditable):

  1. Select a representative sample of each file type (CSV, SAS, JSON, PDF, SDTM packages, etc.).
  2. Verify each sampled file’s checksum against the index (the bits are unchanged), then open it with the reader toolkit (see below).
  3. Run smoke checks: row/record counts, column presence/types, ability to render PDFs, ability to decompress archives.
  4. Record the outcome in a readability report (stored in the archive and referenced in the upgrade’s change control record): tested files, checksum results, tools used (name + version), date, pass criteria (set before testing), pass/fail, notes.
  5. If a file fails, fix the tooling, not the archive content. Only migrate content as a controlled change with approvals: keep the original, checksum the new file, and verify and document that it holds the same information (e.g., row counts, key values, variable names and types), so the new file is a verified (certified) copy.

• Reader toolkit:

  • Maintain a small virtual machine (VM) or container with pinned versions of libraries (e.g., specific SAS/SPSS runtimes, Java/.NET, Python packages) and viewers (e.g., a specific PDF viewer version).
  • Keep the built image itself (e.g., an exported container or VM image), its build recipe (e.g., Dockerfile or VM notes) and their checksums in the archive; a recipe alone may not rebuild once old packages disappear from public sources.

Worked examples

Example 1 — Build a minimal archive index

Goal: create a searchable list of all archived artifacts with fixity.

Steps:

1. For each file, calculate a SHA‑256 checksum and save it as <filename>.sha256.
2. Append a new line to archive_index.csv with: path, description, study_id, data_cut_date, format, size_bytes, checksum, system, version.
3. Store both the file and its checksum in the archive; save the updated index as a new version with its own checksum (earlier versions are kept).

Result: Anyone can verify a file by recomputing its checksum and matching it to the index entry.

Example 2 — Readability test after a major EDC upgrade

Scenario: Your EDC platform upgraded from 2024.3 to 2025.1.

Plan:

• Sample up to 5 files per type (all of them where fewer exist): EDC export ZIP, site‑level CSV, audit trail CSV, coding listings (PDF), and derived analysis datasets (SAS7BDAT or CSV).

• Verify each file’s checksum against the index, then use the reader toolkit container (tagged reader-tools:1.4) to open it.

• Run checks:

  • CSVs: confirm delimiter, encoding (e.g., UTF‑8), number of rows/columns, and presence of key variables (subject_id, visit, dt_collected).
  • ZIPs: decompress successfully and verify internal file checksums.
  • PDFs: open without rendering errors.
  • SAS datasets: open; row counts and variable names and types match the dataset specifications.

• Record outcomes in readability_report_2025-03-15.pdf and add to the archive.

Decision: All pass → no content migration. Keep the report with the archive.

Example 3 — Media refresh (no content change)

Scenario: Moving the archive to new cloud storage.

Actions:

1. Copy files with a tool that supports checksum verification (source and destination).
2. Re‑generate checksums on the destination and compare to the index.
3. Record the new storage location in a new version of the archive index (paths within the archive, content and checksums remain the same).
4. Keep a short media refresh memo describing dates, tool versions, and verification counts.

Outcome: Storage is modernized; archived bits are unchanged and proven identical.

Example 4 — Post‑lock correction using supersession

Scenario: A lab vendor resends 12 corrected results after lock.

Packaging:

• Place corrected file under /addenda/2025-04-02/.
• Add a manifest describing the change and link it to the change control ID.

Manifest snippet (YAML):

change_id: CHG-2417
reason: "Corrected unit conversion for 12 potassium results"
scope:
  source_file: "/as_received/labs/2024-10-29/LAB_ACME_000345.csv"
  corrected_file: "/addenda/2025-04-02/LAB_ACME_000345_fix.csv"
actions:
  - supersede_prior_rows: true
  - no_deletions: true
verification:
  original_file_checksum: "9b2c...772e"
  corrected_file_checksum: "a38d...9011"
  records_affected: 12
approvals:
  unlock_relock: "Approved by lock approvers 2025-04-03"
  dm_lead: "Signed 2025-04-03"
  qa: "Signed 2025-04-04"

Rule: The original remains in the archive. Lock removed all edit rights, so whether the correction enters the analysis is a documented decision by the lock approvers: unlock and re‑lock with written approval, or record it as a known issue and leave the locked data unchanged. If it is applied, analytic datasets reflect the correction through supersession, not overwrite.

Corrections, addenda & supersession

• Corrections after lock are packaged as addenda with their own checksums, manifests, and approvals. Whether a correction changes the locked analysis data is a separate, documented unlock decision (see Example 4).
• Retain both original and corrected deliveries.
• Mark prior rows as superseded using metadata (e.g., a superseded_by field or a linkage in the manifest) so lineage is traceable.
• Never delete or replace the original file; it is legal evidence of what existed at lock.

Pitfalls & misconceptions

• Anti‑pattern: “Keeping up with formats” by converting everything during an upgrade—without checksums, manifests, or sample compares.

  • Fix: Do readability tests; migrate only specific problem files under change control.

• Misconception: “Our vendor keeps the source, so we don’t need our own as‑received copies.”

  • Fix: Always archive your own as‑received files and an index. Vendor retention policies can change.

• Anti‑pattern: Skipping periodic fixity checks, allowing silent corruption to go undetected.

  • Fix: Schedule checksum verification (e.g., quarterly) and log results in the archive.

Practice checklist

• Produce and maintain an archive index with checksums and context.
• Store as‑received files and promotion manifests.
• Run readability tests on major upgrades; store the report with the archive.
• Use supersession for corrections; never overwrite or delete originals.
• Keep a reader toolkit (VM/container) with pinned versions for legacy formats.
• Perform periodic fixity checks and record outcomes.

Key points to remember

• Your archive must be complete, immutable, and readable years later.
• Integrity = immutability + fixity + an index. Generate checksums (e.g., SHA‑256) on write and verify on read/periodically; store the checksum file and the value in the index.
• Keep as‑received source files exactly as delivered, plus promotion manifests that show movement from staging to curated; never rely solely on vendor retention.
• Use retention‑locked/WORM storage, tight write permissions, and audit logging; no in‑place edits—use supersession if anything changes post‑lock.
• Maintain a single archive index (CSV/JSON) listing at minimum: path, description, study_id, data_cut_date, format, size_bytes, checksum, source system, and version.
• Prefer readability tests (the no‑change principle) over mass conversions. Fix the tools, not the archived bits.
• Know the distinction: media refresh = same bits to new storage (verify with checksums); content migration = bits change (only under controlled change).
• On major upgrades or decommissioning, run a documented readability verification using a pinned toolset; migrate only specific problem files with approvals and traceability.
• Keep a lightweight reader toolkit (VM/container) with pinned runtimes/viewers in the archive: the built image and its build recipe, each with a checksum.
• Package post‑lock corrections as addenda with their own checksums, manifests, and approvals; if the lock approvers decide to apply the correction (documented unlock and re‑lock), corrected rows supersede prior rows; originals stay either way.
• Retain the complete audit trails in a readable export (focused reports for key windows, e.g., around lock or dictionary up‑versioning, are extras, not substitutes) and keep approvals/completion evidence with the archive.
• Schedule periodic fixity checks and record results in the archive to detect and prove against silent corruption.