Business Continuity & Disaster Recovery (DR)

Why this matters

Downtime happens. A good plan keeps the study running, protects participants, and lets you prove to inspectors that data stayed complete and trustworthy. ICH E6 expects documented backup, recovery and contingency procedures for trial computerized systems, and 21 CFR Part 11 expects electronic records to stay retrievable throughout retention.

Plain‑language terms

• Business Continuity: How you keep critical study activities going during a disruption (for example, a cloud outage or vendor delay).
• Disaster Recovery (DR): How you restore systems and data after a serious incident.
• RTO (Recovery Time Objective): The maximum acceptable time a system can be down before it must be working again.
• RPO (Recovery Point Objective): The maximum acceptable data loss, expressed as time. Example: an RPO of 1 hour means you may lose at most 1 hour of data changes, so backups or replication must run at least hourly.

Your job as a Clinical Data Manager (CDM) is to make these targets explicit, test that you can meet them, and keep vendors and sites aligned while you recover.

Targets: RTO & RPO by system

Set targets that match study risk (safety‑critical vs routine), site operating hours, and vendor cut schedules. Use these as starting points and tighten where justified.

• EDC (Electronic Data Capture)

  • RTO: 4–8 hours (aim to cover same‑day clinic sessions).
  • RPO: 1 hour during business hours (tighter if safety reporting relies on EDC).
  • Why: Sites need to enter adverse events (AEs) and serious adverse events (SAEs), dosing, and visit data with minimal interruption.

• Integration platform (labs, ePRO (electronic patient‑reported outcomes), IxRS/IRT (interactive response technology), imaging)

  • RTO: within 24 hours.
  • RPO: “last successful delivery” (use vendor cut timing to pick a safe resend point).
  • Why: Batch feeds can be re‑run; the priority is avoiding gaps or duplicates. The IRT itself randomizes and supports emergency unblinding in real time, so it needs shorter targets of its own and a manual fallback.

• Archive (regulatory record copy, immutable)

  • RTO: 48–72 hours, read‑only is acceptable during recovery.
  • RPO: zero (archives are the fixed record of truth).
  • Why: Archives prove history; losing archive content is not acceptable.

Tip: If you must prioritize during a major incident, restore in this order: participant safety data → operational EDC access → vendor feeds → reporting/analytics → long‑term archives.

Evidence: backup & restore

Inspectors don’t accept “we had backups” without proof that you can restore them. Build an evidence pack each time you test or perform a real restore.

1) Controlled procedures

• Approved SOPs/work instructions define: backup scope (data, audit trails, metadata, system configuration), frequency, retention, encryption, off‑site/offline storage, access controls, and roles.
• Document where backups live (cloud region, vault, tape) and how you rotate keys (old keys stay available so older backups can still be decrypted).
• If the EDC or archive is vendor‑hosted, the vendor runs backups and restores but the sponsor stays responsible: set RTO/RPO, restore testing and evidence access in the contract or quality agreement, and review (or audit) the vendor’s restore‑test reports.

2) Restore tests (scheduled and event‑driven) Run at least quarterly and after major changes (upgrades, schema changes, new storage). Capture:

• Request/ticket with start/end timestamps.
• Exact backup used (date/time, location, identifier).
• Steps performed (who did what, in order).
• Verification results (see below).
• Screenshots or logs that a reviewer can follow.
• Sign‑off by QA/CSV (Computer System Validation) with pass/fail and deviations.

3) What “good” verification looks like Verify both data integrity and system fitness for use:

• Record counts by key tables (subjects, visits, forms, lab results) match expected.
• Hashes/checksums or export comparisons for critical tables/files.
• Audit trail is readable and complete; no gaps in who/what/when.
• Date/time stamps and time zone handling are correct after restore.
• User access and roles restored correctly (principle of least privilege); removals and role changes made after the backup point are re‑applied before users log in.
• Application opens, queries can be created/closed, reports run, and eSign works.
• If a DR environment is used, confirm it stays read‑only until this verification is signed off; the DR environment itself is validated in advance, not during the incident.

4) DR drills Simulate loss of a database, cloud region, or integration queue. Time each step and compare the total to the RTO and the data lost to the RPO. Show that your chosen cut timing prevents duplicate or missing data when feeds resume.

Vendor continuity & interim capture

Continuity includes your vendors. Write down exactly how each partner helps you recover.

Vendor playbook

• Named contacts and escalation path (support → manager → on‑call).
• Re‑delivery rules: what triggers a resend, formats accepted, SFTP/API endpoints, and where to send it (primary or DR environment).
• Cut timing: the “go‑back‑to” point for reprocessing (for example, “resend everything after the last file we loaded and reconciled”). A later point leaves holes; an earlier one creates double loads unless the reload is de‑duplicated on record keys (e.g., accession number plus test code).
• Evidence to retain: delivery receipts, file manifests, checksums, and reconciled counts.

Interim capture when EDC is down

• Approved paper or local eSource forms for critical data (AEs/SAEs, dosing, consent, end‑of‑treatment). SAEs never wait for EDC: the site sends each SAE straight to PV (drug safety) on the paper SAE form immediately (usually within 24 hours of learning of it; the protocol sets the timeline) and enters it in EDC later.
• Assign subject/visit identifiers, date/time with time zone, and the person recording.
• Back‑entry procedure with source verification and an audit of back‑entry (who entered, from which source, when). Site staff back‑enter their own data (DM never transcribes); the site keeps the interim forms as source documents.
• Reconciliation: compare interim forms to EDC once restored; query the site on discrepancies and document the resolutions.

Incidents: roles & communications

Clear roles and predictable updates calm sites and sponsors.

Core roles

• Incident Commander: owns decisions and timelines.
• Technical Lead: executes restore/failover and confirms fitness for use.
• Comms Lead: drafts and sends updates to sites, vendors, sponsor, internal leaders.
• QA/CSV Observer: ensures controlled process, gathers evidence, logs deviations.

Runbooks (per system)

• Triggers (for example, “EDC not accessible for 15 minutes”).
• First actions (confirm scope, stop non‑essential jobs).
• Decision points (read‑only mode? failover to DR? start interim capture?).
• Escalation ladder with names and time limits for each rung.

Communications

• Start a timestamped incident log immediately.
• Commit to a cadence (for example, every 60 minutes until resolved, then daily until all reconciliations close).
• Send “what we know / what we’re doing / what you should do now / next update time.”
• Keep a communication tree (who notifies whom, with a backup for each contact) for sites, vendors, and governance bodies.

Pitfalls & misconceptions

• Myth: “Backups are enough.” Reality: Inspectors expect restore evidence that proves you can use the data again.
• Gap: DR plan ignores vendors. Impact: delivery gaps or duplicates with no authoritative re‑run path.
• Gap: No interim capture. Impact: missed AEs/SAEs, dosing, and protocol deviations during outages.
• Overreach: Uncontrolled emergency access during DR. Fix: temporary access must be approved, time‑bound, and documented, given only to trained staff on personal accounts (no shared logins), and removed afterwards.
• Blind spot: Time zones after restore. Fix: verify timestamps and date logic in validations and reports.

Practice checklist

• Publish RTO/RPO per system; tie to risk and site hours.
• Keep SOPs current for backups, encryption, off‑site storage, and restores.
• Run quarterly restore tests; file complete evidence packs.
• Define cut timing and re‑delivery with each vendor.
• Approve interim capture and back‑entry reconciliation steps.
• Maintain incident runbooks, roles, and an escalation ladder.
• After any real incident, complete reconciliations and a CAPA (corrective and preventive action) if needed.

Worked examples

These scenarios are self‑contained and can be practiced as mini‑drills.

Example 1 — EDC outage during clinic hours

• Terms: RTO is the maximum time to restore service. RPO is the maximum data loss window.

• Targets: EDC RTO 8 hours; RPO 1 hour.

• Event: At 10:00, sites report EDC login failures.

• Actions

  1. Incident Commander declares an incident; Comms Lead tells sites to use paper AE/SAE and dosing forms until further notice and to keep sending SAE reports straight to PV within the usual timeline.
  2. Technical Lead restores the 09:30 backup to the DR environment and brings EDC online in read‑only by 12:00 for viewing.
  3. Full write access restored by 16:00, 6 hours after the 10:00 report (meets the 8‑hour RTO).
  4. RPO check: entries made after the 09:30 backup and before the failure at about 10:00 (30 minutes, within the 1‑hour RPO) are lost with their audit trail. Sites are told the restore point and re‑enter the missing entries from their own source documents (DM never transcribes).

• Evidence: timestamps, backup ID, record counts pre/post, audit‑trail check, screenshots, and QA sign‑off.

• Close‑out: sites back‑enter data; CDM reconciles paper vs EDC and queries the site on any differences; protocol deviations are logged where the outage caused something the protocol requires to be missed or done out of window (e.g., an assessment, a dose, randomization); late EDC entry alone is not one.

Example 2 — Lab feed corruption

• Terms: Cut timing is the agreed point to resend data from to avoid gaps/duplicates.

• Targets: Integration RTO 24 hours; RPO is “last successful delivery.”

• Event: Tuesday’s 02:00 lab file had header errors; no results posted.

• Actions

  1. Pause ingestion; notify lab vendor to re‑deliver everything after Monday’s file, the last successful delivery (the cut, matching the RPO); a 00:00 Tuesday cut would miss any pre‑midnight results the failed file carried.
  2. Vendor sends a corrected file and a manifest with counts and checksums.
  3. CDM loads to a quarantine area, validates counts/hashes, then promotes to EDC.

• Evidence: vendor ticket, manifests, counts by subject/visit, and ingestion logs.

• Close‑out: reconcile EDC vs lab portal; the lab corrects and re‑sends its own values, load errors (e.g., duplicates from the replay) are fixed by a controlled reload or an approved data‑correction script, and the site gets queries only for site‑entered fields (e.g., collection dates).

Example 3 — Archive restore proof

• Terms: Archive RPO zero means archived content must be fully restorable without loss.

• Event: Quarterly test of the regulatory archive.

• Actions

  1. Restore a chosen month of archived PDFs/exports to a clean environment.
  2. Check that the file count and every file’s checksum match the archive index (a spot‑check cannot show zero loss), then confirm the files open and are readable.
  3. Confirm metadata (study, site, subject, date) is intact and searchable.

• Evidence: restore log with timestamps, file‑count and checksum comparison against the archive index, and QA approval.

• Close‑out: file the evidence pack; note any improvements to the runbook.

Key points to remember

• Continuity = clear RTO/RPO targets + runnable runbooks + solid restore evidence.
• Apply RTO vs RPO per system: EDC, integrations, and archives (archives should have zero data loss).
• Restore proofs beat backup logs: capture counts and hashes/checksums, screenshots/logs, exact backup IDs and timestamps, and obtain QA/CSV sign‑off that the system is fit for use.
• Treat vendors as inside your DR boundary: define re‑delivery and cut timing, retain manifests/receipts, and reconcile counts after any replay.
• Plan approved interim capture (paper or local eSource) for critical data; perform controlled back‑entry by site staff with audit trail and documented reconciliation to EDC; SAEs still go from the site straight to PV.
• Enforce access control in DR: bring environments up read‑only until post‑restore verification is signed off; any temporary access must be approved, time‑bound, documented, limited to trained staff on personal accounts, and removed afterwards.
• After any restore, verify audit trails, timestamps/time zones, user roles/permissions, and check for duplicates or gaps from replayed feeds.
• Avoid common pitfalls: assuming backups are sufficient without restore testing, skipping back‑entry reconciliation, overlooking time‑zone effects, and allowing unmanaged duplicates after vendor re‑delivery.