Direct Acquisition & Integration Readiness (ePRO/Devices/Labs)

Why this matters

In one sentence: Direct electronic capture—from ePRO apps (electronic patient‑reported outcomes), medical devices, or central labs—can cut transcription errors and increase data resolution only if the capture is validated, the feed is specified, and the pipeline enforces gatekeeping (conformance + provenance).

Manual re‑keying from paper or portals introduces delays, rounding, and copy errors. Direct feeds move data faster and at higher granularity (e.g., second‑by‑second heart rate) but shift the risk to specification, validation, and governance. Poor specs and “load first, fix later” approaches create reconciliation debt that is expensive to clear near database lock.

Acronyms used: CRF = case report form; DM = data management; SFTP = secure file transfer protocol; API = application programming interface; UTC = Coordinated Universal Time; ETL = extract, transform, load; QC = quality control; KPI = key performance indicator; PHI = protected health information.

When to use direct capture

Use direct capture when any of the following apply:

• High frequency or time‑sensitive measures. Examples: continuous glucose monitors, ambulatory blood pressure, arrhythmia monitors, spirometer sessions with second‑level traces.
• Complex timing windows or longitudinal trends. You care about exact timestamps, diurnal patterns, or adherence curves (not just “did it happen”).
• Trusted, validated streams exist. Vendor/device platform follows a quality system; audit trails and security are in place; privacy risks (e.g., location data) are controlled. For ePRO, the participant’s entry is the source, so the investigator must keep access to it (and get a copy at study end); the sponsor or vendor never holds sole control.

Consider manual transcription or hybrid approaches when:

• The signal is rare/low volume and would incur heavy build/integration cost for minimal benefit.
• The vendor cannot meet validation, audit, or privacy requirements (for example, missing audit trails, unverifiable time sources).
• Sites must curate context that devices cannot provide (e.g., clinical rationale or free‑text clarifications), in which case direct data can still be linked but not replace CRF fields.

Tip: Even with direct capture, you usually still need slim CRF fields (e.g., “device used?”, “collection performed?”) for monitoring and protocol compliance.

Integration specs & conformance

Your data transfer specification (DTS; sometimes part of a data transfer agreement, DTA) is the contract. It must be explicit enough that a third party could validate a file without ambiguity. At minimum include:

A. Structure & naming

• File/container naming pattern (with timezone and sequence indicators).
• Transport method (SFTP/API), encryption, and packaging (CSV/JSON/Parquet, compression).
• Delivery cadence and cut logic (e.g., “daily incremental since last watermark” or “weekly full replacement”).

B. Fields & types

• Column list with name, type, format (e.g., ISO 8601 timestamps), nullability, and controlled terminologies (e.g., units, analyte codes).
• Identity keys: subject identifier(s), sample ID, device ID, visit context.
• Primary key or upsert key definition and duplicate policy (e.g., supersession rules).

C. Timing & units

• Time basis (UTC vs local), time source (device clock vs server time), daylight‑saving handling, and allowed clock drift.
• Units and normalization rules; permissible conversions and rounding.

D. Quality & lineage

• Quality flags from the source (e.g., motion artifact, hemolysis, calibration status).
• File‑level checksum and record counts (expected vs actual).
• Error reporting channel and SLA (service‑level agreement) for corrections.
• Sign‑off by sponsor and vendor, version control and change control; a test transfer must pass the gate end to end before the first production delivery.

Structural conformance gate (must pass before load)

Enforce a fail‑closed gate. If a file or API payload fails, quarantine and reject—do not “fix in place” downstream. Typical checks:

• File can be opened/decompressed; checksum matches manifest. (Record counts against the manifest, then by subject and visit, are checked right after the gate.)
• All required columns present; no unexpected extras unless allowed by versioning.
• Types and formats are valid (dates parse; numbers fit the declared type and precision; controlled terms match).
• Primary key constraints hold; duplicates handled per spec.
• Delivery metadata (batch ID, generation time, vendor version) present.

Range and plausibility checks are logic checks: they run on loaded data after the gate (with counts by subject and visit) and raise queries; they do not reject the file.

As‑received archive: Store the original file/payload immutable, with checksum and metadata. Keep a separate stage → curated path with transformation logs so you can prove what changed, when, and why.

Example – excerpt from a clear spec

feed_name: central_lab_chemistry_v2
delivery:
  cadence: daily_incremental
  transport: sftp
  file_pattern: "chemistry_{site}_{YYYYMMDD_HHMM}Z_{seq}_inc.csv.gz"  # generation time in UTC + sequence number
  checksum: sha256
schema:
  primary_key: [subject_id, sample_id, analyte, collection_ts_utc]
  columns:
    - name: subject_id        # string, non-null
    - name: sample_id         # string, non-null
    - name: analyte           # string, term: [ALT, AST, ALP, BILI_T]
    - name: value             # decimal(12,4), non-null
    - name: unit              # string, term: [U/L, mg/dL, µmol/L]
    - name: collection_ts_utc # datetime, ISO 8601, non-null
    - name: result_flag       # string, term: [NORMAL, HIGH, LOW], null allowed
    - name: qc_flag           # string, term: [PASS, FAIL, HEMOLYZED], non-null
versioning:
  allow_extra_columns: false
  supersession: "duplicate primary_key within one file fails the gate; across files, the later file by its name timestamp and sequence wins (never by arrival time)"

Reconciliation patterns

Reconciliation answers four questions across presence, identity, timing, and value. Normalize units and time zones before you compare.

1. Presence – Did we get the record we expected?

   • Compare expected events (from schedule or device plan) to delivered records.
   • Missing? Create a vendor‑routed query when the feed is the source of truth; site‑routed when the activity didn’t happen.

2. Identity – Is the record linked to the right person/sample/device?

   • Mismatched subject ID or sample ID ⇒ vendor‑routed if it is a mapping or key error at the source (lab/device); site‑routed if the site wrote the wrong ID on the requisition or kit label (the vendor then corrects and re‑sends).
   • Site transposed subject in the CRF? Then site‑routed.

3. Timing – Did it occur when the protocol allows?

   • Normalize timestamps to UTC (or a chosen standard) and apply visit windows.
   • Device clocks drifting or wrong daylight‑saving handling ⇒ vendor‑routed.
   • “Procedure done on wrong visit day” ⇒ site‑routed.

4. Value – Do values agree across sources?

   • Differences from rounding or unit conversion → check spec first; if feed agrees with spec, consider CRF transcription error (site).
   • Implausible physiologic values or QC flags FAIL → vendor (or lab) to investigate instrument run.

Owner‑routed queries – quick guide

The owner of the source corrects it. Vendor issues go through the spec’s error‑reporting channel (an issue log, not an EDC query); the vendor corrects and re‑sends (changing a participant’s ePRO entry itself also needs a documented request from the investigator). Site issues are EDC queries, and the site corrects its own entries. DM routes and verifies but never edits either source; a bug in your own pipeline is fixed under change control and reprocessed from the as‑received archive.

Normalization first: Unit and timezone normalization must happen before any presence/timing/value checks; otherwise you will create false positives. Keep normalized values in separate fields; never overwrite the value as received or the site’s CRF entry.

Pitfalls & misconceptions

• “Devices are plug‑and‑play.” They still need validation, audit trails, and clear mappings.
• Loading before schema checks. “We’ll fix later” creates silent drift and contaminated analytics.
• Treating unit normalization as optional. Inconsistent units look like outliers; they are not.
• Ignoring daylight‑saving or device clock drift. One‑hour offsets can break visit windows.
• No immutable archive. Without the as‑received file, you cannot defend provenance during audits.
• Ambiguous supersession rules. Not defining “latest” leads to oscillating values near lock.
• Over‑collecting PHI. Capture only what you need; strip location or free text that adds privacy risk unless explicitly required.

Worked examples

Example 1 — ePRO daily symptom diary across time zones

Context: Participants use a phone app to report symptoms each evening. The vendor provides a daily CSV via SFTP.

Spec essentials (excerpt):

• report_ts_utc in ISO 8601; vendor converts from device local time to UTC.
• One row per subject per calendar day in the subject’s local time (stored also as local_date).
• Primary key: [subject_id, local_date].

Conformance gate:

Reject the file if any row is missing report_ts_utc or local_date, or if duplicate [subject_id, local_date] appears without a supersession marker.

Reconciliation:

• Presence: Compare expected diary days to delivered rows. A day the vendor’s portal shows as submitted but missing from the file → vendor (transfer or cut‑logic gap); a day the participant did not submit → site, to follow up diary compliance with the participant (after the late‑entry window, the day stays missing).
• Timing: Late entries allowed up to 24 hours per protocol; enforce window on the normalized UTC timestamp.
• Value: Sites should not re‑key ePRO scores into the CRF; if a study does, a transcribed total that differs from the direct value → site to correct its CRF entry (the participant’s ePRO entry is the source and stays unchanged).

Takeaway: Define the local‑day concept explicitly and store both local date and UTC timestamp to satisfy windows and trend charts. For example, an entry at 23:30 on 3 March in New York (UTC−5) has report_ts_utc 04:30 on 4 March; grouping by UTC date would make 3 March look missing.

Example 2 — Wearable heart‑rate monitor (high frequency)

Context: A device streams second‑level heart rate. The platform exports minute‑level aggregates (mean_hr, min_hr, max_hr) every 24 hours.

Spec essentials:

• epoch_start_utc (minute boundary), epoch_end_utc, mean_hr, min_hr, max_hr, artifact_flag.
• Quality flags must be present; epochs with artifact_flag != PASS are excluded from derived values (e.g., daily mean heart rate) per the SAP (statistical analysis plan).

Conformance gate:

• Validate that each epoch is one whole minute on a minute boundary, with no overlaps (gaps are checked under Presence).
• Enforce types (e.g., mean_hr, min_hr and max_hr are numeric, as the spec declares). Implausible values do not fail the gate; they load and go to the vendor as value issues.
• Reject if artifact_flag contains unknown terms.

Reconciliation:

• Presence: Expected minutes = enrolled monitoring periods minus declared “device off” durations. Gaps → vendor to investigate upload failures.
• Timing: Compare the device clock with the platform’s server time at each upload; drift beyond the spec’s allowance, or a one‑hour daylight‑saving jump, → vendor.
• Value: No CRF double entry exists. The CRF field “monitor worn?” explains gaps: “No” with no data needs no query; “No” with data, or “Yes” with a gap the vendor cannot explain, → site to confirm.

Takeaway: For high‑frequency signals, aggregate upstream (keeping the raw second‑level data at the source) and carry quality flags through; never infer quality post‑hoc.

Example 3 — Central lab chemistry vs CRF

Context: Lab sends ALT/AST results daily. Sites also transcribe results into the CRF.

Spec essentials:

• Primary key [subject_id, sample_id, analyte, collection_ts_utc].
• Units enumerated; conversions defined in the spec; rounding to the lab’s significant digits.

Conformance gate:

• A duplicate primary key within one file fails the gate; a key resent in a later file is a correction and replaces the earlier row.
• unit must be in the allowed list; qc_flag required.

Reconciliation:

• Presence: Expected sample shipped per accession log, but no lab record → vendor/lab.
• Identity: Sample ID mismatch between lab feed and CRF → depends on where the error sits (often site if CRF entry wrong; vendor if accession label mapping wrong).
• Value: CRF value differs from lab by more than the permitted rounding tolerance → site to correct transcription.
• Timing: Collection time beyond visit window → site to confirm the time against source and, if it is truly out of window, document the protocol deviation.

Takeaway: Declare the lab feed as source of truth for numeric results; CRF is secondary for review context and auditability. Better still, do not have sites re‑key central lab results; keep slim CRF fields (e.g., sample collected?, collection time).

Practice in the wild (checklist)

• Decide on direct capture using a simple screen: high frequency? strict timing? vendor validated? privacy acceptable?
• Write the transfer spec: structure, fields, keys, units, time, quality flags, versioning, supersession, delivery cadence, and SLAs.
• Enforce structural conformance on every delivery: fail‑closed; quarantine; notify vendor with actionable errors.
• Archive as‑received (immutable + checksum), separate from stage/curated layers; log every transform for lineage.
• Normalize units & time zones before any comparisons; document conversions and rounding.
• Run reconciliation by dimension (presence, identity, timing, value) and route queries to the owner (vendor vs site) with evidence.
• Monitor for schema drift (new columns, changed types) and version feeds explicitly.
• Review privacy scope (minimize PHI; mask where not needed); confirm consent covers the capture.
• Test replays (backfills/corrections) and ensure supersession rules produce stable results near lock.

Key points to remember

• Prefer direct electronic acquisition (ePRO/devices/central labs) when streams are validated and the signal is high‑frequency or timing‑sensitive; use manual/hybrid when volume is low, the vendor cannot meet validation, audit trail, or privacy requirements, or sites must add clinical context that devices cannot capture.

• Treat the transfer specification as a contract. Define primary/upsert keys, units, time standard (UTC) and timezone handling, supersession rules, delivery cadence, and required quality flags; include error reporting and SLAs.

• Enforce a fail‑closed structural conformance gate before any load, counts, logic, or reconciliation. Quarantine non‑conformant files (schema/type/key/format violations) rather than “fixing later.” Do not compute KPIs or reconcile on non‑conformant data.

• Preserve provenance. Keep immutable as‑received copies with checksums, and maintain stage → curated transformations with auditable logs and versioning.

• Normalize first, then compare. Standardize units and time zones prior to running any checks or metrics.

• Reconcile in a consistent order: presence → identity → timing → value (after normalization).

• Route queries to the owner with evidence. The owner corrects; DM never edits either source.

  • Vendor/Lab for source/timing/format issues (e.g., device clock, UTC conversion, schema adherence, feed generation/cut logic).
  • Site for transcription differences, sample or kit labelling errors, visit adherence, or CRF context problems.

• Keep results stable near lock. Make supersession rules explicit and consistently applied so late replays/corrections don’t oscillate values.

Memory cue: Structure → Normalize → Reconcile → Route.