Unit Normalization & Time Zones

Why this matters

Clinical data arrive from labs, devices, and sites using different units (mg/dL vs mmol/L) and different time conventions (local time, mixed time zones, daylight saving shifts). If you compare values or apply windows without normalizing first, you get:

• false discrepancies (looks “off” only because the units differ),
• incorrect visit window decisions (a reading appears late/early due to time‑zone confusion),
• and fragile audits (you cannot show how a number was derived).

Goal: a repeatable, versioned process that converts once, rounds once, and always preserves the original value and context so anyone can reconstruct what happened.

Unit normalization

Plain‑English definitions

• Canonical unit: The single “home” unit you use for a test across the study (for example, glucose in mmol/L).
• Inbound whitelist: The set of units you promise to accept for that test (for example, glucose accepts mg/dL or mmol/L; anything else is rejected or flagged).
• Versioned conversion algorithm: A named rule that turns a value in an allowed inbound unit into the canonical unit. The name includes a version (for example, glucose_to_mmol_v1), so you can prove which logic you used. A published version is never edited: a changed factor or rounding rule becomes v2, each stored value keeps the version that produced it, and re‑deriving old values under v2 is a controlled backfill.
• Rounding rule: The one way you round after conversion (for example, round half‑away from zero to 1 decimal place).

What “good” looks like

• One canonical unit per test. Declare it up front in a Unit Policy table.

• Document accepted inbound units per test and how each is handled (convert or keep as‑is).

• Convert once, round once. Perform the conversion, then apply the rounding rule a single time.

• Store both raw and canonical (they map to SDTM’s original and standardized results, for example LBORRES/LBORRESU and LBSTRESN/LBSTRESU):

  • value_raw, unit_raw
  • value_canonical, unit_canonical
  • conversion_algorithm_name and conversion_algorithm_version

• Convert reference ranges too. Convert the lab’s normal‑range limits with the same algorithm and version, so out‑of‑range flags compare like with like.

• No silent changes. If an inbound unit is not on the whitelist, do not convert and do not change the unit yourself. In a vendor file it fails the conformance gate and the vendor re‑delivers; in site‑entered data (for example, a local lab), DM queries the site, which confirms or corrects the entry. A confirmed new unit is converted only after it is added to the Unit Policy (whitelist and algorithm version) through change control.

Unit Policy (example fragment)

Tip: Even when no numeric change occurs (identity), still record the algorithm name and version. That makes audits simple and consistent.

Rounding policy (be explicit)

• Use round half‑away from zero (example: 2.25 → 2.3; −2.25 → −2.3). Many tools round half to even by default (Python’s round(2.25, 1) gives 2.2), and binary floating point stores some values slightly low (1.005 is stored as 1.00499…), so implement the rule explicitly, for example with decimal arithmetic, and test it on halfway values.
• Specify precision per test (for example, glucose 1 decimal; sodium integer).
• Never round more than once.

Time zones & timestamp policy

What to capture at the source

• Local timestamp as recorded at the device/site (for clinical context).
• Offset from UTC at the moment of capture (in minutes, for example, −300 for UTC−5).
• Time zone name from the IANA time zone database (for example, America/New_York) to disambiguate daylight saving transitions.
• Data origin (device ID, lab system, site code) for traceability.

Canonical storage pattern

• Canonical time: store in UTC.
• Keep context: alongside UTC, store tz_offset_minutes, timezone_name, and the original timestamp_local_raw.

Fields to keep together

• timestamp_utc
• timestamp_local_raw
• tz_offset_minutes
• timezone_name
• time_derivation_algorithm_name + version (for example, local_to_utc_v2)
• cut_applied (yes/no) and cut_window_label (for batch/vendor cut rules)

Daylight saving time (DST) and cut rules

• DST‑aware conversion: Keep the time zone name (not only a numeric offset) so the system knows when DST starts/ends, can check each captured offset, and can derive the offset when none was captured.
• Ambiguous or missing hours: During “fall back,” 01:30 can occur twice; the captured offset tells the two apart, and the time zone name lets you check that each offset is valid for that date. During “spring forward,” 02:xx may not exist; devices often shift to 03:00—document this behavior, and query any recorded 02:xx time with the data’s owner.
• Batch cuts: If a vendor’s daily delivery (a cut) holds records up to a data cut‑off (for example, 00:00 UTC), store which records fell before/after the cut‑off, judged on UTC, for reproducible extracts.

Implementation flow

1. Inventory all tests and identify common inbound units. Draft the Unit Policy.
2. Write and validate conversion functions. Name them and set versions. Record rounding per test.
3. Normalize in staging. Persist raw and canonical values side‑by‑side with algorithm metadata.
4. Standardize timestamps. Convert local to UTC with the offset captured at the source (UTC = local time − offset, so −240 adds 4 hours); save local time, offset, and time zone with the UTC value. If no offset was captured, derive it from the time zone name, and query local times in the repeated “fall back” hour or the skipped “spring forward” hour with the data’s owner (site or vendor) instead of guessing.
5. Quality checks. Run delta listings that show raw vs canonical, including edge cases (very low/high values, negative values if applicable, DST boundaries).
6. Governance. Publish the policy; update versions through change control; backfill only when justified and traceable.

Worked examples (CV trial)

These examples are self‑contained. You can walk through them without reading other sections.

Example 1 — Lab glucose conversion (mg/dL → mmol/L)

Context: Canonical unit for glucose is mmol/L. Accepted inbound units: mg/dL and mmol/L.

Rounding: 1 decimal, round half‑away from zero.

Algorithm: glucose_to_mmol_v1 (applies only when inbound is mg/dL).

Inbound record

• value_raw: 125
• unit_raw: mg/dL
• specimen_collected_local: 2025‑06‑14 09:12
• timezone_name: America/New_York
• tz_offset_minutes: −240 (DST in effect)

Processing

• Convert mg/dL to mmol/L by dividing by 18 (from glucose’s molar mass, 180.16 g/mol). The factor is part of v1: switching to 18.016 would make v2, even though 125 mg/dL still gives 6.9 mmol/L.
• 125 mg/dL becomes 6.944… mmol/L → 6.9 mmol/L after rounding once.

Stored output (key fields)

• value_raw: 125
• unit_raw: mg/dL
• value_canonical: 6.9
• unit_canonical: mmol/L
• conversion_algorithm_name: glucose_to_mmol
• conversion_algorithm_version: v1
• timestamp_local_raw: 2025‑06‑14 09:12
• timestamp_utc: 2025‑06‑14 13:12
• tz_offset_minutes: −240
• timezone_name: America/New_York
• time_derivation_algorithm_name: local_to_utc
• time_derivation_algorithm_version: v2

Example 2 — Sodium with identity conversion (mEq/L = mmol/L)

Context: For sodium, mEq/L and mmol/L are numerically equivalent for monovalent ions. Canonical unit is mmol/L.

Rounding: integer (0 decimals).

Algorithm: sodium_identity_v1 (records the decision even though the number does not change).

Inbound record

• value_raw: 138
• unit_raw: mEq/L
• specimen_collected_local: 2025‑01‑15 23:03
• timezone_name: America/New_York
• tz_offset_minutes: −300 (standard time)

Processing

• Keep the same numeric value; set canonical unit to mmol/L; round to an integer (already integer).

Stored output (key fields)

• value_raw: 138
• unit_raw: mEq/L
• value_canonical: 138
• unit_canonical: mmol/L
• conversion_algorithm_name: sodium_identity
• conversion_algorithm_version: v1
• timestamp_utc: 2025‑01‑16 04:03
• Local and time zone fields as received.

Example 3 — Device timestamp at DST “fall back”

Context: On 2025‑11‑02 in America/New_York, clocks move from DST (UTC−4) to standard time (UTC−5). The hour from 01:00 to 02:00 occurs twice.

Inbound device readings (two distinct moments with the same local clock time)

1. timestamp_local_raw: 2025‑11‑02 01:30, tz_offset_minutes: −240, timezone_name: America/New_York
2. timestamp_local_raw: 2025‑11‑02 01:30, tz_offset_minutes: −300, timezone_name: America/New_York

Processing

• Convert using the offset in force at capture:

  • Record 1 → timestamp_utc: 2025‑11‑02 05:30
  • Record 2 → timestamp_utc: 2025‑11‑02 06:30

Why it matters

• Without the offset, both would look identical and could be deduplicated incorrectly or mis‑sequenced. The offset is what tells them apart (both carry the same time zone name); the name lets you check that each offset is valid for that date.

Example 4 — Vendor daily cut at 00:00 UTC

Context: A wearable vendor’s daily batches have a data cut‑off at 00:00 UTC: each batch holds one UTC day and is labeled with that date. Batch membership is decided on canonical UTC.

Inbound

• Reading at site: timestamp_local_raw: 2025‑07‑01 23:58, timezone_name: America/Chicago, tz_offset_minutes: −300 (DST).
• Converted to UTC → 2025‑07‑02 04:58 (23:58 + 5 hours). That is after the July 1 batch’s cut‑off (2025‑07‑02 00:00 UTC), so the reading is in the July 2 batch (cut‑off 2025‑07‑03 00:00 UTC), although its local date is July 1.
• For a day‑based diary or visit window, the reading still counts on July 1 (its local date), unless the protocol or SAP defines windows in UTC.
• Persist cut_applied: yes and cut_window_label: 2025‑07‑02.

Pitfalls & misconceptions

• Overwriting raw values. Never replace the original. Keep raw and canonical side‑by‑side with the algorithm and version used.
• Multiple rounds of rounding. Rounding early and then again later changes results: 2.249 rounded to 2 decimals (2.25) and then to 1 gives 2.3; rounded once to 1 decimal it is 2.2. Convert once and round once.
• Assuming one time zone. Sites, labs, and devices may differ. Always capture timezone_name and tz_offset_minutes.
• Offset‑only storage. Storing only “UTC−5” loses DST context. Keep the time zone name too.
• Silent unit drift. If you do not whitelist units, a device that switches units can quietly corrupt analysis. Whitelist and validate.
• Hidden identity conversions. Even when no numeric change occurs, still record the identity algorithm for traceability.

Practice checklist

• Publish a Unit Policy: canonical unit per test, accepted inbound units, rounding rules.
• Implement versioned conversion functions; test with edge cases and document examples.
• Store raw + canonical + algorithm name/version for both units and time.
• Convert timestamps to UTC and persist local time, offset, and time zone name.
• Document DST and vendor cut behavior; keep cut_applied and labels.
• Run delta listings that show before/after values around DST and unit boundaries.
• Put unit whitelists and timestamp formats in the transfer spec; enforce at the conformance gate.

Key points to remember

• Normalize before reconciliation, window checks, or analytics.
• Declare one canonical unit per test and maintain an accepted‑unit whitelist; reject/flag anything outside it.
• Convert once, round once, using the study’s defined rounding rule/precision; keep the originals.
• Record full traceability for units: store raw + canonical values/units and the conversion algorithm name + version—including identity conversions.
• Capture full time context at source: local timestamp + UTC offset + time zone name (and origin/device); store UTC as canonical.
• Record time derivation metadata: time conversion algorithm name + version alongside timestamp_utc and the preserved local fields.
• Be DST‑aware: the captured offset fixes the instant (it separates the repeated “fall back” hour); the time zone name lets you check offsets and spot skipped “spring forward” times. Store both; never rely on either alone.
• Evaluate event order, elapsed‑time windows and batch logic on canonical UTC; assign day‑based visit windows from the local date unless the protocol or SAP says otherwise; persist vendor cut details (cut_applied, cut_window_label).
• No silent changes: if an inbound unit isn’t on the whitelist, don’t convert—send it back to its owner: a vendor file fails the gate, a site entry gets a query.
• Put unit whitelists, rounding rules, and timestamp formats in the transfer spec and enforce them at the conformance gate.
• Use delta listings to verify raw→canonical results and to probe edge cases (extremes, negatives, DST boundaries).
• Govern changes: version algorithms and policies, publish updates via change control, and ensure backfills are justified and traceable.