ezmsg.nwb.clockmodel#
Clock-jitter reconstruction for NWB timestamps.
Some acquisition streams carry per-sample timestamps that are correct in the
aggregate (averaged over enough samples they give the true sample rate – note
the median does not, when stamps are quantised; see
infer_nominal_rate()) but jittery sample-to-sample: the
recording pipeline stamps each sample at arrival time
rather than from the ADC clock, so neighbouring intervals swing by roughly a
whole sample period and occasionally hop backwards. The iterator’s gap-splitter
then shatters such a stream into tens of thousands of tiny gap-free runs – one
recorded file here fragments an 8.2 M-sample stream into ~285 k messages.
The fix is to replace those timestamps with a smoothed, strictly monotone version that preserves the true (slowly drifting, non-linear) clock while discarding the jitter. Two regimes:
Self-fit – fit a monotone piecewise-linear curve of
dataset_timevssample_indexand evaluate it at every index. Removes jitter and gaps on a single stream with no help from siblings.Shared-clock (PTP group) – when several streams share one device clock (e.g. two PTP-synchronised acquisition hubs), the device -> dataset clock conversion is one smooth function common to all of them. Estimate that conversion
Cfrom the cleanest member, clean the target’s device time from its own sample index, then map throughC. A jittery stream is thereby rescued by a clean sibling; only the target’s fixed per-stream latency is taken from the target itself (via a robust median anchor).
Everything here is pure numpy and file-agnostic so it can be driven both at read
time (see NWBSlicer) and by an offline tool that
rewrites corrected timestamps into an NWB file. The fit family is pluggable via
method; "piecewise_linear" is the only one implemented today and needs
no dependency beyond numpy. A monotone cubic (PCHIP) variant can drop in later
behind an optional scipy extra without changing callers.
Module Attributes
Number of knots for the piecewise-linear fit. |
|
Percentile of the interval deviation used as the jitter-envelope estimate. |
|
sample timestamping jitter is heavy-tailed (on real hardware the worst interval can be tens of times the 99.9th percentile), and a small persistent clock-sync step is not a data gap -- it should be smoothed, not preserved. |
|
Floor for the auto threshold's margin above one sample period (s), so a near-perfect stream doesn't get a threshold so tight that sub-sample noise reads as a gap. |
|
Half-window (samples) for the net-displacement check that separates a real gap (a sustained level shift) from a lone jitter spike (self-cancelling). |
|
A group member is reconstructed via the shared clock model only when its own jitter exceeds the cleanest member's by at least this factor. |
|
a target must also be jittier than this before a sibling's clock is borrowed. |
|
Where reconstructed timestamp vectors are cached across file opens, alongside remfile's |
Functions
- cache_lookup(signature, key, params)[source]#
Return a cached reconstruction, or
Noneon miss / unreadable cache.signatureidentifies the source file (path + size + mtime);keythe stream;paramsthe fit parameters. Any read error degrades to a miss so a corrupt cache entry never breaks a load.
- cache_store(signature, key, params, arr)[source]#
Persist a reconstruction. Failures are swallowed – caching is best-effort.
- find_real_gaps(times, period, threshold, win=16)[source]#
Left-edge indices of genuine data gaps in
times.A candidate is any interval exceeding
thresholdseconds. Each is then verified as a sustained level shift, not a lone jitter spike: the mean detrended level (time - period * index) over a window just after the jump must sit at least half a threshold above the window just before. A jitter spike self-cancels (its neighbours compensate), so its before/after levels match and it is rejected; a real gap shifts every later sample and survives.
- fit_map(x, y, n_knots=40, method='piecewise_linear')[source]#
Fit a monotone
ClockModelmappingx -> y.x/yare index-ordered paired samples. Only"piecewise_linear"is implemented; other names raise so a typo fails loudly rather than silently using the default.- Return type:
- Parameters:
- group_clocks(members, center_tol=1.0, rate_tol=0.01)[source]#
Partition streams into groups that share a device clock.
Each member is
{"key", "t0", "tend", "n"}– only the device-time endpoints and sample count, so the caller can probe them cheaply (two scalar reads per stream) without materialising millions of timestamps.Two streams are judged to share a clock when their device-time spans overlap, their nominal rates agree to within
rate_tol(fractional), and their span centres agree to withincenter_tolseconds. PTP-synchronised devices land within a millisecond; an unrelated clock (a different epoch, or a file-relative axis near zero) is off by seconds or more, so this separates cleanly while staying conservative – a missed grouping merely falls back to per-stream self-fit, which is always safe.
- reconstruct_group(members, n_knots=40, method='piecewise_linear', gap_threshold_s=None)[source]#
Reconstruct every member of a shared-clock group.
membersis a list of{"key", "device", "dataset"}dicts (devicemay beNonefor a lone stream). The cleanest member – least jitter about its own self-fit – is dejittered directly and can supply the clock modelCfor the rest. A member is reconstructed via that shared model only when it carries device timestamps and is jitterier than bothSHARED_MODEL_JITTER_RATIOtimes the clean source andSHARED_MODEL_JITTER_FLOOR_PERIODSsample periods; otherwise it is self-fit, which never imposes a sibling’s clock. A single-member group falls back to a self-fit. Genuine data gaps are preserved per member (gap_threshold_sforwarded;None= auto), and segment joins are kept monotone (_enforce_join_monotonicity()). Returns{key: reconstructed_dataset_times}.This is the file-agnostic entry point shared by the read-time slicer and any offline NWB-correcting tool.
- reconstruct_self(dataset_times, n_knots=40, method='piecewise_linear', gap_threshold_s=None)[source]#
Dejitter a stream from its own timestamps: fit
index -> dataset_time.Returns a monotone, jitter-free timestamp vector the same length as
dataset_timesthat tracks the stream’s slow clock drift. Genuine data gaps are preserved: the series is split at gaps detected on its own timestamps (seefind_real_gaps();gap_threshold_sis the jump threshold in seconds, orNoneto auto-derive per stream), and each segment is fit independently so the true jump between segments survives. Used when a stream has no clean sibling to borrow a clock model from.
Reconstruct a target stream’s dataset timestamps via a shared clock.
clean_device -> clean_datasetdefines the smooth device->dataset clock conversionC(estimated from the cleanest group member). The target’s own device time is cleaned from its sample index (index -> device) and pushed throughC. Whentarget_datasetis supplied, a robust median offset re-anchors the result onto the target’s own timeline, preserving the target’s fixed per-stream latency while still borrowing the drift shape from the clean sibling.Genuine gaps in the target’s device clock are preserved: the stream is split at gaps detected on
target_deviceand each segment is cleaned and re-anchored on its own, so the true jump survives while jitter is smoothed.Cstays global – it is continuous over device time regardless of gaps.Returns a monotone vector the length of
target_device.
Classes
- class ClockModel(knots_x, knots_y, method='piecewise_linear')[source]#
Bases:
objectA monotone map from one time base to another, evaluated by interpolation.
knots_xmust be strictly increasing (see_binned_knots()). Inputs outside the knot range clamp to the endpoint values – appropriate here, where the domain is always the fitted stream’s own span.