ezmsg.nwb.slicer#

NWBSlicer: Shared NWB file handling for iterators and clock-driven producers.

Functions

find_gaps(timestamps, gain, gap_tol)[source]#

Indices i where a gap separates sample i from sample i + 1.

A gap is an inter-sample interval exceeding (1 + gap_tol) * gain — i.e. the timestamps jump by more than (1 + gap_tol) nominal sample periods. Returns the left-edge indices as an int array (empty when there are fewer than two samples, no usable gain, or no gaps). Shared by the iterator (which splits chunks at gaps) and the slicer’s read_by_time (which switches to a CoordinateAxis when a window spans one).

Return type:

ndarray

Parameters:
infer_nominal_rate(dts)[source]#

Sample rate from per-sample intervals: the mean of the non-outlier ones.

Deliberately not 1 / median(dt), which is biased whenever the timestamps are quantised more coarsely than the jitter they carry. The median snaps to whichever grid point holds the most intervals; the true period generally falls between two grid points, so the estimate lands on one of them and stays there no matter how many samples are averaged. On real 30 kHz recordings whose stamps step in 4e-8 s, intervals split between 3.332e-5 and 3.336e-5 and the median reports 30012.005 Hz for a stream delivering 30000.104 – 0.04%, which is ~0.8 ms of misplacement by the end of a 2 s chunk. The mean averages across the grid and recovers the true period.

The trim is what makes the mean usable: a raw mean is dragged by real gaps (one 10 ms hole in the example above pulls the estimate to 29970 Hz). Keeping only intervals within [RATE_TRIM_LO, RATE_TRIM_HI] * median drops gaps, backward steps from clock sync, and the forward excursions that pair with them, leaving the intervals that actually describe the sample period.

Returns 0.0 for input from which no rate can be estimated; callers treat 0.0 as “irregular, use a CoordinateAxis”.

Return type:

float

Parameters:

dts (ndarray)

is_zarr_store(filepath)[source]#

An NWB file stored as an hdmf-zarr store: a .zarr path (local or URL), or a local directory holding a Zarr root.

Return type:

bool

Parameters:

filepath (PathLike | str)

Classes

class NWBSlicer(filepath, reference_clock=ReferenceClockType.SYSTEM, reref_now=False, stream_keys=None, rdcc_nbytes=67108864, rdcc_nslots=10007, dejitter=True, clock_groups=None, dejitter_cache=True, real_gap_threshold=None, structured_ch_axis=False, storage_options=None)[source]#

Bases: object

Shared NWB file handling: open, discover streams, and slice data.

Parameters:

  • filepath (Union[PathLike, str]) – Path to NWB file (local or remote URL).

  • reference_clock (ReferenceClockType) – Which reference clock to use for timestamps.

  • reref_now (bool) – If True, re-reference timestamps to the current time.

  • stream_keys (Optional[list[str]]) – Optional filter for which streams to discover.

  • rdcc_nbytes (int) – HDF5 raw data chunk cache size in bytes. Larger values reduce re-decoding when consecutive reads cross HDF5 chunk boundaries.

  • rdcc_nslots (int) – HDF5 raw data chunk cache slot count. Should be a prime number roughly 10x the number of chunks expected to be hot at once.

  • dejitter (bool) – When True (default), reconstruct smoothed, strictly-monotone timestamps for continuous streams that carry a paired *_device_ts sibling (the structural mark of a re-timestamped fixed-rate acquisition). This collapses the per-sample clock jitter that would otherwise shatter such a stream into thousands of gap-split chunks. Streams that share a device clock (PTP-synchronised hubs) are detected automatically and the cleanest one supplies the clock model for the rest; see clockmodel. Set False to read raw timestamps unchanged.

  • clock_groups (Optional[list[list[str]]]) – Override auto clock-group detection. None auto-detects; a list of key-lists forces exactly those groupings (unlisted correctable streams become singleton self-fits). Ignored when dejitter is False.

  • dejitter_cache (bool) – Cache reconstructed timestamp vectors on disk (default True) so repeated opens of the same file – e.g. multi-pass training – reconstruct once and reload thereafter.

  • real_gap_threshold (Optional[float]) – Guard against smoothing over genuine data gaps (dropped packets). None (default) auto-derives a per-stream jump threshold from that stream’s jitter envelope, so only jumps unambiguously above jitter are treated as real gaps; a float sets an explicit threshold in seconds. Real gaps are preserved (each inter-gap segment is reconstructed independently) so the iterator still splits a chunk there. Pass float("inf") to disable the guard and smooth everything. Ignored when dejitter is False.

  • structured_ch_axis (bool) – Build the ch axis as a structured CHANNEL_DTYPE array – position, label, bank, headstage, array identity – from the electrodes table, matching what a live acquisition source emits, so a graph replaying a file sees the axis it was written against. Default False: the columns this reads are an acquisition-stack convention rather than part of the NWB schema, and most files carry none of them. Raises for a file that has none rather than inventing geometry. See electrodes.

  • storage_options (Optional[dict])

Note

Data comes back as the file stores it – integer ADC counts, typically, not the unit the file declares. Every message carries what it takes to convert under its nwb_scaling_* attrs; put a NWBScalingUnit in the graph (or call NWBScalingTransformer directly) to apply it. Doing it there rather than here is both cleaner and marginally faster – see scaling.

disk_cache = <remfile.DiskCache.DiskCache object>#
DEFAULT_RDCC_NBYTES: ClassVar[int] = 67108864#
DEFAULT_RDCC_NSLOTS: ClassVar[int] = 10007#
__init__(filepath, reference_clock=ReferenceClockType.SYSTEM, reref_now=False, stream_keys=None, rdcc_nbytes=67108864, rdcc_nslots=10007, dejitter=True, clock_groups=None, dejitter_cache=True, real_gap_threshold=None, structured_ch_axis=False, storage_options=None)[source]#
Parameters:
property ts_off: float#

Timestamp offset applied to all timestamps.

property stream_names: list[str]#

Names of all discovered streams.

property start_time: float#

Global start time across all streams (with ts_off applied).

property stop_time: float#

Global stop time across all streams (with ts_off applied).

get_stream_info(key)[source]#

Return the StreamInfo for a given stream key.

Return type:

StreamInfo

Parameters:

key (str)

read_by_index(stream_key, start_idx, stop_idx)[source]#

Read continuous data by sample index range.

For rate-only continuous streams. Returns an AxisArray with data[start_idx:stop_idx].

Return type:

AxisArray

Parameters:

  • stream_key (str)

  • start_idx (int)

  • stop_idx (int)

read_by_time(stream_key, t_start, t_end, gap_tol=0.5)[source]#

Read data by time window [t_start, t_end).

For timestamped continuous streams and event/interval tables. t_start and t_end are in the same reference frame as the stored timestamps (i.e., file-relative, before ts_off).

gap_tol is the gap threshold as a fraction of the nominal sample period (see find_gaps()). When a continuous window’s samples are regularly spaced the result carries a cheap LinearAxis; when the window straddles a gap, the time axis becomes a CoordinateAxis carrying the true per-sample timestamps so the gap is represented faithfully instead of silently flattening post-gap samples onto a uniform axis. Ignored for event streams.

Return type:

AxisArray

Parameters:
close()[source]#

Close the underlying HDF5 file.

getattr rather than self._io: __del__ also runs on an object whose __init__ raised (an unusable argument combination, say), and that object never reached the assignment. Reading the attribute directly turns the real error into an AttributeError raised from a destructor, where it surfaces as an unrelated warning at an unrelated time.

Return type:

None

class StreamInfo(dset, template, fs=0.0, t0=0.0, n_samples=0, timestamps=None, has_timestamps=False, is_event=False, table_ref=None, scaling=None)[source]#

Bases: object

Per-stream metadata discovered from an NWB file.

Parameters:
dset: Any#

h5py dataset or numpy array containing the data.

template: AxisArray#

AxisArray with zero-length time dim, used as a template for output messages.

fs: float = 0.0#

Nominal sample rate (0 for events/irregular).

t0: float = 0.0#

Starting time (relative to file, before ts_off adjustment).

n_samples: int = 0#

Total sample count in the stream.

timestamps: Any = None#

h5py dataset or numpy array of explicit timestamps, or None for rate-only.

has_timestamps: bool = False#

Whether the file has explicit per-sample timestamps for this stream.

is_event: bool = False#

Whether this stream is an interval table (events) vs timeseries.

table_ref: Any = None#

Reference to the interval table (events only).

scaling: StreamScaling | None = None#

What this stream’s stored samples mean – gain, offset, and the unit they would be in once those are applied.

The file’s own numbers verbatim, including its unit string, which is frequently wrong; interpreting it here would hide that. Correcting and applying it is convert’s job. Copied onto every message’s nwb_scaling_* attrs. None only for text streams, which have no scaling to describe.

__init__(dset, template, fs=0.0, t0=0.0, n_samples=0, timestamps=None, has_timestamps=False, is_event=False, table_ref=None, scaling=None)#
Parameters:
Return type:

None