ezmsg.baseproc.util.segment#

Detect breaks in a stream’s time base and number its segments.

A source stamps every message with segment, and changes the value on the first message that does not continue the previous one. Stateful processors reset when they see it change (BaseStatefulProcessor._reset_segment), so gap detection happens once, at the source, instead of in every processor.

Functions

end_of_stream(message)[source]#

The message a source emits after message, its last, to say its stream has ended.

For a source that knows: a file reader at the end of the file, or at the end of a section before a gap it knows about. Downstream units see the new segment and flush what they hold now, instead of when (or if) more data arrives.

The result has message’s layout, config_id, key and attrs, but no samples along its stream dimension. A LinearAxis there is moved to just after message’s last sample; a CoordinateAxis there is emptied. Its segment is message.segment + 1, and nothing else may share it: a source keeping its own count continues it from the result, so that its next message, even data that continues in time, takes result.segment + 1 or more. A source using SegmentTracker calls SegmentTracker.end_of_stream() instead.

Parameters:

message (AxisArray) – The last message the source emitted, or a template whose stream axis already starts where the next sample would have.

Return type:

AxisArray

find_gaps(timestamps, period, tolerance)[source]#

Indices of the samples that do not follow the sample before them.

Sample i follows sample i - 1 when timestamps[i] - timestamps[i - 1] is within tolerance of period. A step backward breaks the run just as a jump forward does. Use the result to split a chunk before it is given a LinearAxis, which cannot describe a gap: np.split(data, find_gaps(...)).

Parameters:
  • timestamps (ArrayLike) – Per-sample times, in seconds.

  • period (float) – Expected time between samples, in seconds (1 / fs).

  • tolerance (float) – Largest deviation from period, in seconds, that is still continuous. Size it to the jitter of the timestamps, not to the sample period: a slip of a sample or two costs a filter far less than the reset a reported gap causes.

Return type:

NDArray[int_]

Returns:

Sorted indices in [1, len(timestamps)); empty when the chunk is continuous.

Classes

class SegmentTracker(tolerance)[source]#

Bases: object

Numbers the segments of one stream from the offsets of its chunks.

Counting starts at 0 and goes up by one per break.

Call update() once per chunk, in order, and put the result in the message’s segment. Each chunk is compared with the one before it only, so neither clock drift nor one late chunk accumulates into a false gap. Call bump() for a break the offsets cannot show, such as a device reopened with known data loss. A changed configuration alone is not a break: it changes config_id, not the segment. End a segment with end_of_stream(), not the module function, so that the next chunk is numbered after the marker.

Parameters:

tolerance (float) – Largest difference, in seconds, between a chunk’s offset and where the previous chunk ended that is still continuous. Two clock-stamped chunks can each be off by the source’s jitter, so allow twice that.

__init__(tolerance)[source]#
Parameters:

tolerance (float)

Return type:

None

update(offset, n_samples, gain)[source]#

Account for a chunk of n_samples starting at offset; return its segment.

Return type:

int

Parameters:
bump()[source]#

Start a new segment regardless of the next chunk’s offset; return it.

Return type:

int

end_of_stream(message)[source]#

end_of_stream() of message, numbered by this tracker.

The marker takes the next segment, and the next chunk (or bump()) the one after it, whatever its offset: the marker says its segment has ended.

Return type:

AxisArray

Parameters:

message (AxisArray)