Neo RawIO

Neo RawIO API

For performance and memory consumption reasons a new layer has been added to Neo.

In brief:
  • neo.io is the user-oriented read/write layer. Reading consists of getting a tree of Neo objects from a data source (file, url, or directory). When reading, all Neo objects are correctly scaled to the correct units. Writing consists of making a set of Neo objects persistent in a file format.
  • neo.rawio is a low-level layer for reading data only. Reading consists of getting NumPy buffers (often int16/int64) of signals/spikes/events. Scaling to real values (microV, times, …) is done in a second step. Here the underlying objects must be consistent across Blocks and Segments for a given data source.

The neo.rawio API has been added for developers. The neo.rawio is close to what could be a C API for reading data but in Python/NumPy.

Not all IOs are implemented in neo.rawio but all classes implemented in neo.rawio are also available in neo.io.

Possible uses of the neo.rawio API are:
  • fast reading chunks of signals in int16 and do the scaling of units (uV) on a GPU while scaling the zoom. This should improve bandwidth HD to RAM and RAM to GPU memory.
  • load only some small chunk of data for heavy computations. For instance the spike sorting module tridesclous does this.
The neo.rawio API is less flexible than neo.io and has some limitations:
  • read-only
  • AnalogSignals must have the same characteristics across all Blocks and Segments: sampling_rate, shape[1], dtype
  • AnalogSignals should all have the same value of sampling_rate, otherwise they won’t be read at the same time.
  • Units must have SpikeTrain event if empty across all Block and Segment
  • Epoch and Event are processed the same way (with durations=None for Event).
For an intuitive comparison of neo.io and neo.rawio see:
  • example/read_file_neo_io.py
  • example/read_file_neo_rawio.py

One speculative benefit of the neo.rawio API should be that a developer should be able to code a new RawIO class with little knowledge of the Neo tree of objects or of the quantities package.

Basic usage

First create a reader from a class:

>>> from neo.rawio import PlexonRawIO
>>> reader = PlexonRawIO(filename='File_plexon_3.plx')

Then browse the internal header and display information:

>>> reader.parse_header()
>>> print(reader)
PlexonRawIO: File_plexon_3.plx
nb_block: 1
nb_segment:  [1]
signal_channels: [V1]
spike_channels: [Wspk1u, Wspk2u, Wspk4u, Wspk5u ... Wspk29u Wspk30u Wspk31u Wspk32u]
event_channels: []

You get the number of blocks and segments per block. You have information about channels: signal_channels, spike_channels, event_channels.

All this information is internally available in the header dict:

>>> for k, v in reader.header.items():
...    print(k, v)
signal_channels [('V1', 0,  1000., 'int16', '',  2.44140625,  0., 0)]
event_channels []
nb_segment [1]
nb_block 1
spike_channels [('Wspk1u', 'ch1#0', '',  0.00146484,  0., 0,  30000.)
('Wspk2u', 'ch2#0', '',  0.00146484,  0., 0,  30000.)
...

Read signal chunks of data and scale them:

>>> channel_indexes = None  #could be channel_indexes = [0]
>>> raw_sigs = reader.get_analogsignal_chunk(block_index=0, seg_index=0,
                    i_start=1024, i_stop=2048, channel_indexes=channel_indexes)
>>> float_sigs = reader.rescale_signal_raw_to_float(raw_sigs, dtype='float64')
>>> sampling_rate = reader.get_signal_sampling_rate()
>>> t_start = reader.get_signal_t_start(block_index=0, seg_index=0)
>>> units =reader.header['signal_channels'][0]['units']
>>> print(raw_sigs.shape, raw_sigs.dtype)
>>> print(float_sigs.shape, float_sigs.dtype)
>>> print(sampling_rate, t_start, units)
(1024, 1) int16
(1024, 1) float64
1000.0 0.0 V

There are 3 ways to select a subset of channels: by index (0 based), by id or by name. By index is unambiguous 0 to n-1 (included), whereas for some IOs channel_names (and sometimes channel_ids) have no guarantees to be unique. In such cases, using names or ids may raise an error.

A selected subset of channels which is passed to get_analog_signal_chunk, get_analog_signal_size, or get_analog_signal_t_start has the additional restriction that all such channels must have the same t_start and signal_size.

Such subsets of channels may be available in specific RawIOs by using the get_group_signal_channel_indexes method, if the RawIO has defined separate group_ids for each group with those common characteristics.

Example with BlackrockRawIO for the file FileSpec2.3001:

>>> raw_sigs = reader.get_analogsignal_chunk(channel_indexes=None) #Take all channels
>>> raw_sigs1 = reader.get_analogsignal_chunk(channel_indexes=[0,  2, 4])) #Take 0 2 and 4
>>> raw_sigs2 = reader.get_analogsignal_chunk(channel_ids=[1, 3, 5]) # Same but with there id (1 based)
>>> raw_sigs3 = reader.get_analogsignal_chunk(channel_names=['chan1', 'chan3', 'chan5'])) # Same but with there name
print(raw_sigs1.shape[1], raw_sigs2.shape[1], raw_sigs3.shape[1])
3, 3, 3

Inspect units channel. Each channel gives a SpikeTrain for each Segment. Note that for many formats a physical channel can have several units after spike sorting. So the nb_unit could be more than physical channel or signal channels.

>>> nb_unit = reader.spike_channels_count()
>>> print('nb_unit', nb_unit)
nb_unit 30
>>> for unit_index in range(nb_unit):
...     nb_spike = reader.spike_count(block_index=0, seg_index=0, unit_index=unit_index)
...     print('unit_index', unit_index, 'nb_spike', nb_spike)
unit_index 0 nb_spike 701
unit_index 1 nb_spike 716
unit_index 2 nb_spike 69
unit_index 3 nb_spike 12
unit_index 4 nb_spike 95
unit_index 5 nb_spike 37
unit_index 6 nb_spike 25
unit_index 7 nb_spike 15
unit_index 8 nb_spike 33
...

Get spike timestamps only between 0 and 10 seconds and convert them to spike times:

>>> spike_timestamps = reader.spike_timestamps(block_index=0, seg_index=0, unit_index=0,
                    t_start=0., t_stop=10.)
>>> print(spike_timestamps.shape, spike_timestamps.dtype, spike_timestamps[:5])
(424,) int64 [  90  420  708 1020 1310]
>>> spike_times =  reader.rescale_spike_timestamp( spike_timestamps, dtype='float64')
>>> print(spike_times.shape, spike_times.dtype, spike_times[:5])
(424,) float64 [ 0.003       0.014       0.0236      0.034       0.04366667]

Get spike waveforms between 0 and 10 s:

>>> raw_waveforms = reader.spike_raw_waveforms(  block_index=0, seg_index=0, unit_index=0,
                    t_start=0., t_stop=10.)
>>> print(raw_waveforms.shape, raw_waveforms.dtype, raw_waveforms[0,0,:4])
(424, 1, 64) int16 [-449 -206   34   40]
>>> float_waveforms = reader.rescale_waveforms_to_float(raw_waveforms, dtype='float32', unit_index=0)
>>> print(float_waveforms.shape, float_waveforms.dtype, float_waveforms[0,0,:4])
(424, 1, 64) float32 [-0.65771484 -0.30175781  0.04980469  0.05859375]

Count events per channel:

>>> reader = PlexonRawIO(filename='File_plexon_2.plx')
>>> reader.parse_header()
>>> nb_event_channel = reader.event_channels_count()
nb_event_channel 28
>>> print('nb_event_channel', nb_event_channel)
>>> for chan_index in range(nb_event_channel):
...     nb_event = reader.event_count(block_index=0, seg_index=0, event_channel_index=chan_index)
...     print('chan_index',chan_index, 'nb_event', nb_event)
chan_index 0 nb_event 1
chan_index 1 nb_event 0
chan_index 2 nb_event 0
chan_index 3 nb_event 0
...

Read event timestamps and times for chanindex=0 and with time limits (t_start=None, t_stop=None):

>>> ev_timestamps, ev_durations, ev_labels = reader.event_timestamps(block_index=0, seg_index=0, event_channel_index=0,
                    t_start=None, t_stop=None)
>>> print(ev_timestamps, ev_durations, ev_labels)
[1268] None ['0']
>>> ev_times = reader.rescale_event_timestamp(ev_timestamps, dtype='float64')
>>> print(ev_times)
[ 0.0317]

List of implemented formats

neo.rawio provides classes for reading with low level API electrophysiological data files.

neo.rawio.rawiolist provides a list of successfully imported rawio classes.

Functions:

neo.rawio.get_rawio_class(filename_or_dirname)

Return a neo.rawio class guess from file extention.

Classes:

class neo.rawio.AlphaOmegaRawIO(dirname='', lsx_files=None, prune_channels=True)

AlphaOmega MPX file format 4 reader. Handles several segments.

A segment is a continuous record (when record starts/stops).

Only files in current dirname are loaded, subfolders are not explored.

Parameters:
  • dirname (str or Path-like) – folder from where to load the data
  • lsx_files (list of strings or None) – list of lsx files in dirname referencing mpx files to load (optional). If None (default), read all mpx files in dirname
  • prune_channels (bool) – if True removes the empty channels, defaults to True

Warning

Because channels must be gathered into coherent streams, channels names must be the default channel names in AlphaRS or Alpha LAB SNR software.

extensions = ['lsx', 'mpx']
class neo.rawio.AxographRawIO(filename, force_single_segment=False)

RawIO class for reading AxoGraph files (.axgd, .axgx)

Args:
filename (string):
File name of the AxoGraph file to read.
force_single_segment (bool):
Episodic files are normally read as multi-Segment Neo objects. This parameter can force AxographRawIO to put all signals into a single Segment. Default: False.
Example:
>>> import neo
>>> r = neo.rawio.AxographRawIO(filename=filename)
>>> r.parse_header()
>>> print(r)

>>> # get signals
>>> raw_chunk = r.get_analogsignal_chunk(
...     block_index=0, seg_index=0,
...     i_start=0, i_stop=1024,
...     channel_names=channel_names)
>>> float_chunk = r.rescale_signal_raw_to_float(
...     raw_chunk,
...     dtype='float64',
...     channel_names=channel_names)
>>> print(float_chunk)

>>> # get event markers
>>> ev_raw_times, _, ev_labels = r.get_event_timestamps(
...     event_channel_index=0)
>>> ev_times = r.rescale_event_timestamp(
...     ev_raw_times, dtype='float64')
>>> print([ev for ev in zip(ev_times, ev_labels)])

>>> # get interval bars
>>> ep_raw_times, ep_raw_durations, ep_labels = r.get_event_timestamps(
...     event_channel_index=1)
>>> ep_times = r.rescale_event_timestamp(
...     ep_raw_times, dtype='float64')
>>> ep_durations = r.rescale_epoch_duration(
...     ep_raw_durations, dtype='float64')
>>> print([ep for ep in zip(ep_times, ep_durations, ep_labels)])

>>> # get notes
>>> print(r.info['notes'])

>>> # get other miscellaneous info
>>> print(r.info)
extensions = ['axgd', 'axgx']
class neo.rawio.AxonaRawIO(filename)

Class for reading raw, continuous data from the Axona dacqUSB system: http://space-memory-navigation.org/DacqUSBFileFormats.pdf

The raw data is saved in .bin binary files with an accompanying .set file about the recording setup (see the above manual for details).

Usage:

import neo.rawio
r = neo.rawio.AxonaRawIO(filename=os.path.join(dir_name, base_filename))
r.parse_header()
print(r)
raw_chunk = r.get_analogsignal_chunk(block_index=0, seg_index=0,
                                     i_start=0, i_stop=1024,
                                     channel_names=channel_names)
float_chunk = reader.rescale_signal_raw_to_float(
    raw_chunk, dtype='float64',
    channel_indexes=[0, 3, 6]
)
extensions = ['bin', 'set', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32']
class neo.rawio.AxonRawIO(filename='')
extensions = ['abf']
class neo.rawio.BiocamRawIO(filename='')

Class for reading data from a Biocam h5 file.

Usage:
>>> import neo.rawio
>>> r = neo.rawio.BiocamRawIO(filename='biocam.h5')
>>> r.parse_header()
>>> print(r)
>>> raw_chunk = r.get_analogsignal_chunk(block_index=0, seg_index=0,
                                         i_start=0, i_stop=1024,
                                         channel_names=channel_names)
>>> float_chunk = r.rescale_signal_raw_to_float(raw_chunk, dtype='float64',
                                                channel_indexes=[0, 3, 6])
extensions = ['h5']
class neo.rawio.BlackrockRawIO(filename=None, nsx_override=None, nev_override=None, nsx_to_load=None, verbose=False)

Class for reading data in from a file set recorded by the Blackrock (Cerebus) recording system.

Upon initialization, the class is linked to the available set of Blackrock files.

Note: This routine will handle files according to specification 2.1, 2.2, and 2.3. Recording pauses that may occur in file specifications 2.2 and 2.3 are automatically extracted and the data set is split into different segments.

The Blackrock data format consists not of a single file, but a set of different files. This constructor associates itself with a set of files that constitute a common data set. By default, all files belonging to the file set have the same base name, but different extensions. However, by using the override parameters, individual filenames can be set.

Args:
filename (string):
File name (without extension) of the set of Blackrock files to associate with. Any .nsX or .nev, .sif, or .ccf extensions are ignored when parsing this parameter.
nsx_override (string):
File name of the .nsX files (without extension). If None, filename is used. Default: None.
nev_override (string):
File name of the .nev file (without extension). If None, filename is used. Default: None.
nsx_to_load (int, list, ‘max’, ‘all’ (=None)) default None:
IDs of nsX file from which to load data, e.g., if set to 5 only data from the ns5 file are loaded. If ‘all’, then all nsX will be loaded. Contrary to previous version of the IO (<0.7), nsx_to_load must be set at the init before parse_header().
Examples:
>>> reader = BlackrockRawIO(filename='FileSpec2.3001', nsx_to_load=5)
>>> reader.parse_header()
Inspect a set of file consisting of files FileSpec2.3001.ns5 and FileSpec2.3001.nev
>>> print(reader)
Display all informations about signal channels, units, segment size….
extensions = ['ns1', 'ns2', 'ns3', 'ns4', 'ns5', 'ns6', 'nev']
class neo.rawio.BrainVisionRawIO(filename='')
extensions = ['vhdr']
class neo.rawio.CedRawIO(filename='', take_ideal_sampling_rate=False)

Class for reading data from CED (Cambridge Electronic Design) spike2. This internally uses the sonpy package which is closed source.

This IO reads smr and smrx files

extensions = ['smr', 'smrx']
class neo.rawio.EDFRawIO(filename='')

Class for reading European Data Format files (EDF and EDF+). Currently only continuous EDF+ files (EDF+C) and original EDF files (EDF) are supported

Usage:
>>> import neo.rawio
>>> r = neo.rawio.EDFRawIO(filename='file.edf')
>>> r.parse_header()
>>> print(r)
>>> raw_chunk = r.get_analogsignal_chunk(block_index=0, seg_index=0,
                    i_start=0, i_stop=1024, stream_index=0, channel_indexes=range(10))
>>> float_chunk = reader.rescale_signal_raw_to_float(raw_chunk, dtype='float64',
                    channel_indexes=[0, 3, 6])
extensions = ['edf']
class neo.rawio.ElanRawIO(filename=None, entfile=None, posfile=None)
extensions = ['eeg']
class neo.rawio.IntanRawIO(filename='')
extensions = ['rhd', 'rhs']
class neo.rawio.MaxwellRawIO(filename='', rec_name=None)

Class for reading MaxOne or MaxTwo files.

extensions = ['h5']
class neo.rawio.MEArecRawIO(filename='')

Class for “reading” fake data from a MEArec file.

Usage:
>>> import neo.rawio
>>> r = neo.rawio.MEArecRawIO(filename='mearec.h5')
>>> r.parse_header()
>>> print(r)
>>> raw_chunk = r.get_analogsignal_chunk(block_index=0, seg_index=0,
                    i_start=0, i_stop=1024,  channel_names=channel_names)
>>> float_chunk = reader.rescale_signal_raw_to_float(raw_chunk, dtype='float64',
                    channel_indexes=[0, 3, 6])
>>> spike_timestamp = reader.spike_timestamps(unit_index=0, t_start=None, t_stop=None)
>>> spike_times = reader.rescale_spike_timestamp(spike_timestamp, 'float64')
extensions = ['h5']
class neo.rawio.MicromedRawIO(filename='')

Class for reading data from micromed (.trc).

extensions = ['trc', 'TRC']
class neo.rawio.NeuralynxRawIO(dirname='', filename='', exclude_filename=None, keep_original_times=False, **kargs)

Class for reading datasets recorded by Neuralynx.

This version works with rawmode of one-dir for a single directory of files or one-file for a single file.

Examples:
>>> reader = NeuralynxRawIO(dirname='Cheetah_v5.5.1/original_data')
>>> reader.parse_header()
Inspect all files in the directory.
>>> print(reader)
Display all information about signal channels, units, segment size….
extensions = ['nse', 'ncs', 'nev', 'ntt']
class neo.rawio.NeuroExplorerRawIO(filename='')
extensions = ['nex']
class neo.rawio.NeuroScopeRawIO(filename='', binary_file=None)
extensions = ['xml', 'dat', 'lfp', 'eeg']
class neo.rawio.NIXRawIO(filename='')
extensions = ['nix']
class neo.rawio.OpenEphysRawIO(dirname='')

OpenEphys GUI software offers several data formats, see https://open-ephys.atlassian.net/wiki/spaces/OEW/pages/491632/Data+format

This class implements the legacy OpenEphys format here https://open-ephys.atlassian.net/wiki/spaces/OEW/pages/65667092/Open+Ephys+format

The OpenEphys group already proposes some tools here: https://github.com/open-ephys/analysis-tools/blob/master/OpenEphys.py but (i) there is no package at PyPI and (ii) those tools read everything in memory.

The format is directory based with several files:
  • .continuous
  • .events
  • .spikes
This implementation is based on:

In contrast to previous code for reading this format, here all data use memmap so it should be super fast and light compared to legacy code.

When the acquisition is stopped and restarted then files are named *_2, *_3. In that case this class creates a new Segment. Note that timestamps are reset in this situation.

Limitation :
  • Works only if all continuous channels have the same sampling rate, which is a reasonable hypothesis.
  • When the recording is stopped and restarted all continuous files will contain gaps. Ideally this would lead to a new Segment but this use case is not implemented due to its complexity. Instead it will raise an error.
Special cases:
  • Normally all continuous files have the same first timestamp and length. In situations where it is not the case all files are clipped to the smallest one so that they are all aligned, and a warning is emitted.
extensions = []
class neo.rawio.OpenEphysBinaryRawIO(dirname='', load_sync_channel=False, experiment_names=None)

Handle several Blocks and several Segments.

dirname : str
Path to Open Ephys directory
load_sync_channel : bool
If False (default) and a SYNC channel is present (e.g. Neuropixels), this is not loaded. If True, the SYNC channel is loaded and can be accessed in the analog signals.
experiment_names : str or list or None
If multiple experiments are available, this argument allows users to select one or more experiments. If None, all experiements are loaded as blocks. E.g. experiment_names=”experiment2”, experiment_names=[“experiment1”, “experiment2”]

For multi-experiment datasets, the streams need to be consistent across experiments. If this is not the case, you can select a subset of experiments with the experiment_names argument.

# Correspondencies Neo OpenEphys block[n-1] experiment[n] New device start/stop segment[s-1] recording[s] New recording start/stop

This IO handles several signal streams. Special event (npy) data are represented as array_annotations. The current implementation does not handle spiking data, this will be added upon user request

extensions = []
class neo.rawio.PhyRawIO(dirname='')

Class for reading Phy data.

Usage:
>>> import neo.rawio
>>> r = neo.rawio.PhyRawIO(dirname='/dir/to/phy/folder')
>>> r.parse_header()
>>> print(r)
>>> spike_timestamp = r.get_spike_timestamps(block_index=0,
... seg_index=0, spike_channel_index=0, t_start=None, t_stop=None)
>>> spike_times = r.rescale_spike_timestamp(spike_timestamp, 'float64')
extensions = []
class neo.rawio.PlexonRawIO(filename='')
extensions = ['plx']
class neo.rawio.RawBinarySignalRawIO(filename='', dtype='int16', sampling_rate=10000.0, nb_channel=2, signal_gain=1.0, signal_offset=0.0, bytesoffset=0)
extensions = ['raw', '*']
class neo.rawio.RawMCSRawIO(filename='')
extensions = ['raw']
class neo.rawio.Spike2RawIO(filename='', take_ideal_sampling_rate=False, ced_units=True, try_signal_grouping=True)

This implementation in neo read only old smr files. For smrx files you need to use CedRawIO which is based on sonpy.

extensions = ['smr']
class neo.rawio.SpikeGadgetsRawIO(filename='', selected_streams=None)
extensions = ['rec']
class neo.rawio.SpikeGLXRawIO(dirname='', load_sync_channel=False, load_channel_location=False)

Class for reading data from a SpikeGLX system

dirname:
The spikeglx folder containing meta/bin files
load_sync_channel=False/True
The last channel (SY0) of each stream is a fake channel used for synchronisation.
extensions = []
class neo.rawio.TdtRawIO(dirname='', sortname='')
extensions = []
class neo.rawio.WinEdrRawIO(filename='')
extensions = ['EDR', 'edr']
class neo.rawio.WinWcpRawIO(filename='')
extensions = ['wcp']