Neo IO

Preamble

The Neo io module aims to provide an exhaustive way of loading and saving several widely used data formats in electrophysiology. The more these heterogeneous formats are supported, the easier it will be to manipulate them as Neo objects in a similar way. Therefore the IO set of classes propose a simple and flexible IO API that fits many format specifications. It is not only file-oriented, it can also read/write objects from a database.

At the moment, there are 3 families of IO modules:
  1. for reading closed manufacturers’ formats (Spike2, Plexon, AlphaOmega, BlackRock, Axon, …)
  2. for reading(/writing) formats from open source tools (KlustaKwik, Elan, WinEdr, WinWcp, PyNN, …)
  3. for reading/writing Neo structure in neutral formats (HDF5, .mat, …) but with Neo structure inside (NeoHDF5, NeoMatlab, …)

Combining 1 for reading and 3 for writing is a good example of use: converting your datasets to a more standard format when you want to share/collaborate.

Introduction

There is an intrinsic structure in the different Neo objects, that could be seen as a hierachy with cross-links. See Neo core. The highest level object is the Block object, which is the high level container able to encapsulate all the others.

A Block has therefore a list of Segment objects, that can, in some file formats, be accessed individually. Depending on the file format, i.e. if it is streamable or not, the whole Block may need to be loaded, but sometimes particular Segment objects can be accessed individually. Within a Segment, the same hierarchical organisation applies. A Segment embeds several objects, such as SpikeTrain, AnalogSignal, IrregularlySampledSignal, Epoch, Event (basically, all the different Neo objects).

Depending on the file format, these objects can sometimes be loaded separately, without the need to load the whole file. If possible, a file IO therefore provides distinct methods allowing to load only particular objects that may be present in the file. The basic idea of each IO file format is to have, as much as possible, read/write methods for the individual encapsulated objects, and otherwise to provide a read/write method that will return the object at the highest level of hierarchy (by default, a Block or a Segment).

The neo.io API is a balance between full flexibility for the user (all read_XXX() methods are enabled) and simple, clean and understandable code for the developer (few read_XXX() methods are enabled). This means that not all IOs offer the full flexibility for partial reading of data files.

One format = one class

The basic syntax is as follows. If you want to load a file format that is implemented in a generic MyFormatIO class:

>>> from neo.io import MyFormatIO
>>> reader = MyFormatIO(filename="myfile.dat")

you can replace MyFormatIO by any implemented class, see List of implemented formats

Modes

IO can be based on a single file, a directory containing files, or a database. This is described in the mode attribute of the IO class.

>>> from neo.io import MyFormatIO
>>> print MyFormatIO.mode
'file'

For file mode the filename keyword argument is necessary. For directory mode the dirname keyword argument is necessary.

Ex:
>>> reader = io.PlexonIO(filename='File_plexon_1.plx')
>>> reader = io.TdtIO(dirname='aep_05')

Supported objects/readable objects

To know what types of object are supported by a given IO interface:

>>> MyFormatIO.supported_objects
[Segment , AnalogSignal , SpikeTrain, Event, Spike]

Supported objects does not mean objects that you can read directly. For instance, many formats support AnalogSignal but don’t allow them to be loaded directly, rather to access the AnalogSignal objects, you must read a Segment:

>>> seg = reader.read_segment()
>>> print(seg.analogsignals)
>>> print(seg.analogsignals[0])

To get a list of directly readable objects

>>> MyFormatIO.readable_objects
[Segment]

The first element of the previous list is the highest level for reading the file. This mean that the IO has a read_segment() method:

>>> seg = reader.read_segment()
>>> type(seg)
neo.core.Segment

All IOs have a read() method that returns a list of Block objects (representing the whole content of the file):

>>> bl = reader.read()
>>> print bl[0].segments[0]
neo.core.Segment

Lazy option (deprecated)

In some cases you may not want to load everything in memory because it could be too big. For this scenario, some IOs implement lazy=True/False. With lazy=True all arrays will have a size of zero, but all the metadata will be loaded. The lazy_shape attribute is added to all array-like objects (AnalogSignal, IrregularlySampledSignal, SpikeTrain, Epoch, Event). In this case, lazy_shape is a tuple that has the same value as shape with lazy=False. To know if a class supports lazy mode use ClassIO.support_lazy. By default (if not specified), lazy=False, i.e. all data is loaded. The lazy option will be removed in future Neo versions. Similar functionality will be implemented using proxy objects.

Example of lazy loading:

>>> seg = reader.read_segment(lazy=False)
>>> print(seg.analogsignals[0].shape)  # this is (N, M)
>>> seg = reader.read_segment(lazy=True)
>>> print(seg.analogsignals[0].shape)  # this is 0, the AnalogSignal is empty
>>> print(seg.analogsignals[0].lazy_shape)  # this is (N, M)

Details of API

The neo.io API is designed to be simple and intuitive:
  • each file format has an IO class (for example for Spike2 files you have a Spike2IO class).
  • each IO class inherits from the BaseIO class.
  • each IO class can read or write directly one or several Neo objects (for example Segment, Block, …): see the readable_objects and writable_objects attributes of the IO class.
  • each IO class supports part of the neo.core hierachy, though not necessarily all of it (see supported_objects).
  • each IO class has a read() method that returns a list of Block objects. If the IO only supports Segment reading, the list will contain one block with all segments from the file.
  • each IO class that supports writing has a write() method that takes as a parameter a list of blocks, a single block or a single segment, depending on the IO’s writable_objects.
  • each IO is able to do a lazy load: all metadata (e.g. sampling_rate) are read, but not the actual numerical data. lazy_shape attribute is added to provide information on real size.
  • each IO is able to save and load all required attributes (metadata) of the objects it supports.
  • each IO can freely add user-defined or manufacturer-defined metadata to the annotations attribute of an object.

If you want to develop your own IO

See IO developers’ guide for information on how to implement a new IO.

List of implemented formats

neo.io provides classes for reading and/or writing electrophysiological data files.

Note that if the package dependency is not satisfied for one io, it does not raise an error but a warning.

neo.io.iolist provides a list of successfully imported io classes.

Functions:

neo.io.get_io(filename, *args, **kwargs)

Return a Neo IO instance, guessing the type based on the filename suffix.

Classes:

class neo.io.AlphaOmegaIO(filename=None)

Class for reading data from Alpha Omega .map files (experimental)

This class is an experimental reader with important limitations. See the source code for details of the limitations. The code of this reader is of alpha quality and received very limited testing.

Usage:
>>> from neo import io
>>> r = io.AlphaOmegaIO( filename = 'File_AlphaOmega_1.map')
>>> blck = r.read_block()
>>> print blck.segments[0].analogsignals
class neo.io.AsciiSignalIO(filename=None)

Class for reading signal in generic ascii format. Columns respresents signals. They all share the same sampling rate. The sampling rate is externally known or the first columns could hold the time vector.

Usage:
>>> from neo import io
>>> r = io.AsciiSignalIO(filename='File_asciisignal_2.txt')
>>> seg = r.read_segment()
>>> print seg.analogsignals
[<AnalogSignal(array([ 39.0625    ,   0.        ,   0.        , ..., -26.85546875 ...
class neo.io.AsciiSpikeTrainIO(filename=None)

Class for reading/writing SpikeTrains in a text file. Each Spiketrain is a line.

Usage:
>>> from neo import io
>>> r = io.AsciiSpikeTrainIO( filename = 'File_ascii_spiketrain_1.txt')
>>> seg = r.read_segment()
>>> print seg.spiketrains     
[<SpikeTrain(array([ 3.89981604,  4.73258781,  0.608428  ,  4.60246277,  1.23805797,
...
class neo.io.AxonIO(filename)

Class for reading data from pCLAMP and AxoScope files (.abf version 1 and 2), developed by Molecular device/Axon technologies.

  • abf = Axon binary file
  • atf is a text file based format from axon that could be read by AsciiIO (but this file is less efficient.)
class neo.io.BCI2000IO(filename)

Class for reading data from a BCI2000 .dat file, either version 1.0 or 1.1

class neo.io.BlackrockIO(filename, nsx_to_load=None, **kargs)

This IO reads .nev/.nsX files of the Blackrock (Cerebus) recording system.

class neo.io.BrainVisionIO(filename)

Class for reading data from the BrainVision product.

class neo.io.BrainwareDamIO(filename=None)

Class for reading Brainware raw data files with the extension ‘.dam’.

The read_block method returns the first Block of the file. It will automatically close the file after reading. The read method is the same as read_block.

Note:

The file format does not contain a sampling rate. The sampling rate is set to 1 Hz, but this is arbitrary. If you have a corresponding .src or .f32 file, you can get the sampling rate from that. It may also be possible to infer it from the attributes, such as “sweep length”, if present.

Usage:
>>> from neo.io.brainwaredamio import BrainwareDamIO
>>> damfile = BrainwareDamIO(filename='multi_500ms_mulitrep_ch1.dam')
>>> blk1 = damfile.read()
>>> blk2 = damfile.read_block()
>>> print blk1.segments
>>> print blk1.segments[0].analogsignals
>>> print blk1.units
>>> print blk1.units[0].name
>>> print blk2
>>> print blk2[0].segments
class neo.io.BrainwareF32IO(filename=None)

Class for reading Brainware Spike ReCord files with the extension ‘.f32’

The read_block method returns the first Block of the file. It will automatically close the file after reading. The read method is the same as read_block.

The read_all_blocks method automatically reads all Blocks. It will automatically close the file after reading.

The read_next_block method will return one Block each time it is called. It will automatically close the file and reset to the first Block after reading the last block. Call the close method to close the file and reset this method back to the first Block.

The isopen property tells whether the file is currently open and reading or closed.

Note 1:
There is always only one ChannelIndex. BrainWare stores the equivalent of ChannelIndexes in separate files.
Usage:
>>> from neo.io.brainwaref32io import BrainwareF32IO
>>> f32file = BrainwareF32IO(filename='multi_500ms_mulitrep_ch1.f32')
>>> blk1 = f32file.read()
>>> blk2 = f32file.read_block()
>>> print blk1.segments
>>> print blk1.segments[0].spiketrains
>>> print blk1.units
>>> print blk1.units[0].name
>>> print blk2
>>> print blk2[0].segments
class neo.io.BrainwareSrcIO(filename=None)

Class for reading Brainware Spike ReCord files with the extension ‘.src’

The read_block method returns the first Block of the file. It will automatically close the file after reading. The read method is the same as read_block.

The read_all_blocks method automatically reads all Blocks. It will automatically close the file after reading.

The read_next_block method will return one Block each time it is called. It will automatically close the file and reset to the first Block after reading the last block. Call the close method to close the file and reset this method back to the first Block.

The _isopen property tells whether the file is currently open and reading or closed.

Note 1:
The first Unit in each ChannelIndex is always UnassignedSpikes, which has a SpikeTrain for each Segment containing all the spikes not assigned to any Unit in that Segment.
Note 2:
The first Segment in each Block is always Comments, which stores all comments as an Event object.
Note 3:
The parameters from the BrainWare table for each condition are stored in the Segment annotations. If there are multiple repetitions of a condition, each repetition is stored as a separate Segment.
Note 4:
There is always only one ChannelIndex. BrainWare stores the equivalent of ChannelIndexes in separate files.
Usage:
>>> from neo.io.brainwaresrcio import BrainwareSrcIO
>>> srcfile = BrainwareSrcIO(filename='multi_500ms_mulitrep_ch1.src')
>>> blk1 = srcfile.read()
>>> blk2 = srcfile.read_block()
>>> blks = srcfile.read_all_blocks()
>>> print blk1.segments
>>> print blk1.segments[0].spiketrains
>>> print blk1.units
>>> print blk1.units[0].name
>>> print blk2
>>> print blk2[0].segments
>>> print blks
>>> print blks[0].segments
class neo.io.ElanIO(filename)

Class for reading data from Elan.

Elan is software for studying time-frequency maps of EEG data.

Elan is developed in Lyon, France, at INSERM U821

https://elan.lyon.inserm.fr

class neo.io.IgorIO(filename=None, parse_notes=None)

Class for reading Igor Binary Waves (.ibw) or Packed Experiment (.pxp) files written by WaveMetrics’ IGOR Pro software.

It requires the igor Python package by W. Trevor King.

Usage:
>>> from neo import io
>>> r = io.IgorIO(filename='...ibw')
class neo.io.KlustaKwikIO(filename, sampling_rate=30000.0)

Reading and writing from KlustaKwik-format files.

class neo.io.KwikIO(filename)

Class for “reading” experimental data from a .kwik file.

Generates a Segment with a AnalogSignal

class neo.io.MicromedIO(filename)

Class for reading/writing data from Micromed files (.trc).

class neo.io.NeoHdf5IO(filename)

Class for reading HDF5 format files created by Neo version 0.4 or earlier.

Writing to HDF5 is not supported by this IO; we recommend using NixIO for this.

class neo.io.NeoMatlabIO(filename=None)

Class for reading/writing Neo objects in MATLAB format (.mat) versions 5 to 7.2.

This module is a bridge for MATLAB users who want to adopt the Neo object representation. The nomenclature is the same but using Matlab structs and cell arrays. With this module MATLAB users can use neo.io to read a format and convert it to .mat.

Rules of conversion:
  • Neo classes are converted to MATLAB structs. e.g., a Block is a struct with attributes “name”, “file_datetime”, …
  • Neo one_to_many relationships are cellarrays in MATLAB. e.g., seg.analogsignals[2] in Python Neo will be seg.analogsignals{3} in MATLAB.
  • Quantity attributes are represented by 2 fields in MATLAB. e.g., anasig.t_start = 1.5 * s in Python will be anasig.t_start = 1.5 and anasig.t_start_unit = 's' in MATLAB.
  • classes that inherit from Quantity (AnalogSignal, SpikeTrain, …) in Python will have 2 fields (array and units) in the MATLAB struct. e.g.: AnalogSignal( [1., 2., 3.], 'V') in Python will be anasig.array = [1. 2. 3] and anasig.units = 'V' in MATLAB.

1 - Scenario 1: create data in MATLAB and read them in Python

This MATLAB code generates a block:

block = struct();
block.segments = { };
block.name = 'my block with matlab';
for s = 1:3
    seg = struct();
    seg.name = strcat('segment ',num2str(s));

    seg.analogsignals = { };
    for a = 1:5
        anasig = struct();
        anasig.signal = rand(100,1);
        anasig.signal_units = 'mV';
        anasig.t_start = 0;
        anasig.t_start_units = 's';
        anasig.sampling_rate = 100;
        anasig.sampling_rate_units = 'Hz';
        seg.analogsignals{a} = anasig;
    end

    seg.spiketrains = { };
    for t = 1:7
        sptr = struct();
        sptr.times = rand(30,1)*10;
        sptr.times_units = 'ms';
        sptr.t_start = 0;
        sptr.t_start_units = 'ms';
        sptr.t_stop = 10;
        sptr.t_stop_units = 'ms';
        seg.spiketrains{t} = sptr;
    end

    event = struct();
    event.times = [0, 10, 30];
    event.times_units = 'ms';
    event.labels = ['trig0'; 'trig1'; 'trig2'];
    seg.events{1} = event;

    epoch = struct();
    epoch.times = [10, 20];
    epoch.times_units = 'ms';
    epoch.durations = [4, 10];
    epoch.durations_units = 'ms';
    epoch.labels = ['a0'; 'a1'];
    seg.epochs{1} = epoch;

    block.segments{s} = seg;

end

save 'myblock.mat' block -V7

This code reads it in Python:

import neo
r = neo.io.NeoMatlabIO(filename='myblock.mat')
bl = r.read_block()
print bl.segments[1].analogsignals[2]
print bl.segments[1].spiketrains[4]

2 - Scenario 2: create data in Python and read them in MATLAB

This Python code generates the same block as in the previous scenario:

import neo
import quantities as pq
from scipy import rand, array

bl = neo.Block(name='my block with neo')
for s in range(3):
    seg = neo.Segment(name='segment' + str(s))
    bl.segments.append(seg)
    for a in range(5):
        anasig = neo.AnalogSignal(rand(100)*pq.mV, t_start=0*pq.s, sampling_rate=100*pq.Hz)
        seg.analogsignals.append(anasig)
    for t in range(7):
        sptr = neo.SpikeTrain(rand(40)*pq.ms, t_start=0*pq.ms, t_stop=10*pq.ms)
        seg.spiketrains.append(sptr)
    ev = neo.Event([0, 10, 30]*pq.ms, labels=array(['trig0', 'trig1', 'trig2']))
    ep = neo.Epoch([10, 20]*pq.ms, durations=[4, 10]*pq.ms, labels=array(['a0', 'a1']))
    seg.events.append(ev)
    seg.epochs.append(ep)

from neo.io.neomatlabio import NeoMatlabIO
w = NeoMatlabIO(filename='myblock.mat')
w.write_block(bl)

This MATLAB code reads it:

load 'myblock.mat'
block.name
block.segments{2}.analogsignals{3}.signal
block.segments{2}.analogsignals{3}.signal_units
block.segments{2}.analogsignals{3}.t_start
block.segments{2}.analogsignals{3}.t_start_units

3 - Scenario 3: conversion

This Python code converts a Spike2 file to MATLAB:

from neo import Block
from neo.io import Spike2IO, NeoMatlabIO

r = Spike2IO(filename='spike2.smr')
w = NeoMatlabIO(filename='convertedfile.mat')
blocks = r.read()
w.write(blocks[0])
class neo.io.NestIO(filenames=None)

Class for reading NEST output files. GDF files for the spike data and DAT files for analog signals are possible.

Usage:
>>> from neo.io.nestio import NestIO
>>> files = ['membrane_voltages-1261-0.dat',
         'spikes-1258-0.gdf']
>>> r = NestIO(filenames=files)
>>> seg = r.read_segment(gid_list=[], t_start=400 * pq.ms,
                     t_stop=600 * pq.ms,
                     id_column_gdf=0, time_column_gdf=1,
                     id_column_dat=0, time_column_dat=1,
                     value_columns_dat=2)
class neo.io.NeuralynxIO(dirname, use_cache=False, cache_path='same_as_resource')

Class for reading data from Neuralynx files. This IO supports NCS, NEV, NSE and NTT file formats.

NCS contains signals for one channel NEV contains events NSE contains spikes and waveforms for mono electrodes NTT contains spikes and waveforms for tetrodes

class neo.io.NeuroExplorerIO(filename)

Class for reading data from NeuroExplorer (.nex)

class neo.io.NeuroScopeIO(filename)

Reading from Neuroscope format files.

Ref: http://neuroscope.sourceforge.net/

neo.io.NeuroshareIO

alias of NeurosharectypesIO

class neo.io.NixIO(filename, mode='rw')

Class for reading and writing NIX files.

class neo.io.NSDFIO(filename=None)

Class for reading and writing files in NSDF Format.

It supports reading and writing: Block, Segment, AnalogSignal, ChannelIndex, with all relationships and metadata.

class neo.io.PickleIO(filename=None, **kargs)

A class for reading and writing Neo data from/to the Python “pickle” format.

Note that files in this format may not be readable if using a different version of Neo to that used to create the file. It should therefore not be used for long-term storage, but rather for intermediate results in a pipeline.

class neo.io.PlexonIO(filename)

Class for reading the old data format from Plexon acquisition system (.plx)

Note that Plexon now use a new format PL2 which is NOT supported by this IO.

Compatible with versions 100 to 106. Other versions have not been tested.

class neo.io.PyNNNumpyIO(filename=None, **kargs)

(DEPRECATED) Reads/writes data from/to PyNN NumpyBinaryFile format

class neo.io.PyNNTextIO(filename=None, **kargs)

(DEPRECATED) Reads/writes data from/to PyNN StandardTextFile format

class neo.io.RawBinarySignalIO(filename, dtype='int16', sampling_rate=10000.0, nb_channel=2, signal_gain=1.0, signal_offset=0.0, bytesoffset=0)

Class for reading/writing data in a raw binary interleaved compact file.

Important release note

Since the version neo 0.6.0 and the neo.rawio API, argmuents of the IO (dtype, nb_channel, sampling_rate) must be given at the __init__ and not at read_segment() because there is no read_segment() in neo.rawio classes.

So now the usage is:
>>>>r = io.RawBinarySignalIO(filename=’file.raw’, dtype=’int16’,
nb_channel=16, sampling_rate=10000.)
class neo.io.StimfitIO(filename=None)

Class for converting a stfio Recording to a Neo object. Provides a standardized representation of the data as defined by the neo project; this is useful to explore the data with an increasing number of electrophysiology software tools that rely on the Neo standard.

stfio is a standalone file i/o Python module that ships with the Stimfit program (http://www.stimfit.org). It is a Python wrapper around Stimfit’s file i/o library (libstfio) that natively provides support for the following file types:

  • ABF (Axon binary file format; pClamp 6–9)
  • ABF2 (Axon binary file format 2; pClamp 10+)
  • ATF (Axon text file format)
  • AXGX/AXGD (Axograph X file format)
  • CFS (Cambridge electronic devices filing system)
  • HEKA (HEKA binary file format)
  • HDF5 (Hierarchical data format 5; only hdf5 files written by Stimfit or stfio are supported)

In addition, libstfio can use the biosig file i/o library as an additional file handling backend (http://biosig.sourceforge.net/), extending support to more than 30 additional file formats (http://pub.ist.ac.at/~schloegl/biosig/TESTED).

Example usage:
>>> import neo
>>> neo_obj = neo.io.StimfitIO("file.abf")
or
>>> import stfio
>>> stfio_obj = stfio.read("file.abf")
>>> neo_obj = neo.io.StimfitIO(stfio_obj)
class neo.io.TdtIO(dirname)

Class for reading data from from Tucker Davis TTank format.

Terminology: TDT holds data with tanks (actually a directory). And tanks hold sub blocks (sub directories). Tanks correspond to Neo Blocks and TDT blocks correspond to Neo Segments.

class neo.io.WinEdrIO(filename)

Class for reading data from WinEdr, a software tool written by John Dempster.

WinEdr is free: http://spider.science.strath.ac.uk/sipbs/software.htm

class neo.io.WinWcpIO(filename)

Class for reading data from WinWCP, a software tool written by John Dempster.

WinWCP is free: http://spider.science.strath.ac.uk/sipbs/software.htm

Logging

neo uses the standard Python logging module for logging. All neo.io classes have logging set up by default, although not all classes produce log messages. The logger name is the same as the full qualified class name, e.g. neo.io.hdf5io.NeoHdf5IO. By default, only log messages that are critically important for users are displayed, so users should not disable log messages unless they are sure they know what they are doing. However, if you wish to disable the messages, you can do so:

>>> import logging
>>>
>>> logger = logging.getLogger('neo')
>>> logger.setLevel(100)

Some io classes provide additional information that might be interesting to advanced users. To enable these messages, do the following:

>>> import logging
>>>
>>> logger = logging.getLogger('neo')
>>> logger.setLevel(logging.INFO)

It is also possible to log to a file in addition to the terminal:

>>> import logging
>>>
>>> logger = logging.getLogger('neo')
>>> handler = logging.FileHandler('filename.log')
>>> logger.addHandler(handler)

To only log to the terminal:

>>> import logging
>>> from neo import logging_handler
>>>
>>> logger = logging.getLogger('neo')
>>> handler = logging.FileHandler('filename.log')
>>> logger.addHandler(handler)
>>>
>>> logging_handler.setLevel(100)

This can also be done for individual IO classes:

>>> import logging
>>>
>>> logger = logging.getLogger('neo.io.hdf5io.NeoHdf5IO')
>>> handler = logging.FileHandler('filename.log')
>>> logger.addHandler(handler)

Individual IO classes can have their loggers disabled as well:

>>> import logging
>>>
>>> logger = logging.getLogger('neo.io.hdf5io.NeoHdf5IO')
>>> logger.setLevel(100)

And more detailed logging messages can be enabled for individual IO classes:

>>> import logging
>>>
>>> logger = logging.getLogger('neo.io.hdf5io.NeoHdf5IO')
>>> logger.setLevel(logging.INFO)

The default handler, which is used to print logs to the command line, is stored in neo.logging_handler. This example changes how the log text is displayed:

>>> import logging
>>> from neo import logging_handler
>>>
>>> formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
>>> logging_handler.setFormatter(formatter)

For more complex logging, please see the documentation for the logging module.

Note

If you wish to implement more advanced logging as describe in the documentation for the logging module or elsewhere on the internet, please do so before calling any neo functions or initializing any neo classes. This is because the default handler is created when neo is imported, but it is not attached to the neo logger until a class that uses logging is initialized or a function that uses logging is called. Further, the handler is only attached if there are no handlers already attached to the root logger or the neo logger, so adding your own logger will override the default one. Additional functions and/or classes may get logging during bugfix releases, so code relying on particular modules not having logging may break at any time without warning.