.. _io_dev_guide:

====================================
Adding support for a new file format
====================================

If you would like to use Neo with a file format that is not yet supported, you have two options:

1. Make an enhancement request (see :doc:`bug_reports`).
   There is no guarantee that a Neo developer will have the time to work on this,
   but the more information you can give about the format, and the more people upvote your request,
   the more likely it will be implemented.
2. Implement it yourself. If you'd like to try this approach, first read :doc:`contributing`,
   then return and read the rest of this page.


There are two ways to implement a new IO module:

1. by directly adding a new IO class in a module within :mod:`neo.io`: the reader/writer will deal directly with Neo objects;
2. by adding a RawIO class in a module within :mod:`neo.rawio`: the reader should work with raw buffers from the file and provide
   some internal headers for the scale/units/name/...
   You can then generate an IO module simply by inheriting from your RawIO class and from :class:`neo.io.BaseFromRaw`.

For read-only classes, we encourage you to write a :class:`RawIO` class because it allows slice reading,
and is generally much quicker and easier than implementing a full IO class.
For read/write classes you can mix the two levels: ``neo.rawio`` for reading and ``neo.io`` for writing.

.. _section-raw-io-recipe:

Recipe to develop an IO module for a new data format
====================================================

1. Fully understand the object model. If in doubt ask the `mailing list`_.
2. Fully understand :mod:`neo.rawio.examplerawio`, which is a fake IO module to explain the API. If in doubt ask the list.
3. Copy/paste ``examplerawio.py`` and choose clear file and class names for your IO.
4. implement all methods that raise :class:`NotImplementedError` in :mod:`neo.rawio.baserawio`. Return None when the object is not supported (e.g., spikes, waveforms)
5. Write good docstrings. List dependencies, including minimum version numbers.
6. Add your class to :mod:`neo.rawio.__init__`. Keep imports inside ``try/except`` for dependency reasons.
7. Create a class in :file:`neo/io/`
8. Add your class to :mod:`neo.io.__init__`. Keep imports inside ``try/except`` for dependency reasons.
9. Create an account at https://gin.g-node.org and deposit files in `NeuralEnsemble/ephy_testing_data`_.
10. Write tests in :file:`neo/rawio/test_xxxxxrawio.py`.
    You must at least pass the standard tests (inherited from :class:`BaseTestRawIO`). See :file:`test_examplerawio.py`
11. Write a similar test in :file:`neo.tests/iotests/test_xxxxxio.py`. See :file:`test_exampleio.py`
12. Make a pull request when all tests pass.

Additional suggestions
----------------------

- If your IO supports several versions of a format (like ABF1, ABF2), upload to the gin.g-node.org test file repository all file versions possible (for test coverage).
- :py:func:`neo.core.Block.create_many_to_one_relationship` offers a utility to complete the hierarchy when all one-to-many relationships have been created.
- In the docstring, explain where you obtained the file format specification, especially if it is a closed one.
- If your IO is based on a database mapper, keep in mind that the returned object MUST be detached,
    because this object can be written to another url for copying.


Tests
=====

:py:class:`neo.rawio.tests.common_rawio_test.BaseTestRawIO` and :py:class:`neo.test.io.commun_io_test.BaseTestIO` provide standard tests.
To use these you need to upload some sample data files at `gin-gnode`_. They will be publicly accessible for testing Neo.
These tests:

- check compliance with the schema: hierarchy, attribute types, ...
- for IO modules able to both write and read data, compare a generated dataset with the same data after a write/read cycle.

The test scripts download all files from `gin-gnode`_ and store them locally in a temporary directory.
Subsequent test runs use the previously downloaded files, rather than trying to download them each time.

Each test must have at least one class that inherits ``BaseTestRawIO`` and that has 3 attributes:

- ``rawioclass``: the class
- ``entities_to_download``: a list of files or directories to download
- ``entities_to_test``: a list of files (or directories) to be tested one by one

Here is an example test script taken from the distribution: :file:`test_axonrawio.py`:

.. literalinclude:: ../../neo/test/rawiotest/test_axonrawio.py


Logging
=======

All IO classes by default have logging using the standard :mod:`logging` module: already set up.
The logger name is the same as the fully qualified class name, e.g. :class:`neo.io.nixio.NixIO`.
The :attr:`class.logger` attribute holds the logger for easy access.

There are generally 3 types of situations in which an IO class should use a logger:

- Recoverable errors with the file that the users need to be notified about.
  In this case, please use :meth:`logger.warning` or :meth:`logger.error`.
  If there is an exception associated with the issue, you can use :meth:`logger.exception` in the exception handler
  to automatically include a backtrace with the log.
  By default, all users will see messages at this level, so please restrict it only to problems the user
  absolutely needs to know about.
- Informational messages that advanced users might want to see in order to get some insight into the file.
  In this case, please use :meth:`logger.info`.
- Messages useful to developers to fix problems with the io class.
  In this case, please use :meth:`logger.debug`.

A log handler is automatically added to :mod:`neo`, so please do not use your own handler.
Please use the :attr:`class.logger` attribute for accessing the logger inside the class rather than :meth:`logging.getLogger`.
Please do not log directly to the root logger (e.g. :meth:`logging.warning`), use the class's logger instead (:meth:`class.logger.warning`).
In the tests for the io class, if you intentionally test broken files, please disable logs by setting the logging level to `100`.


ExampleIO
=========

.. autoclass:: neo.rawio.ExampleRawIO

.. autoclass:: neo.io.ExampleIO

Here are the entire files:

.. literalinclude:: ../../neo/rawio/examplerawio.py

.. literalinclude:: ../../neo/io/exampleio.py


.. _`mailing list`: http://neuralensemble.org/community
.. _gin-gnode: https://gin.g-node.org/NeuralEnsemble/ephy_testing_data
.. _`NeuralEnsemble/ephy_testing_data`: https://gin.g-node.org/NeuralEnsemble/ephy_testing_data