Skip to end of metadata
Go to start of metadata

The sio_mule_common.py dataset agent parser is a common parser which can be used to parse data for all instruments which have been wrapped in an SIO header (see sio_mule_common for a description of the header). 

Recent Changes

The file format that is being parsed by the sio mule common has changed from a single file which changes over time to a set of fixed files (similar to how recovered sio parsers worked previously).  This was changed because the new architecture does not handle file state.  The original file is still created, but keeping track of the state has moved into a new program which will be run on the shore server, and keep track of the state there.  The new files are separated by instrument group.  It is instrument 'group' rather than just instrument because there may be multiple sio header blocks associated with a parser, so they are grouped into files by parser.  For example the ctdmo_ghqr_sio parser receives both 'CT' and 'CO' sio header IDs in its .ctdmo.dat files. The reversing of escape characters (\x18\x6b -> \x2b and \x18\x58 -> \x18) has also been moved to the program on the shore server, so this is no longer required by the parser.  

Migrating a Parser

To migrate a parser from using the old sio parser to the new one, the following should be performed:

If the driver is one that is located in the mflm directory, we are moving the files to the locations matching the IDD names.  For example mflm/adcp is now moved to adcps_jln/sio.  This just requires moving any _init_.py files and any files still needed in the resource directory to the new location.  This may also require renaming the classes in the parser to match the IDD name.  

If the driver is not in the mflm directory, no copying or renaming is needed.

The resource files may no longer be relevant due to the file format changing (nodeXXp1.dat files are no longer used, this has gone to nodeXXp1_N.instrumentgroup.dat).  The new file format can only be found in the IDDs currently, not on the acquisition server.  It is possible that the data that was used to generate the files in the new format may have samples matching the old data file, so check if any of the .yml files can be reused.

The sio common class is now named 'SioParser' which is used by both telemetered and recovered parsers.  The 'SioMuleParser' class no longer exists.  The new SioParser class no longer handles state.  The only input arguments required are config, stream_handle, and exception_callback now, so this needs to be changed in the classes extending SioParser.  In parse_chunks, the chunk_sample_count is no longer used, so this needs to be removed from existing parsers.

The unit tests should have all tests handing changing state removed (starting in the middle, changing state, starting and stopping).  If this is one of the mflm parsers, the import paths and class names will need updating in the tests too. The existing non-state tests for telemetered need to be updated to use the new example file from the IDD, and test comparing to .yml files rather than other methods.  The data in the new format of files may match up to some data in the old file, so it may be possible to reuse some of the created .yml files.  The recovered files were already in a fixed format and can remain.  To transition the recovered tests it helps to look at the old driver test to determine which input test files match up to the output .yml files.  

For an example, the adcps_jln/sio and dosta_abcdjm/sio parsers have been migrated and are committed into oceanobservatories (as of 11/6/14).

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.