Skip to end of metadata
Go to start of metadata

Dataset Driver Development

Development of a dataset driver with uFrame involves development of one or more Python scripts which will instantiate a parser which will process a specific and return the particles produced as a result of parsing that file.

Dataset Driver Script Development

Driver scripts are placed under configuration management control in the following mi-dataset repo folder area:

mi/dataset/driver/<driver_pathing>

Example:

The full pathing under the uFrame development directory structure for driver development scripts is:

~/ooi/edex/com.raytheon.uf.edex.ooi.decoder.dataset/utility/edex_static/base/ooi/parsers/mi-dataset/mi/dataset/driver/<driver_pathing>

Decoder Dataset Spring XML Development

All dataset decoder XML files should be contained within the following Java package:

com.raytheon.uf.edex.ooi.decoder.dataset.spring

The XML files should reside under res/spring folder.

A "...decode.xml" file will need to exist for each specific driver set that will handle producing the recovered and telemetered particle types.

The .xml file should be generated using the spring_generator.py script in the ooi-tools repository.  See Spring Test Case for instructions on how to run this. 

The following elements in that should be created:

1) A bean for each recovered and telemetered parser where applicable.  The specifics for each "processor" bean that make it unique are the "name" and the "scriptPath". An example is below.

2) A camelContext which will contain the endpoints and routes to activate a specific driver.  An endpoint is required for each recovered and telemetered harvest path.  Two routes are required for each recovered and telemetered driver.  Important aspects here are the from endpoint, the header constants, the queue Ingest "to uri".  An example is below.

EDEX Build and Deploy

Upon completing any changes of Python or XML files, you need to rebuild and redeploy the newlybuilt artifacts:

ooi-build.build.deploy

- In Eclipse, right select deploy-install.xml, select "Run as", select "Ant Build...", and then "Run"

EDEX Execution

cd ~/uframes/ooi/bin

./edex-server all status

Make sure nothing is running (i.e. no PIDs)

./edex-server all start

./edex-server all status

Make sure PIDs are listed for all

./edex-server edex watch

Wait to see:

EDEX ESB is now operational

EDEX Data Purge

To purge the postgres sensorreading table and sensorreading HDF5:

cd ~/uframes/ooi/bin
./edex-server purgeall
Enter y

EDEX Driver Test

Store a file in the expected Harvester Directory

cp <INPUT_FILE> ~/uframes/ooi/uframe-1.0/edex/data/ooi/<HARVEST_DIR>

Examine the edex output for errors in the terminal in which "./edex-server edex watch" is running.

Postgres DB Examination

Use pgadmin3 to examine the contents of the sensorreading db table and verify all of the metadata parameters exist that should exist.

HDF5 Data Examination

Use h5dump to examine the contents of the sensorreading hdf5 output.  Look in ~/uframes/ooi/uframe-1.0/edex/data/hdf5/sensorreading

Stopping EDEX

cd ~/uframes/ooi/bin

./edex-server edex stop

./edex-server all stop

Useful Information for Development

Master Parameters File

The master parameters file can be found in Java package com.raytheon.edex.plugin.ooidata

The path to the file under that package is utility/common_static/base/ooi-param/masterParameters.xml

We shouldn't need to change that file, but changes will probably be given to us as more parameters are defined.

IMPORTANT!  If you ever change the "masterParameters.xml" file, you will need to reset the EDEX server database.  Ensure EDEX is stopped, and execute the command below.

Stream Definitions

Stream definitions are generated via a tool.  Access the following link for more info on the tool: https://confluence.oceanobservatories.org/x/BIBuAw

Stream definitions are defined in Java package:

com.raytheon.uf.edex.ooi.parameter.util

The stream definition xml files are located in folder:

utility/edex_static/base/ooi/streams

Sometimes there are cases when the stream definition may not exist, or the stream definition may have issues.  If you want to add ones temporarily until preload is fixed or the CM controlled streams.xml file is fixed, you can add additional stream XML file into that folder.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.