Development of a dataset driver with uFrame involves development of one or more Python scripts which will instantiate a parser which will process a specific and return the particles produced as a result of parsing that file.
Driver scripts are placed under configuration management control in the following mi-dataset repo folder area:
The full pathing under the uFrame development directory structure for driver development scripts is:
All dataset decoder XML files should be contained within the following Java package:
The XML files should reside under res/spring folder.
A "...decode.xml" file will need to exist for each specific driver set that will handle producing the recovered and telemetered particle types.
The .xml file should be generated using the spring_generator.py script in the ooi-tools repository. See Spring Test Case for instructions on how to run this.
The following elements in that should be created:
1) A bean for each recovered and telemetered parser where applicable. The specifics for each "processor" bean that make it unique are the "name" and the "scriptPath". An example is below.
2) A camelContext which will contain the endpoints and routes to activate a specific driver. An endpoint is required for each recovered and telemetered harvest path. Two routes are required for each recovered and telemetered driver. Important aspects here are the from endpoint, the header constants, the queue Ingest "to uri". An example is below.
Upon completing any changes of Python or XML files, you need to rebuild and redeploy the newlybuilt artifacts:
- In Eclipse, right select deploy-install.xml, select "Run as", select "Ant Build...", and then "Run"
./edex-server all status
Make sure nothing is running (i.e. no PIDs)
./edex-server all start
./edex-server all status
Make sure PIDs are listed for all
./edex-server edex watch
Wait to see:
EDEX ESB is now operational
To purge the postgres sensorreading table and sensorreading HDF5:
Store a file in the expected Harvester Directory
cp <INPUT_FILE> ~/uframes/ooi/uframe-1.0/edex/data/ooi/<HARVEST_DIR>
Examine the edex output for errors in the terminal in which "./edex-server edex watch" is running.
Use pgadmin3 to examine the contents of the sensorreading db table and verify all of the metadata parameters exist that should exist.
Use h5dump to examine the contents of the sensorreading hdf5 output. Look in ~/uframes/ooi/uframe-1.0/edex/data/hdf5/sensorreading
./edex-server edex stop
./edex-server all stop
The master parameters file can be found in Java package com.raytheon.edex.plugin.ooidata
The path to the file under that package is utility/common_static/base/ooi-param/masterParameters.xml
We shouldn't need to change that file, but changes will probably be given to us as more parameters are defined.
IMPORTANT! If you ever change the "masterParameters.xml" file, you will need to reset the EDEX server database. Ensure EDEX is stopped, and execute the command below.
Stream definitions are generated via a tool. Access the following link for more info on the tool: https://confluence.oceanobservatories.org/x/BIBuAw
Stream definitions are defined in Java package:
The stream definition xml files are located in folder:
Sometimes there are cases when the stream definition may not exist, or the stream definition may have issues. If you want to add ones temporarily until preload is fixed or the CM controlled streams.xml file is fixed, you can add additional stream XML file into that folder.