View Source

h1. Dataset Driver Development

Development of a dataset driver with uFrame involves development of one or more Python scripts which will instantiate a parser which will process a specific and return the particles produced as a result of parsing that file.

h2. Dataset Driver Script Development

Driver scripts are placed under configuration management control in the following mi-dataset repo folder area:

mi/dataset/driver/<driver_pathing>

Example:

{code}
mi-dataset/mi/dataset/driver/WFP_ENG/STC_IMODEM/wfp_eng_stc_imodem_recovered_driver.py

{code}

The full pathing under the uFrame development directory structure for driver development scripts is:

\~/ooi/edex/com.raytheon.uf.edex.ooi.decoder.dataset/utility/edex_static/base/ooi/parsers/mi-dataset/mi/dataset/driver/<driver_pathing>

{code}
~/ooi/edex/com.raytheon.uf.edex.ooi.decoder.dataset/utility/edex_static/base/ooi/parsers/mi-dataset/mi/dataset/driver/WFP_ENG/STC_IMODEM/wfp_eng_stc_imodem_recovered_driver.py

{code}


h2. Decoder Dataset Spring XML Development

All dataset decoder XML files should be contained within the following Java package:


com.raytheon.uf.edex.ooi.decoder.dataset.spring

The XML files should reside under res/spring folder.

A "...decode.xml" file will need to exist for each specific driver set that will handle producing the recovered and telemetered particle types.

The .xml file should be generated using the spring_generator.py script in the ooi-tools repository. &nbsp;See&nbsp;[Spring Test Case|https://docs.google.com/document/d/1j3dMXm4pG8q2aCVKVGAZrU-OWIHNEC7AcNqDz9a8-po/edit?pli=1#]&nbsp;for instructions on how to run this.&nbsp;

The following elements in that should be created:


1) A bean for each recovered and telemetered parser where applicable.&nbsp; The specifics for each "processor" bean that make it unique are the "name" and the "scriptPath". An example is below.

{code}
<bean id="NutnrBRecoveredProcessor">
<constructor-arg name="name" value="Nutnr_b_Recovered" />
<constructor-arg name="basePythonCodePath" value="#{basePythonCodeLocalizationPath}" />
<constructor-arg name="scriptPath" value="#{basePythonCodeLocalizationPath}/mi/dataset/driver/nutnr_b/nutnr_b_recovered_driver.py" />
</bean>


{code}

2) A camelContext which will contain the endpoints and routes to activate a specific driver.&nbsp; An endpoint is required for each recovered and telemetered harvest path.&nbsp; Two routes are required for each recovered and telemetered driver.&nbsp; Important aspects here are the from endpoint, the header constants, the queue Ingest "to uri".&nbsp; An example is below.

{code}
<camelContext id="nutnr-b-decode-camel"
xmlns="http://camel.apache.org/schema/spring" errorHandlerRef="errorHandler">

<endpoint id="NutnrBRecoveredFileEndpoint"
uri="file:${edex.home}/data/ooi/nutnr_b_recovered?delete=true&amp;delay=5000&amp;maxMessagesPerPoll=1000&amp;exclusiveReadLockStrategy=#fileChangedStrategy" />

<route id="NutnrBRecoveredFileConsumerRoute">
<from ref="NutnrBRecoveredFileEndpoint" />
<bean ref="fileToString" />
<setHeader headerName="deliveryType">
<constant>recovered</constant>
</setHeader>
<setHeader headerName="sensor">
<constant>XX00XXXX-XX00X-00-NUTNRBDCLCONCR00</constant>
</setHeader>
<to uri="jms-durable:queue:Ingest.nutnr_b_recovered" />
</route>

<route id="NutnrBRecoveredIngest">
<from uri="jms-durable:queue:Ingest.nutnr_b_recovered" />
<doTry>
<pipeline>
<bean ref="NutnrBRecoveredProcessor" />
<to uri="direct-vm:persistIndexAlert" />
</pipeline>
<doCatch>
<exception>java.lang.Throwable</exception>
<to uri="log:ooi.nutnr_b?level=ERROR" />
</doCatch>
</doTry>
</route>

</camelContext>

{code}


h2. EDEX Build and Deploy

Upon completing any changes of Python or XML files, you need to rebuild and redeploy the newlybuilt artifacts:

ooi-build.build.deploy

\- In Eclipse, right select deploy-install.xml, select "Run as", select "Ant Build...", and then "Run"

h2. EDEX Execution

cd \~/uframes/ooi/bin

./edex-server all status

Make sure nothing is running (i.e. no PIDs)

./edex-server all start

./edex-server all status

Make sure PIDs are listed for all

./edex-server edex watch


Wait to see:


EDEX ESB is now operational

h3. EDEX Data Purge

To purge the postgres sensorreading table and sensorreading HDF5:

cd \~/uframes/ooi/bin
./edex-server purgeall
Enter y

h3. EDEX Driver Test

Store a file in the expected Harvester Directory

cp <INPUT_FILE> \~/uframes/ooi/uframe-1.0/edex/data/ooi/<HARVEST_DIR>

Examine the edex output for errors in the terminal in which "./edex-server edex watch" is running.


h3. Postgres DB Examination

Use pgadmin3 to examine the contents of the sensorreading db table and verify all of the metadata parameters exist that should exist.

h3. HDF5 Data Examination

Use h5dump to examine the contents of the sensorreading hdf5 output.&nbsp; Look in \~/uframes/ooi/uframe-1.0/edex/data/hdf5/sensorreading

h3. Stopping EDEX

cd \~/uframes/ooi/bin

./edex-server edex stop

./edex-server all stop

h2. Useful Information for Development


h3. Master Parameters File

The master parameters file can be found in Java package com.raytheon.edex.plugin.ooidata

The path to the file under that package is utility/common_static/base/ooi-param/masterParameters.xml

We shouldn't need to change that file, but changes will probably be given to us as more parameters are defined.

IMPORTANT\!&nbsp; If you ever change the "masterParameters.xml" file, you will need to reset the EDEX server database.&nbsp; Ensure EDEX is stopped, and execute the command below.

{code}
./edex-server resetdb

{code}


h3. Stream Definitions

Stream definitions are generated via a tool.&nbsp; Access the following link for more info on the tool: [https://confluence.oceanobservatories.org/x/BIBuAw]


Stream definitions are defined in Java package:

com.raytheon.uf.edex.ooi.parameter.util

The stream definition xml files are located in folder:

utility/edex_static/base/ooi/streams
\\

Sometimes there are cases when the stream definition may not exist, or the stream definition may have issues.&nbsp; If you want to add ones temporarily until preload is fixed or the CM controlled streams.xml file is fixed, you can add additional stream XML file into that folder.