compared with
Current by Michael Meisinger
on Apr 07, 2014 16:22.

Key
This line was removed.
This word was removed. This word was added.
This line was added.

Changes (8)

View Page History
h2. Overview

Data agents (aka Dataset agents, External dataset agents) act as data producers (see [CIAD MI Dataset Agent Data Acquisition and Ingestion]) in the ION system for data sources providing data in electronic form. The play a similar role than [instrument agents|syseng:CIAD MI OV Instrument and Platform Agents], but lack the ability to command and control the actual device itself. In Release 3, instrument and data agents may be harmonized under one unifying framework. In Release 2, data agents are applied in 2 forms:
Dataset agents (aka Data agents, External dataset agents) act as data producers (see [CIAD MI Dataset Agent Data Acquisition and Ingestion]) in the OOINet system for OOI instruments and platforms providing data in electronic form. The play a similar role than [instrument and platform agents|syseng:CIAD MI OV Instrument and Platform Agents], but lack the ability to command and control the actual device itself. Dataset agents can also be used in an external observatory integration (EOI) context, pulling data from external non-OOI data sources. In general, dataset agents can be applied in these 2 scenarios:

* Representing OOI instrumentation without direct connection or CI presence (e.g. gliders, subsurface moorings, disconnected instruments) for which data arrives in file form from the Marine IOs.
* Representing external data sources via the External Observatory Integration (EOI) effort, e.g. NOAA, IOOS, Neptune CA data sources

Dataset agents responsibilities include management, decomposition and packaging of data from the producer and communication of data producer events.

The figure below shows the various way in which instrument, platform and dataset agents can be deployed.

* Case 1 shows the case of real-time connected OOI instruments for which instrument agents exist. Instrument agents produce granules to be ingested into science coverages and presented as DataProducts to the user. Dataset agents are NOT used in this case. Note: Platform agents are integrated in a similar way, only that they produce engineering data instead of science data.
* Case 2 shows the case in disconnected OOI instruments without CI presence. Data is received from Marine IOs via a file exchange interface (using iRODS servers) in file formates defined in IDD documents. Data agents publish granules to be ingested into science coverages and presented as DataProducts to the user
* Case 2 shows uncabled OOI instruments. Data is received from Marine IOs via a file exchange interface (using rsync or iRODS servers) in file formats defined in IDD documents. Dataset agents publish granules to be ingested into science coverages and presented as DataProducts to the user. Dataset agents exist for every device on the instrument, child node and platform (mooring) levels.
* Case 3 (UNUSED in Release 2) shows external data sources hosting one or multiple external data sets. Dataset agents fetch data in regular intervals from these data sources and granules to be presented as DataProducts to the user. These granules may not be ingested and live only transiently in ION OOINet for data consumers that require these data products. A data source agent may coordinate all dataset agents that fetch data from the same data source.

!https://docs.google.com/drawings/d/1M853R2aLdwLCn3_IxLgQ-qGqoauT-7Xs_K20nPJjL4Y/pub?w=1039&h=720!