|This page provides high level overviews related to information and data flows within the OOINet system. Please refer to linked pages for details.|
Figure 1 shows abstracted flows of data and control within the OOINet system, supported by the infrastructure elements. The figure shows a sequence of data flow steps from the instrument to the end user
Figure 1. OOI Data and Control Flows (OV-1)
For a detailed overview of data flows and integration scenarios related to OOI Marine Instrument and Platform Integration, please see here:
Figure 2 depicts the flexible data processing mechanism. Sensor and application specific data processing pipelines and workflows can be developed by sequencing individual processing, distribution and transformation steps that receive data and data products from the Data Distribution Network and make new products available as resources to the Data Distribution Network. This way, any interested party can tap into the process at any point. In addition, buffering, caching and archiving can be managed transparently by the CI infrastructure. A typical sequence involves raw data acquisition from instruments, automated calibration and unit transformation steps, human-in-the loop metadata association and QA/QC steps, and subsequent interactive analysis and visualization.
Figure 2. Data Product Generation (OV-5)
The following pages provide further details and related topics:
- CIAD SA OV Data Product Generation
- CIAD SA OV Data Processing
- CIAD SA OV Data Calibration
- CIAD SA OV Data Validation
- CIAD DM OV Data Transformation
- CIAD DM OV Distribution
The table below defines information elements that are exchanged between ION system services and agents. See also the OOI Reference Module for a definition of common OOI terms.
|Information Element||Product Definition|
|Data||'Data' is a term the CI tries not to use in its unqualified form.|
|Raw Data||Any set of information, as it is provided to the CI by an information source, that is associated with the metadata required of CI information resources. Raw data can include observed data, model outputs, and human-entered information. The CI can persist any raw data it receives from data providers in the raw data's unmodified form. The content that the CI receives and considers "raw data" might have already undergone transformation, filtering and correction by the provider before it arrives at the CI (for example, observed values transformed inside the sensor), but any such modifications are outside of the control of the CI.|
|Observation Data||Any set of information that has been produced by sensing a property or phenomenon in a way that produces an estimate of the corresponding physical quantity(ies), and is associated with the metadata required of CI information resources. Observation data is typically generated by an instrument capable of sensing the real world.|
|Engineering Data|| Any set of information that has been produced by sensing the infrastructure or the system itself, not the environment. It is associated with the metadata required of CI information resources. Examples include power level.
| Calibrated Data
|| Data that has been converted from raw measurement counts, e.g. raw voltage measurements, into engineering units. A well-specified transformation algorithm produces calibrated data, for instance in real-time on the arrival of a Granule.
| Qualified Data
|| Data that has passed an evaluation or test, and is associated with the metadata required of CI information resources. (Quality control is one example of an evaluation or test that can lead to qualified data.)
|Derived Data||Any set of information that has been produced by processing other information resources, and is associated with the metadata required of CI information resources.|
|Data Set|| A self contained CI information resource with a specified set of characteristics, associated with the metadata required of CI information resources. Data Sets may change over time as new incremental updates arrive (for instance, recent observation data is added (ingested) to the Data Set).
| Data Stream
|| A CI information resource representing a flow of data messages from producers to consumers, with a set of characteristics agreed upon by producing and consuming parties, where the individual packets and the overall flow are associated with the metadata required of CI information resources.
| Data Message
|| A coherent unit of information, with specific characteristics, that can be passed around the system and, when combined with 0 or more units with the same characteristics, form a data stream. Also called a Granule
|Data Product|| External form of presenting a Data Stream and/or Data Set with associated metadata to describe the nature of the underlying Information Resources. A Data Product does not guarantee that data is kept persistent in the system; data may be retrieved on demand from external sources, or recomputed from lower level Data Products
|| A unit of information that contains instructions for a taskable resource, can be interpreted by the resource's agent. The agent may apply policy and translate it into device specific interaction.
|Granule|| See Data Message. Also called supplement, increment.
The figure below shows the hierarchical relationship of some of the resources as concerns of the relevant subsystem
Figure 3. 2650-00038 CI Data Elements (OV-7)
The OOI Integrated Observatory Network defines an internal representation for science data and metadata, applicable to other information as well. This internal representation is independent of the structure, feature type and content of data and can represent data of any such form.
The main elements of the Common Data and Metadata Model (please see the linked page for details) are:
- Granule: a uniform, flexible container for information of any structure (domain) and value (range), used to transport data in data streams
- Coverage: a flexible persistent representation of science data and metadata
- Parameters: the value (definitions) referenced by both granule and coverage
- Common Object Model: the encoding of information in system messages
The figure below illustrates a central capability of the OOINet: Data streaming. The basic concepts are data sources that produce data increments, also called supplements, published as messages with metadata that describe the message content precisely. A message router (Exchange Point) that is configured with routing rules (by content metadata and by geographical location) processing all published messages and routing them to message queues for various consumers. Data Consumers that take messages when ready from their exclusive queue. All messages are cataloged and some are persisted for subsequent historic data access.
Data Transformers are processes executed by execution engines that consume messages, transform them and produce output messages that are fed back into the message router for further consumption, cataloging, storage or processing.
Figure 4. Data Streaming (OV-1)
The following pages provide further details and related topics:
The figure below shows how these data collection activities are located in defined logical deployment packages "points". The Instrument Point hosts the direct instrument/sensor access and can for instance be deployed on a buoy hosting a sensor, or on a mobile platform, such as an AUV. The Acquisition Point hosts the logic that transforms instrument specific command and data packets into the CI observatory services, for instance to perform data processing. The Ingest Point hosts the services that take external data (from sensors or other observatories) and add them to CI registries and storage repositories. The Storage Point hosts the services to store and retrieve data from disk. The Application Point hosts the services to manipulate data and derive advanced data products, such as visualizations and numerical models. The Access Portal hosts the presentation elements, such as a portal server.
Figure 5. 2650-00021 Data Collection Flow, logical deployment points (OV-2)