|This is the central page describing OOINet data distribution capabilities. This includes science data message streaming and subscription to data streams. See child pages for details.|
The Distribution service provides the pub/sub/notify mechanism to manage data streams and all related resources.
The figure below illustrates the core concepts related to messaging and data streaming:
- Producer (process) - Publishes messages to streams
- Stream - A predefined identifier for message sequences from a publisher
- Message/Granule - A data message
- Subscription - Query to receive messages matching streams and other criteria in queue
- Queue - A protected buffer on the broker holding messaging in FIFO order until consumed and ack'ed
- Consumer (process) - Taking messages in FIFO order off a queue
Figure 1. Messaging Concepts and Message Processing Patterns (OV-1)
Figure 2. DM Distribution Services Overview (OV-2)
The Pubsub Management Service manages Publish/Subscribe message routing. Subscriptions are defined by what a consumer wants to receive. In the case of subscriptions to streams (there are also subscriptions to events, see Event Subscriber) these are characterized by queries against stream topics. Internally, these queries are realized using COI infrastructure as a set of broker routing rules. The broker delivers messages selected by the query to a queue specified by the subscription.
The following resources are managed by the data distribution services. They are essential for the data streaming pubsub architecture.
Identifies a resource that can be mapped to a messaging pubsub topic, such that the sequence of messages published to this stream can be received in the same form in real-time by all subscribers to this data stream. See here for details:
A detailed description of the content of science data stream messages (granules), in particular the science data parameters they transport. See here for details:
- CIAD DM SV Granule Data Container
- CIAD DM OV Common Data and Metadata Model
- Resource Spec for StreamDefinition
A standing request to the system to receive science data messages related to a data stream or derived concept. Subscriptions are owned by the subscriber. Multiple subscriptions can exist for the same stream and all subscribers will receive independent copies of the data messages. The following types of subscriptions exists and further types may be added in the future:
- Simple stream subscription: A standing subscription to a message stream, owned by the subscriber. All subscribers receive copies of all data messages in (near) real-time.
- Derived stream subscriptions: A standing subscription to receive a derived parameter in form of a sequence of data messages. The modality of the subscription, e.g. the frequency of updates can be requested.
- Event subscription: A stranding subscription to system events, filtered by common criteria, such as origin type, origin, sub-type or other filters. See CIAD DM SV Events and Notifications
See here for details:
The figure below shows the science data messaging flows and services to manage the related resources..
Figure 3. Data distribution flow and managing services (OV-1)
See here for details:
Data transformation capabilities take incoming data and derive data from these. This can be in streaming form or on demand. See here for details:
Ingestion is the process of cataloging metadata and data from an incoming science data stream for inventory purposes and for historical data retrieval. See here for details: