Skip to end of metadata
Go to start of metadata
This is the central page describing OOINet data distribution capabilities. This includes science data message streaming and subscription to data streams. See child pages for details.

Overview

The Distribution service provides the pub/sub/notify mechanism to manage data streams and all related resources.

The figure below illustrates the core concepts related to messaging and data streaming:

  • Producer (process) - Publishes messages to streams
  • Stream - A predefined identifier for message sequences from a publisher
  • Message/Granule - A data message
  • Subscription - Query to receive messages matching streams and other criteria in queue
  • Queue - A protected buffer on the broker holding messaging in FIFO order until consumed and ack'ed
  • Consumer (process) - Taking messages in FIFO order off a queue

Figure 1. Messaging Concepts and Message Processing Patterns (OV-1)

Decomposition

Figure 2. DM Distribution Services Overview (OV-2)

The Pubsub Management Service manages Publish/Subscribe message routing. Subscriptions are defined by what a consumer wants to receive. In the case of subscriptions to streams (there are also subscriptions to events, see Event Subscriber) these are characterized by queries against stream topics. Internally, these queries are realized using COI infrastructure as a set of broker routing rules. The broker delivers messages selected by the query to a queue specified by the subscription.

Publish-Subscribe (Pubsub) Resources

The following resources are managed by the data distribution services. They are essential for the data streaming pubsub architecture.

Stream

Identifies a resource that can be mapped to a messaging pubsub topic, such that the sequence of messages published to this stream can be received in the same form in real-time by all subscribers to this data stream. See here for details:

Stream Definition

A detailed description of the content of science data stream messages (granules), in particular the science data parameters they transport. See here for details:

Subscription

A standing request to the system to receive science data messages related to a data stream or derived concept. Subscriptions are owned by the subscriber. Multiple subscriptions can exist for the same stream and all subscribers will receive independent copies of the data messages. The following types of subscriptions exists and further types may be added in the future:

  • Simple stream subscription: A standing subscription to a message stream, owned by the subscriber. All subscribers receive copies of all data messages in (near) real-time.
  • Derived stream subscriptions: A standing subscription to receive a derived parameter in form of a sequence of data messages. The modality of the subscription, e.g. the frequency of updates can be requested.
  • Event subscription: A stranding subscription to system events, filtered by common criteria, such as origin type, origin, sub-type or other filters. See CIAD DM SV Events and Notifications

See here for details:

Behavior

Science Data Messaging and Managing Services

The figure below shows the science data messaging flows and services to manage the related resources..

Figure 3. Data distribution flow and managing services (OV-1)

Stream Registration and Subscription to Streams

See here for details:

Data Transformation

Data transformation capabilities take incoming data and derive data from these. This can be in streaming form or on demand. See here for details:

Data Ingestion

Ingestion is the process of cataloging metadata and data from an incoming science data stream for inventory purposes and for historical data retrieval. See here for details:

Labels

ready-for-review ready-for-review Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.