Skip to end of metadata
Go to start of metadata

The Exchange is the fundamental communication capability of the OOINet. It strongly influences the overall operations of the system (see Figure 1). The Exchange provides a message-based communication mechanism used between the OOINet processes such as services, instrument agents, the service gateway connecting the UI presentation platform and external service clients.

The Exchange and its associated resources are managed by the Exchange Management Service. Capability Containers provide a configurable messaging interface. All messaging complies with the Common Message Format.

Background: The Exchange is inspired by the Rich Service architectural pattern.

Overview

Message-Based System Integration

The Exchange uses messaging as the central paradigm of inter-application information exchange. Message-oriented systems and message-oriented middleware (MOM) technology realize a system architecture pattern that enables the flexible integration of systems of systems with loose coupling. Loose coupling is an important architectural property that has beneficial influences on maintainability, extensibility, robustness, scalability and other quality properties of the system and its individual software components.

Messaging infrastructures are based the concept of a message as the exclusive means of information exchange between the distributed components of the system. All information that is passed between two components is contained in the exchanged messages. Message exchange is asynchronous in that the sender of a message does not wait for the message to be delivered or returned. It only waits for the MOM to acknowledge the hand-over of the message. Delivering messages between participants uses the concept of queues. A client in a message-oriented architecture only knows the incoming queues where it receives messages from as well as the outgoing queues it delivers messages to, and has a notion of the message formats that pertain to these queues. The messaging infrastructure provides the capability for system integrators to connect these queues to known endpoints in the network; it thereby handles routing, storing and delivery of messages to the intended recipients across the network. The concepts of message exchange as the only means of inter-component communication, of logical, configurable queues and of message storing and routing are the main enablers for loose coupling of distributed components, such as in wide-area network connected, federated systems with intermittent network connections.

Messaging infrastructures typically provide two styles of addressing messages: point-to-point and publish-subscribe. The first case connects one sender with one receiver through a queue and thus decouples both components while still providing a peer-to-peer communication link. The second model enables one or multiple components to publish messages to queues that are then submitted to all components that subscribe to the queue.

The messaging infrastructure provides a number of quality of service guarantees that enable robust application design. Once the MOM acknowledges the hand-over of a message from the sending component, it guarantees delivery of the message to the intended recipient or alternatively notification of delivery failure to the sender. Persistent, transactional storage (e.g., a database) provides this basic capability. Similarly, for message delivery the MOM guarantees delivery of a message exactly once (and re-delivery in case of a failure) until the receiving component acknowledges its correct handling. Certain messaging infrastructures enable configuration of further quality-of-service parameters such as priority delivery level.

The messaging paradigm is strongly supported by modern development environments. For instance, Java supports messaging through the JMS (Java Message Service) API that enables access to any compliant messaging service provider. JMS is part of the Java EE (Enterprise Edition) capabilities. A recent movement to standardize the underlying transport has resulted in the Advanced Message Queuing Protocol (AMQP) defining the behavior of a messaging server and its client so that implementation is truly interoperable. The protocol provides point-to-point and publish/subscribe messaging, or combinations of the two.

Exchange Concepts

Figure 1 depicts the general architecture of the ION Exchange using Message Brokers as central infrastructure elements, and logical concepts such as Exchange Points responsible for the routing and delivery of messages. Message Clients provide the interfaces to the application logic.

Figure 1. 2660-00007 Exchange Architecture (SV-2)

Figure 2 specifies the roles and responsibilities of the different actors that are part of the ION Exchange.

Figure 2. 2660-00008 Exchange Actors (SV-2)

Exchange Spaces and Exchange Points are the resources that the Exchange service manages and provides. In short, Exchange Points are managed resources, where messages can be sent to and received from, without publishers and consumers having to know directly one another's existence. Exchange spaces are the organizational entities that group a number of Exchange points and there permitted users.

For instance an Exchange Space for unprocessed science data coming from all instruments of an observatory, such as the Coastal-Global (CG) scale observatory of the OOI. Publishers and consumers need to register with the Exchange Space beforehand, before being permitted to publish to or consume from an exchange point.

An ION process publishes, queries and subscribes to Exchange Points, binds Exchange Points through subscriptions, owns and manages the life cycle of Exchange Points & Exchange Spaces, communicates with Brokers using Sessions, manages the life cycle of sessions, has as identity (an address, name, and owner) and is a Finite State Machine.

A Broker creates, persists and federates Exchange Points & Exchange Spaces, communicates with Brokers & Clients using Sessions, manages the life cycle of Sessions, has an identity and is a Finite State Machine.

A Session manages message transfer, manages the life cycle of network connections, provides connection authentication and security, has an identity, has end point addresses, has a network protocol strategy and is a Finite State Machine.
An example for message transfer routing is

  • C1-1 transfers a message through the Exchange Point P1 on Exchange X to C1-2: C1-1 -> B1.X.P1 -> C1-2
  • C2-1 subscribes to X.P1 created on B2
  • C1-1 publishes Message to X.P1 created on B1
  • X.P1 delivers C2-1 subscription: C1-1 -> B1.X.P1 -> B2.X.P1 -> C1-2

Figure 3 shows a data product generation scenario using the entities defined above. The scenario makes use of the physical deployment sites of the designed OOI observatory network infrastructure. Observational data from a sensor is accepted by the Portland CyberPoP Acquisition Point via the instrument agent. The messaging infrastructure services provided by the CI Capability Container wrap the acquired data in form of self-contained, self-descriptive messages of type "raw data" and make them available to the Exchange. The sequence of data messages containing the raw data and descriptive metadata realize a data stream that can be consumed by any interested party connected to the Exchange. The Exchange transparently provides routing and distribution, for instance to data processing (in the Amazon cloud), the repository (in San Diego) and event detection (at a research team's institution) consumers connected to the Exchange. All consumers are based on the CI Capability Container and its messaging services. All of the data stream targets can be located at different points in the network. The Exchange realizes the data stream distribution network. It is internally comprised of message brokers that route and relay the message and ensure security and enforce policy.

Figure 3. CI Data Product Generation Scenario (SV-2)

Decomposition

Figure 4 shows the highest level of decomposition of the Exchange service into operational nodes.

Figure 4. Exchange Decomposition (OV-2)

The Exchange Management Service is used to manage the higher level functions of resources which make up an exchange; such as exchange spaces, exchange names, brokers, vhosts and federations etc. The use of a specialized management service is important to ensure secure access and management of exchange resources separate from process related activities, as well as, to ensure a consistent configuration of clusters of brokers that make up a federation or domain of authority. Essentially, the Exchange Management Service is responsible to managing and configuring lists of resources that make of the exchange space, which is synonymous with an Org or Domain of Authority.

Use of Exchange Spaces

There are several uses of how the Exchange Space framework is used in ION. Important cases include:

(1) Exchange Spaces represent communication namespaces for independent domains of authority (Orgs)

See CIAD COI OV Org

(2) Exchange Spaces represent intermittently connected communication networks - out of scope

The terrestrial side (always available) and multiple remote side platform networks are intermittently connected. Addressing communicating processes within and across these networks is an important function of the two component architecture:

See CIAD MI SV Two-Component Agent Architecture

Exchange Resources

Figure 5 below shows the resources managed within the Exchange. Data distribution (e.g. publish/subscribe) capabilities are based directly on the Exchange. The Exchange consists of two abstraction layers (see also Common Message Format):

Integration Level:

  • The Exchange consists of many Exchange Spaces
  • Each Exchange Space belongs to one Org (= Domain of Authority)
  • Each Exchange Space is a namespace of registered Exchange Names
  • Exchange Points exist in Exchange Spaces, providing pubsub routing capabilities
  • Communicating processes (such as service processes and agents) and Exchange Points have an Exchange Name

Messaging Level:

  • The Exchange is a Message Broker or Message Broker Federation
  • Many Message Brokers are within one Message Broker Federation
  • Each Message Broker has several sets of Broker Credentials
  • Message Brokers may either be AMQP (software) brokers or Solace (hardware) message routers. Logical concepts are mapped to messaging implementations, such as to topics, routing keys, queues

Figure 5. Exchange and Pubsub Resources (OV-1)

Exchange Entities

The following table shows the various OOINet entities using the Exchange. See also CIAD COI OV Exchange Management Service.

Entity 
System restart behavior
Description, Commands
Service request queue
Keep the queue and drain the messages
A queue that contains the service requests, shared across the service workers. Messages in service queues typically have a short timeout.
Process request queue
Keep the queue and drain the messages
A queue that is unique to one process (id), used to command this process. UNUSED in Release 2
Process RPC response queue
Cleanup old resources
A queue out of a pool that will receive responses to a request, for protocols such as RPC. Challenge: the queue is shared and incoming messages have to be tracable to a single process.
Stream subscription queue
Restart processes and let them process durable messages
A queue with bindings to data streams of interest. Note: ingestion is a special case of this
Event subscription queue
Restart process and have them decide what to do with events
A queue with bindings to events of interest, based on the inherent characteristics of events (event type, base type, subtype, origin)
Visualization queue
  A queue created to hold messages temporarily for certain applications, such as real-time visualization. The main characteristics is that at times there is no consumer for the queue.

Exchange Implementation

AMQP Implementation

AMQP 0.9.1 is the messaging standard/protocol of use for OOINet Releases 2 and 3. RabbitMQ is the message broker technology providing the AMQP implementation

See here for details:

Messaging Infrastructure

Message Broker

See here for details:

Exchange Federation (out of scope)

Federating messaging servers or appliances is out of scope for the OOINet. See Exchange Federation for the original designs and the intent.

References

See Also:

  • See here for governance interactions (negotiations) between the distributed entities interacting when joining an Exchange Space, using an Exchange Point and connecting to an Exchange Broker.

Labels

read-coi read-coi Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.