Skip to end of metadata
Go to start of metadata

(Note the child pages of this page)


The organizational concept here is to provide a structure to coordinate the top down view of elaboration. Section one introduces the context of the elaboration phase.

Section two is a place holder for the overall product description - the bullets from Michael's LCO slides. (See also; note the Product Description is being generated independently of this page.)

Section three breaks out each subsystem: what the goals are for the elaboration phase? what is a 'functional system' for each of the subsystems? Starting with a description of what capabilities the demonstration at LCA should include (also being covered as part of product description). Child pages link to the dependencies and interfaces for each subsystem. These pages should describe in detail the OV2 diagram of Logical Architecture Dependencies. In this context, interfaces means the gray areas between subsystems where responsibilities are shared and developement must be closely coupled. Each subsystem has a link to what it provides to others as well as the other subsystems on which it depends. There are only four pages, but they are represented under each subsystem where the dependency exists. It will be a collaborative effort to fill out these pages - for each X marked in the table the leads and architects need to coordinate their activities during the elaboration period.


  1. Guideline to Development Phases
  2. Overall Product Description:
    • Core deliverables/technologies: what is the application going to provide
  3. Elaboration Goals by Subsystem:
    • A description of the subsystem product at LCA: what is the demo?
    • Links for:
      • specific infrastructure technologies
      • Inter-subsystem interfaces and dependencies
  4. Integration plan, time line (rough)

1. Guideline to Development Phases:

  • Inception phase product is a viable architecture
    • Success at LCO marks the transition to the next phase
  • Elaboration phase product is a functional system
    • Success at LCA marks the transition to the next phase
  • Construction phase product is an operational system

2. Core Deliverables: Overall Product Description

This includes the service architecture,l the technologies involved, and the necessary user & external interfaces. For us, this mostly speaks to the DM functions for the IOOS use case.

Release 1 OOI CI Funcitons - From Michael's LCO presentation:
  • Instrument and platform interfaces
  • Instrument and platform control from the observatory
  • Direct access to instruments and platforms
  • Data acquisition from sensors and external observatories
  • Management of science data and general observatory information
  • Data inventory and metadata annotation
  • Data ingestion, i.e. canonicalization and metadata annotation
  • Data discovery, cataloging and presentation
  • Data distribution, replication, streaming
  • Data persistence, assurance, retrieval
  • Process and service execution
  • Virtualization of computing
  • Elastic scaling to demand
  • High availability services
  • Service-orientation (not a function)
  • Reliable message based asynchronous communication
  • Security: identity and policy management
  • Multi facility system integration
Release 1 Data Distribution Network

what does 'Data Distribution Network' mean here? the last 3 don't fit. jbg

"Ingest" data (products) from external data sources; both streaming and cataloged
Characterize data sources with their metadata attributes (format, structure, meaning)
Transform external data into internal OOI data formats, for stream-based distribution and persistence
Distribute data via streaming and DAP servers to data analysts and numerical modelers

Provide initial instrument integration, control and sensor data acquisition
Provide instrument and platform direct access
Provide service integration and execution platform

Need more details - links to child pages for each of these!

3. Use Cases and the LCA Demo:

Each Subsystem is listed here - most to be filled in. The dependencies to other subsystems are listed below as links.

3.1 COI Subsystem: End to End functional Demo

Specific demo opportunities (e.g. operator defines xy resource access policy and then approves a request of another user to join the group leading to different policy and results).

COI scope in R1
  1.  Capability container
    • Provide a reliable capability container for service deployment. This capability container should provide all agent/governance/policy and identity management as necessary for service interaction.
      • We can go through LCA withouth governance/IdM fully in place, but we have to show that we have hooks in place.
      • Must have the OOI message header in place by LCA
      • Whether we implement the Facility model in release 1 has still to be decided. We don't need the edge of domain of authority in release 1, but if we don't implement it in R1 we have to redo a lot of code to support it in R2, R3.
    • Technologies:
      • Capability container -Python, Twisted, txAMQP
        • In Elaboration phase COI has to bring up a Java capability container! We need to support NetCDF at LCA, which will require Java capability container
      • Messaging Service - Exchange Spaces, RabbitMQ AMQP broker
      • DIF (application networks - VLAN - Network Interface)
        • What is the connection between the networking architecture and the messaging arch?
  2. Resource Agent
    • Provide an initial resource agent implementation with integration into the capability container and the control of policy enforcement
      • Policy & Governance - Jess/Pyke rules engine
  3. Resource registry
    • Together with DM, implement the core resource registry service and enable specialization by resource type specific services, for instance a service (resource) registry
    • In R1 we have to solve the multi-process, multi-site, single domain of authority data store.
    • Resource Registry Data Model - implemented on top of Data Store Data Model
      • Resource Registry: aka Data Store Redis, Cassandra, other? 'Eventual Consistency' is our target for R1
      • Propose to use CUAHSI architecture as point of departure for the evaluation. Project the CUAHSI data model on the Redis dm and Cassandra dm to evaluate (define metrics for the evaluation). Look at how the querries work on the data models.
      • Should we consider using both Redis & Cassandra? Each is optimized for a different purpose. Example architecture: each worker process works on a cached data, which has a relation with the data store. Assess what is viable and what we can decide in R1
  4. Resource description
    • Together with the DM subsystem, define the attributes and lifecycle that each resource follows, so that resource management can occur uniformly across the system.
  5. Identity Management/Secure Messaging
    • Ability for user to register and authenticate using InCommon identity provider.
    • Ability for an entity to create authenticated messages.
      • For the sake of being minimal in what we promise, I am assuming confidentiality is out of scope.
    • Ability for an entity to receive an authenticate message and verify that identity.
Interfaces and Dependencies to other subsystems:

What does COI Provide? (Capability Container)

What does COI need from CEI? (Process Execution)

  • Idm: Nothing.

What does COI need from DM? (Repository)

  • Idm: Idm needs a "principal registry/identity repository" for use by the Authentication and Registration services. This is basically a repository to store bindings from external to internal identities (tuples).

3.2 CEI Subsystem: End to End functional Demo

Specific demo opportunities (e.g. start an EPU and run a DM service or operation...)

Interfaces and Dependencies to other subsystems:

What does CEI Provide? (Process Execution)

What does CEI need from COI? (Capability Container)

3.3 DM Subsystem: End to End functional Demo

At LCA the DM will demonstrate subscription to a data source with streaming updates. The data is ingested (Propose: CDAT python library used in pydap - closest to Unidata CDM implemented in python or NetCDF Java - the CDM reference implementation) into the CI native encoding (propose: Google Protocol Buffers) and stored in the inventory. The inventory is an expanded (propose: REDIS or Cassandra) attribute store, a prototype for the COI resource registry. The CI Data model will support one feature type (propose: Point Feature), as an end to end example, full Unidata CDM compliance will be added by R1. A data set can be found in the inventory using the ontology of the data source and retrieved by the user with a DAP request. A data stream can be found in the inventory using the ontology of the data source and the user can subscribe to the exchange point for the stream. The user will receive the stream in the CI native encoding.  The currently available (LCO DX Demo) transparent proxy service will provide DAP access to data which is currently in the inventory. A cache miss on a proxy dap request will pass through to the original source and register that data set for review by the operator since there is insufficient Metadata (No owner or policy info) to automatically ingest it with such a request. The demo will include running across multiple EC2 instances, showing fault tolerance and concurrent execution of requests.

Elaboration Period Work Plan:

1. The service architecture of the DX must be re-factored and filled out to reflect the DM architecture and use the latest version of magnet.
2. The resource registry service and resource descriptions must be implemented to store and retrieve information resources in the inventory.
3. The ingestion and transformation services need to be implemented to convert from DAP to the CI native encoding on ingestion and back to DAP for external presentation. We may also need to handle other encodings used in IDD.
4. Example transformation services for at least one feature type from the Unidata Common Data Model must be implemented to provide an end to end demo by LCA.
5. The persistent archive service must be prototyped and the integration plan specified.

DM Task planning for encoding & Resource registry:

Issues to resolve:
Encoding - Google Protocol Buffers, Fudge?
File system

Approach - Subsystem must provide their data model first 
Then we can pick an encoding

Data - cuahsi
Instruments- SSDS
ID - Globus/Linux/PAM model
VM image - amazon or eucalyptus (ubuntu enterprise cloud)
Instance - UEC - ubuntu cloud, eucalyptus
Services/processes - tree structure, with dependecies, a DAG

Tasks for DM

Present above models as strawman to subsystems
Map above data model to abstract data model

Tasks for COI

Create abstract data model and encoding rules

How dow we access atomic level science data? Other subsystems just work with objects.

Interfaces and Dependencies to other subsystems:

What does DM provide? (Repository, Data pub/sub, Ingestion/Transformation/Presentation, DM Control and Status, Operation Control and Status)

Got this text from  2650-00008 OV2 CI Services Networks - not sure about the last two items?

What does DM need from COI? (Capability Container)
What does DM need from CEI? (Process Execution)

3.4 S&A Subsystem: End to End functional Demo

Only the basic functionalities will be demonstrated for LCA. Features such as scheduling and governance will not be in place.

  • Pick most commonly used sensor (e.g CTD)
  • Simple UI (shell type) which will provide features for services such as;
    • Register an instrument
    • Register data processes
    • Pick an instrument from the inst. registry
    • Pick the data type from data process registry 
    • Request services - Direct Access or Data Acquisition
  • Working code encapsulating above features to demonstrate on how a sensor such as a CTD is connected to CI and operated in Direct access and data acquisition modes, parsing data to the ingestion services in Data Management subsystem. The demonstration will also capture hand-offs, such as from Data Acquisition (Observatory) mode to Direct Access mode, and vice-versa.
Interfaces and connections to other subsystems:

What does S&A Provide? (Observatory Control and Status)

What does S&A need from COI? (Capability Container)
What does S&A need from CEI? (Process Execution)
What does S&A need from DM? (Repository, Data pub/sub, Ingestion/Transformation/Presentation, DM Control and Status, Operation Control and Status)

Instrument Agent---Steve Foley

Will have to use the agent architecture designed by COI/CEI/DM.

Needed for Sensor prototype

Interfaces and connections to other subsystems:

What does I A Provide?
What does I A need from COI? (Capability Container)
What does I A need from CEI? (Process Execution)
What does I A need from DM? (Repository, Data pub/sub, Ingestion/Transformation/Presentation, DM Control and Status, Operation Control and Status)

External Observatories integration---*David

Needed for IOOS collaboration

Are the tasks and work plan for this component due at the end of Design Period?

User interfaces/user experience---*Susanne Jul

  • behind on this
  • won't ask end users of R1 (not administrators) to download something in R1
  • operator in R1 is our own operations team, process of deployable types may address this (CEI deliverable).

4. Goals for Alignment of Architecture and Implementation

- There is feedback between service architecture (names, dependencies) and the service names in the implementation in both directions
- Service names, service interactions (message types & sequences), message types are specified (in EA or Confluence) for the core elements and the implementation adheres to these specs.
- There is a standard development environment (GIT, doxygen, unit tests) applied for all packages in the LCA prototypes
- There is a build process that can build a dependency of packages and build an integrated package, with test cases
- COI and CEI infrastructure packages can evolve within limits as the apps evolve
- There is a deployment mechanism that can start a set of all the processes for an integrated prototype almost automatically (locally on one machine, in the cloud, in a cluster, wherever)
- There is a package that defines the example data used (datasets, test cases, etc) for demo scriptability and repeatability


Design period list the work that needs to be done

Iteration 1 develop the plan

Iteration 2 Teams implement the plan

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Mar 09, 2010

    William Pritchett says:

    We also need to figure out what UIs will need to be prototyped (I didn't see tha...

    We also need to figure out what UIs will need to be prototyped (I didn't see that explicitly mentioned anywhere) and I think it needs to be an explicit task so it's not treated as an afterthought.

  2. Mar 10, 2010

    David Stuebe says:

    Thanks Bill, I have added a section for the user interface and user experience. ...

    Thanks Bill, I have added a section for the user interface and user experience. We know we are behind on that and hope to have it better fleshed out soon. Our work won't do much good if no one can use it!