Skip to end of metadata
Go to start of metadata

Ocean Observatories Infrastructure Cyberinfrastructure Prototype Development

This page describes CI prototype development activities and their results. The OOI Cyberinfrastructure Subsystem Teams use prototyping to evaluate and advance system component designs and to reduce technical risk. The tables below list the OOI CI prototypes. Prototypes may involve external partner organizations and explore the wide range of technologies that are part of the CI design.

OOI CI Construction Prototypes (Sept 2009-Mar 2015)

Prototypes are performed during the inception and elaboration phases of releases 1-4. Designs and deliverables are documented on the respective subsystem development pages. These prototypes are integral part of the release engineering process, instrumental to the definition, refinement and evaluation of the OOI CI System Architecture. The following tables provides an overview of the CI prototypes.

Release 3 Prototypes

Elaboration R3I1 Prototypes

PageSubsystemTypePriorityIterationDatesStatusDescriptionLeadRisksResultsJira Task
Prototype Kepler workflow connected to Integrated ObservatoryASApproach AssessmentHighR3I1Sep - Dec 2012completedPrototype Kepler workflow connected to Integrated Observatory.Ilkay AltintasCIDEVAS-77
Prototype federation between OrgsCOIApproach & Design ConsolidationMediumR3I1Sep - Dec 2012plannedDemonstrate an affiliation between Orgs and the ability to share resources.Stephen HenrieCIDEVCOI-636
Prototype framework for callback style service operationsCOIApproach AssessmentMediumR3I1Sep - Dec 2012dropped no impactdetermine if possible to implement a callback-style mechanism that responds to messagesStephen HenrieCIDEVCOI-634
Prototype generating service interfaces automatically from Scribble conversation specsCOITechnology RiskMediumR3I1Sep - Dec 2012in progressPrototype generating service interfaces automatically from Scribble conversation specsNobuko YoshidaCIDEVCOI-633
Prototype integrating governance and conversation specsCOIApproach & Design ConsolidationHighR3I1Sep - Dec 2012in progressMonitor messaging to fully integrate governance and commitment framework to influence commitments.Stephen HenrieCIDEVCOI-635
Prototype negotiation agreementsCOIApproach & Design ConsolidationHighR3I1Sep - Dec 2012proposedShow how to represent agreements and their negotiationsStephen HenrieCIDEVCOI-637
Prototype ontology registration solutionDMApproach AssessmentMediumR3I1Sep - Dec 2012completedDemonstrate solution for registering ontologiesJohn GraybealReferenced MMI ORR as reference solutionCIDEVDM-629
Prototype MOOS asset operation in simulated environmentP&PApproach AssessmentMediumR3I1Sep - Dec 2012completedDemonstrate operation of one autonomous MOOS-controlled asset in simulation environment.Arjuna BalasuriyaCIDEVPP-5
Prototype MOOS simulation in ION virtual machineP&PApproach AssessmentHighR3I1Feb - May 2013completedSet up a simulation environment for one full vehicle MOOS community within a suitable ION virtual machine.Arjuna BalasuriyaCIDEVPP-10
Prototype ocean coverage model with simulation environmentP&PApproach AssessmentLowR3I1Sep - Dec 2012in progressGet recording from ROMS integrated into simulator, using a dynamic data set modeled/recorded over a time period.Arjuna BalasuriyaCIDEVPP-7
Prototype data processing sequence with KeplerSAApproach & Design ConsolidationHighR3I1Sep - Dec 2012in progressPrototype a processing chain for performing a typical OOI data processing sequence using Kepler.Brian SchliningCIDEVSA-442
Prototype integration of PUCK-enabled instrument with Integrated ObservatorySAApproach AssessmentHighR3I1Sep - Dec 2012in progressPrototype integration of a PUCK'd instrument with the Integrated Observatory.Tom O'ReillyCIDEVSA-441
Prototype minimum and desired PUCK contentsSAApproach AssessmentMediumR3I1Sep - Dec 2012completedProvide specification and examples of 'minimum' and 'desired' PUCK contents for OOI.Tom O'ReillyRecommendation to stick with minimal (factory data sheet) contents onlyCIDEVSA-440

Release 2 Prototypes

Inception R2I2 Prototypes

PageSubsystemTypePriorityIterationDatesStatusDescriptionLeadRisksResults
Prototype licensing solutions to run MatlabCEIFunctional RiskMediumR2I2Aug 2011completedInvestigate licensing solutions to run Matlab in a cloud (in general we will need to develop and prototype any "quirks" associated with execution engines).Tim Freeman, John Bresnahan2317 Cloud Computing Strategy
Prototype gevent based Python Capability ContainerCOITechnology RiskHighR2I2July - Aug 2011completedProvide a Python Capability Container using gevent for inner container concurrency, e.g. for concurrent execution of multiple CC-processes. Remove Twisted as the event handler.Adam Smith2312 Capability Container Robustness
2316 COI Integration
2313 Core Infrastructure Scalability
Prototype GovernanceCOIFunctional RiskHighR2I2June - Aug 2011partially completedDemonstrate in the context of ION a working application of governance of ION-based resources within ION facilities (domains of authority). Include all aspects of the governance life cycle in the demonstration software. Emphasize the compatibility of the governance approach with the ION system. It isn't yet clear what the rule engine will be that we will be able to use in Python (PyKE has some limitations that we could not easily overcome in a previous effort).NCSU Team2309 Governance Approach Readiness
2315 Collaboration Agreement Complexity
Prototype common object model implementation improvementDM, COIApproach AssessmentHighR2I2July - Aug 2011completedPrototype improvement of the current object model API and handing of the ion-object-definition package repository and package dependencies.Adam Smith2314 Distributed State Framework
2316 COI Integration
2313 Core Infrastructure Scalability
Prototype CI-SIAM on embedded platformMIApproach AssessmentMediumR2I2June-Aug 2011completedDeploy SIAM on the CG TS-7370 board, install actual instruments and SIAM driversCarlos Rueda2329 Marine Architecture Description
2322 Instrument Integration Strategy
Prototype embedded platform software deploymentMITechnology RiskHighR2I2May - Aug 2011completedPrototype deploying CI software on the CG TS-7370 board. CI software should include a messaging infrastructure , a simplified capability container or process framework and several processes performing recurring tasks within the embedded deployment. Goal is to find out which languages (C, Java, Python) work in embedded environments. Stability, resource use, etc.Edward Hunter2329 Marine Architecture Description
2322 Instrument Integration Strategy
Prototype Instrument Life cycleSAApproach AssessmentHighR2I2July - Aug 2011completedPrototype instrument life cycle using the R1 system infrastructure. Work with representatives of the Marine IOs and with knowledge form previous CI Workshop reports and UX user engagements to integrate life cycle knowledge into the ION.Alon Yaari2322 Instrument Integration Strategy
2308 Resource Life Cycle Services

Elaboration R2E1 Prototypes

PageSubsystemTypePriorityIterationDatesStatusDescriptionLeadRisksResults
Prototype visualization pipeline standaloneASApproach AssessmentMediumR2E1Sept - Dec 2011completedDevelop a standalone prototype integrating a set of visualization technologies into a simulated message driven processing pipeline (without integrating with the messaging system, the container or the process execution framework)Raj Singh
Prototype Process ExecutionCEIApproach AssessmentHighR2E1Sept - Oct 2011not started (realized in product)Prototype executing a pre-registered algorithm within an existing execution engine (e.g. Matlab script inside Matlab, or C program controlled by CC), connected to input queue and producing into output queueTim Freeman2318 Resource and Execution Management Services
2317 Cloud Computing Strategy
2313 Core Infrastructure Scalability
in production R2E3
Prototype Integrated GovernanceCOIApproach AssessmentHighR2E1Sept 2011 - Jan 2012completedDemonstrate in the context of ION a working application of governance of ION-based resources within ION facilities (domains of authority). Include all aspects of the governance life cycle in the demonstration software. Emphasize the compatibility of the governance approach with the ION system. It isn't yet clear what the rule engine will be that we will be able to use in Python (PyKE has some limitations that we could not easily overcome in a previous effort).Stephen Henrie2309 Governance Approach Readiness
2315 Collaboration Agreement Complexity
Prototype Integrated Policy EngineCOIApproach AssessmentHighR2E1Sept - Oct 2011completedThe intent of this prototype is to integrate a python based XACML implementation (ndg project)  available here: http://pypi.python.org/pypi/ndg-xacml/0.4.0 as a Policy Decision Point (PDP) within the container.Prashant Kediyal2309 Governance Approach Readiness
2315 Collaboration Agreement Complexity
Prototype Resource LifecycleCOIApproach AssessmentHighR2E1Sept - Nov 2011completedDemonstrate in the context of ION a working application of managing the lifecycle of an ION-based resources within ION facilities (domains of authority).Michael Meisinger2308 Resource Life Cycle Services
Prototype Data StreamingDMApproach AssessmentHighR2E1Sept - Dec 2011completedBased on the mechanisms of pubsub and the Exchange, prototype describing, registering and using data streams such that arbitrary consumers can (a) find and (b) subscribe to data streams. Consumers get a certain metadata context before actual real-time streaming would occur.David Stuebe2313 Core Infrastructure Scalabilityin production R2E3
Prototype Direct AccessSAApproach AssessmentMediumR2E1Sept 2011 - Feb 2012completedPrototype direct instrument access building upon the R1 system infrastructure. The approach allows authorized users to access an instrument directly (as if it were connected to a local serial port) but while maintaining connectivity to the CI. The CI also should log all traffic with the device during direct access mode.Alon Yaari, Bill Bollenbacher2322 Instrument Integration Strategyin production R2E3

Elaboration R2E2 Prototypes

PageSubsystemTypePriorityIterationDatesStatusDescriptionLeadRisksResults
Prototype for Web Application FrameworkCOIApproach AssessmentHighR2E2Nov-Dec 2011completedImplement a Python flask based webapp framework, integrated with CILogon and pyon messaging as replacement for Grails.Todd Boteler2311 Identity management technology integration
2339 R1 code refactoring
in production
Prototype Service BridgeCOIApproach AssessmentHighR2E2Nov-Dec 2011completedImplement a service bridge as service in a Python CC, bridging between browser side JSON/AJAX requests to ION common messages. Do not implement authentication and sessions yet.Todd Boteler2339 R1 code refactoringin production
Prototype GridFields and Scientific Data ModelingDMTechnology RiskMediumR2E1Sept - Oct 2011not startedPrototype "GridFields" as a better abstraction to model and perform computational activities over scientific data sets. Define a data model representation which can express the fundamental science data concepts described by the ISO 19123 coverage types as structured objects that can be operated on, persisted and transported.David Stuebe
Prototype Science Data Transport and PersistenceDMTechnology RiskHighR2E2Sept - Dec 2011completedPrototype how science data files are stored, accessed, transferred and manipulated. This prototype uses a SWE-based object model and HDF5 as data representation.David Stuebe2314 Distributed State Framework
2313 Core Infrastructure Scalability

Elaboration R2E3 Prototypes

PageSubsystemTypePriorityIterationDatesStatusDescriptionLeadRisksResults
ION Release 2 Architecture DemonstrationAllArchitecture DemonstrationHighR2E3Nov 2011 - Mar 2012completedImplement a service bridge as service in a Python CC, bridging between browser side JSON/AJAX requests to ION common messages. Do not implement authentication and sessions yet.Michael Meisinger, Tim Ampe2316 COI integration
2313 Core Infrastructure Scalability
in production
Prototype Conversation Layer, Monitor and ScribbleCOIFunctional RiskHighR2E3-R3C1February 2012 - July 2012 (three stages: for the detailed development plan, see below)partially completedDevelop the versatile and efficient communication layer and the corresponding monitor in the ION which enable execution and monitoring of diverse form of structured conversations catering for current and future application needs in the ION.Imperial College, Queen Mary College Team2310 Interaction Management Readiness
2338 Integration of governance/conversations
NB: See the "Risks" section below for detailed description of the perceived risks including the urgent ones and the mitigations means currently being taken.
There are three major stages of deliveries for this component:
  • Stage 1 deliverables were delivered in March 2012, reported in this page.
  • Stage 2 deliverables: 
    • the first part delivered in June 2012, reported in this page.
    • the second part delivered in July 2012, reported in this page.
  • Stage 3 deliverables: planned for July/August 2012.
Prototype Mesh Data ModelDMArchitecture DemonstrationHighR2E3-R2C1Feb 2012 - Mar 2012in progressPrototype implementation(s) that simultaneously inform, constrain, and validate the various concepts of the emergent OOICI Science Common Data ModelChris Mueller, Matt Arrott, Michael Meisinger2314 Distributed State Framework

Elaboration R2C1 Prototypes

PageSubsystemTypePriorityIterationDatesStatusDescriptionLeadRisksResults
Indexing with ElasticSearchDMApproach & Design ConsolidationHighR2C1April 2012Nearly CompleteDesign for ElasticSearch integrationLuke CampbellEliminated
SciCDM ConsolidationDMApproach & Design ConsolidationHighR2C1May - June 2012StartedComplete the design of the Data Model APIDon BrittainTechnology integration choices
Prototype CG-CI Common DriverMIApproach AssessmentHighR2C1Apr - May 2012completedProve that common driver is viable in the CG architectureBill French

Release 1 Prototypes

Prototype Name Subsystem Construction Phase
Dates Status Reports etc.
Governance and Policy Rules Engine Integration COI R1 Inception Sept 09 to Feb 10
completed
Summary
Capability Container with Service Integration COI R1 Inception Sept 09 to Feb 10 completed
Summary
Messaging Service Namespaces COI R1 Inception Sept 09 to Feb 10 completed Summary
Release 1 Operational Architecture Prototype (LCA Demo)
COI, DM, SA
R1 Elaboration
Mar 10 to Aug 10
completed
 
ANF-SQLstream Integration COI-CEI-DM R1 Construction Nov 10 to Jan 11 completed  
SIAM-CI Integration SA, IPAA R1 Construction Jan 11 to June 11 ongoing SIAM Integration Results Presentation
Instrument and Platform Agent Integration SA, IPAA
R1 Construction Jan 11 to May 11 completed IPAA Integration Results Presentation, Inst. State Models

Compressed Pilot Period Prototypes (Jan-Dec 2009)

During the OOI pilot period (Jan to Dec 2009), the emphasis is on prototype development and community deployments. The main purpose of pilot prototype projects is risk mitigation for the OOI construction period, starting Sept 2009. Many of the projects follow a 6 week iteration cycle. The compression of the pilot period due to the early construction of the OOI lead to a shift in the prototype schedule and the delay of some prototypes.

Prototype Name Participants Technical Scope, Comments Dates
Status SoW
Reports etc.
Agent Contract Network UCSD, NCSU
Researching the fundamentals of contracts and commitments in distributed applications spanning multiple domains of authority Feb 09 to Dec 09
completed SoW Deliverables, COI-Report
Cloud Provisioning Environment UCSD, ANL
Deploying and managing cloud computing resources automatically and based on demand Mar 09 to Oct 09 completed SoW
Poster, CEI-Report
Collaboration Tools Cloud Deployment UCSD, OL
Deploying the OOI collaboration tools (Confluence, Jira, Alfresco) in the cloud for increased availability Mar 09 to Jul 09 completed
  CEI-Report
Data Exchange UCSD, NOAA (IOOS)
Developing a cloud deployed science data distribution and processing infrastructure. This infrastructure should serve numerical modeling communities and provides cloud based data set caching, cataloging and processing. Server-side processing options include Ferret, Gridfields and Matlab, covering structured and unstructured gridded data Feb 09 to Feb 10
preliminary prototype available
SoW
Poster, DM-Report, see Signell e.a. JMS 2008
Global Load Balancing UCSD
Investigating a reliable global network routing strategy based on global site selectors
Mar 09 to Aug 09 completed

CEI-Report
Messaging Service: Client Adapter UCSD Developing an architecture and Python implementation for a framework that adapts any application and service to the Messaging service for reliable communication
Feb 09 to Oct 09 completed (continued in COI)
COI-Report
Messaging Service: Distributed IPC Facility
UCSD, Infinetics, RabbitMQ Investigating and developing a fundamental infrastructure for distributed inter-process communication (IPC) that requires enrollment of all participants and  explicit allocation of networking resources before any communication, i.e. read/write on distributed information state. Sep 09 to Dec 09 completed
SoW Final Report, Architecture ReportAPI-doc, COI-Report
Messaging Service: Broker Infrastructure UCSD, RabbitMQ
Develop the core transport and communication mechanism of the distributed system, based on a group of AMQP message broker installations. Virtualizes AMQP broker capabilities throughout the distributed system spanning multiple domains of authority.
Sep 09 to Dec 09 completed SoW Final Report, API-doc, COI-Report
OMF Collaboration Prototype UCSD, MBARI, NCSA, Lindquist
Integrating RedHat MRG and QMF with Instrument Proxies and Identity Management. Based on the OMF prototype using Enterprise Service Bus technology. Replace ESB with MRG. Provide an instrument agent platform that interfaces with Antelope ORB (via a to-be-developed ORB2AMQP bridge) and with existing SIAM instrument drivers fronting a serial and an Ethernet instrument
Aug 09 to Dec 09 completed
SoW
SA-Report
OSSE - Ocean System Simulation Experiment UCSD, JPL, MIT, Rutgers Integrating observational data and forecast model data products for the purpose of tasking autonomous resources, such as gliders. Based on the Data Exchange infrastructure for science data distribution. Applies the NASA JPL tools CASPER and ASPEN for observation resource planning and MIT's MOOS-IvP for autonomous control of mobile assets such as AUVs. The experiments and deployments are carried out in the Mid Atlantic Bight. Feb 09 to Dec 09 completed OSSE Pilot Plan OSSE-Report
OSSE MaRCOOS Model Integration OSSE team, MaRCOOS Integrating historical and operational model data products as well as observational data (SST, ocean color, glider profiles) for the OSSE in the MaRCOOS region Feb 09 to Dec 09 completed
OSSE Pilot Plan OSSE-Report
Semantic Framework Prototype UCSD, MBARI, MMI
Providing access to semantically enabled science data. Based on the Data Exchange prototype and technologies. Uses OpenDAP and the VSTO semantic framework and applies MMI (Marine Metadata Initiative) vocabularies and tools. Sep 09 to Dec 09 completed
SoW
DM-Report
SQLstream Prototype UCSD, SQLstream
Evaluating the SQLstream engine for science data stream processing. Takes the EarthScope Array Network Facility (ANF) processes and recreates them using SQL technologies and the SQLstream engine. Aug 09 to Dec 09 completed SoW
SA-Report
Unstructured Grid Services UCSD, OPeNDAP, UW Integrate Gridfields library and operators for manipulating unstructured gridded data in a data stream based network. Integrate the Hyrax data server backend (C++) with AMQP messaging. Sep 09 to Dec 09 completed SoW
DM-Report
Visualization Workflow Pipeline
UCSD, UW, Microsoft
Deploying a Microsoft Trident workflow engine leveraging the Microsoft Azure cloud with workflows feeding ocean model data through a visualization pipeline for UW COVE client. The architecture demonstrates an application architecture that spans client, server, and cloud using the Microsoft Platform.
  completed SoW
DM-Report

Previous Completed Prototypes

Most of the prototypes prior to FDR (Final Design Review, Nov 2008) were exploratory and conducted as a means of risk mitigation for the risk of integrating heterogeneous leading edge technologies. The attached reports contain findings and lessons learned. 

Prototype Name Participants Technical Scope, Comments Status Results
ESB & Instrument Integration Strategy OOI/Looking,
MBARI,
NCSA
A service and message oriented communication platform based on Mule ESB and ActiveMQ. Integrates Kepler workflows, SIAM and Antelope based instruments, Google Maps web interfaces, data source selection. Executed in a VMware virtual machine environment. Includes a distributed ESB deployment installation providing secure access to streaming instruments (MBARI, NCSA installation).
completed
2007
COI-Prototype Report, COI-Prototype Design
Continuous Visualization Pipeline OOI/Looking,
SCCOOS
A ROMS numerical ocean model (JPL) provides output data products via OpenDAP to a pipeline of VTK (Visual Tool Kit), OpenSceneGraph, OSSIMPlanet, and to Google Earth, using COLLADA. completed
2008
Prototype Report
High Availability Strategy Phase 1: Virtual Cluster OOI All phases of the high availability strategy prototypes address high availability, local clusters, cloud deployments, in a global network network environment. Phase 1 provides a deployment of multiple instances of a MySQL clustered database system, Confluence and Alfresco applications and the ajproxy load balancer within VMware and XEN virtual machine environments. Other technologies involved include Tangsaw, Spring framework
completed
2008
Prototype Report
High Availability Strategy Phase 2: Cloud Deployment OOI,
Globus/Nimbus (ANL),
CFT
Based on phase 1 technologies. Works out the deployment pipeline and how to package and contextualize images. Adds Globus/Nimbus (ANL) for contextualization of instantiated virtual machine images, the CohesiveFT Elastic Server platform for virtual machine image generation from software packages and Amazon EC2 and S3 services as cloud execution environments.
completed
2008
Prototype Report
High Availability Strategy Phase 3: Global Traffic Mgt & Resource Load balancing OOI,
Optiputer
Based on phases 1 and 2. Combines the virtual cluster and cloud deployment prototypes as execution platforms. Combined with Cisco global site selection and load balancing network technologies completed
2008
Prototype Report
Instrument Coupling with Ocean Modeling OOI/Looking,
SCCOOS
CalPoly mobile sensor network (Remus) and ROMS numerical models
completed
2008
Prototype Report
Data Distribution Network Version 1 OOI,
NOAA (IOOS, TBD)
Initial version of the DDN, released Jan 2009. Combines the technologies and implementations of ERDDAP, THREDDS Data Server (TDS), RabbitMQ. Deployed redundantly and scalably in a cloud execution environment based on Amazon EC2/S3. completed 2009
DDN Design Draft

Appendix: Prototype Concepts (not pursued)

See:

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.