- 1. Introduction
- Managed Instrument Network (Primary Theme)
- CI Infrastructure and Network Management (Secondary Theme)
- Enhanced Data Interaction and Visualization (Secondary Theme)
- Early Prototyping (Secondary Theme)
- 2. Product Management Activities
- 2.1 Release 2 Product Specification Activities
- 2.2 Release 2 Product Description Activities
- 2.3 Release 2 Adoption and Engagement Activities
- 3. Requirements and Engineering Activities
- 4. Design and Development Activities
- 4.1. Overview of Activities
- 4.2. Architecture Activities
- 4.3 Subsystem Activities
- 4.4. Cross-Subsystem Integration and Dependencies
- 4.5. Integration and Test
- 5. Implementation Activities
- 5.1 Marine Integration
- 5.1.1 Instrument Development Kit
- 5.1.2 Drivers for Sensor Set 1
- 5.1.3 Platform and Instrument Integration
- 5.1.4 Data Related to Marine IO Activities
- 5.2 CyberPoP Development and Deployment
- 6. Highlighted Activities
- 7. Appendices
This document outlines the plan for Release 2 Construction Phase. It provides a high level overview of all aspects of the construction phase and will be used as the basis for planning all Release 2 Construction Phase activities.
- Release 2 Product Specification
- Release 2 Product Descritption
- Release 2 Use Cases
- Release 2 Acceptance Scenarios
- Release 2 Elaboration Execution Plan
- Release 2 Construction Plan
- ION architecture specification
The following schedule will be followed for the three Construction phase iterations:
- R2C1: 03/26/12 - 05/18/12
- R2C2: 05/21/12 - 07/13/12
- R3C3: 07/16/12 - 09/07/12
The The Initial Operating Capability (IOC) review period will occupy the two weeks following the third iteration of Construction, with the review itself tentatively planned for the week of September 17, 2012.
- IOC: 09/10/12 - 09/21/12
This section is drawn from the Release 2 Construction Plan.
CI Release 2 is designated the Managed Instrument Network release, reflecting our primary theme of developing integrated support for OOI Marine Observatory instrument providers and operators. Secondary themes are support for operational management of the CI infrastructure and network, integration of computational tools for data manipulation, analysis, and visualization, and early prototyping of release 3 capabilities.
The goal of the primary theme is to provide end-to-end control of data collection and advanced control of managed instruments for OOI Marine Observatory instrument providers and operators. The features planned under this theme are:
This includes activities such as managing policies for marine resources (observatories, platforms, instruments, etc.), managing resources and their use, and tracking and auditing access to resources and services
This includes activities such as configuring and deploying instruments, monitoring platform and instrument state and usage, and performing platform- or instrument-specific commands
This includes most instrument lifecycle activities such as commissioning and decommissioning, testing for marine network qualification, and managing instrument participation in the OOI network.
This includes activities such as applying test procedures to test and troubleshoot an instrument, testing its data for OOI format compliance, and accessing supporting materials for resources in the OOI network (manuals, pictures, logs etc.).
This includes activities related to data acquisition and data product generation processes, including accessing and processing an instrument’s raw data, and controlling acquisition, dissemination, and persistence of data from platforms, instruments, or observatories.
The goal of this theme is to support those parts of operations and management of the CI portion of the OOI network that are not covered by existing network management tools. The features planned under this theme are:
This includes activities related to determining location, repair state, etc. throughout the life of the resource
This introduces personalized profiles for individual users, and supports association between a single user id and multiple external identities
This theme is aimed at providing more powerful support to science end-users, particularly in the areas of locating and working with data of interest. The features planned under this theme are:
This expands the tools provided for search-by-query of OOI data resources, introducing more sophisticated metadata search capabilities, filtering and organizing search results, saved query and prospective query capabilities
This introduces concepts of versions of data resources, and expands concepts of provenance
This allows externally created tools for generating data visualizations to be registered and used through the OOI ION interface, focusing specifically on developer-created visualization workflows and tools, and MatLab-based visualizations
The Early Prototyping theme allows for experimentation with features or technologies to be introduced in a later release. So far only one feature is planned:
Experiment with the opportunities presented by use of the OOI on handheld devices such as smartphones or tablet computers, and develop understanding of user desires and development requirements
Release 2 Product Specification activities in the Construction phase are limited to corrections to identified errors in the Product Specification.
During Construction phase, Product Specification materials for Release 3 and Release 4 will be developed.
In Construction phase Iteration 1, the Release 2 Use Cases will be updated to reflect team consensus developed through the Acceptance Test Scenarios development and review. These modifications will be submitted to CI Configuration Control upon completion, and will complete specification of the Use Cases for the Release 2 system.
The CI team will develop an Engagement and Adoption Road Map during the Construction phase, in consultation and coordination with Ocean Leadership and marine Implementing Organizations. The principals for this Road Map will be discussed and refined during the development of the document in the Construction phase.
The initial document will be released for review by Ocean Leadership and Implementing Organizations in Construction Iteration 1.
The Construction Phase calls for "completing formal integration, test, and verification to the greatest possible extent" with the CI Systems Engineer responsible for all verification activities. At the end of the phase, the verification procedures should be completed and execution of those procedures should be completed to the greatest possible extent.
To complete these activities, the CI Systems Engineer and supporting team members will perform or coordinate the following tasks during this phase:
- development of Integration and Verification procedures
- execution of Verification procedures against the applicable requirements; and
- continued support of Risk Analysis and Configuration Control activities.
- support for validation and acceptance of release leading to PMO acceptance
- support ongoing quality audits
The principal goals of the Construction phase are:
- Optimizing resources and minimizing rework so that development cost and schedule plans can be met,
- Maintaining quality,
- Producing final code, subject to testing and refinement, as quickly as possible,
- Completing formal integration, test and verification to the greatest possible extent,
- Documenting the release.
Essential activities during the Construction phase include:
- Management of resources and development processes to meet cost, schedule and quality constraints,
- Complete component development and unit testing,
- To the best extent possible, complete subsystem development through integration of components according to the Integration and Verification Plan and Integration Procedures,
- To the best extent possible, integration of subsystems into a release,
- To the best extent possible, verification of the components through integrated release against the applicable requirements,
- Producing candidate user documentation and training materials,
- Planning for defect correction and beta testing during the Transition phase, and deployment into the marine infrastructure thereafter.
These activities must be performed for Release 2 components (as appropriate).
|Applies to ALL service components||Applies to ALL use cases||Product content||Produced to these criteria||In this artifact|
|X||X||complete working code and unit tests checked in||all tests pass; stubs/remaining work detailed||Test Plans, Procedures and Reports|
|X||component's pending tasks are detailed (if Iterations remain)||described as explicit TestITs of 5d scope, assigned to iteration||Iteration task specifications|
|X||related Use Cases||Components and Requirements mappe; formal mapping exists following our described process||Use cases on Confluence linked to requirements in DOORS linked to services and their elements on Confluence|
|X|| all Integration Tests completed
||test written and passes||Test Procedures and Reports|
|X||X||related User/Operator documentation complete||component functionality visible in UI fully documented||Document Register|
|X||X||required User Interfaces coded, integrated, reviewed, and tested||reviews of both mockup and functional code complete||UX specification|
|X||any risks associated with the component are updated||reflecting results of work performed on component during phase||Risk register in SAF|
|| changes and refinements to system architecture and technology list as found out by production quality implementation and integration
||reviewed and baselined for milestone||Architecture Specification in Confluence and Enterprise Architect|
The high-level service architecture was baselined in the Elaboration phase, and the core system interfaces established. The operational architecture prototype, demonstrated at LCA, as well as accompanying performance and scalability tests proved the viability of this baseline architecture.
In Construction, the Architecture team will develop detailed design specifications for any remaining services and will support the integration of the subsystem services and the user interface as needed. The architecture team will reduce effort on Release 2 beginning with R2C2 and start preparing for Release 3. During R2C3, the architecture team will transition their effort completely to Release 3 and will only support integration needs.
The key areas of design effort are by iteration:
- Mesh data model integration and persistence (DM, MI, EOI, AS)
- Data discovery and catalog (DM, EOI)
- Different visualization methods (AS)
- Governance negotiation integration (COI)
- Elastic computing service integration (CEI)
- Dynamic service, object, resource and agent management (COI)
- Robust multi-site deployment (Integration)
- As needed for system integration
Note: some of the architects act in dual role also also as subsystem developers or development leads. They will continue acting in this capacity as needed, but reduce their effort on the Release 2 implementation in favor of preparing Release 3 prototypes to the degree possible. Where development efforts is required by the architecture team to complete Release 2, it will remain as high priority.
For each subsystem, a set of activities will be performed in Construction that accomplish the necessary technical progress for each component of the subsystem. The broad set of activities to be performed is described in detail in the project pages referenced in Table 4.1-1.
The broad scope of work for each subsystem is described here in Table 4.3-1 for the key subsystems of Release 2: COI, CEI, DM, SA, AS, EOI, and UX, with specific subsystem construction plans referenced by the link in the first column.
Finally, detailed tasks for each subsystem are reflected in the Appendix.
|Page||Subsystem Name||Subsystem Description||Release 2 Focus|
|MI Construction Plan||Marine Integration|
This Marine Integration subsystem is responsible for the design and implementation of the Instrument Platform Agent Architecture (IPAA). The IPAA is part of the CI Marine Network Integration which will implement, integrate and activate each of the Marine Nodes into the Integrated Observatory Network. The IPAA is responsible for the specification, development and prototyping of the OOI standard instrument agent and instrument/device driver strategy.
The focus of MI is to deliver the design and implementation of the Instrument Platform Agent Architecture (IPAA). The IPAA is part of the CI Marine Network Integration which will implement, integrate and activate each of the Marine Nodes into the Integrated Observatory Network. The IPAA is responsible for the specification, development and prototyping of the OOI standard instrument agent and instrument/device driver strategy.
|R2 AS Construction Plan||Analysis & Synthesis|
The Analysis & Synthesis subsystem supports a wide variety of data and data product analysis, manipulation, generation and presentation, especially through visualization. It also provides the virtual collaboration platform that will be applied to realize virtual observatories, classrooms and laboratories.
The focus of A&S-R2 is delivery of the following services:
|R2 CEI Construction Plan||Common Execution Infrastructure|
The Common Execution Infrastructure subsystem is responsible for the services to manage the distributed, immediate mode execution of processes.
The focus of CEI-R2 is to extend and utilize the CEI-R1 functionality by delivering the following services:
|R2 COI Construction Plan||Common Operating Infrastructure|
The Common Operating Infrastructure subsystem is a task and product account responsible for providing the services and distributed infrastructure to build a secure, scalable, fault-tolerant federated system of independently operated observatory components.
The focus of COI-R2 is to extend and utilize the COI-R1 functionality by delivering the following services:
|R2 DM Construction Plan||Data Management|
The Data Management Subsystem is responsible for providing life cycle management, federation, preservation and presentation of OOI data holdings and associated metadata via data streams, repositories and catalogs.
The focus of DM-R2 is to extend and utilize the DM-R1 functionality by delivering the following services:
|R2 EOI Construction Plan||External Observatories Integration|
This subsystem ensures the Integrated Observatory Network (ION) interoperates with Integrated Ocean Observing System (IOOS), Neptune Canada and the World Meteorological Organization (WMO). It is responsible for exploiting the CI integration strategy for external interoperations with independent observatory systems.
The focus of EOI-R2 is to deliver the functionality to interoperate with a core set of applications and services used widely within the IOOS community. This effort is developed in close association with the NOAA IOOS office and its Data Integration Framework project. This effort is also the top level application concern for ION-R2 that provides the external application integration context for the design and testing of the four active ION subsystem development efforts in ION-R2; Sensing and Acquisition, Data Management, Common Operating Infrastructure, and Common Execution Infrastructure.
|R2 SA Construction Plan||Sensing & Acquisition|
The Sensing and Acquisition subsystem is a task and product account responsible for providing the life cycle and operational management of sensor network environments as well as observing activities (i.e., scheduling, collecting, processing) associated with sensor data acquisition.
The focus of S&A-R2 is to deliver the functionality to access, control, modify, monitor, and document the complete operational life cycle of all instruments and external data sources within the OOI domain of responsibility. This effort is developed in close association with the “Instrument and Platform Agent Architecture” Control Account. This effort is also the top level application concern for this release and provides the application context for the design and testing of the four active infrastructure subsystem development efforts in ION-R2; Data Management, Common Operating Infrastructure, and Common Execution Infrastructure.
|R2 UX Construction Plan||User Experience|
User Interfaces, designs, templates presenting the capabilities of the Integrated Observatory to its users. Every release has a different target user audience and adds incrementally to the capabilities of preceding releases.
The focus of UX-R2 is on the following areas:
As part of the Construction phase, task deliverables from each subsystem will be continuously integrated and tested to maintain a full working system throughout. Each task deliverable will identify any cross-subsystem dependencies at the start of each iteration (or as soon as they are identified) to allow for prioritization of work across the entire development team. Cross-subsystem dependencies are maintained and tracked by the System Development Manager and are the responsibility of the appropriate subsystem leads.
The CI software will be integrated in an iterative way, building from the unit level up to the subsystem level and then to the CI system level. Integration testing will be planned for, designed, and built along with the subsystem functionality being integrated and tested. This will allow for integration as early as possible during the construction phase, and will ensure working and integrated functionality by the end of each iteration.
The subsystem development teams are responsible for integration and testing of components or services within their subsystems. It is the responsibility of the ITV team (with the assistance of the subsystem teams) to test multiple subsystems together to ensure an integrated system.
The integration plan for Release 2 can be found here: Release 2 Integration Plan
In addition to subsystem development activities, a number of implementation activities will take place in parallel to the Construction phase for Marine Integration and CyberPoP Development and deployment. While these activities are phased in parallel to the system development phases, they are nonetheless connected to the system development activities, and so are reviewed here.
The Marine integration team will finish build and test of the instrument development kit (IDK), develop drivers as part of the sensor set 1 construction, provide for platform and instrument integration, and facilitate the integration of data related to marine IO activities.
The Instrument Development Kit is a software and hardware environment for developing instrument drivers. For developing drivers, the IDK provides access to the instrument driver development framework, a collection of modules containing samples, utilities, and existing patterns from which to base new drivers. For testing drivers, the IDK emulates a node on a stand-alone CI implementation. Since the IDK updates itself from the main CI code base, it remains up-to-date with recent CI changes and therefore provides a valid suite of qualification tests to verify instrument OOI system compatibility. The IDK should also be usable for developing instrument drivers that are run from deployed platforms (such as the TS-7370). This will involve a system for deploying software from the IDK to the platform where it can be tested.
With the availability of the Instrument Development Kit, the team will begin developing drivers for Sensor Set 1 (as part of the Sensor Set 1 Construction phase). As circumstances and resources warrant, the team may begin developing drivers for Sensor Set 2 (running the Sensor Set 2 Construction phase in parallel).
During the Construction phase, the team will conduct extended discussions with the marine Implementing Organizations to verify the use cases and requirements for platform and instrument integration. Draft procedures will be established to support all phases of the instrument and platform life cycles. The instrument information collection and management plan will be monitored and updated during the Construction phase.
Several data resources will be produced by the OOI before the completion of Release 2. For these resources, interim strategies must be developed to capture and manage the data in the short term, and transition it to the working system upon completion of Release 2. While ION is not scheduled to be formally commissioned until the end of Release 3, the team expects to make a good faith effort to manage OOI data arriving before that milestone.
The data types known at this time to be arriving before completion of Release 2 include:
- data from test mooring deployments by the marine implementing organizations
- cruise data from cruises conducted by the marine implementing organizations
- science and engineering data from deployed gliders
- metadata associated with procured or developed sensors and platforms
- data associated with deployed cable infrastructure
- images associated with operations and development of OOI systems
In each case, interim measures may already be in place to at least capture the data being produced — for example, the CGSN IO is already capturing and publishing data from a test mooring deployment, the Sensor Database application written by PMO is being modified to capture additional metadata about sensors, and interim measures are being defined to capture and replicate data from deployed gliders. In some cases (for example, the test mooring data) this data is being acquired by ION, though possibly not in the final desired form.
In Construction, the Marine Integration team will work with the External Observatory Integration, the Product Management team, and the marine IOs to propose and agree on data paths for managing each of these assets. Most of the data paths will involve interim collections of the data, which will be transitioned to permanent collections once Release 2 is operational.
The primary deployment element of the OOI Integrated Observatory is the Cyberinfrastructure Point of Presence (CyberPoP). CyberPoPs provide hardware environments, deployed within a protected data center facility, that provide computation, storage and network resources.
A Terrestrial CyberPoP is the primary physical hardware and software deployment location of the OOI Integrated Observatory. Terrestrial CyberPoPs are OOI Configuration Items and are commissioned according to the Transition to Operations plan, including 2 Observatory Acquisition Points (OAP), 2 Observatory Distribution Points (ODP), and 1 Operations Management Point (OMP). The Observatory Execution Point (OEP) is a form of Terrestrial CyberPoP which is operated on the infrastructure of marine and cloud computation and cloud providers.
Marine CyberPoPs are OOI deployment sites in the Marine Networks, operated by the CGSN and RSN Marine Observatories (see External Interfaces), hosting CI developed Integrated Observatory Network software deployments, such as Instrument and Platform agents.
CyberPoPs provide the central connection points to the OOI National Internet Infrastructure (NII), as specified in the network architecture.
There are three types of OOI-hosted Terrestrial CyberPoPs:
- Observatory Acquisition Point (OAP): provides the primary point of access for Marine observatories to the CI, data acquisition, initial data processing such as Quality Assurance and Quality Control, and data preservation
- Observatory Distribution Point (ODP): provides data distribution across the ION distribution network and for peering with external network providers, cloud execution and storage environments such as the Amazon Elastic Cloud and the Teragrid
- Operations Management Point (OMP): provides observatory network and resource operations and state of health monitoring capabilities, and is deployed close to marine observatory and CI control centers
In addition, there is a Terrestrial CyberPoP hosted on the infrastructure of computation and cloud service providers:
- Observatory Execution Point (OEP): provides computational capabilities that can be provisioned on demand, to support the elastic provisioning in cloud environments (such as the Amazon EC2 and Scientific Clouds based on the TeraGrid) of Common Execution Infrastructure and other OOI services needed for execution of user processes such as numerical models and data visualizations.
Key infrastructure components need to be in place before the ION application can be used. The OOI National Internet Wide Area Network must be deployed to connect each of the CyberPoP locations by a high speed 10 Gbps network managed by third party provider. The selection of the provider will be done through an RFP process starting in Elaboration Iteration 1 with an award planned for Elaboration Iteration 2.
Another key component is the deployment of a high speed messaging system using the AMQP protocol and running on the national network. This provides direct application-level connectivity between capability containers in ION at users' sites. The selection of the vendor to provide this service will be done by a RFP process starting in Elaboration Iteration 3 with an award planned for Construction.
There will be five terrestrial CyberPoPs with the functions, locations, and deployment schedule as shown in Table 5.3-1. In Construction phase, the initial hardware configurations of the CyberPoPs will be built at UCSD and the ION software configurations will be loaded on the equipment. Upon verification that all components are operational, we will ship the rack to the CyberPoP location and connect it to the high speed network.
|CyberPoP Function||Location||R2 Deployment Phase|
|Acquisition Point||Pittock Building, Portland, OR||Construction|
|Acquisition Point||Woods Hole Oceanographic Institution||Construction|
|Distribution Point||Data Center at Univ. of Washington, Seattle, WA||Construction|
|Distribution Point||Starlight Facility, Northwestern University, Chicago, IL||Construction|
|Management Point||University of California San Diego, San Diego, CA||Operational|
This section describes the ION operational and integration activities taking place during the Construction phase of Release 2. Although not available for general public use, the Release 1 software will be tested by early adopters and members of the OOI team, and used by members of the CI team to evaluate and demonstrate operational features.
During Release 2 Construction phase, the Release 1 software will continue to be supported in a production configuration on one CyberPoP of the Integrated Observatory Network. This will provide the opportunity for users and operational managers to become experienced with the nature of the core Release 1 Integrated Observatory Network, and to develop and refine operational practices for this system. Changes to the Release 1 software will be minimized, to favor Release 2 development achievements.
In addition, Release 2 software will be deployed in a development configuration on multiple CyberPoPs simultaneously. This presents the opportunity for early construction, integration, and testing of a distributed ION system, and gives the operations team the chance to verify the operational configurations of the newly deployed CyberPoPs.
The schedule for the deployment of the Release 2 system to multiple distributed CyberPoPs will depend on availability of the appropriately configured hardware and software.
The following bullets describe the support attributes planned for Release 1 during the Release 2 Construction phase and beyond. The focus is on acquiring necessary expertise and maintaining a viable and compelling test system, while devoting the maximum possible resources to Release 2 feature development.
- Once launched, Release 1 will be operated as a 24x7 system, targeting the availability requirements of the formally released ION system in Release 3.
- Backup processes and equipment will be designed, deployed, and tested as needed during the Construction phase to achieve the targeted operational uptime.
- The CI Operations team will be the initial interface with the user community for any questions or problems encountered. They will dispatch and provide initial responses for all user inquiries, and will address most problems within the Operations team.
- JIRA will be the primary tool for documenting all problems and user communications, and tracking any follow-on work and disposition.
- An R1 Feedback committee made up of the Operations Manager, System Development Manager, Product Manager, Architecture Manager and User Experience Manager will meet regularly to review tickets that represent possible Release 1 scope increase, and identify appropriate resolution strategies. Any significant scope impacts will be identified and discussed with Ocean Leadership as appropriate. Tickets to be reviewed will include
- requests for new features (primary assessor: Product Manager);
- bug fixes that do not imminently impact the usability or acceptance of the Release 1 system (primary assessor: System Development Manager); and
- requests for operational scope beyond that budgeted for Release 1 (primary assessor: Operations Manager).
- The Release 1 system will be operated within a single CyberPoP, which will be deployed to Portland as soon as that site is fully configured.
During Construction Iteration 1, the CI team will devise and deploy usage measurement software to enable the OOI to monitor key metrics of system usage. Since the Release 1 system is not a commissioned system, it is not targeted as a high-usage system with a wide range of capabilities and mature user interfaces. Therefore, metrics will be targeted to obtain simple overall metrics of system use, and to give the operations, product management, and user experience teams experience with these metrics.
The Operations will monitor and report on collected Release 1 metrics during R1 Feedback committee meetings.
No major prototypes are planned during the construction phase.
No workshops are planned during Construction phase.
No conferences, meetings, or presentations are planned for Product Management purposes during Construction. The Cyberinfrastructure will participate in key
This PDF contains a list of all subsystem construction tasks: