- 1. Introduction
- 2. General Transition Activities
- 2.1 Defect correction
- 2.2 Unit test extensions
- 2.3 Alpha testing of the end-to-end R2 system
- 2.4 Performance, stress and stability testing
- 2.5 Miscellaneous testing
- 2.6 Verification
- 2.7 Integration Testing
- 2.8 Packaging and assembly
- 2.9 Documentation
- 2.10 Acceptance
- 2.11 Transition to operations planning
- 3. Integrated Product Teams (IPT)
- 4. San Diego Development and Testing Environment
- 5. Release 2 Deployment Plans
- Release 2 Deployment Overview
- Assumptions and Constraints
- Core Infrastructure
- Statically Operated Dependencies
- ION system
- External Interfaces
- System User Access
This document outlines the plan for Release 2 Transition Phase. It provides a high level overview of all aspects of the transition phase and will be used as the basis for planning all Release 2 Transition Phase activities.
Because some functionality for Release 2 will be delivered in a follow-on point release, references to Release 2 will encompass the initial deliverable for release 2 (i.e. R2.0). Follow-on functionality for release 2 will be delivered in a release described as the point release (R2.1). Functionality for the 2.1 point release will comprise product changes due to feedback from early users of release 2.0 as well as important R2 features not completed in time for 2.0.
- Release 2 Product Specification
- Release 2 Product Descritption
- Release 2 Use Cases
- Release 2 Acceptance Scenarios
- Release 2 Elaboration Execution Plan
- Release 2 Construction Execution Plan
- Release 2 Construction Plan
- ION architecture specification
The following schedule will be followed for the two transition phase iterations:
- R2T1: 11/19/2012 - 12/11/2012 (3 Weeks excluding Thanksgiving holiday)
- R2T2: 12/12/2012 - 01/09/2013 (3 Weeks excluding Christmas/New Years holidays/Closures)
The Product Readiness Review (PRR) review period will occupy the two weeks following the second iteration of Transition:
- PRR Period: 01/10/2012 - 01/23/2012
- Review: 01/22/2012 - 01/23/2012
The Transition phase of the OOI system life cycle is predominantly focused on testing and defect correction.
Management of Transition activities resides with the System Development Manager and System Engineer. The former manages all day-to-day test and defect correction activities, and coordinates work assignments to the developers and Integration and Test Team. The System Engineer is responsible for delivering a quality and tested system to the PMO at the end of Transition for the acceptance process, and manages all verification testing. The Project Scientist is responsible for validation of the release. The QA Manager is responsible for auditing the processes and products of Transition on an ongoing basis.
Code freeze occurs at the end of the Construction phase (11/02/2012) for R2. Except for defect correction, code features are not expected to be added after this date (with the exception of the R2 point release 2.1).
Defect correction is the core activity of Transition. At the end of Construction and prior to the PRR milestone, the developers are required to enter remaining code bugs into Jira as defects. Defects will be carefully tracked using Jira (i.e., added to or closed) throughout the Transition phase in a dynamic manner. See: Bug Tracking During Transition for details on the process.
Developers, members of the I&T team, and alpha/beta testers are all required to report bugs in a formal manner using Jira as they are detected and/or corrected. The System Development Manager will conduct multiple weekly scrum meetings and coordinate bug triage with the Senior Developers and Integration and Test team. The System Engineer and System Architect will also participate in these meetings.
The defects in Jira constitute the Defect List that the System Engineer is required to report to the PMO at bi-weekly intervals during Transition, and defects will continue to be tracked once R2 is complete.
At the end of Construction, unit tests are expected to provide about 60% coverage of the R2 code base. However, industry best practices require about 80% (or more) coverage, and consequently the effort will be made during Transition to expand the unit test base to achieve this level. The System Development Manager will coordinate the activities of developers between defect correction and unit test extension throughout Transition, placing priority on the former but ensuring that developer time is used efficiently to achieve a broader test base. See current code coverage levels for pyon Capability Container: http://ooici.net/coverage/pyon_combine/ and ION Services: http://ooici.net/coverage/coi_combine/
Alpha testing by representative users is an important input to the defect detection process. The target users for Release 2 are each of the OOI Implementing Organizations (IOs) as well as the CI Operations team. Each target user will assign one or more alpha testers to run through their own set of tests on the R2 end-to-end system to make sure it satisfies their requirements. They will be assisted by the System Development Manager and Senior Developers, who will use the input of alpha testers to detect defects and track improvement requests and enter them into Jira. The alpha testing cycle will be initiated early in the transition phase with a demonstration and walk-through of the system.
Besides all the functional testing that is performed on the system, non-functional testing will also be performed during transition which includes: Performance, Load, Stress, and Stability testing. The System Development Manager will coordinate the non-functional testing effort with assistance from the I&T lead. As part of this effort additional tests will be created and automated for regression testing purposes. As part of this effort, a tool similar to "Chaos Monkey" will be used or written to randomly kill instances and services within a running system to simulate unexpected outages.
As part of end-to-end testing, live test users will perform ad hoc testing of the end-to-end running system. This type of testing is used to uncover additional defects not normally encountered by automated or manual scripted testing.
Verification is testing against the L3 System and L4 Subsystem requirements, and consitutes the process of asking "was the system built right". There are three verification approaches used in R2:
- Inspection-- comparison of an item to documentation to confirm compliance with requirements. Inspection is typically used to verify properties that can be confirmed by examination. It may also be used to verify an L3 System requirement as a roll-up of all linked L4 and lower level requirements.
- Demonstration-- the action of showing that the requirements are fulfilled through direct evidence or observation of the response of an item of software. Demonstration utilizes a set of selected test activities and inputs to show that the response is correct.
- Test-- a process by which the functionality or performance of an item of software is assessed under specified, controlled conditions that may either be simulated or real. This may require a special test harness and post-test analysis.
- Analysis-- the processing of accumulated data obtained from other qualification methods. Examples are reduction, interpretation, or extrapolation of test results.
A procedure using one of these approaches is assigned to each requirement in R2, is documented in the Release 2 Verification Overview , and is an IOC deliverable. Verification is carried out by the I&T team and/or Senior Developers, and is the responsibility of the System Engineer, who delivers a Verification Report for the PRR milestone at the end of Transition.
End-to-end Integration testing will continue during the transition period to make sure that all aspects of the system are fully functioning. Integration test procedures are located here Release 2 Integration Procedures and is the responsibility of the System Development Manager and the I&T lead. Results from the testing will be provided within the Integration Report delivered at the PRR milestone.
Software development teams deliver their components, services and subsystems in the form of software packages and assemblies of packages. Each software package or assembly has an associated Git source code repository. Packages may have dependencies on other delivered packages that must be specified as part of the package. Each package or assembly delivery is uniquely identified by a set of attributes, including package name and version number. At the end of Transition, the release will constitute a set of packages and assemblies that must be available to the R2 user community. The System Development Manager is responsible for this activity with the assistance of the Senior Developers.
See Release 2.0 Packaging and Assembly page.
At the end of Transition, a set of documentation of the release is a deliverable at the PRR milestone. This is comprised of the following elements:
- Release 2 User Manual that describes the capabilities, workflows and user interfaces of the release
- Release 2 Service Guide that describes the service interfaces callable by developers
- Release 2 Operators Manual that includes build instructions, descriptions of what can be modified in run-time and how to do that, and general operations information
The System Engineer and System Development Manager will plan for the acceptance process that follows the PRR milestone. This primarily requires ensuring the the required documentation is provided to the PMO for acceptance.
The System Engineer, System Development Manager and Operations Manager must plan for the transition to operations after acceptance of the CI release by the PMO. This includes planning for "hot fixes" or patches to the code once Release 2 goes live. The transition to operations process is described in 2110-00002 Transition to Operations Plan.
The following links provide details of transition activity on a subsystem by subsystem basis:
- AS Release 2 Transition
- CEI Release 2 Transition
- COI Release 2 Transition
- DM Release 2 Transition
- SA Release 2 Transition
- ITV Release 2 Transition
- UX Release 2 Transition
During Transition, the ITV team drives the testing of the CI Release 2 integrated software on the San Diego Testing Environment. This is comprised of the following:
- 3 Dell 810 servers running VMWare and hosting:
- A cluster of static VMs hosting BigCouch and Elastic Search databases
- A cluster of static VM hosting the rabbitmq broker
- A static VM hosting the ZooKeeper database
- A static VM hosting Apache server and flask plugin hosting the R2 Web Application
- A static VM hosting ERDDAP server and entry point to datasets
- A static VM hosting gralog syslog server
- A static VM hosting fault-tolerant RSN Port Agents
- 5 Dell 810 servers running KVM and hosting:
- A set of dynamically launched VMs (via Nimbus) running R2 Capability Containers hosting sets of Release 2 ION Services
NOTE: See next section for additional details on the R2 Deployment Environment
|This page describes the deployment of Release 2|
Figure 1 shows an overview of the high-level Release 2 deployment architecture.
Figure 1. Release 2 Deployment
- The launched ION system will only be running in Portland
- ION persisted will be replicated by operations tools to the Seattle CyberPoP
- In addition, data will be replicated at acquisition time on remote platforms, Marine IO data servers and similar
- Woods Hole Acquisition Point will act as network interchange only to secure CI
- Internet and cloud access occurs via Seattle and Chicago CyberPoPs
- May host firewalls, load balancers and web servers
- Monitoring of the ION system will occur from San Diego Engineering Center and from other locations
- No hardware messaging appliances in R2
- Portland CyberPoP has Nimbus (KVM) and VMware hypervisors
- CyberPoP network and hardware infrastructure includes
- Compellent storage system accessible via NFS and VMware mounts on VMs
- A10 Load balancer
- VMware hypervisors will host
- Apache web server with Flask plugin
- RabbitMQ brokers (cluster, federation) - one or multiple
- CouchDB cluster or BigCouch cluster - one or multiple
- EasticSearch cluster
- ZooKeeper cluster
- ERDDAP data externalization server
- RSN port agents
- iRODS server installation (for data acquisition from CG)
- Nimbus installation on Portland CyberPoP
- Contains ContextBroker
- Messaging system maintains persistent state (durable queues)
- Launched via cloudinit.d using CEI software stack
- Services are launched first, then agents, then transform processes
- Services are bootstrapped by boot level in dependency order
- Restart cleans up persistent information and messaging state before system start
- ION system contains web service gateways (http/https servers fronting services and agents)
- ION system contains pyDAP OpenDAP server, fronting the internal science data persistence
- To CG Marine IO
- CG will operate 3 OMCs (Operations and Management Center) in Woods Hole (WHOI), Corvallis (OSU) and San Diego (SIO)
- OMCs are designed redundantly but each nominally fulfills a dedicated responsibility
- An iRODS data sync exists with each OMC
- Each OMC has a CI accessible software API for core command & control (e.g. establish telemetry) and status
- To RSN Marine IO
- RSN will operate the OMS system out of Portland, OR
- All ports on the RSN cable are accessible via IP to CI software
- To EPE IO
- To ION operations
- Scripting tools, such as preload, automatic load
- To the general public (end users)
- See below
- Via central ION domain names on ports 80, 443
- Authentication via users' home identity providers using the CIlogon service
- CG and RSN marine operators can connect via virtual serial port software for direct instrument access
- Science end users can access the ION system OpenDAP servers (ERDDAP, pyDAP)