![]() | This is the home page for the Data Management (DM) subsystem Integrated Product Team. This subsystem is constructed in Releases 1 through 3 of OOI CI construction. |
Welcome to the Data Management Subsystem Page
The information distribution services of the Data Management subsystem provide information distribution, persistence and access across space and time, making data accessible across the observatory network over extended periods of time. Information includes observational data and derived data products along with other information resources required for the operation of the integrated observatory, such as ancillary instrument information, user identities, workflow definitions, and executable virtual machine images. All information products are managed together with their metadata. Data collection and processing activities are based on the distributed exchange framework, supporting flexible access to data at all steps of the activities.
Team
Name | Org | Status |
Role |
---|---|---|---|
Campbell, Luke | RPS ASA | active | Senior Software Engineer |
Subsystem Objectives
Release 1 Objectives
- Provide data distribution services leveraging publish-subscribe communication
- Define the common OOI data and metadata model and provide common object model tool chain
- Provide basic Persistent Archive Services
Release 2 Objectives
(see also R2 DM Construction Plan)
- Supporting advanced but essential components of resource and data life cycle: versioning, association and provenance; archiving; semantic tagging and vocabulary management; data processing services - in a way that is tuned to large scale data processing, large catalogs, using the messaging infrastructure
- Support data processing workflows and life cycle elements in a way that meets science requirements
- Support SA to provide data product generation workflows
- Support AS to provide visualization product generation workflows
- Prepare for AS to provide a generic workflow execution architecture in Release 3
- Work with CEI on process execution for real-time data transformations
- Refine and extend the OOI Common data and metadata model, in particular geared towards resource life cycle, more feature types, provenance, versioning and semantic capabilities
- Define feature type integration strategy for new/additional feature types
- Provide support for most relevant feature types to convert in and out of the common data model
- Expand the tools provided for search-by-query of OOI data resources, introducing more sophisticated metadata search capabilities, filtering and organizing search results, saved query and prospective query capabilities
- Enable interoperability with external communities and applications by providing integration capabilities and transformation processes.
- Provide flexible, robust manageable, multi-domain of authority support for persistent data stores, leveraging existing storage infrastructures and archives
Release 3 Objectives
See the R3 DM Construction Plan.
Documentation
Requirements
[DM R1 Requirements]
[DM R2 Requirements]
[DM R1 User Scenarios]
DM R2 User Scenarios
Architecture and Design
The System Engineering section of this wiki contains the architectural materials for the DM subsystem.
Development and Implementation Progress
Release 1
- Inception
- Elaboration
- Construction
- Transition
Release 2
The finalized iteration task plan (13 May) for Data Management and Distribution is here. The evolving list of tasks is on this GoogleDocs spreadsheet.
- Inception
- Elaboration
- Construction
DM R2 Designs
R2 Status Report
References and Reading
- DM Reading For R2
- Exploring HDF
- Notes on Fenics
- iTAPS and MOAB
- Setting up a launch_plan to launch services
- Data Model Risk Analysis
Release 3
The evolving list of tasks is on this Google spreadsheet.
- Inception
Prototypes
- [Blog Integration]
- [DM Elaboration Integration]
- Redis Coordination
- Data Streaming Load Test
- Indexing with ElasticSearch
- SciCDM Consolidation
Labels
Page: Data Management system references
Page: Data Streaming Load Test
Page: DM Persistent Archive Services Research
Page: DM R2 Elaboration Design
Page: DM R2 User Scenarios
Page: DM R2 Workshop Proposals
Page: DM Reading For R2
Page: DM Release 1 Construction 1
Page: DM Release 1 Construction 2
Page: DM Release 1 Construction 3
Page: DM Release 1 Construction Plan
Page: DM Release 1 Elaboration 1
Page: DM Release 1 Elaboration 2
Page: DM Release 1 Elaboration 3
Page: DM Release 1 Elaboration Plan
Page: DM Release 1 Inception 1
Page: DM Release 1 Inception 2
Page: DM Release 1 Transition
Page: DM Release 2 Construction 1
Page: DM Release 2 Construction 2
Page: DM Release 2 Construction 3
Page: DM Release 2 Construction 4
Page: DM Release 2 Elaboration 1
Page: DM Release 2 Elaboration 2
Page: DM Release 2 Elaboration 3
Page: DM Release 2 Inception 1
Page: DM Release 2 Inception 2
Page: DM Release 2 Transition
Page: DM Release 3 Inception 1
Page: Etherpad Collection
Page: example sequence diagram
Page: Indexing with ElasticSearch
Page: Introduction to the DM Subsystem
Page: Marine IO Data Handler Status
Page: R2 DM status breakdown by Subsystem Service Group
Page: Redis Coordination
Page: SBE16 Preload Walkthrough
Page: Setting up a launch_plan to launch services
Page: VEL3D Example Walkthrough