Skip to end of metadata
Go to start of metadata

See the Release 3 Milestone Summary page.

WBS 1.2.3.21.53.02
ID M166
Description Leverage alternate information store technologies to increase system reliability, performance and query capabilities.
Deliverable ID D073
Deliverable Information Store Infrastructure
Development Owner M. Meisinger
Development Team Distributed OS
Developers M. Meisinger
Release R3
Status Activated
Start Date 10/1/2013
End Date TBD
Comments  

Milestone Scoping and Requirements

Requirements and Capabilities Tracing

html-include: java.net.UnknownHostException: architecture.oceanobservatories.org

Dependencies and Gaps

Use Cases

Assumptions and Prerequisites

  • Postgres can serve as a drop in replacement for CouchDB/BigCouch given the following modifications:
    • Minor datastore framework refactoring to enable multiple concurrent datastore backends
    • Additional datastore backend
    • Modification to a few tools that use low level datastore access or exploit CouchDB specifics
  • Postgres can be combined with ElasticSearch for OOINet resource integration

Steps

  • Operations Team prepares hardware and instances for production Postgres deployment
  • Operations Team brings OOINet down for maintenance
  • Operations Team perform data migration, as developed in milestone M168 Data Migration of R2 to Postgres
    • Data migration of the operational database
  • Operations Team restarts OOINet and check for
    • correct operations
    • correct construction of ElasticSearch indexes
    • correct warm/hot replication
    • correct archiving
  • Operations Team verifies that performance of the OOINet has increased and UI pages load quicker

Milestone Tasks

  • Gather datastore requirements and candidate technologies
  • Investigate postgresql cluster solutions
  • Investigate postgresql snapshot/archive/restore solutions
  • Scope datastiore implementation/migration capabilities
  • Add postgresql handler to pyon framework will all necessary indexes
    • The core backend has been implemented and needs to be merged
    • Some adjustments to indexes and attribute mappings need to be performed
    • Optimize postgresql pyon backend to Postgres 9.3 and optimal use of JSON datatype
  • Refactor pyon datastore to support various database backends
    • This has been implemented during R2 and needs to be merged
    • The primary idea of mapping the OOI data model to postgres is explained below.
  • Define postgresql river for ElasticSearch
  • ElasticSearch index modifications
  • Add postgres and extensions to base image
  • Postgres operational monitoring
  • Integrate and test with production environment

Milestone Design

Design References and Context

Design Notes

Basic Design

Design Goals:

  • Provide PostgreSQL as a drop-in replacement for CouchDB.
    • Given a pyon framework extension to support multiple datastore backends and minor framework tweaks as necessary
  • Map the key-value object store paradigm used with Couch to a relational schema where objects remain schema-less as JSON but can be indexed by common attributes
  • Enable Postgres replication (clustering)
  • Enable better indexes and query performance.
    • Potentially even enable load-balanced read from mirrored instances, vs. update only to the master instance
  • Create the pathway towards transactions

The following figure depicts the integration of PostgreSQL as a drop-in replacement of CouchDB. It shows how the various datastores are mapped to relational tables and lists the CouchDB views vs. PostgreSQL indexes.

Figure 1. Mapping of resources to CouchDB and PostgreSQL

Implementation Notes

References

See specific pages:

Labels

r3milestone r3milestone Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Oct 14, 2013

    Michael Meisinger says:

    Etherpad: http://etherpad.oceanobservatories.org/R3DatabaseMigration