Skip to end of metadata
Go to start of metadata

Description

This is the main page for the OOI pilot period prototype project Data Exchange. This project is a collaboration between IOOS (NOAA) and the OOI.

The goal of the Data Exchange is to make oceanographic science easier by accelerating the transfer of data, providing cloud-based server-side computation and the propagation of standards for data formats and conventions. Whenever possible, we will reuse existing work, adhere rigorously to standards and produce open-source code for the community. The code should scale from a small system that can run on a laptop to a multi-continent archipelago of caches, servers, exchange points and networks.

We expect to be moving large amounts of data, and have spent time learning the tradeoffs of different encoding and transport methods. DAP, with its underlying use of XDR, is quite bandwidth efficient and widely used for remote access to large datasets. Layered on top of DAP are standards and programs that allow transformation, aggregation, computation and visualization. (Thredds, Ferret, NetCDF, COARDS, CF, GridFields, Matlab, Erddap, pydap and many others.)

Here's a diagram, taken from the 2009 kickoff meeting:

Components (briefly)

  • Proxy is a lightweight HTTP proxy that takes the users' DAP requests and forwards them to the controller
  • The fetcher is the mirror of the proxy, and pulls DAP data and sends it to the proxy and/or persister as directed.
  • The persister and cache save and serve data, respectively.
  • The controller handles coordination between all the components.
  • The attribute store maintains the list of datasets and their status. (Actually, it does more than that, but from the DX point of view that's its role.)
  • The management interface is a web-based app for viewing, registering and deleting datasets, and for setting caching policy.

Background and requirements

For a good introduction to some of the user requirements, technological constraints as well as OpenDAP, please see
"Collaboration tools and techniques for large model datasets" by Signell et al, Journal of Marine Systems, Jan 2008. It does a good job of explaining DAP, the alternatives, how it's used and more.

Preceding work and references

Please see also

Tasks and Deliverables

Development Iterations

Topics

Source code and documentation:

Server-side computation:

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.