November 12, 2010
- Tim L.
- Charly recap of last week
- EOI Task Status Report
- Nearly complete set of agents
- Discussion of protobuffer CDM implementation
Charly on last week in California: time spent on calibration on current events - ended up thinking about co-development - much more complicated than a use-case intersection. Many things need doing before it could happen - but the idea is welcomed.
Casual discussion between Charly, Jeff & Rich - suggest that a tiger team spun off of the DMAC steering team take 3-4 months and explore the pros/cons of doing co-development. Matthew, John Orcut, and Jean McGovern all like the idea.
EOI Task Status:
All agents have functional "prototypes" with exception of Altimeter (RADS). Functionally: agents support acquiring data and translating to CF/CDM compliant form
Protocol Buffers (OOI CDM):
DM subsystem (David Stuebe) has written Google Protocol Buffers representation of the unidata CDM - EOI has implemented an interface between the OOI CDM and the NetCDF Java implementation of the Unidata CDM.
Structure of dataset [context] is primarily pointers. Data contains arrays which may be fractured.
Scheme is flexible: Can send a number of messages to specify data slices, instead of a
singular large request. Similar to OpenDap through NetCDF java: loads headers initially
but leaves data aquisition to requests.
Somewhat distributed. Everything is referenced by link (piece of array etc.), via an identifier.
Gives ability to send the structure of dataset with linkage to the requested data (very low level for the request, however the entire dataset is available). There are restrictions on how data can be requested from OPeNDAP, such restrictions have been uplifted (such as appending time slices or geographic slices to the begining end or even in the middle of the data????).
Charly: How do we benefit?
Reduce volume across wire?
Make requests/delivery simpler?
Response [Chris]: Not sure since the implementation is in its infancy (between Chris M and David Stuebe)
Protocol buffers by nature are very small/low overhead. They have a very efficient packaging
structure and are optimized to be pushed to the wire. Ultimately, it will do all of those things.
"Think XML but smaller, faster, better" - from Google Protocol Buffers website
Google uses protocol buffers everywhere, such is why everything they run is so fast – "otherworldly".
- When data is requested from OpenDAP client must wait for the entire stream to return. Protocol buffer implementation of CDM, responses recieved in packets whose reciept may be resumed. Because of the arrangement of the packets, a partial download is useable.
- Proto-buffers are what are actually being stored.
- NcStream bits from John ?? uses protocol buffers. Issue: implementation is essentially just a wrapper around the NetCDF Java objects (i.e. Variable). Idea is the same but the implementation is incomplete. [compare our proto-buffer CDM to Johns implementation]
The use of protobuffers is not revolutionary since its been around for a bit and proven to work. The only difference is our application is in a new context.
- Jeff - Will OOI be providing software to allow Org to use Proto-buffers to serve the community?
- Response: Chris thinks OOI will be providing technologies to fill-in the gaps, but may not necessarily be providing the actual software solution.
- Rich on Radial data (HFRADAR):
Eric Terrel, Brian Powel, ...., afraid of loosing access to radial data. Did they [who] say that
the data will/must be available
Money goes to research etc, but this is not exactly what the grant was for. There are concerns with how this can be constrained/controlled.
Next meeting is Dec 10th