February 25, 2011
opic: Replacement IOOS OOI Meeting (Mueller/Bauer)
Date: Friday, February 25, 2011
Time: 11:00 am, Eastern Standard Time (New York, GMT-05:00)
Meeting Number: 551 786 910
Meeting Password: eoiioos1
- Tim L
- Charles Alexander
- Rich Signell
- Matt Arrott
- John Graybeal
- Jeff DLB
- Construction 2 Tasks/Plan - https://confluence.oceanobservatories.org/display/CIDev/EOI+Release+1+Construction+2
- NAVY update
- NSF + NOAA onboarding
- Rich (& chris?) to speak with Frank and Doug about onboarding options (passing along the dispatcher or using the UI)
- Matt and Charly to find openings to have a meeting with Tim, Jon Orcutt and Zdenka.
EOI Construction 2 (other teams on C3)
- Technical aspects in place (many tech tasks front-loaded)
- Ensure interaction with infrastructure possible by current implementations
- Address dependencies accordingly
- Develop additional agents for UH + upper ocean
- Push working relationships with modeling groups
Charly: Release 1 pushed for June?
Chris: Yes. Tasks which slide to July time-frame are mostly testing – shouldn't impact technical work. Most bases covered (nothing novel popped up).
Rich: (@Stennis) Along with Walt McCall (NDBC), met with with Frank Bub, Doug May, Danny Illac and Kyle Rushing at NAVO. Frank is the head of the modeling group, Doug is in charge of acquiring the data that feeds the models. Danny and Kyle work for Doug. Frank and Doug were very interested in being the "3rd customer" for the EOI. They are very interested in getting their hands on data that is coming from the IOOS regions that is NOT on the GTS. They differ from our other two cutomers in that much of the data they do ingest is coming from their GTS feed - such as HFRADAR, Glider from SOS, etc.
Gave briefing on OOI - interested in participating. Interested in getting data into their system... getting their data out is a diff issue (mostly concerned with "FTPing it")
Matt: How much data + on what period does this represent?
Rich: Not sure. Probably ~100GB/day
Matt: Is there any historical archive of this data?
Rich: Yes. New protocol is to FTP data up to NCEP. NCEP serves data for some period (maybe week or two) and NCDDC (in Stennis) will FTP data back from NCEP and archive it. [??] IOOS modeling test bed project. This isn't ideal – In the future would be great to address issues with this protocol.
NCDDC - National Coastal Data Dev Center part of NODC
Matt: Can we just get data from NCEP.
Matt: Will this solve the issue.
Rich: Sure. We'll have to talk to NCEP about it.
Matt: Why can't we just be distributor. We'll want to id source.
Chris: Already IDed..
John: For many institutions – web traffic is a primary measure of effectiveness.
Scott Cross would probably agree that there is a better way to do this.
Matt: NCEP is planning to serve this data to the public?
Matt: We are public. This doesn't req any coordination, we are just getting it and distributing to our community. @John, how would you recommend as a distributor to satisfy aforementioned condition.. what is solution?
John: prominent labeling of the data that is being reserved so that it's source institution providers are well represented.
Matt: How can we provide value? For the providers that want to monitor their effectiveness, we actually have an op to provide them with continuous reporting on the consumption of their data products.
Charly: In terms of a vehicle for serving their products – (relative to DMAC work) – the key reason to talk to Frank is that there is some assimilation pathway we can help with similar to work with Wilkin + Powell. Navy has a data stream that ends up at NCEP etc.. why is this the topic?
Rich: Conversation is relative to the future. Navy is interested in OOICI – opportunity to standardize their data stream for non-gts streams into their modeling system.
Matt: Navy data was a part of the dataset we wanted to provide in this first solution. Is this not correct.
Jeff: Thought it was because Frank Bubb need data to start his model.
Rich: John Wilkin does use the navy model output to drive his model. Special FTP arrangment with NRL is how he gets it now.. They are not in favor of these special arrangements.
Matt: Right, they wanted to find a stable long-term channel for providing that data
Rich: NRL is the research and development arm for NAVO (Frank). NRL is more experimental products (similar to the ioos regional associations). Running experimental models (quasi-operationally) – sometimes better. If they are runnng hycom they send data to hycom group in Florida (lag time). Not sure if that preoperational data goes to NCEP. Both of these are of interest to regional modelers. We should work on the part Navy is most interested in to get them on board to start. This might test OOI in ways it hasn't been tested (as far as volume of data)
Jeff: Which data is Frank interested in (that hes not already getting)?
Charly: Tagging telemetry data (from tagging seals + sharks) gives him addl profiles in western pac.
Rich: Interesting. Guys running 4d var models – these have a large impact on assimilations
Workshop next week in Santa Cruz
Andy Moore has good slides showing impact of marine mammal observations
Matt: Is HFRadar on todo list? Chris: We need to reassess. Whole vector..
Matt: Brian said whole vector has no value to him.. there is a political step to get the component vector. This process has not been started.
John Wilkin however does consume the whole vector results.
Brian uses radials. We do have access to OPeNDAP to radial data on a continual basis from UH which we can uses as samples?
Matt: For the purpose of solving Brians problem.. we do have a source for radials..
Rich: Yes. And this does solve the problem of pushing radials through OOI
Matt: Chris, are you working with bringing in radials from Hawaii?
Chris: Must review listings to confirm which data will be supported according to UH.
Matt: Sounds as though HFR is coming online for John Wilkin + bring regional radial through DAP for UH.
Chris: Caveat - large dataset support - must do more research on datasets to ensure we can support them.
Charly: Is Frank a viable customer to help in this iteration, and if so does it fit in this iteration without bringing us off schedule - or push past June?
Rich: Thinks so. We don't have to take similar approach and handle all data streams.. We could just pass some code and see if it works with them. Bringing them into the conv would be pretty helpful
Rich: They talked about gliders looks like they didnt know it was avail through NDBC.
Whats the status of glider data (NDBC)?
Charly: Some is being exposed - check out their website http://www.ndbc.noaa.gov/gliders.php. Jeff?
Jeff: Certainly not all is there.
Rich: NAVO spent 2 years on glider data automated QC tool - Lager. Before they used this tool, 90% of their glider data was refused. Once they got this tool 90% goes in, 5% marked for QC (toolbox provides mechanism for manual QC). This tool was of extreme interest to NDBC. Hopefully setting up agreements with NDBC to use this tool before pushing data to GTS. The LAGER software is licensed by NRL, but there will be a request letter from Bill Burnett (NDBC) to Gregg Jacobs (NRL) requesting this software. They want a formal agreement so as to not arbitrariily pass code back and forth.
Charly: Sounds like Frank is a go... Chris?
Chris: At this point adding additional new datasources is not really an option (per time constraints). What seems to be possible and beneficial is to provide him with the Dispatcher (receives notifications on dataset changes) which would allow him to subscribe to existing OOI data (such as SOS). Bringing him onboard looks like something for next iteration.
Rich: Who responds to Frank and Doug?
Charly: Me and Rich?
Chris: I dont know that we have time to support modifications to workflows etc.. easiest pathway is to ONLY pass the dispatcher so they can automate data acquisition from OOI.
Matt: After R1 is out onboarding is one of our ongoing tasks. The question is: does onboarding Frank benefit us.
Rich: Id be happy to follow up the Frank/Doug and explain this situation.
Matt: and set expectation that we will revisit the scope of their involvement around [??]. Does this look good Chris?
Chris: Yeah, Dispatcher needs polish (which is going to happen for Brian anyhow)
Matt: The understanding is that things are still in dev (not well enough documented/bug free) and they will be coming back to us for implementing.
Chris: Don't see a whole lot of things going wrong with the dispatcher going forward. Rich, did you get the sense that they would have the open door (security) to get this dispatcher running? The Dispatcher needs an incoming connection. Rich: Looks like it could be a concern.
Chris: Another potential path. We have two externalization requirements: dispatcher for John/Brian and the ability to download datasets directly from website. Looks like the former would be an option for them. In which case they are a normal user rather than an early adopter.
Rich: Lets have the conversation before making this choice.
NSF IOOS Talk
Matt: Tim and I W| Anthony, we are now at a place where Tim, Jon Orcutt and Zdenka
Charly: Since it looks like its our job to frame the call/agenda etc. the soonest we can have the call is mid/late march – otherwise we can have some discussions next week to decide if we can do this earlier.
Charly: Will look into finding a window for himself and Zdenka. Matt to find window for remaining. This sound good? Matt: Yes.
Matt: We just setup a commitee between us and navy to address nat sec issues.
- Meeting next week **