Skip to end of metadata
Go to start of metadata

This page documents the efforts of the Sensing and Acquisition (S&A) Team to refine the requirements and tasks needed to develop the Sensing and Acquisition sub-component of the OOI Cyber Infrastructure. Following this refinement, the result is a Scope of Work (SOW) that defines the estimation of the level of effort that will be required to complete each task.

Important Dates:
  1. Subsystem Requirement Review: 9/14/2009
  2. Refined L3/L4 Requirements: 9/21/2009
  3. Scope of Work Document: 9/28/2009
  4. Refined Task List: 10/5/2009
  5. MRG Training: 10/6/2009 - 10/8/2009
  6. SE Process Meeting: 10/9/2009
Logistics Information
  1. Meetings: The team has telecons every Monday and Thursday at 11:00 AM Pacific
    Topic: Sensing and Acquisition Team Meeting
    Date: Every Monday, Thursday, from Thursday, September 17, 2009 to Thursday, October 15, 2009
    Time: 11:00 am, Pacific Daylight Time (GMT -07:00, San Francisco)
    Meeting Number: 555 849 940
    Meeting Password: Sensing1
    
    Please click the link below to see more information, or to join the meeting.
    
    -------------------------------------------------------
    To join the online meeting (Now from iPhones too!)
    -------------------------------------------------------
    1. Go to https://ooinetwork.webex.com/ooinetwork/j.php?ED=111322702&UID=1021885547&PW=7fcbfb008368090f000d045e46
    2. Enter your name and email address.
    3. Enter the meeting password: Sensing1
    4. Click "Join Now".
    
    -------------------------------------------------------
    To join the teleconference only
    -------------------------------------------------------
    Provide your phone number when you join the meeting to receive a call back. Or, call the number below and enter the meeting number.
    Call-in toll-free number (US/Canada): 866-469-3239
    Call-in toll number (US/Canada): 1-650-429-3300
    Global call-in numbers: https://ooinetwork.webex.com/ooinetwork/globalcallin.php?serviceType=MC&ED=111322702&tollFree=1
    Toll-free dialing restrictions: http://www.webex.com/pdf/tollfree_restrictions.pdf
    
    -------------------------------------------------------
    For assistance
    -------------------------------------------------------
    1. Go to https://ooinetwork.webex.com/ooinetwork/mc
    2. On the left navigation bar, click "Support".
    
    You can contact me at:
    rmbauer@ucsd.edu
    
  2. Email List: cisagroup at ooici.ucsd.edu
  3. Lead: The interim contact person is Kevin Gomes.
  4. Meeting Notes:
    1. September 21, 2009
    2. September 24, 2009
    3. September 28, 2009
    4. October 1, 2009
    5. October 5, 2009

Resources

In order for the S&A team to refine the requirement and tasks and accurately scope the SOW, much research was needed to understand enough of the other sub components to define which component was responsible for meeting which requirement in the architecture.  The documents we used are shown here (with our understanding of the flow of how one document informs another, click for larger image):

Current Requirements and Status

Interfaces

Common Operating Infrastructure <-> Instrument Service Network
  1. Message Handling
    1. ?
  2. Identity Management
    1. ?
  3. Governance
    1. ?
  4. Policy Enforcement
    1. ?
  5. Life Cycle Management
    1. ?
  6. Capability Container <-> Instrument Agent
    1. ?
Common Execution Infrastructure <-> Instrument Service Network
Data Management Service Network <-> Instrument Service Network
Analysis and Synthesis <-> Instrument Service Network
Planning and Prosecution <-> Instrument Service Network

Assumptions

Everything in Assumptions and Dependencies is consistent with John G's understanding except as noted.

  1. Sensing and Acquisition is responsible for:
    1. Interfaces to Sensors (instruments)
    2. Interfaces to instrument platforms
    3. Interfaces to power ports
    4. Interfaces to "other" observatory infrastructure.
    5. Representing the above physical resources as stateful, controllable agents
    6. Interfaces with external observatory infrastructure management systems such as:
      1. Power
      2. Bandwidth allocation
      3. State-of-health monitoring
  2. Primary Programming Language is Java (2130-00002 pg. 1)
JBG
Michael M has told me verbally that these language decisions are not yet made. Not sure whether that or document holds. JBG

    

  1. Specific purpose applications for user interfaces will be in Python (2130-00002 pg. 1)
  2. Instrument drivers and embedded components will be written in C/C++ (2130-00002 pg. 1)
  3. Capability Containers will be deployed on a compute node which will then contain the execution of the instrument agents (2130-00002 pg. 1)
  4. Integration happens through messaging and service orchestration (2130-00002 pg. 2)
  5. Instrument agents will not have to manage the messaging exchange, the COI will manage the exchange and the instrument agent only deals with messages to and from the COI.
  6. All data from instruments is packaged in a message (with associated metadata) and put on the exchange
  7. The interactions between S&A and all other system components will occur over the exchange.
  8. The exchange handles all the issues of connectivity (mooring case).
  9. In the case of instrument direct access, the other service networks (i.e. not S&A) will establish the communication channel and the VPN so that direct access can occur (i.e. the IO's will provide VPN management). The same is true for the serial access case and the user will simply connect the virtual serial port software to an IP address.
  10. During the VPN (direct access case). Other than some form of authentication, all other CI functionality will not be available due to this being an out-of-band mechanism. 
    Is the above comment still the case?  We discussed various strategies here. J.Curcio 01_Dec_09
  11. There is a one-to-one relationship between IP address and an instrument agent. [There is no relation between instrument agent and IP address]
  12. If the DN Information Container Overview on 2130-00002 pg. 12 is correct, the Instrument agent will be responsible for:
    1. Constructing the message
    2. Supplying L1 Metadata
    3. Supplying L2 Metadata
    4. Building the Header of the message
    5. Building the Body of the message
Diagrams
Title Diagram
SA Component Diagram (PDF) SA_Component_Diagram.pdf
Physical Deployment
Functional Components
Functional Components 2
Resource Agent
Data Management Dependencies
Infrastructure As A Service (IAAS)
Messaging Integration Design
Physical/Protocol Level Interactions

Dependencies

  1. The COI provides the integration substrate for all functional capabilities (2130-00002 pg. 2)
  2. The COI provides consistent identity management (2130-00002 pg. 2)
  3. The COI provides consistent governance (2130-00002 pg. 2)
  4. The COI provides consistent policy enforcement (2130-00002 pg. 2)
  5. The COI provides uniform life-cycle management and access for resources (instruments are a resource) (2130-00002 pg. 2)
  6. The Data Management (DM) Service Network (SN) will provide data distribution (2130-00002 pg. 2)
    JBG
    This and the following line are uncertain in my mind, I haven't seen these tasks in DM yet (but they may be there).
  7. The DM SN will provide data persistence (buffering, caching, and archiving) (2130-00002 pg. 2 & pg. 7)
  8. The DM SN will provide data access including ancillary instrument information (2130-00002 pg. 2)
  9. The DM SN will handle information and data ingest (2130-00002 pg. 7)
  10. The DM SN will handle query and access (2130-00002 pg. 7)
  11. The DM SN will handle the mediation and transformation between different syntactic and semantic representations (2130-00002 pg. 7 & pg. 12). This includes:
    1. parsing
    2. metadata extraction
    3. syntactical format conversion
    4. mediation based on data semantics using ontologies
    5. information verification
    6. information validation
  12. The DM SN will provide a common information model (including data and metadata model) to automate information distribution with pervasive and universal access (2130-00002 pg. 8)
  13. The DM SN Ingestion service acts as a bridge to the Instrument Service Network (2130-00002 pg. 8).
  14. The DM SN Ingestion service will provide initial data parsing (2130-00002 pg. 8)
  15. The DM SN Ingestion service will provide initial metadata extraction (2130-00002 pg. 8).
  16. The DM SN Ingestion service will provide registration of data products (2130-00002 pg. 8).
  17. The DM SN Ingestion service will provide versioning of data products (2130-00002 pg. 8).
  18. Analysis and Synthesis will be responsible for user defined data analysis and manipulation processes (2130-00002 pg. 7)
  19. Analysis and Synthesis will be responsible for event detection (2130-00002 pg. 7)
  20. Analysis and Synthesis will be responsible for model integration (2130-00002 pg. 7)
  21. Analysis and Synthesis will be responsible for data manipulation (2130-00002 pg. 7)
  22. Analysis and Synthesis will be responsible for synthesis of data products (2130-00002 pg. 7)
  23. Analysis and Synthesis will be responsible for visualization (2130-00002 pg. 7)
  24. Analysis and Synthesis will be responsible for definition and management of virtual observatories which may include instruments (2130-00002 pg. 7)
  25. The Common Operating Infrastructure (COI) provides message-based exchange. (2130-00002 pg. 7)
  26. The COI ensures pervasive and consistent governance (2130-00002 pg. 7)
  27. The COI ensures pervasive and consistent policy enforcement (2130-00002 pg. 7)
  28. The COI ensures pervasive and consistent identity management (2130-00002 pg. 7)
  29. The COI provides resource state management (2130-00002 pg. 7)
  30. The COI provides a uniform way of representing and manipulating observatory resources throughout their lifecycle (2130-00002 pg. 7)
  31. There will exist (outside of the S&A SN) an Instrument Process Repository where instrument agent information is stored.(2130-00002 pg. 14) Information such as:
    1. Vendor Specific Driver
    2. Calibration
    3. Output Data Products
    4. Static Sensor Metadata
    5. Instrument Configuration
  32. There will exist (outside of the S&A SN) a Resource Catalog where information related to physical resources (such as manuals) will be stored. (2130-00002 pg. 14)
  33. The infrastructure for the network will be provided (satellite, LAN, etc.) by the CI already. The S&A will build on top of an existing network.
  34. Coding any prototypes that succeed in acquiring real data will REQUIRE exact, real-world instruments (type, manufacturer, model numbers) to be specified and perhaps samples provided.
  35. We need the list of instruments to support, rumored to exist and to have been sorted in order of complexity by Frank Vernon, for us to start bridging the gap between suggested ontologies / messaging systems and actual real-world data sources.

Outstanding Issues

  1. Is the assumption that we are making about the instrument agents running inside a capability container always true so that we can depend the existence of the CC for all instrument agents?
    JBG
    Yes
  2. 2130-00002 pg. 2 states Mule will be the ESB, I thought we were not using an ESB?
  3. Does the instrument agent ever interact direcly with CEI or is it all through messaging?
    JBG
    All through messaging.
  4. Is the assumption that the instrument agent only interacts with the COI true (i.e. does it just exchange messages with the COI and the COI handles interactions with other resources (like other instruments))?
    JBG
    Yes, true (except possibly for direct access.
  5. (2130-00002 pg. 6) What is meant by "other" observatory infrastructure?
    JBG
    Controllable resources like lights, docking systems, maybe network controls ...
  6. (2130-00002 pg. 6) states "performs observatory management and oversight", I would assume that is referring to the external observatory management system and not to sensing and acquisition (as it sounds), correct?
  7. Is the DN Information Container Overview(2910-00010 OV7 shown on 2130-00002 pg. 12) the form that all messages on the Exchange will take or only those for the DM SN?
    JBG
    All
    1. Related: Can you give an example of a message that would have no body (it is listed as optional)?
      JBG
      In my copy, 'Header' is listed as optional. I expect many messages might not have a header. Uhh, no examples come to mind in particular though.
  8. Is the assumption that the VPN connection is handled by the IO's true?
  9. If the direct instrument access occurs over VPN connection, are we foregoing some of the CI functionality? For example, will logging still need to be done, but in a different way? Will authentication happen on the VPN and will that be synch'd/linked with the CI policies, etc.?
    JBG
    Yes, I think you are forgoing all the CI functionality. This is why direct access via VPN would be A Bad Thing. IMHO.
  10. Is the idea implement IP over AMQP with OOI managed user authentication? Likely to be VPN that's not aligned with the other security or policy components. Look up the requirements -> do we try to use AMQP -> This requirement needs to be looked up.
    JBG
    I think we can envision the debate here. IO: "I need total fidelity in the direct access connection." CI: "We really want it all to go through COI. What is total fidelity anyway." IO: "Just give us a direct connection already." At least, that's what I'd say if I were the IO (or the CI).
  11. For direct instrument access, does the OOI CI still need to enforce policy when things are going through direct access? For example, you don't want somebody turning on a noise generating machine when a passive acoustic experiment is occurring. There is really no way for the end user to know about the impacts his/her actions on other resources. This seems extraordinarily difficult.
  12. Why is the data process repository in the sensing and acquisition service network? I would think that the vast majority of data processing happens outside of sensing and acquisition and it seems like the responsibility is misplaced.
    I agree, Arjuna and Curcio discussed this topic earlier - not resolved. J.Curcio 01_Dec_09

[Requirements Refinement]

Task List Refinement for Release 1 Sensing and Acquisition Services

[Scope of Work Definition]

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.