This page documents the efforts of the Sensing and Acquisition (S&A) Team to refine the requirements and tasks needed to develop the Sensing and Acquisition sub-component of the OOI Cyber Infrastructure. Following this refinement, the result is a Scope of Work (SOW) that defines the estimation of the level of effort that will be required to complete each task.
- Subsystem Requirement Review: 9/14/2009
- Refined L3/L4 Requirements: 9/21/2009
- Scope of Work Document: 9/28/2009
- Refined Task List: 10/5/2009
- MRG Training: 10/6/2009 - 10/8/2009
- SE Process Meeting: 10/9/2009
- Meetings: The team has telecons every Monday and Thursday at 11:00 AM Pacific
- Email List: cisagroup at ooici.ucsd.edu
- Lead: The interim contact person is Kevin Gomes.
- Meeting Notes:
In order for the S&A team to refine the requirement and tasks and accurately scope the SOW, much research was needed to understand enough of the other sub components to define which component was responsible for meeting which requirement in the architecture. The documents we used are shown here (with our understanding of the flow of how one document informs another, click for larger image):
- Message Handling
- Identity Management
- Policy Enforcement
- Life Cycle Management
- Capability Container <-> Instrument Agent
Everything in Assumptions and Dependencies is consistent with John G's understanding except as noted.
- Sensing and Acquisition is responsible for:
- Interfaces to Sensors (instruments)
- Interfaces to instrument platforms
- Interfaces to power ports
- Interfaces to "other" observatory infrastructure.
- Representing the above physical resources as stateful, controllable agents
- Interfaces with external observatory infrastructure management systems such as:
- Bandwidth allocation
- State-of-health monitoring
- Primary Programming Language is Java (2130-00002 pg. 1)
Michael M has told me verbally that these language decisions are not yet made. Not sure whether that or document holds. JBG
- Specific purpose applications for user interfaces will be in Python (2130-00002 pg. 1)
- Instrument drivers and embedded components will be written in C/C++ (2130-00002 pg. 1)
- Capability Containers will be deployed on a compute node which will then contain the execution of the instrument agents (2130-00002 pg. 1)
- Integration happens through messaging and service orchestration (2130-00002 pg. 2)
- Instrument agents will not have to manage the messaging exchange, the COI will manage the exchange and the instrument agent only deals with messages to and from the COI.
- All data from instruments is packaged in a message (with associated metadata) and put on the exchange
- The interactions between S&A and all other system components will occur over the exchange.
- The exchange handles all the issues of connectivity (mooring case).
- In the case of instrument direct access, the other service networks (i.e. not S&A) will establish the communication channel and the VPN so that direct access can occur (i.e. the IO's will provide VPN management). The same is true for the serial access case and the user will simply connect the virtual serial port software to an IP address.
- During the VPN (direct access case). Other than some form of authentication, all other CI functionality will not be available due to this being an out-of-band mechanism.
Is the above comment still the case? We discussed various strategies here. J.Curcio 01_Dec_09
There is a one-to-one relationship between IP address and an instrument agent. [There is no relation between instrument agent and IP address]
- If the DN Information Container Overview on 2130-00002 pg. 12 is correct, the Instrument agent will be responsible for:
- Constructing the message
- Supplying L1 Metadata
- Supplying L2 Metadata
- Building the Header of the message
- Building the Body of the message
|SA Component Diagram (PDF)||SA_Component_Diagram.pdf|
|Functional Components 2|
|Data Management Dependencies|
|Infrastructure As A Service (IAAS)|
|Messaging Integration Design|
|Physical/Protocol Level Interactions|
- The COI provides the integration substrate for all functional capabilities (2130-00002 pg. 2)
- The COI provides consistent identity management (2130-00002 pg. 2)
- The COI provides consistent governance (2130-00002 pg. 2)
- The COI provides consistent policy enforcement (2130-00002 pg. 2)
- The COI provides uniform life-cycle management and access for resources (instruments are a resource) (2130-00002 pg. 2)
- The Data Management (DM) Service Network (SN) will provide data distribution (2130-00002 pg. 2)
This and the following line are uncertain in my mind, I haven't seen these tasks in DM yet (but they may be there).
- The DM SN will provide data persistence (buffering, caching, and archiving) (2130-00002 pg. 2 & pg. 7)
- The DM SN will provide data access including ancillary instrument information (2130-00002 pg. 2)
- The DM SN will handle information and data ingest (2130-00002 pg. 7)
- The DM SN will handle query and access (2130-00002 pg. 7)
- The DM SN will handle the mediation and transformation between different syntactic and semantic representations (2130-00002 pg. 7 & pg. 12). This includes:
- metadata extraction
- syntactical format conversion
- mediation based on data semantics using ontologies
- information verification
- information validation
- The DM SN will provide a common information model (including data and metadata model) to automate information distribution with pervasive and universal access (2130-00002 pg. 8)
- The DM SN Ingestion service acts as a bridge to the Instrument Service Network (2130-00002 pg. 8).
- The DM SN Ingestion service will provide initial data parsing (2130-00002 pg. 8)
- The DM SN Ingestion service will provide initial metadata extraction (2130-00002 pg. 8).
- The DM SN Ingestion service will provide registration of data products (2130-00002 pg. 8).
- The DM SN Ingestion service will provide versioning of data products (2130-00002 pg. 8).
- Analysis and Synthesis will be responsible for user defined data analysis and manipulation processes (2130-00002 pg. 7)
- Analysis and Synthesis will be responsible for event detection (2130-00002 pg. 7)
- Analysis and Synthesis will be responsible for model integration (2130-00002 pg. 7)
- Analysis and Synthesis will be responsible for data manipulation (2130-00002 pg. 7)
- Analysis and Synthesis will be responsible for synthesis of data products (2130-00002 pg. 7)
- Analysis and Synthesis will be responsible for visualization (2130-00002 pg. 7)
- Analysis and Synthesis will be responsible for definition and management of virtual observatories which may include instruments (2130-00002 pg. 7)
- The Common Operating Infrastructure (COI) provides message-based exchange. (2130-00002 pg. 7)
- The COI ensures pervasive and consistent governance (2130-00002 pg. 7)
- The COI ensures pervasive and consistent policy enforcement (2130-00002 pg. 7)
- The COI ensures pervasive and consistent identity management (2130-00002 pg. 7)
- The COI provides resource state management (2130-00002 pg. 7)
- The COI provides a uniform way of representing and manipulating observatory resources throughout their lifecycle (2130-00002 pg. 7)
- There will exist (outside of the S&A SN) an Instrument Process Repository where instrument agent information is stored.(2130-00002 pg. 14) Information such as:
- Vendor Specific Driver
- Output Data Products
- Static Sensor Metadata
- Instrument Configuration
- There will exist (outside of the S&A SN) a Resource Catalog where information related to physical resources (such as manuals) will be stored. (2130-00002 pg. 14)
- The infrastructure for the network will be provided (satellite, LAN, etc.) by the CI already. The S&A will build on top of an existing network.
- Coding any prototypes that succeed in acquiring real data will REQUIRE exact, real-world instruments (type, manufacturer, model numbers) to be specified and perhaps samples provided.
- We need the list of instruments to support, rumored to exist and to have been sorted in order of complexity by Frank Vernon, for us to start bridging the gap between suggested ontologies / messaging systems and actual real-world data sources.
- Is the assumption that we are making about the instrument agents running inside a capability container always true so that we can depend the existence of the CC for all instrument agents?
- 2130-00002 pg. 2 states Mule will be the ESB, I thought we were not using an ESB?
- Does the instrument agent ever interact direcly with CEI or is it all through messaging?
All through messaging.
- Is the assumption that the instrument agent only interacts with the COI true (i.e. does it just exchange messages with the COI and the COI handles interactions with other resources (like other instruments))?
Yes, true (except possibly for direct access.
- (2130-00002 pg. 6) What is meant by "other" observatory infrastructure?
Controllable resources like lights, docking systems, maybe network controls ...
- (2130-00002 pg. 6) states "performs observatory management and oversight", I would assume that is referring to the external observatory management system and not to sensing and acquisition (as it sounds), correct?
- Is the DN Information Container Overview(2910-00010 OV7 shown on 2130-00002 pg. 12) the form that all messages on the Exchange will take or only those for the DM SN?
- Related: Can you give an example of a message that would have no body (it is listed as optional)?
In my copy, 'Header' is listed as optional. I expect many messages might not have a header. Uhh, no examples come to mind in particular though.
- Related: Can you give an example of a message that would have no body (it is listed as optional)?
- Is the assumption that the VPN connection is handled by the IO's true?
- If the direct instrument access occurs over VPN connection, are we foregoing some of the CI functionality? For example, will logging still need to be done, but in a different way? Will authentication happen on the VPN and will that be synch'd/linked with the CI policies, etc.?
Yes, I think you are forgoing all the CI functionality. This is why direct access via VPN would be A Bad Thing. IMHO.
- Is the idea implement IP over AMQP with OOI managed user authentication? Likely to be VPN that's not aligned with the other security or policy components. Look up the requirements -> do we try to use AMQP -> This requirement needs to be looked up.
I think we can envision the debate here. IO: "I need total fidelity in the direct access connection." CI: "We really want it all to go through COI. What is total fidelity anyway." IO: "Just give us a direct connection already." At least, that's what I'd say if I were the IO (or the CI).
- For direct instrument access, does the OOI CI still need to enforce policy when things are going through direct access? For example, you don't want somebody turning on a noise generating machine when a passive acoustic experiment is occurring. There is really no way for the end user to know about the impacts his/her actions on other resources. This seems extraordinarily difficult.
- Why is the data process repository in the sensing and acquisition service network? I would think that the vast majority of data processing happens outside of sensing and acquisition and it seems like the responsibility is misplaced.
I agree, Arjuna and Curcio discussed this topic earlier - not resolved. J.Curcio 01_Dec_09
Page: 20090924 Sensing and Acquisition Team Meeting Notes
Page: 20090928 Sensing and Acquisition Team Meeting Notes
Page: 20091001 Sensing and Acquisition Team Meeting Notes
Page: 20091005 Sensing and Acquisition Team Meeting Notes
Page: Task List Refinement for Release 1 Sensing and Acquisition Services