compared with
Current by Michael Meisinger
on Jul 11, 2014 12:32.

Key
This line was removed.
This word was removed. This word was added.
This line was added.

Changes (25)

View Page History
|| Development Owner | M. Manning ||
|| Development Team | Data Processing ||
|| Developers | B. McKenna ||
|| Release | R3 ||
|| Status | Activated Done ||
|| Start Date | 1/6/2014 ||
|| End Date | TBD 4/2014 ||
|| Comments | ||
{metadata-list}
h3. Requirements and Capabilities Tracing

The requirements in this section come from the requirements spreadsheet finalized and agreed to between the CI Development Manager and CI System Engineer. That document is [M112 Requirements Spreadsheet|^M112_Requirements_2014-02-18_ver_1-02.xls]


h5. The following table contains all of the Milestone Service Requirements

|| Reqt ID || Requirement Text || Rationale and Description || Proposed Change || ||
| L4-CI-DM-RQ-124 | Search & Navigation | | | |
| L4-CI-DM-RQ-219 | Search and navigation shall implement resource discovery through complex query  | For example, resource type and a specific metadata attribute | | |
| L4-CI-DM-RQ-91 | Search and navigation shall implement geospatial query for resources | For example, by specifying a lat/lon/depth box | | |
| L4-CI-DM-RQ-194 | Search and navigation shall combine resource discovery with resource access | Subject to policy | | |
| NEW DM-1 | | | Search and Navigation Geospatial Query shall support bounding box spatial searches using the following spatial query operators from the OGC Reference Model:  a. overlaps; b. contains;  c. within; and d. disjoint. | \[https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L332-L421 \\
\] |
| NEW DM-3 | | applies to all types of searches, including geospatial and temporal. | Search and Navigation shall provide a count of the result-set from a search in near real-time. | \[https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L310-L321 \\
\] |


h5. The following table contains the Milestone UX requirements

|| Reqt ID || Requirement Text || Proposed Change ||
| L4 UX RQ 117 | A user interface for search and navigation shall be provided | |
| L4 UX RQ 119 | The search and navigation user interface shall present all supported query options | |
| NEW UX-1 | | A user interface for geospatial search shall be provided. |
| NEW | | The Geospatial Search user interface shall allow a user to search for geolocated data products by specifying the latitude, longitude, and depth at which the data were collected. |
| NEW | | The Geospatial Search user interface shall allow a user to search for deployed marine infrastructure resources by specifying the latitude, longitude, and depth at which the resource is deployed. |
| NEW | | The Geospatial Search user interface shall allow a user to specify depth by entering or selecting values, units, and whether to measure up or down from -from the ocean floor or- the ocean surface. |
| NEW | | The Geospatial Search user interface shall allow a user to specify a  latitude and longitude bounding box  by entering values of north and south latitude and east and west longitude. |
| NEW | | The Geospatial Search user interface shall allow a user to specify a latitude and longitude bounding box by drawing the bounding box on a map. |
| NEW | | The Geospatial Search user interface shall allow a user to perform bounding box spatial searches using the following spatial query  operators from the OGC Reference Model:  a. overlaps; b. contains;  c. within; and d. disjoint. |
| NEW | other search types include temporal search, search by name, and search by type | The Geospatial Search user interface shall allow a user to combine a geospatial search with one or more other search types, by "and'ing" the geospatial search criteria with the search criteria from the other search type(s). |
| NEW | | The Geospatial Search user interface shall provide contextual help. |
| NEW | | The Geospatial Search user interface shall provide help documentation. |
| NEW | | The search and navigation user interface shall allow a user to limit the number of results returned from a search query. |
| NEW | | The search and navigation user interface shall present to  a user the number of results returned from a search query. |


{hidden-data}
{html-include:url=http://architecture.oceanobservatories.org/ooinet/milestonereq/M112.html}
{hidden-data}

h3. Use Cases
h4. Steps


h2. Milestone Testing

For skip (offset). First we [create a large number of resources|https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L236-L238] then we test using the [QueryLanguage|https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L290-L296] and the [Discovery Intermediate Format|https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L302-L308]


For the count of total results in database, [should return 1 value|https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L310-L315]


For the enhanced geospatial, we run two sets of tests:

The [first|https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L393-L410] uses a WKT Polygon object to represent a box and test the overlaps, contains and within operators.

The [second|https://github.com/ooici/coi-services/blob/master/ion/services/dm/presentation/test/discovery_test.py#L412-L421] demonstrates the ability to specify a latitude/longitude point and an associated buffer (AKA radius) and search relative to the circle formed by that radius.



h4.



h2. Milestone Pull Requests
{panel}

h5. Additional search parameters (skip) and query for total results in DB
{panel}

* h3. [https://github.com/ooici/coi-services/pull/1773]
* h3. [https://github.com/ooici/coi-services/pull/1773]
* h3. [https://github.com/ooici/coi-services/pull/1776]
* h3. [https://github.com/ooici/coi-services/pull/1779]

{color:#000000}{*}Milestones - Tasks - Actions addressed{*}{color}
* M112-T131-A002: Return number of total results available in datastore from DiscoveryService
* M112-T131-A005: DiscoveryService processes an offset (skip) parameter for pagination

DiscoveryService currently supports a '*limit*' parameter, allowing the client to specify how many results to return from server. An additional '*skip*' parameter is implemented to allow the client to request the next set of n results, skipping the first m results. This addition will allow the UI to implement pagination. For example, to get results 501 to 600 from the server:

{noformat}
{'limit': 100, 'skip': 500}
{noformat}

DiscoveryService can now also be used to obtain the '*total number of results available*' in the datastore. Using the standard query object and adding

{noformat}
search_args={'count': True}
{noformat}

will return an array containing a single integer value, the total number of results available in the datastore. This call should be followed (or preceded) by the same query object without the {{*\{'count':True\}{*}}} to obtain the actual results with the limit and/or offset applied.
{panel}

h5. Implementation of WKT string for geospatial query (with optional buffer specified), this supports the point with radius search
{panel}

* h3. [https://github.com/ooici/coi-services/pull/1795]
* h3. [https://github.com/ooici/coi-services/pull/1815]
* h3. [https://github.com/ooici/pyon/pull/469]

*Milestones - Tasks - Actions addressed*

* M112-T133-A001: Implement WKT qmatcher in ds_discovery
* M112-T133-A002: Pyon DatastoreQuery language support for WKT
* M112-T133-A003: Pyon PostgresQuery language support for WKT

DiscoveryService query can now recognize a request of the form:

{noformat}
{'wkt': <WKT_STRING>, 'buffer': <decimal_degrees>}  
{noformat}

This capability supports the requirement for point based (with radius) search to be made available in the UI. The point must be encoded in WKT format, eg. POINT(10.0, 10.0) and the associated buffer parameter will be used for the radius. This geospatial object (point with buffer) is passed directly to the PostGIS layer making it the most efficient and optimal to reference a circle against the stored locations in the database

h5. Update geospatial Device records upon deployment

* h3. [https://github.com/ooici/coi-services/pull/1857]

*Milestones - Tasks - Actions addressed*

* M112-T132-A001 - Upon deployment, updates Devices with Site and Deployment attributes


h2. Milestone User Interface

[M112 - Wireframe v1 (030114)|^M112-Wireframe.jpg]

[M112 - Wireframe v2 (031714)|^New-Wireframe-circle.jpg]



h2. Milestone Tasks

See more details here: [https://confluence.oceanobservatories.org/display/CIDev/Postgres+Datastore]


h5. Enhance discovery to use geospatial information in data store

[https://confluence.oceanobservatories.org/display/CIDev/Postgres+SQL+Snippets]


Indexes are inherently used when available in PostgreSQL, without an index a brute-force or exhaustive search is used.




h5. Enhance business logic for geospatial resources

[http://etherpad.oceanobservatories.org/r3elasticsearch]


h4. PostGIS and Location Aware Resources

h4. Application Resources that are geo-aware

|| Resource || Location type || Notes ||
| Observatory | point or polygon | |
| PlatformSite | point | |
| PlatformSite InstrumentSite | point | |
| InstrumentSite | point | |
| DataProduct | point or polygon | polygon for glider |
| Deployment | point or polygon | |
| Deployment Site | point or polygon | |
| Site | point or polygon | |

h4. Application Resources that are geo-searchable
|| Resource || Path || Notes ||
| Instrument Device | via Deployed Site | |
| Platform Device | via Deployed Site | |

h5. Add geospatial and temporal attributes to the Device resource

* Deployment activation/deactivation processing can assign the location from the deployed site and the temporal extent from the deployment resource to the device.
** [https://github.com/ooici/ion-definitions/blob/master/objects/data/shared.yml#L214]
*** this is subclassed by InstrumentDevice, PlatformDevice, SensorDevice (which is not used)
** transfer the geospatial information (if any is provided) for the associated Site resource (PlatformSite, InstrumentSite) which is in the FrameOfReference parent class in the geospatial_point_center attribute (do we need to update this attributes?) into the geospatial attribute of the Device.
*** update or remove the geo / temporal attributes at the same time that the hasDevice association is made.
*** At that time you know the Site and device pairing so you dont need to 'find' anything:
**** [https://github.com/ooici/coi-services/blob/master/ion/services/sa/observatory/observatory_management_service.py#L625-L633]
( line 628 and 633 has the pairing)
*** In the remove case (633, 681) you dont need to pull anything from the site, just blank out the geo/temporal attributes in the device as it is no longer deployed.
**** [https://github.com/ooici/coi-services/blob/master/ion/services/sa/observatory/observatory_management_service.py#L676-L682]
(line 681 has the pairing)
** transfer the temporal information in the Deployment resource
*** in the FrameOfReference subclass inside the constraint_list attribute, if provided, there would be a TemporalBounds instance)



h2. Implementation Notes

[https://confluence.oceanobservatories.org/display/CIDev/Postgres+Datastore]


---


h3. Status Discussion 31 Jan with Luke, Brian, Michael, Tim and Maurice

* Issues working thru the several layers of discovery search. Brian feels he has a handle on it now.
* Need to focus on a full suite of integration tests that demonstrate various search types ( overlap, bounding box, etc) for multiple search types
** There are 3 weeks allocated to integration with the UI, this is the time that will test UI-created searches
* Must define which application level resources needed for search
** which need location attributes
** which would be found by searching for an associated resource and how would that work
*** A device does not have location attributes but it is deployed to a site which does have location. Search for all devices in a bounding box.
** Temporal and geo search scenarios?

h3. Geospatially indexed types email thread. MMeisinger 4 Feb

I suggest to run ooi_beta.yml preload for a full set of resources to investigate for search. Hint: you can run it once and then just restart the system after code changes without a new reload.

(1) Current Situation

Currently we have geospatial information for these resources:
- Observatory - only geospatial
- PlatformSite - only geospatial
- InstrumentSite - only geospatial
- DataProduct

Site is an abstract base type and will never exist as a resource in the system. Subsite is not used for OOI deployments.

Currently these resource types have temporal information:

- Deployment - only temporal
- DataProduct

(2) Desirable information

- Deployment is associated with a Site via a hasDeployment association. This means that the Deployment has access to a unique set of geospatial metadata and could be queried this way, even historically or for the future.

- Devices when deployed as primary are associated with a site via a hasDevice association. This means that the Device has access to a unique set of geospatial metadata and could be queried this way for certain epochs (note: not historically)

- Devices obviously have a physical life span (date of purchase until date of decommissioning/retiring). This information could be added as metadata with an easy change but it's not currently in the YML/preload

- DataProducts have metadata that is manually set. Obviously it may be desirable to update this metadata recurringly (e.g. in a batch process) every so often given the actual coverage. The question is semantical: The DataProduct metadata has the nominal temporal range inside, not the actual. The actual data may have gaps so you are always dealing with bounding boxes, not with exact matches.

- It may be desirable to issue historic queries: Find all devices that were deployed at area1 in historic temporal range. We have this information through Deployment resources but the query may be a bit more complicated

- Dataset resources may be interesting too, but maybe less so to the user

- There is a dependency between the derived DataProducts for a device (e.g. L1) and the parsed DataProduct. These would need to be kept in sync

- There will be additional complication with site DataProducts that have different device associations that are vaild for certain time periods only.

(3) Next steps

I believe we should discuss these directions:

a: Can we enhance the resource metadata for more resource types?

b: Can we define processes that update resource metadata recurringly to make it more reliable? Or on certain service calls, e.g. activate_deployment

c: Can we extend the discovery service / datastore query classes with abstracted SQL queries that use associations to find more resources - careful here.

d: What are the user expectations for searching and finding, for navigating resource trees and for maintaining resource metadata and do we meet these right now.



h3. Design and planning discussion. MMeisinger, TAmpe, MManning 11 Feb

Given the remaining time on this milestone we will need to provide a trimmed set of geosearch capabilities that still allows the user to locate critical resources.

* Add geospatial and temporal attributes to the Device resource
** Deployment activation/deactivation processing can assign the location from the deployed site and the temporal extent from the deployment resource to the device.
* Data Product resource geospatial and temporal information will be updated via batch processing.

Note that currently in preload OOI sites have geospatial information but do not have temporal rage. Non-OOI sites currently have both geospatial and temporal information. Ideally this could be align with a script is time allows.

MMeisinger has defined an approach to support circle and polygon shaped bounding boxes in Discovery and will implement this near the end of this milestone delivery date.