Skip to end of metadata
Go to start of metadata
This page provides a high-level "storyboard" describing intent and expected content for an R2E1 prototype. This storyboard is initially provided by the architecture team to the development team to coordinate high-level intent of all prototypes, and is subsequently refined in close collaboration between developers and architects. See Prototyping page.

Prototype Overview

See the Prototyping page for a list of all prototypes.

Subsystem AS
Type Approach Assessment
Priority Medium
Iteration R2E1
Dates Sept - Dec 2011
Status completed
Lead Raj Singh
Description Develop a standalone prototype integrating a set of visualization technologies into a simulated message driven processing pipeline (without integrating with the messaging system, the container or the process execution framework)

Risks Addressed

  • Technology Risk: 
    • Compatibility of UI graphing library with OOI data formats and streams. 
    • Ability Ability of Matplotlib to replace Matlab as an effective open source tool.
    • Compatibility of UI components with existing mapping technologies (Google/Bing)
    • Compatibility with HTML5 for supporting next generation of web technology.
    • Support for mature scientific presentation formats (Line, Bar, scatter, pie etc)
  • Performance Risk: 
    • Ability of the UI components to handle large number of data points without a loss of interaction. 
    • Ability to convert instrument data formats to visualization data-tables without loss of interaction.
    • Ability to receive and update visualization in real-time.
  • Architecture Risk:
    • Ability to work with Pyon's asynchronous data delivery architecture.

Requirements Outline, a high-level overview 

Create a standalone prototype to test components and libraries for implementation of R2 visualization subsystem. Assess flexibility in terms of data representation. Also assess user experience in terms of interaction and wait times. Evaluate compatibility of the prototype with the asynchronous push based data delivery architecture of Pyon.

Placement in Release 2 Architecture

Not applicable as this is a standalone prototype used primarily to assess user experience and stability from chosen components. This prototype will not be integrated as-is in to the R2 architecture as the DM / PubSub components of Pyon were not available at this point in time.

Prototype Goals

The subsystem is expected to transfer data from within the R2 architecture to the user interface using a server-client model. For R2 the client interface will be provided within a web-browser. The server side components are expected to be stable and efficient at translating and streaming data to the UI. The UI visualization components were required to be efficient at loading data tables and rendering representations at interactive rates.

Implementation Details

The prototype implementation can be classified broadly into its UI & Rendering components and backend (data server) pieces. The relationships between the components is shown in the following figure :

[1] UI and Rendering

   A survey of several graphing libraries was conducted ( ) and a decision was made to choose Google Charts API for the rendering component of graphical representations. The prototype used Google Earth as the environment for embedding the geo-referenced data. A MS Excel file containing Lat-lon and other fields about some platforms was obtained from John Graybeal and converted to a KML file using Google Fusion Tables. This step can be omitted by using a KML library. KML is a human readable, XML like format that is easy to create/edit. This KML file was used within Google earth to create markers. For the sake of testing one of the markers was edited to add HTML content. The source of the HTML content were the graphs generated for the prototype.

For graphing, Google's Annotated Timeline tool was specifically chosen because of its ability to interact with large timelines. The tool provides a sliding view of the data and offers zoom controls which are reflected in a 'sliding window' at the bottom of the graph. The API also allows users to specifiy the graph parameters (eg, time period, variables plotted, scale etc ) using external UI widgets. Users are also offered an instantaneous view of the value under the mouse. The annotated timeline also offers a feature by which it can refresh automatically every X seconds and fetches a new data table from the source. This allowed us to construct a real time graphing tool.

The HTML/Javascript component of the prototype is an HTML page with 3 tabs that shows 3 different representations:

  • Overview : This is a view of the data collected by a particular platform in its entirety . It offers an overview of the entire data and allows users to zoom and pan through the timeline. Users can either view multiple variables or turn-off certain ones. 
  • Realtime view : This was a view of the data that was generated by emulating a data source from a NetCDF file. The client received N records every X seconds allowing an update of the graph and causing it to 'slide' in real time.
  • Matplotlib Graphs : This showed  a gallery of pre-rendered graphs generated using Matplotlib. A Python script was used to plot two variables in a 2-axes and 3-axes view. This was to test Matplotlib's capability to render data from timeline data.  

[2] Backend (Data Server) 

   The data source generating the Google DataTables for the UI was constructed as a Python script running under Apache/modpython. The Python script was called by the HTML/Javascript code and causes it to read a NetCDF file from disk. The code then parses the data and creates a DataTable where the first column is the timeline and the subsequest columns represent individual variables. The table is then converted to a JSON string and sent back to the web browser for rendering. 

  • For the case of the Overview data the entire NetCDF file is read, decimated by n values (for efficient loading) and then converted to a data table.
  • For the case of Real-time view, only 2048 values were read from the NetCDF file and for every subsequent request, this window of 2048 values was moved by 5 entries. The last starting position of the window is stored in a scratch file so that the emulation provides clients with only 'fresh' data. When the code reaches the EOF, it wraps around to the beginning of the file
  • For the case of matplotlib graphs, the python script generating them were run manually and the generated images were staged in the web directory. This can be easily changed to incorporated automatic generation of updated graphs every few hours and the output images can be stored in appropriate pre-assigned directories (one per data product).


r2e1prototype r2e1prototype Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.