Process arriving sensor data into a calibrated, quality-controlled data product.
|Actors||Data Process Programmer|
|References||CIAD SA OV Data Stream Processing|
|Is Used By|
|Extends||UC.R2.21 Transform Data in Workflow|
|Is Extended By|
|In Acceptance Scenarios||AS.R2.02C Instrument Life Cycle Support, AS.R2.04A Data Product Leads Drive Core Data Product Creation|
|Technical Notes||This is an exemplar use case. It does not mandate setting up a workflow capability for calibration and QC;
it is just to demonstrate the viability of transforming input data automatically as it arrives.
This use case generates a new data record, rather than annotating an existing data record with comments.
|Primary Service||Data Processing Services|
|UC Status||Mapped + Ready|
This information summarizes the Use Case functionality.
Use the data processing capabilities of the system to apply a calibration and/or quality control algorithm to a data set, and create a new 'quality controlled and/or calibrated data set' on the fly as data arrives.
- The assumptions of UC.R2.21 Transform Data in Workflow apply.
- A Level 1 or Level 2 data set will result, since the process produces new data types (the calibrated, quality controlled data) not present in the original.
- The form of the calibrated, quality controlled product is prototypical, not representative of similar products to be produced in Release 3.
- The specification for the transformation is provided in a Data Product Specification.
A suitably authorized user is logged in and ready to enter the data process definition to perform the calibration and QC transformations.
The italicized steps are adapted from UC.R2.21 Transform Data in Workflow; the non-normative (second level in the outline) comments are unique to this scenario.
- Data Process Programmer (DPP) creates a data process definition specifying each required calibration and quality control transformation.
- Application of calibration coefficients may be one step, and each quality control process a separate step; these are expected to produce separate data products (hence, different definitions).
- This script can be tested and validated off-line, requiring minimal changes for use in the system.
- The data process definition is based on a Data Product Specification provided by OOI scientists.
- DPP defines the actual data transformations.
- Again, one transformation for each definition.
- The DPP configures the data process definition's execution frequencies.
- This should be set to execute whenever the data arrives from the previous transformation (the sequence of transformations is data-driven).
- The DPP defines the resulting output(s) from each transformation.
- In some cases transformation outputs will not be persisted; on other cases the Data Process Specification may call for them to be persisted.
- The outputs from the last transformation in the chain should be kept available for a period, if the data will be accessed by anyone other than the real-time subscribers.
- The DPP activates each data transformation in the chain.
- The arrival of new data causes each data transformation process in turn to be executed, creating the calibrated, quality controlled data.
- Real-time metadata for the produced resources are updated automatically by the data processes.
- Static metadata were updated when the resources were first registered.
- The transformation process issues an event (in addition to publishing resulting data) after each execution's completion.
- The DPP inspects any persisted outputs to ensure they are appropriately transformed and described.
- This check occurs outside of the routine ION processing — it can be performed manually.
- An example verification: Were right data outputs created with appropriate metadata attached? Were values correct?
- Each data process continues operating on newly arriving data supplements, until it is deactivated.
- Confirm that supplements are correctly processed.
- Confirm that processing stops when a data process is deactivated.
The data process definition is registered, is executing successfully according to the prescribed schedule, and is producing the desired products.
These comments provide additional context (usually quite technical) for editors of the use case.
The definition of the comprehensive data quality control algorithms, and of calibration algorithms, will occur as part of the activation of each instrument type (during driver and QC procedure development and configuration). However, the QC framework to execute it will only be in place at the end of R3 (S&A). So this use case is simply to develop an example and identify issues.