Skip to end of metadata
Go to start of metadata
This page describes events and notification capabilities in the ION system. Events are signals in the systems; they published as messages and can be subscribed to. Notifications inform users about the occurrence of events and other system conditions based on user requests.

Terminology

Term
Description
Event
Describes an occurrence of interest that happened at an instant in the system, typically about a device or resource
Event Object
The object describing an event. Has a type and unique ID and other common attributes
Event Message
The real-time message containing one event object
Event Repository
The datastore of event objects that can be queried efficiently. Events are persisted here eventually (not in real-time)
Event Transform A dedicated process receiving real-time input from certain data streams and events, running an event detection algorithm, producing new events.
Notification
Communication of one or multiple events to a user based on a NotificationRequest using a delivery channel. Notifications do not have to be delivered in real-time
Notification Request
A user defined standing rule that triggers notifications based on events
Delivery Channel
The way notifications are delivered to a user, such as email, SMS (text message)
Delivery Modality
The modifications performed to events to create a notification (e.g. collection, digest, transform into a different format) and the requested timing of notifications (immediately, on schedule, on trigger)

Events

Events indicate signals in the system. Event information is contained within event objects and is transported in real-time via event messages.

The following table describes the fields that every event object contains.

Field
Type
Description
Must be set by publishing process? 
Available Subscription Filter? 
type_
str
Event type. This is also the type of Event object. Note: when using IonObjects, use event._get_type()
No. Set implicitly through the class of event object
Yes, optional. Filter on event type
_id
str
The unique identifier for the specific event. This can be used to retrieve the event from the events datastore
No. Generated by the event framework on publish
No
origin
str
Identifies the actual originator of the event. Note: the is not typically the publisher of the event, but the resource that caused the event. In many cases, the origin is a resource id
Yes, mandatory
Yes, optional. Filter on specific origin value
origin_type
str
Indicates the type of origin, where applicable. If origin is a resource id, this is the type of resource
Yes, optional
Yes, optional. Filter on specific origin type value
ts_created
str
Timestamp of event. Milliseconds in UNIX epoch.
Typically No. Set by the event framework on publish but can be provided for special cases. 
No
base_types
list
List of all base types that this event object extends from, with Event being the last element of this list. All events extend from "Event".
No. Set by the event framework on publish Yes, optional implicit. Filter on event base type has this effect.
description
str
Additional description for human readers.
Yes, optional No
sub_type
str
An optional expression of a subtype in addition to the type expressed by the event object. E.g. "CREATE" is a subtype of the ResourceModifiedEvent. Subtypes may contain multiple "words", separated by dots. Yes, optional Yes, optional. Can filter on individual words within sub_type.

Special event types add additional fields.

Event Types

The following diagram shows an overview of the core structure of the event hierarchy.

Figure 1. ION Event Hierarchy (OV-7)

These main base types of events exist:

  • ResourceEvent: Events about the registration of a resource in the registry and its lifecycle
  • DeviceEvent: Events from a resource agent about the manipulation of an external taskable resource (called "device")
  • InformationContentEvent: Events about the digital content of an information resource, not the description of this resource as a resource object.
  • AgentEvent: Events about an agent in the system (but not about the controlled device or the registry entry)
  • Error: Events describing an error condition in the system
  • other base types for infrastructure use, such as ExchangeEvent, OrgEvent, IngestionEvent, PolicyEvent, ExecutionEvent

The following table lists all types of events and their typical origins and sinks.

Category Event Type Super type(s)
Origin
Origin type
Subtypes Produced By  Description
Resource
ResourceLifecycleEvent
ResourceEvent
resource id
Resource type
New state Resource Registry 
The lifecycle state of a resource has changed
Resource ResourceModifiedEvent
ResourceEvent resource id Resource type CREATE, UPDATE, DELETE Resource Registry Something other than lifecycle state about the resource has changed.

(Note the Governance Controller subscribes to this event for Policy updates)
Resource
ResourceOperatorEvent
ResourceEvent
resource id
Resource type   User Notification Service, requested by an operator user through UI 
Indicates a human operator created condition related to a resource. May be subclassed
Device DeviceCommandEvent DeviceEvent device resource id Resource type   Resource Agent This event is published by an agent for a resource (set in the origin) right after a command has been executed.
Device
DeviceSpecificLifecycleEvent
DeviceEvent device resource id 
Resource type
New state Resource Agent
TBD. State in device specific state machine (e.g. managed by the driver) has changed
Device DeviceCommonLifecycleEvent
DeviceEvent device resource id
Resource type New state
Resource Agent TBD. State in device common state machine (e.g. managed by the agent) has changed
Device
DeviceOperatorEvent
DeviceEvent
device resource id
Resource type   User Notification Service, requested by an operator user through UI Indicates a human operator created condition related to a device resource, such as an InstrumentDevice or PlatformDevice.
Information 
InformationContentModifiedEvent
InformationContentEvent 
information resource id
Resource type
  Ingestion  
The content of an information resource (e.g. the data in a coverage) has changed 
Information
InformationContentStatusEvent 
InformationContentEvent
information resource id Resource type   Ingestion
Something of significance of an information resource (other than content and resource attributes) has changed or is abnormal
Information
InformationContentAccessedEvent
InformationContentEvent
information resource id
Resource type
InformationContent AccessEnum
Visualization, download Published when a user accesses some information content, such as a data product through visualization or download.
Execution 
ContainerLifecycleEvent
ExecutionEvent
container id
"CapabilityContainer"
  Capability Container
Container states include START, TERMINATE, ERROR
Execution
ProcessLifecycleEvent
ExecutionEvent
process id
"ContainerProcess"
  Process Dispatcher Service Process states include SPAWN, TERMINATE, ERROR
Ingestion
DatasetIngestionConfigurationEvent ResourceModifiedEvent       Ingestion Management Service
Indicates changes in ingestion configuration
Ingestion
GranuleIngestedEvent
ResourceModifiedEvent resource id
Resource type (dataset)
  Ingestion Worker Process
TBD.
Governance
EnrollmentRequestStatusEvent RequestStatusEvent, NegotiationEvent provider id (org id or resource id)
"Org" or "Resource"
  Org Management Service
 
Governance RoleRequestStatusEvent RequestStatusEvent, NegotiationEvent provider actor id (org id or resource id) "Org" or "Resource"   Org Management Service  
Governance ResourceRequestStatusEvent RequestStatusEvent, NegotiationEvent provider actor id (org id or resource id) "Org" or "Resource"   Org Management Service  
Error
TransformError
ErrorEvent
resource id of transform
      TBD
Error
ProcessError
ErrorEvent process id
      TBD

Event Framework

The event framework is the event publishing/subscribing infrastructure, available to processes in the ION capability container. The event framework is used to publish events using DM Data distribution capabilities, and to subscribe to events. Events are not targeted at any special recipient and can be consumed by any interested process. Processes may react in real-time to events rather than repeatedly querying system state. Events have a defined type, identify the origin of the event and add event specific additional information

Figure 2. Event Framework and related services (OV-2)

Example EventPublisher

Example EventSubscriber

Event field defaults

The event framework EventPublisher sets the following event fields:

  • ts_created: to the timestamp the EventPublisher publish_event method is called
  • base_types: to the base types of the current event type

Mapping to Pubsub and Exchange resources

Exchange Point: pyon.events

Routing Key: Event.<base_types>.<event_type>.<sub_type>.<origin_type>.<origin>

Notes:

  • base_types contains as many (dot separated) words as there base types between Event and the actual event type. Example: Event.ResourceEvent.ResourceModifiedEvent
  • sub_type may contain many (dot separated) words

The event types are defined in the table above. The content of the origin field varies depending on the event type.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Mar 18, 2011

    Michael Meisinger says:

    See a discussion http://etherpad.ooici.org/0318-notifications
  2. Dec 13, 2012

    Michael Meisinger says:

    OLD STUFF - JUST FOR REFERENCE Release 1 event types This is for historic ref...

    OLD STUFF - JUST FOR REFERENCE

    Release 1 event types

    This is for historic reference only
    Category Event Type Event ID Event msg object  ID
    EventPublisher base subclass Origin in topic (eventtype.origin)
    Source Observers Notes
    Lifecycle related
    Resource life cycle
    1001
    ResourceLifeCycleEvent
    ResourceLifecycleEventPublisher resource id (UUID)
    Resource registry service, Datastore service (resource instance)
      New resource created sent by resource registry; other LC state changes by datastore based on pushes.
    Resources in R1 include dataset, datasource
      Container life cycle
    1051 ResourceLifeCycleEvent
    ResourceLifecycleEventPublisher container name
    CC Agent
    ?
    Container state NEW, READY, ACTIVE, TERMINATED, ERROR
      Process life cycle
    1052
    ResourceLifeCycleEvent
    ResourceLifecycleEventPublisher process exchange name
    CC Agent
    ?
    Process state NEW, READY, ACTIVE, TERMINATED, ERROR
    Infrastructure related Application start/stop 1075 AppLoaderEvent InfrastructureEventPublisher application name AppLoader class methods   Contains info from app definition and a STARTED or STOPPED state.
      Container Startup 1076 ContainerStartupEvent InfrastructureEventPublisher
    container id CC twisted service   Contains repeated list of apps or releases that started up.
    Resource related Data source update
    1101
    eg. TriggerEvent
    TriggerEventPublisher
    data source resource id (UUID) External data source (script)
    Dataset agent
    Created by CM script with a one-off message
      Data source unavailable 1102 DatasourceUnavailableEvent ResourceModifiedEventPublisher data source resource id (UUID) Ingest service AIS? With reason of what's going on
      Dataset supplement added 
    1111 DatasetSupplementAddedEvent 
    ResourceModifiedEventPublisher
    dataset resource id (UUID) Ingest service
    ? Contains such info as title, URL, timestamp, dataset_id, datasource_id.
      Business state modification 1112 ResourceModificationEvent ResourceModifiedEventPublisher       Publishes changes in business logic state that a resource maintains, not lifecycle state
      Dataset Change 1113 DatasetChangeEvent ResourceModifiedEventPublisher
    dataset resource id ingestion service AIS Used specifically for clearing the AIS Cache
      Datasource Change
    1114
    DataSourceChangeEvent
    ResourceModifiedEventPublisher
    datasource resource id
    AIS AIS Used specifically for clearing the AIS Cache
      Ingestion Processing 1115 IngestionProcessingEvent ResourceModifiedEventPublisher dataset resource id ingestion service JAW Used to increase JAW's rpc timeout to the ingestion call. Indicates Ingestion is still doing work.
      Dataset Streaming 1116 DatasetStreamingEvent ResourceModifiedEventPublisher dataset resource id DatasetAgent (java) Ingestion Represents data messages from DatasetAgent to Ingestion (recv_dataset, recv_chunk, recv_done).  NOTE this does not have a specific Event Message type - the message types sent are (10001, 2001, and 2005).
      Subscription New/Modification 1201 SubscriptionEvent ResourceModifiedEventPublisher dispatcher_id AIS Dispatcher, email service Handles new/modify/delete for subscriptions.
    Service related Schedule event
    2001
    TriggerEvent
    TriggerEventPublisher [See Schedule table.] Scheduler service
      Pre-registered schedule triggers event
    Logging related Informational log entries 3003 LoggingEvent LoggingEventPublisher       For informational messages that are published
      Error log entries
    3002 LoggingEvent LoggingEventPublisher   Logging framework within container
    Logging Service
    For critical and error log entries (not for lower log levels)
      Critical log entries
    3001
    LoggingEvent
    LoggingEventPublisher        
    Data related Data block 4001 DataEvent DataEventPublisher   Instrument or data ingest agent/service Data set management This carries a block of data into the PubSub system

    Potential Event types

    Category Event Source Observers Notes
    Presence Login * SSI Audit subsystem -
    - User CMDs profile SSI, user registry service Auditing, potentially other users Social networking for science - might be a sales point
    - User CMDs data subscription SSI, pubsub registry Auditing -
    Datasets Registration ** DM controller service Many! User probably registers new source via website or API (MATLAB)
    - Creation * DM controller service, Server-side processing system (Ferret, GridFields) Many, ingester for sure! Ingester must grab and parse the metadata for semantic classification
    - Dataset start - DM Controller, or perhaps syndication service Ingestion service -
    - Dataset block - Fetcher Preservation service, ingestion service -
    - Dataset error - Fetcher, ingestion svc, preservation Logger How do errors back-
    propagate to the user?
    - Dataset end - Fetcher Ditto Possibly creates 'new dataset' event?
    - Dataset preservation completed - Preservation service Dataset registry -
    - Dataset polling - Syndication System monitor DDN1 use-case, periodic polling and look for changes via mdsum(metadata)
    - Dataset change detected ** Fetcher, syndication Controller Who takes receipt and generates a request to pull the new data?
    - Data read Proxy, user, SSC service Metrics (cache hit ratio)
    Metrics Cache hit Controller, presenation service Service monitor, operator interface -
    - Cache miss Controller Ditto -
    - Cache fill Controller Ditto In the DX model, as a matter of policy a miss could trigger an automatic fill. May not be applicable here.
    Data stream Register * Datastream controller or registry Ingestion svc Many more possible consumers once dataset is classified
    - Start of data - Fetcher, instrument service Preservation, pub-sub (&subscribers) -
    - Block of data - Ditto Ditto -
    - End of data - Ditto Ditto May be impossible (infinite/unbounded data stream) or never happen
    - Delisting/removal * Ditto Ditto Gotta have a way to remove a source
    - Transformations Transformation service Unknown Speculative - possible use cases include WMS/KML generation, other ERDDAP-type tranformations.
    Ingestion Dataset tagged/classified Data listeners Is this how the XML data header is created?
    Audit Login OK * SSI Auditor, monitor -
    - Login failure * SSI Ditto -
    - Access OK SSI Ditto -
    - Access denied * SSI Ditto, more urgent though -
    - Role CMD SSI Ditto E.g. 'admin adds abilities to user X'
    Pub-sub & syndication Subscribe * Pub-Sub controller service P-S registry -
    - Unsubscribe * P-S controller svc P-S registry -
    - Query datasets/datastreams P-S controller or registry Metrics and monitoring
    Errors Security related SSI Audit log, sysadmins More on this in the Audit and Presence sections.
    - Filesystem Preservation service, SRM agent, PaaS agent, OS E.g. 'no space on device', 'permission denied', etc.
    - Network Any and all, primarily first seen by COI code Audit & logging AMQP errors, TCP errors, DNS.
    Logging LOG.(DEBUG,ERROR,INFO...) ** All P-S to developer consoles Mr Hyde explains.