Skip to end of metadata
Go to start of metadata
 The lcaarch project is a GIT source code repository and integrated Python code base that provides some basic framework code for bootstrapping a CI-wide prototype and for running prototype services, and the actual service implementations themselves together with unit level and system level test cases. The scope of the services is the full functionality of the OOI CI deliverable of release 1 transcending any subsystem prototypes. Initially, all services start out as stubs with their message sending and receiving interface only and have the potential to grow into a fully functional operational architecture prototype as required for LCA.

Quick Start

Using GIT (need to install first), get a copy of the source code tree with:

See README.txt file in the root directory of the lcaarch project. You need to install a few core dependencies first, see the new developer tutorial.

Run the bootstrap script from the lcaarch directory to see if things work:

Run all test cases recursively from lcaarch directory with:

Summary and Purpose

There are many individual prototype development projects in OOI CI mostly structured by subsystem. Some of them are bigger, some smaller. Some active, some historic. Some on UCSD's gitosis server, some on github. This is good.

This project is an effort to integrate the major work going on in the individual development projects into a coherent system. The emphasis is not so much on solving the functional and technical tasks (e.g. how to interface with the Cassandra datastore, or how to provision a contextualized VM on Nimbus and EC2). Instead, the focus is on the services of the integrated system and their dependencies. What messages do the services react to? What messages do they send in response? How are the messages and their structure defined? What are some system level scenarios that exercise all the services?

In order to achieve this goal, it is sufficient if all services are stubbed out in their basic form only. Additional functionality can be integrated as possible and needed over time. The project has the potential to grow into a fully functional operational prototype that can demonstrate the system's core capabilities during LCA.

We start small. Things are expected to change in the beginning. The initial hurdle to use the lcaarch project should be small (meaning very few needed dependencies).

lcaarch project overview

The following tells you what you should know about the organization and use of the lcaarch project.

How to get it running?

lcaarch runs at least on Macs and in Ubuntu Linux environments. You don't need to run your own instance of a message broker or key/value store. By default the servers on amoeba.ucsd.edu are used.

See above for the quick start instructions. Read the README.txt for installation information about dependencies and for running the core functions and test cases. The test cases in the source code tree are good examples of the functionality and how to use it, and for test driven development. There is a growing number of test cases that demonstrate a wealth of unit level, service level, integration level (multiple services working together) and system level tests.

Directory structure

Dir
Explanation
.
Named "lcaarch". The root of the project source tree. Everything is expected to be started from here (twistd and trial)
ion
Python source code for the Integrated Observatory Network (i.e. ION) prototype
*/test
Anywhere in the source tree: these directories contain trial test cases
ion/agents
Agent processes and base classes for agent processes
ion/core
Common base classes for bootstrapping the system or to represent a running, communicating process in the system
ion/core/cc
Capability Container mechanics
ion/data
Information and data related modules that are not subsystem (COI, DM) services, but actual helper classes to access data stores or to manage ION data
ion/interact
Core classes for managing interactions, conversations, protocols
ion/services
Base classes for services
ion/services/coi
ion/services/coi
ion/services/dm
ion/services/sa
Actual service implementations (as processes) with their tests, structured by subsystem. Subdirectories may be added where necessary. Extensive functional code may be placed elsewhere.
ion/test
Base classes for ION trial unit tests
ion/util
Utility modules, functions and classes
logs
Files in here are generated by the logging mechanism. There are trace level logs (i.e. copies of the console with debug information), explicit logs that were sent via messages to the logger service and other logs, such as all messages sent in the system
res
Any kind of other resource files needed for running the system
res/config
Configuration files for the ION system, such as the central system configuration file (ion.config) and specific config files. Many files are in the form of executable (by eval()) python statements, such as constant definitions of Python directories and lists.
res/logging
Configuration files for Python logging component
res/scripts
Magnet container start scripts

Code Structure and Core Base Classes

Package
Module
Class
Purpose and Explanation
ion.agents
resource_agent.py ResourceAgent
Base class for all resource agent processes. Note that they are not services, but they may call services.
ion.core
base_process.py
BaseProcess
The core definitions for a process running within a capability container. A process has a unique identifier (container id + local id) and can send and receive messages via the container's AMQP messaging interface. Process instances are spawned by another process (called the supervisor process).
The default implementation is that when a message to the process is received with a header "op=performative", the class method "op_performative" is called. Typical service subclasses only have to define the operation methods they support.
ion.core
base_process.py
RpcClient
Client base class for performing synchronous service calls in RPC style (but non-blocking for other processes)
ion.core
bootstrap.py
-
Central class for bootstrapping the ION system in a container. Note, trial test cases have a different way of starting the container.
ion.core
ioninit.py
-
Loaded by the ion package by default. Basic initializations of all code in the system, such as setting up the logging system, or loading the central config file
ion.services
base_service.py
BaseService
Base class for a service process. On initialization of a service process, the "slc_init()" method is called.
ion.services
hello_service.py
HelloService
Example service process implementation to use as reference.
ion.util
procutils.py
-
Utility functions for processes
ion.test
iontest.py
IonTestCase
Base class for a twisted trial test cases. All such classes are executed if you do a "trial ion" from the lcaarch dir. This base class helps to set up the container's operational environment and to start and initialize the core services, if needed for a test case.

Basic conventions

  • package names are lower case only, one word no underscores
  • module.py names are lower case only, with underscores as needed. Keep them short.
  • Class names are CapitalCase (e.g. ServiceClass, MyService)
  • Functions and class methods start with a lower case and then are either func_name or funcName
  • Variables start with a lower case and then are either varname, var_name or varName
  • Constants may be all upper case (e.h. CONSTANT = 'some')
  • test case modules start with test_
  • test case class names have the word Test in them (e.g. SomeTest or TestSomething)
  • test case methods start with test_ (e.g. test_function1)

Auto generated documentation

Doxygen automatically creates source code documentation every 30 minutes. This is VERY useful.

Please see Doxygen lcaarch. This includes an automatically generated plot of the class hierarchy.

How to work with it?

The following tells you how to use the project and develop you own code and contribute back.

Get Involved

Know which part of the system you work on (e.g. on a subsystem service, a specific agent etc). You already know if you work on the framework and base classes. So most likely you can ignore them for now as long as you understand them.

Know where to place your source code in the source tree

Know how to write a trial unit test and where to place it

Communicate frequently and do frequent git pulls from the master remote branch.

Contribute

Make sure you get the source code tree in non-anonymous mode. Otherwise you cannot push to the server based GIT repository. This means use the "git clone git@..." style. If you previously checked out in anonymous mode, you can also edit your .git/config file.

When making local changes, create local GIT branches (see GIT revision control page, tutorial):

Add your new files and commit your work locally with telling commit messages

Once you are confident about the sum of all your changes, merge locally with the remote master branch, and ONLY IF ALL THINGS STILL WORK LOCALLY, push to the server.

Usage Details

Bootstrap

1 Setup the container (logging, load config files, check versions etc)

2 Walk through an ordered list of processes that will be started in this order. Different "installations" may have different bootstrap process sequences.
  - Startup the root supervisor process on the local container
  - The root supervisor spawns all the configured processes (via self.spawn_child)
  - Each process is spawned (with spawn arguments) and immediately after receives an init message, which triggers the slc_init

The core set of processes including the CC agent are spawned in sequence first and then any additional services and processes (also in order). Dependency is (currently) built into the bootstrap order.

Calling services

Every service (note: not a process) has a well known default name. This name is defined in the service declaration in the source code module of the service. For instance the COI datastore service uses the name "datastore" as well known name. All clients of this service can use this name to access the service.

AMQP queue names are determined by "scoping" the well known name of a service with the identifier of the running system. The system is the set of processes that were bootstrapped from the one root supervisor process, and all capability containers that share the system name.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. May 17, 2010

    Paul Hubbard says:

    Note that when running Python 2.5, LCAarch depends on simplejson, so for a new i...

    Note that when running Python 2.5, LCAarch depends on simplejson, so for a new install you also need to install it:

  2. Feb 01, 2011

    Carlos Rueda says:

    Is this lcaarch project still relevant? Seems like https://github.com/ooici/ionc...

    Is this lcaarch project still relevant? Seems like https://github.com/ooici/ioncore-python has superseded it (in fact , there is no lcaarch project at github, afaict). If so, a note at the beginning of this page would be convenient, as well as a corresponding update to https://confluence.oceanobservatories.org/display/CIDev/IONCore+Java (last edited 8/15/10).