Skip to end of metadata
Go to start of metadata
This document intends to serve as a cookbook, with concrete examples, for porting/implementing services in the R2 container.

Important Changes From R1 to R2

R2 introduces the requirement of defining service interfaces via YAML files. Additionally, R2 replaces the mechanism for defining message objects and data model objects via Google Protocol Buffers with a YAML based mechanism.

Service clients are now implicit, not explicit like they were in R1, i.e. developers don't provide client classes for their services anymore - instead these are generated from the service definitions. Services no longer define operations with the 'op_' prefix. Clients now make service calls to operations as if they were invoking an imported method. Messaging is handled internally and is not exposed directly to the client or service. Clients and services should be coded to be messaging agnostic.

Message content is validated on your behalf so you don't need to write redundant chunks of validation logic at the top of each of your methods.

Method entry and exit will be logged on your behalf, removing more boiler plate from your code.

Gone are the yield and @defers (inlineCallbacks) of R1.  R2 utilizes gevent to manage the messaging stack, replacing twisted.  Gevent can be thought of as providing threading functionality to Python vs the callback functionality twisted provided.

Access to configuration is now implicit, not explicit. Every service can access configuration content via the self.CFG attribute. There are no longer process spawn-args. Instead, processes have access to local configuration overrides that may be specific to them.

Though the data store can still be accessed via the service layer, R2 also allows access as a library directly importable by services. This allows services to take advantage of the underlying data store's concurrency model efficiently.

All these changes help to reduce the boiler plate in your code.   If you find yourself copy/pasting large chunks of code, refactor.  Strive to have as concise and clean business logic as possible.

For the lines of code you do write, please conform to Python's PEP-8 coding style.

Code Base Overview

Model Object and Service Interface Definitions

The R2 model object and service interface definitions can be found in the 'ion-definitions' git repository at url:

The repository directory structure is ordered as follows:

|--- objects/
|      |--- data : data model object definition YAML files
|      |--- services : service interface definition YAML files
|--- r1_templates/
|      |--- protodefs : transformed ion-object-definitions GPBs to YAML template files, ordered by both id number and name
|      |--- servicedefs : template service definition YAML files gleaned from every process in ioncore-python that extended class ServiceProcess
|--- res/
|      |--- apps/ : home for all service app files
|      |--- config/ :
|            |--- pyon.yml : main config file (override by defining pyon.local.yml)
|            |--- logging.yml : main logging config file (override by defining logging.local.yml)
|      |--- deploy/ : home for all service startup (deploy) files


The R2 pyon code base can be found in the 'pyon' git repository at url: Generally, only the COI team should be making changes in this repository.

The repository is ordered as follows:

|--- extern/
|      |--- ion-definitions : git submodule referencing ion-definitions
|--- pycc/ : script for launching the container (equivalent to "twistd ..." in R1)
|--- pyon/
|      |--- container
|      |--- core
|      |--- datastore
|      |--- directory
|      |--- ion
|      |--- net
|      |--- service
|      |--- util
|--- obj/ : symbolic link to extern/ion-definitions/objects
|--- res/ : symbolic link to exten/ion-definitions/res
|--- scripts/
|      |--- : container launching module (invoked from root pycc script)
|      |--- : code generation script which converts YAML service interface definitions to Python zope interfaces

Service Repositories

All subsystem teams will work in their own repository. (Note: for now all start work in the coi-services repository). These repositories will depend on only on pyon and can include third party dependencies. The general layout of these repositories is illustrated in the layout of the coi-services repository.

|--- bin/
|      |--- buildout : util to build and install necessary eggs
|      |--- control_cc : script for controlling a running container
|      |--- generate_interfaces : converts service definitions into python interfaces
|      |--- nosetests : runs unit tests in repository
|      |--- pycc : container startup script
|      |--- python : pre-loaded with pyon dependency eggs
|      |--- unittest : rund unitests locally
|--- eggs
|--- extern
|      |--- ion-definitions : git submodule referencing ion-definitions
|--- ion : root of the source hierarchy
|--- obj : symbolic link to ion-definitions/objects
|--- res : symbolic link to ion-definitions/res

Developing with the container

Accessing configuration information at runtime

All configuration information is read at container startup and represented in memory as a Python dict. Configuration information is exposed to services via the 'self.CFG' attribute inside ION processes. You can access configuration values with several notations. Either the traditional dict of dicts notation or via dot notation, or via a get function. Either means is interchangeable. For example, given configuration content like the following...


You can code either one of the following statements in your service or process code:

Explicit config file access:

Service Implementation Overview

To implement/port a service in R2, the following steps need to be followed:

  • Set up your development environment
  • Define any data model objects, either in a separate YAML file under the 'obj/data' directory or at the top of the service interface definition file in the 'obj/services' directory
  • Define the service interface via a YAML file in the 'obj/services' directory
  • Code generate the service interface definition YAML into a Python zope interface, base service class, and service clients for that service
  • Create/edit a service business logic module which extends the base service interface
  • Create a unit test
  • Commit your changes

Implementation Example

The following illustrates all the steps required to create/port a service in R2. This example utilizes simplistic bank and bond trading services.

Step 1 - Set up your development environment

Please follow the install, package download and package dependencies steps in the coi-services README.

The above README assumes installation on a Mac with OS X 10.6 Snow Leopard. For a Linux environment install, certain additional steps need to be performed that are not described here.

At the end of this install, the following should be true

  • You have defined a virtualenv 'coi' or similar and are in the 'coi-services' repository
  • You have run bin/buildout successfully
  • You can run 'bin/pycc --rel res/deploy/examples/hello.yml' successfully
  • You can run 'bin/unittest' successfully
  • You have installed the GIT hooks for submodules
  • You have updated the repository with 'git pull'

You are now ready to start defining data model objects and service definitions.

Step 2 - Gather auto generated interface template input

We have made an attempt to somewhat ease the pain of porting services to the R2 container by auto generating template YAML model object for each GPB and a service definition for each class in the R1 code base that extents 'ServiceProcess'. These template files can be found in the 'ion-definitions' repository under the r1_templates directory. If the templates look useful, copy the template content to the interface definition directory 'objects/services'. Else, construct a new YAML file in the 'objects/services' directory. Note, the 'ion-definitions/objects' directory will be symbolically linked to the 'obj' directory in your repository. It is easiest to perform work on the YAML files via the symbolic link versus making changes directly in a cloned instance of the 'ion-definitions' repository.

Step 3 - Edit the service definition interface YAML file

Service definition files are located in the 'obj/services' directory under your sub-team repository. Service definitions are organized into sub-directories by CI subsystem of the 'obj/services' directory.

Define the service name

Near the top of the interface definition file, you should define the attribute 'name'. The value of this attribute is the name by which the service is known everywhere in the system. The startup configuration will reference this name. Additionally, other services that utilize this service will reference this name when defining their dependencies. This value should be all lower case, optionally with underscores inserted as necessary to make the name more readable. The value of the 'name' attribute for the bond trading service is:

Declare all service dependencies

Just below the 'name' attribute, add a 'dependencies' attribute. The value of this attribute is a comma separated list '[]' comprising names of services that this service depends on. For example, the bond trading service depends on the resource registry service.

Define service operations and their parameters

Now the real interface definition work starts. For each operation in the service, do the following.

If this is a porting effort, you should determine the GPB(s) that were passed as arguments and look for the associated auto generated GPB to YAML conversion file(s) found in the 'r1_templates/protodefs' directory. These files should jump start your conversion of the operation parameter list definitions. These conversion files contain the YAML equivalent of the GPB definition with a type appropriate default value assigned to each attribute. Either copy these object definitions into the data model object directory 'objects/data' or copy the content to the top of the service interface definition file you are working on. Each data model object definition added to the top of the service interface definition file must start with the 'obj:' keyword and be terminated by a document delimiter line "---". If this is a new operation, define the op and the parameters. It is your choice whether to define parameters as fully enumerated or in terms of data model objects. Note, parameter order will be maintained in the resulting generated Python interface file.

See the Doxygen generation reference page for details on adding documentation to services and objects

The following illustrates three different ways to accomplish defining a service interface illustrated below. Note that YAML is a one pass parser so nested data types need to be defined before they are referenced. Also note, the 'objects/data' directory is parsed before the 'objects/services' directory. But there is no guarantee of parsing order of individual files within a specific directory. Take this into consideration when deciding the appropriate place to define data model objects.

Option 1 - Fully enumerate all parameters on operation definition


Option 2 - Define parameters in terms of data model objects in separate data model definition files



Option 3 - Define parameters in terms of data model objects in service interface definition file


Rules for defining model objects and service parameters

- In R2 we are striving for all developers to embrace and code to the PEP-8 standards ( With this in mind, name your data model objects with upper, camel case names. Name your object fields with lower case and underscores.

- Every field should have a default value assigned. One key improvement in R2 is the validation of message content in the messaging stack. The type of the field is presumed from the type of the default value. You can opt to not provide a default value, but then that field cannot benefit from validation.

- Enum types are available to define a choice of labels and values for one field, with a default value. If you assign an enum a label (e.g. ACTIVE), its assigned value (e.g. 1) is actually used and serialized

- Types can be composed of nested types. Keeping in mind YAML is a single pass parser determines the order in which you define nested types.

- Types can extend other types. Keeping in mind YAML is a single pass parser determines the order in which you define extended types. Extension is used for Resource objects, for instance:

Step 4 - Generate the Service Interface YAML into a Python zope interface, base service class, and service clients

Run the interface generation script:

> cd <root of your repo>
> bin/generate_interfaces

For all YAML service interface definition files in the 'obj/services' directory, the generate_interfaces script will build the following for each defined service:

Generated Item Name Comments
Python zope interface IExampleService  
Base service BaseExampleService You inherit from this service in your actual service implementation. It also implements the interface defined above.
Service clients for this service ExampleServiceClient, ExampleProcessServiceClient Two varieties, one has an attached Process (f/e to use the Process interceptor stack), the other does not need a Process.
Dependent service clients ExampleServiceClients Internally used class to set the Base service's "clients" attribute. This class helps IDE auto-completion by defining the dependencies statically.

The output python modules are located in the '<your repo>/interface/services' directory. Modules are named 'i<yaml file base name>.py'.

You can regenerate the service interfaces as often as you need. None of your code will be lost.

Your service implementation class should extend this base class, not the zope interface. This is to ensure that the container can derive the service's name and dependency information automatically.

Before moving on to the next step, validate the content of your data model and service interface YAML files by attempting to start the container (bin/pycc).  Be sure to do this before checking anything into the ion-definitions repository.

Step 5 - Create/edit the implementation service class

The example code shown here can be found in coi-services/examples/bank.  In this case, look in

- At the top of your implementation module, add import 'from<yaml file base name> import Base<your service name>'

- Copy/paste operation method declarations from the generated Python interface file into your implementation module.

- Create/edit operation business logic to utilize the parameters.

This example shows a number of key aspects of the new R2 container.

Value type checking

Note that there isn't boilerplate value type checking code at the top of the operation. Let the built in validation interceptor handle value type checking. By all means add specific value checking as necessary, though.

Calling dependent services

The Trade Service depends on the datastore service. Client calls to the dependent service are made with lines like the following.

From the example above, the resource registry create method is called, returning a tuple.

How does this work? At service startup, the container iterates through the service's dependencies attribute and constructs an RPCClient object for each dependent service. This RPCClient instance is then added as an attribute of the client service under the hierarchy 'self.clients.<service name>'. Any operation implemented by the dependent service is available to be called from the client service.


Note there isn't entry and return logging statements polluting the business logic. Defer to the container to log service operation entry/exit. Use logging to record key branches taken in your business logic.

IonObject and the IonObject repository

Data model objects defined in YAML files the ion-definitions repository are read at container startup and persisted in a repository. Developers can ask the repository to create an instance of one of these objects. The code asks for the type by name and can optionally pass a dict that will populate the object instance.

For more information regarding resource object persistence and access, see the R2 Resource Development guide.
Returning from operations and returning values

As you can see from the coding example, you return values just as if you were returning from a standard method call. No need to invoke a messaging operation.

Error handling

The intention in R2 is for all service operation failures to be reported in terms of HTTP status codes as we have mapped them in the 'pyon/core/' module. On error, your service op should throw one of the exception types from that module with useful, non-cryptic causal text. The messaging stack will catch the exception, return an error message on your behalf and re-raise the exception on the client side. Please do not attempt to circumvent this mechanism as this will only complicate debugging the deployed system. Your service should either run to completion and return valid return data or throw exception to report an error.

No yield/@defers!!!

Now that we have switched over to gevent, there is no need to try and match up your yields and defers.  Purge all yield and @defer statements from your business logic.

Step 6 - Create a unit test

We're striving for 90+% code coverage in the core and 80+% code coverage in general.  In light of that, every new module added should have a unit test associated with it.  Like in R1, unit tests are declared in a 'test' directory subordinate to the directory containing the module to be defined.

Unit tests should focus very specifically on code within a single module.  Do not allow your unit tests to bleed over into other modules.  If you do so, you're really building an integration test.

Currently you have two options for testing your code.  If individual methods do not depend on one or more services, you can just call the methods directly from your test class. See 'pyon/pyon/datastore/test/' for sample code.  Alternatively, if you are testing a service's operations, for now you will need to define a test method that starts the container and all necessary services and then feeds requests to the service via a ProcessRPCClient endpoint.  See 'coi-services/examples/bank/test/' for sample code.  Ultimately, there will be a mock object concept that will allow you to stub out services to eliminate the need for starting the container and using message passing to test your services.

Step 7 - Documentation

Really, you should be doing this all along.

Documenting specifications

Object Specifications (in objects/data/)

  • Use comments to document object types and object attributes
  • The following rules apply to comments
    • Comments need to go in lines directly before an item (same indentation)
    • There can be several lines of comments per item (but no empty lines)
    • Alternatively or additionally, a comment can be in the same line as the item
    • One space between # and text in comment
    • The characters #@ are reserved to decorators
    • Suggested: Two empty lines between objects
    • Suggested: One empty line between attributes

Service Specifications (in objects/service/)

  • Use docstring on the service level
  • Use docstring on the service operation level
  • Use comments for service operation arguments and returns
Doxygen Documentation for service operations

Buildbot use doxygen with doxypy to generate documentation for the project. Here is the page if for some reason you want to generate yourself: Adding Doxygen to your Python project.

In order to take advantage of doxygen generation, you need to use doxygen tags.
Common doxygen tags to use: (Here is a page illustrating the generation effect: Doxygen generation reference.)

Step 8 - Check it all in

Only architects should commit to the ooici/ion-definitions repository. Everyone else needs to create a fork of ion-definitions in their own GitHub account and push commits to this fork.

After creating your personal fork of ion-defintions on github, execute the following from the coi-services project directory:

So now that you've created/modified your data model objects, service definitions and implementation class, how do you commit it to git? Your commit and push involves first committing and pushing the ion-definition submodule changes and then changes to your sub-team repository. From the root of your sub-team repository, do the following.

Handy Hint
Every time you run 'git submodule update', ion-definitions will be in a headless branch.
So, it's important for you to 'git checkout master' if you need make changes to ion-definitions.

For developers:

Now got to GitHub and open a pull request from "myaccount/ion-definitions" to "ooici/ion-definitions" and wait for resolution. Then continue pushing your services code.

For architects:

If something goes wrong

  • If your container fails at startup with an error message like "ImportError: No module named services.idirectory_service", you failed to run the "bin/generate_interfaces" script.
  • If your container fails at startup with an error message like "ImportError: No module named bank.trade_service" and you are sure that your service's processapp definition in the deploy yaml file is correct, you've failed to include an "" in your implementation package directory.
  • If trying to run a python file directly with python ( rather than pycc ) and you het an error like "ImportError: No module named gevent", you need to use the buildout version of python to get all of the eggs; so use bin/python instead.
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.