|This document intends to serve as a cookbook, with concrete examples, for porting/implementing services in the R2 container.|
- Important Changes From R1 to R2
- Code Base Overview
- Developing with the container
- Service Implementation Overview
- Implementation Example
- Step 1 - Set up your development environment
- Step 2 - Gather auto generated interface template input
- Step 3 - Edit the service definition interface YAML file
- Step 4 - Generate the Service Interface YAML into a Python zope interface, base service class, and service clients
- Step 5 - Create/edit the implementation service class
- Step 6 - Create a unit test
- Step 7 - Documentation
- Step 8 - Check it all in
- If something goes wrong
R2 introduces the requirement of defining service interfaces via YAML files. Additionally, R2 replaces the mechanism for defining message objects and data model objects via Google Protocol Buffers with a YAML based mechanism.
Service clients are now implicit, not explicit like they were in R1, i.e. developers don't provide client classes for their services anymore - instead these are generated from the service definitions. Services no longer define operations with the 'op_' prefix. Clients now make service calls to operations as if they were invoking an imported method. Messaging is handled internally and is not exposed directly to the client or service. Clients and services should be coded to be messaging agnostic.
Message content is validated on your behalf so you don't need to write redundant chunks of validation logic at the top of each of your methods.
Method entry and exit will be logged on your behalf, removing more boiler plate from your code.
Gone are the yield and @defers (inlineCallbacks) of R1. R2 utilizes gevent to manage the messaging stack, replacing twisted. Gevent can be thought of as providing threading functionality to Python vs the callback functionality twisted provided.
Access to configuration is now implicit, not explicit. Every service can access configuration content via the self.CFG attribute. There are no longer process spawn-args. Instead, processes have access to local configuration overrides that may be specific to them.
Though the data store can still be accessed via the service layer, R2 also allows access as a library directly importable by services. This allows services to take advantage of the underlying data store's concurrency model efficiently.
All these changes help to reduce the boiler plate in your code. If you find yourself copy/pasting large chunks of code, refactor. Strive to have as concise and clean business logic as possible.
For the lines of code you do write, please conform to Python's PEP-8 coding style.
The R2 model object and service interface definitions can be found in the 'ion-definitions' git repository at url: firstname.lastname@example.org:ooici/ion-definitions.git
The repository directory structure is ordered as follows:
The R2 pyon code base can be found in the 'pyon' git repository at url: email@example.com:ooici/pyon.git. Generally, only the COI team should be making changes in this repository.
The repository is ordered as follows:
All subsystem teams will work in their own repository. (Note: for now all start work in the coi-services repository). These repositories will depend on only on pyon and can include third party dependencies. The general layout of these repositories is illustrated in the layout of the coi-services repository.
All configuration information is read at container startup and represented in memory as a Python dict. Configuration information is exposed to services via the 'self.CFG' attribute inside ION processes. You can access configuration values with several notations. Either the traditional dict of dicts notation or via dot notation, or via a get function. Either means is interchangeable. For example, given configuration content like the following...
You can code either one of the following statements in your service or process code:
Explicit config file access:
To implement/port a service in R2, the following steps need to be followed:
- Set up your development environment
- Define any data model objects, either in a separate YAML file under the 'obj/data' directory or at the top of the service interface definition file in the 'obj/services' directory
- Define the service interface via a YAML file in the 'obj/services' directory
- Code generate the service interface definition YAML into a Python zope interface, base service class, and service clients for that service
- Create/edit a service business logic module which extends the base service interface
- Create a unit test
- Commit your changes
The following illustrates all the steps required to create/port a service in R2. This example utilizes simplistic bank and bond trading services.
Please follow the install, package download and package dependencies steps in the coi-services README.
|The above README assumes installation on a Mac with OS X 10.6 Snow Leopard. For a Linux environment install, certain additional steps need to be performed that are not described here.|
At the end of this install, the following should be true
- You have defined a virtualenv 'coi' or similar and are in the 'coi-services' repository
- You have run bin/buildout successfully
- You can run 'bin/pycc --rel res/deploy/examples/hello.yml' successfully
- You can run 'bin/unittest' successfully
- You have installed the GIT hooks for submodules
- You have updated the repository with 'git pull'
You are now ready to start defining data model objects and service definitions.
We have made an attempt to somewhat ease the pain of porting services to the R2 container by auto generating template YAML model object for each GPB and a service definition for each class in the R1 code base that extents 'ServiceProcess'. These template files can be found in the 'ion-definitions' repository under the r1_templates directory. If the templates look useful, copy the template content to the interface definition directory 'objects/services'. Else, construct a new YAML file in the 'objects/services' directory. Note, the 'ion-definitions/objects' directory will be symbolically linked to the 'obj' directory in your repository. It is easiest to perform work on the YAML files via the symbolic link versus making changes directly in a cloned instance of the 'ion-definitions' repository.
Service definition files are located in the 'obj/services' directory under your sub-team repository. Service definitions are organized into sub-directories by CI subsystem of the 'obj/services' directory.
Near the top of the interface definition file, you should define the attribute 'name'. The value of this attribute is the name by which the service is known everywhere in the system. The startup configuration will reference this name. Additionally, other services that utilize this service will reference this name when defining their dependencies. This value should be all lower case, optionally with underscores inserted as necessary to make the name more readable. The value of the 'name' attribute for the bond trading service is:
Just below the 'name' attribute, add a 'dependencies' attribute. The value of this attribute is a comma separated list '' comprising names of services that this service depends on. For example, the bond trading service depends on the resource registry service.
Now the real interface definition work starts. For each operation in the service, do the following.
If this is a porting effort, you should determine the GPB(s) that were passed as arguments and look for the associated auto generated GPB to YAML conversion file(s) found in the 'r1_templates/protodefs' directory. These files should jump start your conversion of the operation parameter list definitions. These conversion files contain the YAML equivalent of the GPB definition with a type appropriate default value assigned to each attribute. Either copy these object definitions into the data model object directory 'objects/data' or copy the content to the top of the service interface definition file you are working on. Each data model object definition added to the top of the service interface definition file must start with the 'obj:' keyword and be terminated by a document delimiter line "---". If this is a new operation, define the op and the parameters. It is your choice whether to define parameters as fully enumerated or in terms of data model objects. Note, parameter order will be maintained in the resulting generated Python interface file.
|See the Doxygen generation reference page for details on adding documentation to services and objects|
The following illustrates three different ways to accomplish defining a service interface illustrated below. Note that YAML is a one pass parser so nested data types need to be defined before they are referenced. Also note, the 'objects/data' directory is parsed before the 'objects/services' directory. But there is no guarantee of parsing order of individual files within a specific directory. Take this into consideration when deciding the appropriate place to define data model objects.
- In R2 we are striving for all developers to embrace and code to the PEP-8 standards (http://www.python.org/dev/peps/pep-0008/). With this in mind, name your data model objects with upper, camel case names. Name your object fields with lower case and underscores.
- Every field should have a default value assigned. One key improvement in R2 is the validation of message content in the messaging stack. The type of the field is presumed from the type of the default value. You can opt to not provide a default value, but then that field cannot benefit from validation.
- Enum types are available to define a choice of labels and values for one field, with a default value. If you assign an enum a label (e.g. ACTIVE), its assigned value (e.g. 1) is actually used and serialized
- Types can be composed of nested types. Keeping in mind YAML is a single pass parser determines the order in which you define nested types.
- Types can extend other types. Keeping in mind YAML is a single pass parser determines the order in which you define extended types. Extension is used for Resource objects, for instance:
Step 4 - Generate the Service Interface YAML into a Python zope interface, base service class, and service clients
Run the interface generation script:
For all YAML service interface definition files in the 'obj/services' directory, the generate_interfaces script will build the following for each defined service:
|Python zope interface||IExampleService|
|Base service||BaseExampleService||You inherit from this service in your actual service implementation. It also implements the interface defined above.|
|Service clients for this service||ExampleServiceClient, ExampleProcessServiceClient||Two varieties, one has an attached Process (f/e to use the Process interceptor stack), the other does not need a Process.|
|Dependent service clients||ExampleServiceClients||Internally used class to set the Base service's "clients" attribute. This class helps IDE auto-completion by defining the dependencies statically.|
The output python modules are located in the '<your repo>/interface/services' directory. Modules are named 'i<yaml file base name>.py'.
|You can regenerate the service interfaces as often as you need. None of your code will be lost.|
Your service implementation class should extend this base class, not the zope interface. This is to ensure that the container can derive the service's name and dependency information automatically.
Before moving on to the next step, validate the content of your data model and service interface YAML files by attempting to start the container (bin/pycc). Be sure to do this before checking anything into the ion-definitions repository.
The example code shown here can be found in coi-services/examples/bank. In this case, look in trade_service.py.
- At the top of your implementation module, add import 'from interface.services.i<yaml file base name> import Base<your service name>'
- Copy/paste operation method declarations from the generated Python interface file into your implementation module.
- Create/edit operation business logic to utilize the parameters.
This example shows a number of key aspects of the new R2 container.
Note that there isn't boilerplate value type checking code at the top of the operation. Let the built in validation interceptor handle value type checking. By all means add specific value checking as necessary, though.
Calling dependent services
The Trade Service depends on the datastore service. Client calls to the dependent service are made with lines like the following.
From the example above, the resource registry create method is called, returning a tuple.
How does this work? At service startup, the container iterates through the service's dependencies attribute and constructs an RPCClient object for each dependent service. This RPCClient instance is then added as an attribute of the client service under the hierarchy 'self.clients.<service name>'. Any operation implemented by the dependent service is available to be called from the client service.
Note there isn't entry and return logging statements polluting the business logic. Defer to the container to log service operation entry/exit. Use logging to record key branches taken in your business logic.
Data model objects defined in YAML files the ion-definitions repository are read at container startup and persisted in a repository. Developers can ask the repository to create an instance of one of these objects. The code asks for the type by name and can optionally pass a dict that will populate the object instance.
|For more information regarding resource object persistence and access, see the R2 Resource Development guide.|
As you can see from the coding example, you return values just as if you were returning from a standard method call. No need to invoke a messaging operation.
The intention in R2 is for all service operation failures to be reported in terms of HTTP status codes as we have mapped them in the 'pyon/core/exception.py' module. On error, your service op should throw one of the exception types from that module with useful, non-cryptic causal text. The messaging stack will catch the exception, return an error message on your behalf and re-raise the exception on the client side. Please do not attempt to circumvent this mechanism as this will only complicate debugging the deployed system. Your service should either run to completion and return valid return data or throw exception to report an error.
Now that we have switched over to gevent, there is no need to try and match up your yields and defers. Purge all yield and @defer statements from your business logic.
We're striving for 90+% code coverage in the core and 80+% code coverage in general. In light of that, every new module added should have a unit test associated with it. Like in R1, unit tests are declared in a 'test' directory subordinate to the directory containing the module to be defined.
Unit tests should focus very specifically on code within a single module. Do not allow your unit tests to bleed over into other modules. If you do so, you're really building an integration test.
Currently you have two options for testing your code. If individual methods do not depend on one or more services, you can just call the methods directly from your test class. See 'pyon/pyon/datastore/test/test_datastores.py' for sample code. Alternatively, if you are testing a service's operations, for now you will need to define a test method that starts the container and all necessary services and then feeds requests to the service via a ProcessRPCClient endpoint. See 'coi-services/examples/bank/test/test_bank.py' for sample code. Ultimately, there will be a mock object concept that will allow you to stub out services to eliminate the need for starting the container and using message passing to test your services.
Really, you should be doing this all along.
Object Specifications (in objects/data/)
- Use comments to document object types and object attributes
- The following rules apply to comments
- Comments need to go in lines directly before an item (same indentation)
- There can be several lines of comments per item (but no empty lines)
- Alternatively or additionally, a comment can be in the same line as the item
- One space between # and text in comment
- The characters #@ are reserved to decorators
- Suggested: Two empty lines between objects
- Suggested: One empty line between attributes
Service Specifications (in objects/service/)
- Use docstring on the service level
- Use docstring on the service operation level
- Use comments for service operation arguments and returns
Buildbot use doxygen with doxypy to generate documentation for the project. Here is the page if for some reason you want to generate yourself: Adding Doxygen to your Python project.
In order to take advantage of doxygen generation, you need to use doxygen tags.
Common doxygen tags to use: (Here is a page illustrating the generation effect: Doxygen generation reference.)
|Only architects should commit to the ooici/ion-definitions repository. Everyone else needs to create a fork of ion-definitions in their own GitHub account and push commits to this fork.|
After creating your personal fork of ion-defintions on github, execute the following from the coi-services project directory:
So now that you've created/modified your data model objects, service definitions and implementation class, how do you commit it to git? Your commit and push involves first committing and pushing the ion-definition submodule changes and then changes to your sub-team repository. From the root of your sub-team repository, do the following.
Every time you run 'git submodule update', ion-definitions will be in a headless branch.
So, it's important for you to 'git checkout master' if you need make changes to ion-definitions.
Now got to GitHub and open a pull request from "myaccount/ion-definitions" to "ooici/ion-definitions" and wait for resolution. Then continue pushing your services code.
- If your container fails at startup with an error message like "ImportError: No module named services.idirectory_service", you failed to run the "bin/generate_interfaces" script.
- If your container fails at startup with an error message like "ImportError: No module named bank.trade_service" and you are sure that your service's processapp definition in the deploy yaml file is correct, you've failed to include an "_init_.py" in your implementation package directory.
- If trying to run a python file directly with python ( rather than pycc ) and you het an error like "ImportError: No module named gevent", you need to use the buildout version of python to get all of the eggs; so use bin/python instead.