r14 - 18 Apr 2007 - 17:30:31 - BrianMoseleyYou are here: OSAF >  Projects Web  >  CosmoHome > CosmoDevelopmentHome > CosmoAtomPrototype

Cosmo Atom Prototype

These notes decribe a prototype built to demonstrate communication between the web UI and the server using the Atom Syndication Format ("Atom") and the Atom Publishing Protocol ("APP") as described in CosmoFeedServiceSpec. The prototype was constructed by Brian Moseley, Bobby Rullo and Travis Vachon.

Executive Summary

We are pleased with the simplicity of communicating between the web UI and the server using Atom and APP. The code design in both client and server is simplified and becomes easier to maintain. All communications are done in an open, transparent, easily debuggable fashion that facilitates an independently-layered server architecture. The Atom protocols are sufficiently simple but powerful that a wide variety of other programs can use them to access server data in the same manner as the UI. We recommend evolving the prototype work done thus far into a full-featured Atom and APP implementation and completely removing the existing JSON-RPC implementation.

Rationale

The web UI currently interacts with the server using JSON-RPC. This allows the UI to execute method calls on Java objects in the server. These objects are a simplified version of the Cosmo model that lends itself well to being serialized using JSON for remoting purposes.

With JSON-RPC, all RPC operations are targeted to a single URL, /json-rpc. The details of the target object and the method name and arguments are hidden inside the JSON payload in the HTTP request content. Therefore, these details are not available to the server until the request content has been parsed by the JsonRpcServlet. This means that servlet filter-based security and logging facilities are not employed for JSON-RPC requests, and any security and logging to be done when servicing these requests has to be coded by hand.

Furthermore, there are often variations on how the client needs to invoke a remote method. For instance, some methods which return sets of results need to incorporate paging (a mechanism for limiting the size and offset of the slice of the result set that is returned). Some of those same methods allow both tickets and user credentials to be supplied for security negotiation. With JSON-RPC, each "logical" operation requires multiple remote methods, each with a different signature, to be implemented both on the server and the client for each combination of parameters.

By using a REST-style protocol to communicate between the UI and the server, all of these problems are avoided. Operations are targeted at specific resources which map to entities in the server, and different HTTP methods are used to invoke different operations on a particular resource (e.g. GET to retrieve the list of items in a collection, POST to add an item to a collection, DELETE to remove an item from a collection, etc.). Filter-based security and logging may transparently participate in the fulfillment of these operations without any custom logic having to be provided. Standard HTTP mechanisms for parameterizing queries and for providing extra information about a request (extension request and response headers) can be used to specify tickets, Basic credentials and paging parameters.

Atom and APP together define a bidirectional REST protocol for representing HTTP collections as feeds and resources as entries with associated media content. Feeds and entries also carry meta-data about their author and dates of publication and most recent update. Entries can be added to feeds with POST, updated with PUT and deleted with DELETE.

This model maps well to the Chandler/Cosmo domain model. A feed can be generated to describe the contents of a collection identified by a specific URI. The URI can be parameterized to constrain the type and amount of contents that are returned (for example, all items which are event-stamped and start within some period of time, or the first 25 items in a collection sorted by last modified date in ascending order). An entry can contain content in one of the several data formats understood by Cosmo, including iCalendar, EIMML, and EIM serialized with JSON.

Goals

The exercise was meant to explore the architecture of Atom-based communication between the web UI and the server. There are several "hot spots" in this architecture:

Client

  • Converting to and from the client domain model to the interchange model
  • Formatting and parsing entity representations that are processable by the server
  • Constructing and interpreting Atom and APP requests and responses

Server

  • Interpreting and constructing Atom and APP requests and responses
  • Executing parameterized queries specified by GET requests
  • Applying creations and updates specified by POST, PUT and DELETE requests
  • Formatting and parsing entity representations in various formats (EIMML, EIM-JSON, iCalendar)

Constraints

In order to focus the investigation, we chose to implement one "combination" use case: loading a calendar and subsequently updating an event. Recurrence is not supported for simplicity's sake. Although the client specifies a date range query for the search (all events in the current week), the server returns all items in the collection without limiting the results.

EIM-JSON

Data is sent and received using a JSON serialization of EIM record sets. Record sets are packed into a very simple nested data structure that strikes a balance between readability (for development and troubleshooting purposes), compactness and ease of processing. Here is an example (with line breaks inserted for readability):

{"uuid":"326ac16e-8c71-423d-937b-0dd298e92437","records":{
"item":{"ns":"http://osafoundation.org/eim/item/0","key":{"uuid":["text","326ac16e-8c71-423d-937b-0dd298e92437"]},"fields":{"title":["text","Welcome to Cosmo"],"triage":["text","100 -1175285108.00 1"],"hasBeenSent":["integer","0"],"needsReply":["integer","0"],"createdOn":["decimal","1175285108"]}},
"modby":{"ns":"http://osafoundation.org/eim/modifiedBy/0","key":{"uuid":["text","326ac16e-8c71-423d-937b-0dd298e92437"],"userid":["text","bcm@maz.org"],"timestamp":["decimal","1175285108"],"action":["integer","500"]}},
"note":{"ns":"http://osafoundation.org/eim/note/0","key":{"uuid":["text","326ac16e-8c71-423d-937b-0dd298e92437"]},"fields":{"body":["clob","Welcome to Cosmo"],"icalUid":["text","250a7f06-9f66-4dd8-98e7-140d7498cbd9"]}},
"event":{"ns":"http://osafoundation.org/eim/event/0","key":{"uuid":["text","326ac16e-8c71-423d-937b-0dd298e92437"]},"fields":{"dtstart":["text",";VALUE=DATE-TIME:20070328T090000"],"duration":["text","PT1H"],"location":["text",""],"rrule":["text",null],"exrule":["text",null],"rdate":["text",null],"exdate":["text",null],"status":["text",null]}}
}}

This JSON-serialized record set is approximately 60% the size of the equivalent EIMML record set. Many opportunities exist to further optimize for size and processing time.

A more detailed specification for this format can be found here.

Server Details

We use the Apache Abdera library for parsing and generating Atom feed and entry documents. Abdera also provides a framework for APP server implementations, but for expediency's sake we chose not to use it. A production implementation of Atom-based communication would be built on top of the APP framework.

We use the JSON Tools Core library for parsing JSON content and the free Java classes from json.org to write JSON.

Protocol Handling

The AtomServlet class handles the protocol side of things. The method doGet does the following:

  1. Locates the collection with the uid specified in the request URI
  2. Asks the AtomGeneratorFactory to return a FeedGenerator for the projection* and format** specified in the request URI
  3. Directs the FeedGenerator to provide a feed describing all of the NoteItem children of the collection
  4. Writes the feed and returns a 200 OK response

* Projection defines the type of content to return in the feed: basic (suitable for HTML-rendering feed readers), full (containing all known data about an item in an EIM record set), free-busy (an iCalendar VFREEBUSY representing the collection's aggregate free-busy info).

** Format defines the representation of entity content: eim-json, eimml, icalendar; only relevant to the full projection.

The method doPut does the following:

  1. Locates the item with the uid specified in the request URI
  2. If the request content represents an Atom entry, treats the request as a media link entry update, extracting the entry content and media type; otherwise, treats the request as a media update, using the request content and media type directly
  3. Asks the AtomProcessorFactory to return a ContentProcessor for the specified media type
  4. Directs the ContentProcessor to process the provided content, applying it to the item in question
  5. Saves the updated state of the item
  6. Returns a 204 No Content response

Feed Generation

The mechanics of generating an Atom feed document are handled by BaseFeedGenerator. This includes setting the meta-data for the Feed (id, title, updated date, generator, author, links to other representations of the collection) and each Entry (id, title, updated and published dates, edit link). It uses the Abdera Factory class and model interfaces.

Subclasses of BaseFeedGenerator are responsible for setting the content of each Entry based on the corresponding NoteItem. Because full projection feeds can return entry content in various representations, FullFeedGenerator delegates content formatting to AtomContentFactory, which returns a content String in the format specified in the request URI (or eim-json if none was specified).

Content Formatting

AtomContentFactory formats eim-json and eimml content by using ItemTranslator to translate the NoteItem to a EimRecordSet and then serializing the record set with JsonStreamWriter or EimmlStreamWriter.

Content Processing

JsonProcessor takes a Reader containing the provided eim-json content, deserializes it with JsonStreamReader, and then uses ItemTranslator to apply the record set to the NoteItem. EimmlProcessor does the same thing with EimmlStreamReader for eimml content.

What's Not Implemented

Create and Delete Entries

The server should allow entries to be created in feeds with POST and allow entries to be deleted from feeds with =DELETE.

Date Range Queries

The server should respect the start-min and start-max query parameters and return only those event-stamped items which begin within that specified period of time, including items representing occurrences of recurring events.

Paging Feeds

The server should implement the APP paging extensions so that the client can specify that a feed should contain only a subset of the entries that would normally be returned in a feed.

Sorting Feeds

The server should implement query parameters that control the ordering of the entries returned in a feed. In absence of these parameters, the server should return entries such that the most recently updated entry is the first in the feed.

Dashboard Projection or Query

The Chandler dashboard defines complicated rules for sorting and the inclusion of specific occurrences of recurring items. The server should provide a dashboard projection (or perhaps query) that implements these rules to determine which specific entries are included in the feed and in what order.

Free-busy Projection

The server should provide a free-busy projection that returns an iCalendar VFREEBUSY component describing the aggregate free-busy info for the collection (taking into account date-range and other query parameters).

Recurrence

In terms of recurrence, there are three different types of items:

  • Regular items, which represent single-occurrence events
  • Modification items, which represent individual occurrences of recurring events that are exceptions to the event's recurrence rule
  • "Ghost" items, which represent individual occurrences of recurring events defined by the event's recurrence rule

For full and dashboard projection feeds, the server needs to expand recurrence rules to return record sets for all of these types of items.

Conflict Detection

The server should return etags for each feed and entry it sends the client, and it should honor the If-Match and If-None-Match headers so that clients and the server can decide if operations whether or not a resource is cacheable or in the correct state for an update to take place.

Item Relationship Links

Feeds and entries should include links to related items. Examples of related items include:

  • Parent collections of a collection
  • Parent collections of an item
  • Items modified by an item
  • Items that modify an item

Feed Validation

All feeds generated by the server should be validated using industry standard validation tools.

Other

  • On update, the modby record does not seem to be applied correctly; the timestamp and userid are not being saved to the db
  • Events are not indexed on update

Web UI Details

Prototype Implementation

The prototype we have developed has been integrated into the existing Web UI code. Because work on the proposed Web UI infrastructure changes is not mature our prototype code simply serves as a translation layer between the existing model and service code and the experimental Atom server backend.

We have successfully tested our prototype on all supported browsers.

Architecture

Transport Layer

We have implemented two service methods, getEvents and saveEvent. Both are tailored to their primary use in the Web UI code, which are initial calendar load and updates of existing events respectively. As a result, getEvents is designed to be called exclusively synchronously and saveEvent is designed to be called exclusively asynchronously.

Once the new infrastructure work is in place these and all other service methods will be implemented as Conduit methods as described in the Web UI service layer proposal. As described in the linked document, these methods will return dojo.Deferred objects and take a final kwArgs argument for handling optional arguments.

The transport layer suffers from many of the browser quirks we dealt with in our implementation of the current CMP JavaScript module, the largest of which is improper handling of responses with a status code of 204. Fortunately, the code we used to deal with this could be used verbatim. In our production implementation this will likely be abstracted into a "REST Protocol Abstract Class" or something similar.

Translation Layer

Our translation from Atom to our JavaScript object model takes place in two steps. In the first, handled by the browser, the Atom document is parsed into a DOM tree structure. Once our code receives this structure it uses standard DOM methods to get a list of <entry> elements. For each of these elements it then finds the <content> element and gets the value of its internal TextNode?. This value is the serialized EIM-JSON record set, which our code evals and translates to our JavaScript object model.

Unfortunately, the first step (parsing into a DOM tree) does not happen automatically in Internet Explorer 6/7 when server responses have a content type of application/atom+xml. We have currently implemented a workaround for this by creating the appropriate ActiveX object manually and feeding it the Atom string, which should be no slower than if it had done this automatically.

Once the new infrastructure work lands all of these steps will be encapsulated in a Translator object as described in the Web UI service layer proposal.

Model Layer

As mentioned above, our prototype converts EIM-JSON record sets to and from traditional Web UI model objects. This translation is somewhat awkward because of significant differences in models. Happily, our new model, with its support for stamping, maps more cleanly to the EIM-JSON record set model.

Currently, the only attributes supported for updates are time and basic text fields like title and body. Support for recurrence and complex time attributes like all day or any time have either not been implemented, or do not work.

One new issue arising from our prototype work is integration with Atom's <link> elements. As our Web UI infrastructure work has been proposed, transport details are not associated with Model objects, and are instead calculated by the transport layer at the time a service call is made. In contrast, Atom <entry> elements contain <link> elements that contain URLs that can be used to update items they represent. This implies that this information should remain associated with the JavaScript model objects throughout their lifetime, to be extracted by the transport layer during update operations. Further design needs to be done by Bobby Rullo and Travis Vachon to determine how this will be handled.

Feature support

As a result of the extensive design work that has gone into EIM, adoption of EIM-JSON over Atom will open up a number of possibilities that are either not possible or very difficult with our current JSON-RPC data model. These include:
  • Support for complicated sorting and paging requests
  • Support for smart triage capabilities like those found in Chandler
  • Support for new Stamp types by defining new Record Sets
  • Support for items in multiple collections and smart event updates using metadata from =<link>=s

Conclusion

From the server perspective, providing Atom access to stored items is easy. Atom and APP are simple protocols to implement and require a minimal design, especially considering that Abdera's APP framework does much of the legwork in implementing HTTP-level details. We are able to capitalize on existing APIs for translating between the domain model and EIM, which makes Atom data access consistent with Morse Code access. The serialization of EIM to JSON is trivial.

The most difficult part of migrating from JSON-RPC to Atom will be consolidating the different implementations of date-range and free-busy queries and recurrence expansion from the rpc and caldav packages into the service layer - and that is a task which which must be done as part of a larger storage task in the Preview time frame regardless of whether we switch to Atom or continue with JSON-RPC.

Of the three client side layers that will be affected by the move to Atom, Transport will likely be the easiest. The quirks of server communication using XHR are abstracted in the dojo.io.bind module, and additional problems have been largely flushed out in the implementation of our CMP module. Only slightly more difficult from the standpoint of this work should be the Model code. This will be largely unchanged from our model proposal, except for the issue mentioned in the Model Layer section above.

Most difficult on the client side is likely to be the Translation Layer. This will need to conform to the intricacies of the EIM specification, and is likely to be buggy. Extensive unit and functional testing will reduce much of the risk associated with this implementation.

Fortunately, much of this work is likely to be no more time intensive, if not less so, than the client side work that has already been proposed. The additional features we are likely to either gain or make low hanging fruit definitely seem worth this investment.

Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r14 < r13 < r12 < r11 < r10 | More topic actions
 
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.