r76 - 23 Apr 2007 - 08:02:26 - MorgenSagenYou are here: OSAF >  Projects Web  >  DevelopmentHome > ServicesWorkingGroup > SharingProject

Sharing Project

The sharing framework allows repository items to be transported in and out of Chandler in various serialization formats and across various transport mechanisms.


External Information Model

The Chandler Services and Cosmo Server teams are now collaborating on a new intermediate layer -- dubbed the 'External Information Model' (aka EIM) -- between objects and their external serialized representations. Decoupling the two in this manner means either can change format without affecting the other; a parcel developer doesn't need to know about external formats, as long as they write code that can convert items to and from EIM form. Whether we're dumping data to disk or sharing data across the wire, parcels will convert items to this new intermediate form which will then be converted to the appropriate external representation upon export. Having such a model also helps solve the schema evolution problem because it makes it easier for parcel developers to write upgrade code; they simply write code that translates a more-or-less fixed external format into their parcel's current schema. Because the external format is not used for normal application processing, it can be simpler and far more stable than the application-specific schema is likely to be.

EIM Primitive Types

EIM records consist of values of the following primitive types:

  • Bytes [length], where the maximum length must be specified and it must be 1024 or less
  • Text [length], where the maximum length (in bytes of UTF-8 encoding) must be specified and it must be 1024 or less
  • Blob, a Binary Large OBject of arbitrary-length data. Unlike Chandler repository lobs, this type does NOT include encoding or mime-type information; these must be specified as separate fields if needed.
  • Clob, a Character Large OBject of unbounded UTF-8 text.
  • Integer, an unsigned 32-bit integer
  • Datetime, a date and time value, with time zone or without (floating) [Note: EIMML serializes this in ISO8601 format, so while we maintain the right UTC offset, we lose the actual timezone, including daylight savings info]
  • Decimal [digits, decimal_places], a fixed point number where digits specifies the total number of digits, and decimal_places specifies the number of digits to the right of the decimal point

EIM Record Type Definitions

The agreed-upon set of record type definitions that Chandler and Cosmo will exchange is stored in SVN here:

  • Record Type Definitions

The Meaning of Nothing

There are various degrees of Nothingness in the world of EIM. If you want to express the lack of a value in a field, you set the field to None. In EIMML this is serialized to <ns:field eim:type="..." />. However, to express Emptiness, such as an empty string, a field value of "" will be serialized to <ns:field eim:type="..." empty="true"/>. Finally, in the case of recurring event modifications, any field which is meant to be inherited from the master, a field value of Inherit will get serialized to <ns:field eim:type="..." missing="true"/>. On the other hand, if an event modification wants to override something on the master -- say the master has a value for a field and the modification doesn't -- you can set the modification's field to None.

[Originally we named the Python symbol "Missing" but later the Chandler team internally renamed it "Inherit". I'm thinking it might have been better as "Missing" actually, because it can be used internally to let the EIM layer know that an attribute should be deleted from the item.]

Additional EIM Documents


Sharing Protocols

Morsecode

CalDAV

Chandler speaks a limited subset of CalDAV, and is able to subscribe to and publish calendars.

WebDAV

Chandler can use basic WebDAV requests to share a collection of items using DAV collections and resources. Details about each item are stored in XML resources in "CloudXML" format, which is being phased out in favor of EIMML.

HTTP

Simple subscribing to remote monolithic icalendar files is accomplished over HTTP and honoring last-modified headers.

XMPP

Some experimental work has been done to enable sharing over Jabber/XMPP


Sharing Formats

EIMML

XML representation of EIM records - see EimmlSpec for the specification.

CloudXML

Soon to be deprecated XML representation of Chandler items

iCalendar

Standard iCalendar files are parsed/generated by Jeffrey's voject library. Chandler can import/export local .ics files, and can subscribe or publish (one way only) a remote .ics file either over WebDAV (publish) or HTTP GET (subscribe)


APIs

EIM Record API

The API for managing EIM records is documented at EIM.txt.

Edit/Update Sharing API

To support p2p sharing of individual items, the sharing package now has inbound( ) and outbound( ) functions. Each function takes a "peer" parameter which is any Chandler item that represents who you are sharing the item with. For example, this peer might be a Contact or emailAddress item; the inbound/outbound functions don't care what it is as long as it's a Chandler Item with a UUID.

sharing.inbound( ) currently takes an EIMML string (although in the future we will want the Translator and Serializer to be configurable), processes it, and applies any non-conflicting changes to the local item. The item is created if not already existing, and stamped/unstamped as needed. Conflicts are stored in the share item's "conflictingStates" ref collection.

sharing.outbound( ) takes an existing item and returns an EIMML string to transmit. If the item has any pending conflicts, outbound( ) will instead raise a ConflictsPending? exception.

Unlike server-centric sharing, the EIMML generated by outbound( ) always contains the full state of the item. It also contains some additional information, including the sender's repository UUID, plus the sender's version number of the transmitted item. This allows the inbound( ) function to compare the peer repository UUID with the one it previously saw; if it's different, inbound( ) assumes that the peer has re-created its repository, and all the inbound records are considered as changes. If the peer repository UUID is a match, then the inbound records are compared against the previously seen state to determine what's actually changed. Additionally, If the item version embedded in the EIMML is less than or equal to the version number last seen from this peer, the changes are ignored an OutOfSequence? exception is raised.

See the Itemcentric Sharing section of EIMML.txt for example usage.

Conflicts API

A conflict is an inbound change that overlaps with (and has a different value than) a local change not yet sent out. To see if a given item has any conflicts, you can check the contents of the sharing.SharedItem(item).conflictingStates attribute. If that ref collection has any contents, you can call sharing.SharedItem(item).getConflicts( ). That method returns a generator which yields Conflict objects. Alternatively you can call sharing.hasConflicts(item) and sharing.getConflicts(item). The Conflict object attributes that are relevant to a conflict resolution UI are:

  • field -- a string indicating which field was changed
  • value -- a string indicating the inbound value that was not applied because of a local change
  • peer -- an item representing who the conflict is with; in the case of edit/update sharing the peer will be the Contact or EmailAddress? item (or whatever the mail layer chooses to use), and in the case of Cosmo-sharing the peer will be the Share item itself.

There are two methods on Conflict objects:

  • apply( ) -- applies the pending change to the item
  • discard( ) -- doesn't modify the item, but just throws away the inbound pending change

Note: "field" and "value" are not fully baked yet -- "field" currently always has value "Something", and "value" contains a user-unfriendly representation of the conflicting recordset. Real soon now we will produce Conflict objects for each conflicting field with friendlier strings. However, the API won't be changing, and the development of the resolution UI can proceed.

Conflicts can be created for the currently selected items through the "Test->Sharing->Create Conflict" menus item.


Client Design

Architecture

In the new EIM-based sharing framework, the Share object sticks around, but the old Format and Conduit classes give way to a new RecordSetConduit? class plus Translators and Serializers. Translators know how to convert between items and EIM records, while Serializers convert between EIM records and a formats such as XML. The RecordSetConduit? base class implements the logic for synchronizing internal and external changes to a collection by means of merging recordsets. A recordset is a set of records associated with a particular item. A State object tracks the agreed-upon records for an item (per share), as well as any pending conflicts that have come in but not yet applied. The State object contains the logic for merging a single item's records and detecting conflicts.

For "edit/update" sharing, which is item-centric and p2p based, the Share and Conduit objects are not even needed. Each shared item is associated with a sharing.State item (one per peer) which keeps track of the latest agreed-upon state, conflicts, last seen item version, and peer repository id.

Filtering

The following explains how filtering acts in the current implementation:

  • A collection can be read-only or read-write

  • A field can be filtered out or shared

  • From the underlying sharing framework standpoint, the user is allowed to change any field.

  • Filtered fields don't get sent, get applied locally, nor participate in conflicts.

  • Shared fields which have differing local (in your Chandler) and external (on the server) values result in the external value being flagged as a pending conflict. A conflict is resolved when the local and external values are made the same, by:
    1. you applying the pending conflict (thus setting the local value equal to the external one), or...
    2. you discarding the pending conflict (thus setting the external value equal to the local one), or...
    3. someone else changing the external value so that it is equal to your local one

  • The only difference is that for case #2 in a read-only subscribe, since you can't actually modify the external value, if you discard the pending conflict the sharing framework will make the conflict go away and only show another conflict for that field if a differing external change is made later on.

The following is a proposal for Preview with respect to filters during publish and subscribe:

  • Make the default for both publish and subscribe to not have any filters active (in other words, all checkboxes checked; share all fields)
  • If you don't want to share something, you uncheck the appropriate field
  • The UI should make it obvious that the item you are viewing in the detail view is one where changes to it are not shared. Perhaps the "never share" lock icon could be repurposed for this use. [Note: I recently realized I haven't actually implemented the "never share" feature in the new EIM sharing framework]

EIM Sync Algorithm

ChandlerEIMSyncAlgorithm

Collection Subscription Algorithm

ChandlerCollectionSubscriptionAlgorithm


Project Status

0.7 alpha5 goals

  • Done
    • Define an information model for interacting with shared data (External Information Model, EIM) Done
    • Define new sharing format (EIMML?) Done
    • Implement serialize/deserialize functions to convert records to sharing format and back Done
    • Implement new merge code Done
    • Implement edit/update sharing API Done
    • Parsing HTML to find embedded morsecode URL Done
    • Implement new Subscribe algorithm Done
    • Implement conflict resolution API Done
    • Implement client side of new Cosmo sharing protocol ("Morsecode") Done
    • Magic URL support
      • Need to support "external" (what the user passes around) and "internal" (the actual morsecode URL that Chandler uses) urls Done
    • Hook up new filtering stuff to UI, and persist it Done
    • Combine triageStatus and triageStatusChanged Done
    • Treat lastModifiedBy as a change log, adding a new record type Morgen
    • Added "Activity" and "Listener" classes, and changed sharing code/dialogs to use them
    • Added ActivityViewer?
    • When publishing to Cosmo over morsecode, ask that tickets be generated, and store them in conduit
    • CalDAVRecordSetConduit?
    • WebDAVRecordSetConduit?
    • Monolithic recordset conduits for sharing/importing/exporting .ics files
    • FileSystemMonolithicRecordSetConduit?
    • WebDAVMonolithicRecordSetConduit?

  • In progress
    • Define EIM record types Need to define mail message record type and implement translation (bkirsch)
    • Implement import/export functions for each record type to convert items into records and back Need to implement the event-related callbacks (jeffrey)
    • Implement conflict rendering pje
    • Port CalDAV code to work atop EIM framework -- a new ICSSerializer class which implements the same two methods as sharing.eimml.EIMMLSerializer

  • Not started
    • On bgysnc error, what to display?
    • On manually-triggered sync, what to display?
    • Human-readable sharing log
    • Figure out how much dual-fork to morsecode migration to automate
    • Adjustments to "Share" and "Manage share" dialogs to accomodate sharing Triage status (Bug#7411) Morgen

Performance

I've started profiling the reload of a dump file containing the office calendar. Starting a page to document that at SharingPerformance.

Outstanding Chandler/Cosmo Interop Issues

  • Per bcm: on GET responses, X-MorseCode-TicketType is going to be read-only, read-write or free-busy when the GET is via ticket.
  • Need to think about the difference between None and empty string values (email to lists sent) We agreed on how to represent, None, Empty, and Missing
  • Make sure that Cosmo disallows an update when given an out of date sync token, and make sure Chandler handles that correctly
  • Are we going to keep the names of collections in sync when sharing? Especially now that CosmoUI? can change the names.

User Service Document

When the Chandler user first sets up a Cosmo account in the accounts dialog, Chandler needs to fetch the User Service Document (BrianMoseleyCosmoUSD) which will live at a well known place on osaf.us (although a different installation could have it elsewhere). The well known url on osaf.us is /cosmo/cmp/user/username/service (as in /cosmo/cmp/user/morgen/service); Chandler should GET that url using basic auth. It will return an XML document which includes the morsecode base path which Chandler will use to publish collections. For example, "/cosmo/mc". Chandler will need to append "collection/uuid".

Multiple URLs per Collection

In addition to the morsecode (/cosmo/mc) URL, Chandler will need the user-facing (/cosmo/pim) sharing url, and, in order to retrieve tickets the DAV url (/cosmo/dav?).

Additional Documents

toggleopenShow attachmentstogglecloseHide attachments
Topic attachments
I Attachment Action Size Date Who Comment
gifgif SharingFramework.gif manage 6.0 K 31 Mar 2006 - 00:58 MorgenSagen  
pngpng RecordSetConduit.png manage 34.1 K 12 Mar 2007 - 17:53 MorgenSagen A diagram of a RecordSetConduit?
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r76 < r75 < r74 < r73 < r72 | More topic actions
 
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.