r56 - 07 Feb 2006 - 16:30:55 - LisaDusseaultYou are here: OSAF >  Projects Web  >  DevelopmentHome > ServicesWorkingGroup > EmailService

Email Service Project

The Email service project is part of the Services Working Group. This project consists of three evolutionary stages.

  1. Identify and incorporate open source email component solutions in to the Chandler release. Components include an IMAP library, a POP3 library, a SMTP library, a RFC-2882 Message Parser, a Spell Checking library, one or more junk mail filter libraries, and potentially a mail filtering library.
  2. Implement a feature rich mail client within Chandler leveraging open source technology to provide best-of-breed features from current mail clients including support for IMAP, POP3, TLS / SSL, filters, spam control, spell checking, virus protection, and multiple accounts.
  3. Expand upon the current notion of a mail client by leveraging the power of Chandler for search, replication, and sharing. Improve upon current mail client solutions by implementing lacking features such as labels, collapsable threads, etc. One of the main goals for Chandler 1.0 "Canoga" is to allow users to manage large volumes of email more efficiently and effectively than with current email clients. Here is the list of compelling email features for the 1.0 "Canoga" release.


  • BrianKirsch

Scheduled Feature work

The main new feature for the Mail Service in .5 will be Attachment handling in sending and receiving mail. There will also be addition work done to ensure proper I18N conversion of incoming messages from Strings to Unicode. Anyone who has worked on Internationalization and Localization knows how difficult this problem is to handle correctly. This will be an iterative solution with the base layer going in for .5.

Task Status Milestone
Help define proposal for handling IMAP folders after 0.4 In progress n/a
Parse and store email bodies (common multiple MIME parts) cleanly soon 0.5


Unscheduled features/tasks

  • Work with the Python email sig to get a robust Email Address Validator in place
  • Manage multiple IMAP accounts
  • Any support whatsoever for IMAP folders other than the Inbox
  • Add POP3 support, POP3 account information
  • Handling all kinds of wierd email formats will require many iterations over time -- mostly we will deal with these as we encounter them in testing.


_New Email Service details coming soon_

Wow, it has been to long since this page has been updated. I have been off working on Interationalizing Chandler and have not been focusing on mail in the .6 release. To give you a brief overview of details to come POP is now implemented using twisted pop3client with some additions submitted to twisted by myself. IMAP and POP now download messages in batches of 50 to reduce memory foot print and increase UI feedback. In addition, all protocols SMTP, POP, IMAP feature a "Test Account Settings" option which will contact the server and verify the host name, port, SSL option, username, and password.

Thanks for your patience more details to come soon.


0.4 release overview

The main issues that were resolved for the 0.4 release where which libraries Chandler will leverage to perform IMAP, RFC 2822 Message Parsing, and SMTP operations. The IMAP and SMTP libraries must support TLS / SSL or be easily wrapped to do so. We finalized on Twisted for SMTP and IMAP and decided to leverage the new Python Email Message Package for our RFC 2882 Message Parsing. For the .5 release The Mail Service will under go a significant refactoring to clean up dependencies between the CPIA, Services, and Repository layers which were identified in the .4 release.

0.4 tasks

Feature Owner Next Actions SWAG Priority Milestone
Choose protocol library(ies) Brian K Done med now 0.3.15 May 4th
Twisted thread proposal finalized -- see TwistedHome Brian K Done med now 0.3.17
Integrate Twisted into Chandler build Brian K Done low now 0.3.17
Support IMAP for downloading email to repos (inbox only) Brian K Done med now 0.3.19
Code review Twisted integration with John Brian K Done small now 0.3.19
Design and document email content model -- see ContentModelProject, ContentModelToDo20040406 Brian K Done med now 0.3.23
Choose SMTP Library Brian K Done med soon 0.3.21
Integrate SMTP library into build etc. BK Done med soon 0.3.22
On user send email, verify addresses & compose email and send BK Done med soon 0.3.23
A way for incoming email to trigger other code? (e.g. incoming sharing emails) BK Done med soon 0.3.24
Design and implement email API designed for 3rd party devs, particularly send email BK Done med soon 0.3.23

0.4 news items

I am pleased to announce that after a much too long period of reading through source code, chatting via email and IRC, and writing test programs, we have determined the two most crucial libraries for our initial release: RFC 2882 message parsing and IMAP.

During the early part of May Barry Warsaw, with much help from Anthony Baxter, completed the new RFC 2882 message Feedparser that will be part of the standard Python 2.4 distribution. The parser is a great improvement over previous versions with two particular areas standing out. First, it can now process messages line by line instead of as a giant text block. Needless to say this is a tremendous improvement and also lends itself very well to event based programming (more on that in a bit.) Second, with much thanks to Anthony Baxter for his MIME torture test, it now handles parsing of many real world (can anyone say Spam) messages with unusual or plain old broken message bodies. So for Chandler's mail parsing functionality I am pleased to announce we are going with the Python 2.4 email library. If you want more info on the Feedparser, just click here.

Now on to the IMAP library. For sometime now we have known we wanted to use either the Python standard imaplib or Twisted's IMAP4Client. The advantages of imaplib are that it is easy to use, well tested in the real world over the years, part of the standard Python distribution and easy to integrate in to Chandler's threaded model. The main disadvantages are that it is extremely low-level, and as such requires a deep understanding of the IMAP protocol, it is no longer being actively developed and is missing many important features such as encrypted login schemes and pipelining. Twisted on the other hand, has more robust IMAP handling in its IMAPClient library. The main disadvantage to Twisted is it is an asynchronous framework with not as much documentation as one would like and a rather substantial learning curve. The two most important words being asynchronous and framework. First off, Chandler is itself a framework. The Twisted model is "you don't run Twisted, Twisted runs you!" meaning your code is intended to live inside the Twisted framework and not the other way around. Secondly, Chandler is a threaded application and Twisted uses a blocking asynchronous event core called a Reactor. However, despite these two drawbacks, the advantages to choosing Twisted became more and more clear as my research progressed. I wrote several programs to test Twisted's IMAP functionality and more importantly find out how an asynchronous framework could be run inside a threaded application. I am pleased to announce that the results of those tests were very successful. So as you probably already guessed, for Chandler's IMAP library we are going with Twisted. Of course I forgot to mention the best part about using Twisted is that as a framework it has many other elements which we can utilize in Chandler: SOAP, XMPP, SSL, POP3?.

As Twisted's scope reaches far beyond the range of mail, I have taken the liberty of creating an official Twisted Wiki Home Page. Here you will find info on Twisted, links to the example programs I used to test Twisted with threading and some additional documentation on using the Twisted Reactor and Deferred model which is lacking currently in the core Twisted howto's.

For our SMTP Library Twisted was also chosen. The current SMTP Libraries in Twisted 1.3 are very rudimentary and lack may of the features required for ESMTP communication. In addition the authentication scheme for ESMTP in Twisted 1.3 was not correct and there were a number of additional bugs related to TLS / SSL. The code also lacked a high level ESMTP API. As the changes needed to improve the Twisted SMTP code were fairly straight forward, I took some time made the appropriate additions / fixes and with the help of JP Calderone (he coder / maintainer of all things mail in Twisted) we checked in the new code in to the Twisted main trunk on Friday July 23, 2004. So for anyone else requiring a robust SMTP / ESMTP library download the latest Twisted sources via SVN smile

Even Older (and not so relevant) News

Unfortunately, there is no obvious solution. There are number of Python related mail projects currently being developed include Twisted, Quotient, and the new mail libraries for the Python standard distribution. Quotient is built on Twisted and there appears to be much interplay between the Twisted and Quotient folks. Code being developed for Quotient is getting folded back in to Twisted. Making the decision harder is that all three Python solutions Twisted, Quotient, Standard Library are all under going significant development and change.

In six months to a year the choice of mail libraries in Python will be much easier than it is now as development is currently in flux. I am currently testing the IMAP features of Twisted 1.2.0 and Twisted 1.2.1 Alpha (Install notes for Mac OS X). Message parsing remains an open issue for 0.4. I am evaluating the PyCon additions to the Python branch, Spam Bayes and Quotient. More info to come as all this shakes out.

15 October 2004

Two significant patches have been contributed back to the Twisted communitiy which greatly improve the robustness of the Twisted Mail Packages. The first patch among other things adds Timeout logic checking to smtp.py to prevent the client from hanging if the server fails to return a response. The second is a patch to Twisted's imap4.py library which like the smtp patch adds real world Timeout features to prevent the client from hanging. I also created two test servers (SMTP and IMAP) that force race conditions such as timeout and invalid responses to test the strength of the Chandler client in handling real world errors. The servers are written utilizing the Twisted Framework and are located in the parcel.osaf.mail.test.test_servers package.

14 October 2004

Keep you eye out for a new page that will be appearing in the next few days detailing the new and improved Mail Content Model as well as discussing some of the decisions that were made post .4 release in terms of what functionality belongs in the three Chandler layers CPIA, Services, Content Model / Repository. In .4 the divisions between layers were not as clear as one would hope and this is going to be rectified in the .5 release.

7 October 2004

I am back from my honeymoon with some exciting new features for mail in Chandler. The .4b release is almost ready for primetime and features SMTP support, sharing via WEBDAV / SMTP / IMAP, and a much improved Email Content Model. In addition the Twisted SMTP client API has been debugged and upgraded substantially and is in the process of being submitted back to the Twisted community. See the ZeroPointFourEmailSummary for more info regarding the .4B release mail functionality.

21 June 2004

The lastest version of Chandler now incorporates the Twisted Framework in the build. Twisted is a central service in Chandler that runs in it own thread. An initial mail parcel has been checked in that leverages Twisted to retrieve and store mail as Chandler items from multiple IMAP accounts. Incorporating Twisted and IMAP support are major additions in making Chandler a high caliber email solution.

25 May 2004

An update to the new Python RFC 2882 Feedparser has been recommended which would allow large attachment data to be stored on the file system instead of in memory. Menno Smits has taken a first pass at implementing this feature in two classes. DiskMessage and DiskFeedParser. These classes still need to be tested and a clean API suggested for folding the changes in to Python 2.4. For more info click here.

17 May 2004

Twisted 1.3.0 has been released by the Twisted Matrix folks. To download the new version click here (Release Notes).

10 May 2004

Barry Warsaw with major contributions from Anthony Baxter has checked in a new Mail Parser to the Python Core. The Parser handles complex / broken message parsing and can transform MIME messages to Mail Objects on a line by line basis. Here is more info on the checkin.

Chris "Radix" Armstrong the build / release engineer for Twisted provided a heads up to the upcoming release schedule. Here is more info on our chat.

6 May 2004

Brian Warner of Twisted / Petmail fame is coming by today to discuss Twisted Reactor Core principals. The Reactor is the Twisted asynchronous event loop mechanism. Its design and use are some what unique and can be difficult for new users to comprehend and implement correctly. Brian is going to be discussing advanced Reactor logic as well as how to integrate an event based mechanism in to a threaded (Chandler) application.

29 April 2004

After playing Austrialian phone tag for almost a week, Piers Lauder, the creator of Python's imaplib and I chated about imap support in the Python standard distribution. Here are the notes from our conversation.

26 April 2004

Jp "exarkun" Calderone, the creator of Twisted's IMAP support, and I spoke for almost an hour on Friday regarding current and future Twisted development plans. Here are the notes from our conversation.

21 April 2004

Brian Warner of Twisted / Petmail fame graciously stopped by to discuss issues relating to mail and spam. Here are the notes from the visit.

19 April 2004

Lisa Dusseault, Ducky Sherwood, and myself had a very informative conference call today with Python developer Barry Warsaw. Here are the notes from the call.

16 April 2004

A new email Parser was checked in to the Python CVS tree about three weeks ago. This parse solves performance issues as well as handles poorly formed rfc 2882 message better.

On Monday 19 April 2004, OSAF will have a conference call with Barry Warsaw creator of much of Python's current mail support to discuss the roadmap for mail in future Python releases.

On Tuesday 20 April 2004, Brian Warner of Twisted / Petmail fame will by stopping by OSAF to discuss Twisted and mail and to join the OSAF staff for an informal lunch time chat.

14 April 2004

I spoke with the Twisted folks via IRC and was told that Twisted Mail will at some point in the near future be distributed as a single package. Twisted is currently a vary large framework consisting of many components which are not needed for client mail development.

-- BrianKirsch - 14 Apr 2004

PageType ProjectPage?
MaintainedBy BrianKirsch
PageStatus Source of truth -- this page reflects current OSAF thinking?
Trash.CommentsWelcome2 Feel free to contribute comments?, either by adding to the Comments Welcome section of this page, or by posting to the dev list, or by sending mail directly to the person listed as maintaining the page.
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r56 < r55 < r54 < r53 < r52 | More topic actions
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.