Overview

---

Twisted is an asynchronous networking framework written in Python. It has a pluggable main-loop which can run anywhere from poll() on Linux in CPython, to polling emulated with threads on Jython. Twisted is hosted by the twisted matrix website. The current version is 1.3.0 (download).

Why are we using Twisted?

---

Twisted will be a core part of the Chandler architecture. There were several factors looked at in making this decision.

• First, Twisted's networking framework allows the implementation of Servers and Clients in very little time and with very few lines of code. Twisted has support for all the major protocols (FTP, HTTP, SSH, IRC, XMPP) with support for additional protocols added frequently.
  

• Second, Twisted has an excellent IMAP library as part of its mail package. The library supports many features that the standard Python imaplib does not including MD5 Login and pipelining. Twisted has POP3 and SMTP support as well. All protocols including IMAP, POP3, and SMTP can be run with SSL enabled.

• Third, Twisted has a very good Asynchronous model. Exposing, the Twisted "Reactor" event loop in Chandler can be an advantage by limiting the number of threads spawned and the associated locking issues. Twisted also has an excellent Thread model that supports Thread pooling as well as Timer and Scheduling services.

• And finally, the Twisted community has many bright developers and many pre-existing applications that can be made in to Chandler parcels. Incorporating Twisted in to Chandler can help bring community attention and support to the project. Anthony Baxter has already expressed interest in integrating his VOIP Twisted application Shtoom as a Chandler parcel.

Getting Started with Twisted

---

• Getting Twisted
• Twisted Introduction
• What is Twisted?
• Why should I use Twisted?
• Visual Representation of the Twisted Framework

Twisted's Future

---

The next major release of Twisted will focus on dividing Twisted's protocol services (Mail, Web, IRC) in to distinct packages which can be added or removed from Twisted core.

• Twisted Splt ProductFAQ^?

Twisted Basics

---

Twisted is a very large framework containing many components (Web Server, Perspective Broker, etc.). Here are the core Twisted components that Chandler developers should be familiar with:

• Asynchronous Programming
• Reactor Basics
• Using Deferreds
• Writing a Client
• Writing a Server
• Using Threads
• Using Services
• Scheduling Tasks

Extra Credit

• Interfaces and Adapters
• XML-RPC and SOAP
• Using Generators in Twisted

Tutorials

---

• The Evolution of Finger
• Network Programming with Twisted Part 1
• Network Programming with Twisted Part 2
• Network Programming with Twisted Part 3
• Network Programming with Twisted Part 4

Articles

---

• Onlamp Twisted Article
• Twisted Introduction

References

---

• Twisted API Reference
• Twisted Documentation
• Example Twisted Code
• Protocols implemented in Twisted
• Twisted OSX Installation Note

OSAF Twisted Source Code

---

• Console based utility for creating and sending an SMTP Message
• Console based utility for connecting to an IMAP Server and issuing command
• Threaded Reactor / IMAP examples with advanced Deferred logic (Download as zip file)
• Running a Reactor in a thread

Chandler with Twisted Integration

---

* Integrating Twisted in to the current Chandler architecture

Thoughts on why Chandler should adopt an Asynchronous model

---

From:   dizzyd@jabber.org
Subject: Some thoughts on Threading
Date: May 25, 2004 2:53:54 PM PDT
To:   bkirsch@osafoundation.org
Cc:   lisa@osafoundation.org
Greetings,
I've been poking around Chandler a little bit more, reading docs, and just generally
trying to understand what makes Chandler tick.  One thing that stands out to me 
is the lack of a clear threading strategy. I'd like to throw out a strawman idea.
First off, I would submit that it is unnecessary to have more than two threads
running in Chandler at any given time. Python is well known for having a 
GIL (Global Interpreter Lock) that prohibits two threads from executing
Python code simultaneously. Unless the majority of code in Chandler 
is not in Python, you'll time waiting on the GIL when you could be executing code. 
With the appropriate async event loops and primitives, 
threading is overkill for the majority of desktop applications.  
This statement is reinforced by the fact that
most (if not all) of the major graphical toolkits currently in use today have 
at their heart an event loop. Additionally, threading a desktop application
that is meant to be a platform for people to build "parcels" on
means that you are increasing the pre-requisite development 
knowledge necessary to build a stable, comprehensible "parcel". All
of that said, give the technology choices that Chandler has made, I think it
is unavoidable that it should use less than two threads. However,
at least with two threads you can provide a reasonable API that
will hide (for the most part) the fact that code is running on 
different threads.
There has been some indication that the Chandler community is 
interested in using Twisted, and I have some
familiarity with Twisted, so my strawman incorporates this framework.  
Specifically, I would suggest that Chandler should have one thread for the GUI
event loop and one thread for the Twisted event loop. All
graphical operations, callbacks, etc. would happen on the GUI thread,
while agent ops, network ops, etc. would live on the Twisted thread. 
Twisted provides a nice facility for doing timed callbacks and long-lived 
(efficient) network operations. Considering that Chandler is a platform 
for integrating various forms of personal 
information management, and that more and more personal information 
are exchanged via the 'Net, it make sense
to have at the core of your platform a system that allows you to manage
tens of sockets simultaneously without introducing gross locking complexities
(that _will_ emerge if every socket is on its own thread).
The one argument I've heard this far against using an async model (such as Twisted)
versus threads is that threads are a universal concept, whereas Twisted is 
very python specific (esp. once you get into deferreds, etc.) I would note 
that while threads are indeed a universal concept, the semantics and
implementation of them vary wildly from platform 
to platform (language to language, etc.) Just because people understand 
the concept of threads doesn't mean they know 
how to write solid code with them. While Twisted is admittedly 
Python-centric, it's conceivable that an reasonably 
small intermediate API could be developed that would hide most 
of the Python-oddities/specificity.
Admittedly all of this would require a rather dramatic change in the underlying 
architecture of Chandler. However, if you intend to build a serious product, that 
is meant to be the platform for the next generation of personal
information management tools, you're going to have to make solid 
architectural decisions sooner, rather than later. As most everyone knows, 
it's far cheaper to invest development time near the beginning of a project 
rather than at the end of the project.
So, in a nutshell, that's my $0.02. I would be glad to help out in 
the prototyping of the idea, as I have time.

Twisted Issues

---

Is the Reactor itself Thread safe? Can two or more threads added callbacks to the Reactor with out locking?

The Reator is technically not thread safe. All code accessing the Reactor must operate in the Reactor thread. To accomplish this use reactor.callFromThread.

My Deferred or DeferredList never fires, so my program just mysteriously hangs! What's wrong?

It really depends on what your program is doing, but the most common cause is this: it is firing -- but it's an error, not a success, and you have forgotten to add an errback, so nothing happens. Always add errbacks!

The reason this happens is that unhandled errors in Deferreds get printed when the Deferred is garbage collected. Make sure your Deferred is garbage collected by deleting all references to it when you are done with it, e.g. after callback() is called.

How do I use Deferreds to make my blocking code non-blocking?

You don't. Deferreds don't magically turn a blocking function call into a non-blocking one. A Deferred is just a simple object that represents a deferred result, with methods to allow convenient adding of callbacks. (This is a common misunderstanding; suggestions on how to make this clearer in the Deferred Execution howto are welcome!)

If you have blocking code that you want to use non-blockingly in Twisted, either rewrite it to be non-blocking, or run it in a thread. There is a convenience function, deferToThread, to help you with the threaded approach -- but be sure to read Using Threads in Twisted.

#Twisted IRC Chats

---

• Itamar details how to run the Reactor in a thread
• Itamar details how to use Services

-- BrianKirsch - 22 May 2004