Dave Smith | 1 Jun 02:55 2004

Re: Twisted has a home on Chandler Wiki


On May 31, 2004, at 3:38 PM, Andi Vajda wrote:

> With that in mind, my opinion, for what it's worth, is that we should 
> have the
> following threading model:
>
>   - a UI thread
>   - a twisted thread
>   - a task manager thread pool to which tasks are assigned and 
> scheduled
>

As a refinement on that suggestion, I would note that Twisted provides 
built-in thread pooling:

http://www.twistedmatrix.com/documents/current/howto/threading

This would allow you to use Twisted's scheduling system:

http://www.twistedmatrix.com/documents/current/howto/time

and avoid unnecessary duplication/re-creation of an entire scheduling 
system and thread pooling.

D.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev" mailing list
(Continue reading)

John Anderson | 1 Jun 03:03 2004

Re: Twisted has a home on Chandler Wiki


Andi Vajda wrote:

>Summing up my thoughts and understanding about the threading model for
>Chandler, quite possibly flawed at the moment...
>
>  - the UI is in one thread, the main thread. We don't want it to block or
>    appear frozen.
>
>  - the Twisted model strongly advocates against threaded networking and
>    proposes an event loop or upcall programming pattern that serializes all
>    the networking operations on one thread. (This reminds me about network
>    programming in Windows 3.11, before Microsoft had networking in Windows).
>    There are plusses and minusses to this approach, I'm not advocating
>    for or against it.
>  
>
I was thinking Macintosh OS 9, which had everything going through the 
main event loop. Unfortunately, this lead to very clunky multitasking 
(if you can remember that long ago) and didn't work very well. Things 
got much better when OS 10 came around and used threads.

The rational for not using threads in twisted was described int the 
Intro link as:

"The event-loop networking model was chosen over threads as it tends to 
be more scalable, integrates well with GUI applications, and is 
supported by virtually all operating systems."

I thought threads would actually scale better as we move towards 
(Continue reading)

darryl | 1 Jun 05:26 2004
Picon

Re: PyLucene Question

Andi Vajda wrote:

>>Well, so far all i've gotten to work was actually editing
>>IndexReader.java and renaming the delete to deleteTerm and then
>>building everything from there. No unresolved symbols.
>>    
>>
>
>You're right. If we're renaming them on the swig/python side we may as well
>rename them in java too and solve the problem. According to the javadocs:
>
>  - IndexReader.delete(int) is deleting the document for given a number, so I
>    renamed it IndexReader.deleteDocument(int)
>
>  - IndexReader.delete(Term) is deleting all documents containing a given
>    term, so I renamed it IndexReader.deleteDocuments(Term)
>
>I added a new patch for this to PyLucene's patches file, fixed the PyLucene.i
>file and checked it all in again.
>
>Thanks !
>
>Andi..
>  
>

If I happen to add more of the missing classes and functions to PyLucene.i
would you like me to submit patches (even if they are trivial) ?

Or asking another way, are you actively hacking in there so I'd be 
(Continue reading)

Dave Smith | 1 Jun 03:38 2004

Re: Twisted has a home on Chandler Wiki


On May 31, 2004, at 7:03 PM, John Anderson wrote:

> I was thinking Macintosh OS 9, which had everything going through the 
> main event loop. Unfortunately, this lead to very clunky multitasking 
> (if you can remember that long ago) and didn't work very well. Things 
> got much better when OS 10 came around and used threads.

I think that the choice between the threads and async model comes down 
to knowing what problem needs to be solved. Clearly both models have 
their shortcomings and are not appropriate in all circumstances. That 
said, it's my personal opinion that the async model is most appropriate 
for systems that need to deal with multiple network I/O sources. While 
the async model does introduce its own set of programming challenges, 
my experience has been that (with the right toolkit) it's easier for 
the average developer to use/build robustly on an asynchronous system.

> I thought threads would actually scale better as we move towards 
> multprocessor machines, but maybe I'm missing something here. 
> Integrating with GUI apps may be a little different in Chandler since 
> most, if not all, the data in Chandler just shows up in the repository 
> as items and the GUI thread gets notified that new data is available 
> to display. I presume the OS support issue is that not all OS's have 
> good thread implementations. This is no longer a problem for Win32, 
> Mac OSX, or Linux -- our target machines. However, this is really a 
> moot point for Python, however, since it runs all it's threads in a 
> single OS thread..

I've had the opportunity to work on some massively scalable network 
servers and the best model that has emerged is a predominantly async 
(Continue reading)

Anthony Baxter | 1 Jun 03:56 2004
Picon

Re: Twisted has a home on Chandler Wiki

Andi Vajda wrote
> With that in mind, my opinion, for what it's worth, is that we should have the
> following threading model:
> 
>   - a UI thread
>   - a twisted thread
>   - a task manager thread pool to which tasks are assigned and scheduled
> 
> All Twisted-based networking should be done via the twisted thread, which
> involves a fair amount of inter-thread communication, but that's what the
> Twisted model is all about isn't it ?

Note that if you're using wx, you really do need a separate thread
for the wx and twisted event loops. If you try to make the twisted
event loop drive the wx event loop you'll find that menus and popup
dialogs spawn a sub-event-loop, and twisted will block while that's
running. If you try to make the wx event loop drive twisted (using
a wxTimer) you run into the wxTimer's essential suckiness - it
guarantees only 1-100 iterations per second (and on windows you're
lucky to get 100.

In twisted there's a 'callFromThread()' method that allows other
threads to inject stuff into the twisted event loop.

I know about this because we hit this problem with shtoom's wx UI.
Check the shtoom svn trunk for example code.

--

-- 
Anthony Baxter     <anthony@...>
It's never too late to have a happy childhood.
(Continue reading)

Heikki Toivonen | 1 Jun 06:01 2004

Milestone 0.3.17 tomorrow (Tuesday)

According to the calendar, tomorrow we should be marking the end of 
milestone 0.3.17. I'd like to close the tree at 3 pm (PST) tomorrow to 
create the milestone builds - expect 2 hour outage. Please try to leave 
the tree green for that.

If you are one of the following people, please send me a highlighted 
summary of the changes in this past milestone so that I can incorporate 
it in the HISTORY file.

Hook:

heikki 	 13 changes
jed 	8 changes
jeffrey 	2 changes
john 	94 changes
markie 	30 changes
morgen 	4 changes
vajda 	51 changes

This Bonsai-produced table shows all the checkins in the current 
milestone, sorted by person & date, which might prove helpful (this link 
always points to the latest milestone checkins):

http://bonsai.osafoundation.org/showcheckins.cgi?&sort=person,date

--

-- 
   Heikki Toivonen

(Continue reading)

Andi Vajda | 1 Jun 07:08 2004

Re: PyLucene Question


Patches are welcome !

Andi..

On Mon, 31 May 2004, darryl wrote:

> Andi Vajda wrote:
>
> >>Well, so far all i've gotten to work was actually editing
> >>IndexReader.java and renaming the delete to deleteTerm and then
> >>building everything from there. No unresolved symbols.
> >>
> >>
> >
> >You're right. If we're renaming them on the swig/python side we may as well
> >rename them in java too and solve the problem. According to the javadocs:
> >
> >  - IndexReader.delete(int) is deleting the document for given a number, so I
> >    renamed it IndexReader.deleteDocument(int)
> >
> >  - IndexReader.delete(Term) is deleting all documents containing a given
> >    term, so I renamed it IndexReader.deleteDocuments(Term)
> >
> >I added a new patch for this to PyLucene's patches file, fixed the PyLucene.i
> >file and checked it all in again.
> >
> >Thanks !
> >
> >Andi..
(Continue reading)

Brian Kirsch | 1 Jun 19:08 2004

Re: Twisted has a home on Chandler Wiki

Thanks Anthony,
We are going to run Twisted in a thread and use the callFromThread API 
as you suggested:

Here is my basic thread code:

class ReactorException(Exception):
       def __init__(self, *args):
             Exception.__init__(self, *args)

class ReactorThread(threading.Thread):
       """Run the Reactor in a Thread to prevent blocking the
          Main Thread once reactor.run is called"""

       def __init__(self):
               threading.Thread.__init__(self)
               self._reactorRunning = False

       def run(self):
               if self._reactorRunning:
                     raise ReactorException("Reactor Already Running")

               self._reactorRunning = True

               #call run passing a False flag indicating to the
               #reactor not to install sig handlers since sig handlers
               #only work on the main thread

               reactor.run(False)	     	

(Continue reading)

Brian Kirsch | 1 Jun 19:18 2004

Re: Twisted has a home on Chandler Wiki

John,
Although Twisted does advocate using an asynchronous model it has very 
good Thread support and Thread pooling
built in to the Reactor core. If we make the decision to leverage 
Twisted as a core element of Chandler it does not
mean we have to give up threads. I created some sample programs that 
leverage the Twisted thread model and they
have performed quite well.

Using Threads:
http://twistedmatrix.com/documents/current/howto/threading

Sample Programs:
http://aloha.osafoundation.org/~bkirsch/src/IMAPTwistedThreadTests.zip

Brian Kirsch - Email Framework Engineer
Open Source Applications Foundation
543 Howard St. 5th Floor 
San Francisco, CA 94105 
(415) 946-3056 

On May 31, 2004, at 6:03 PM, John Anderson wrote:

>
>
> Andi Vajda wrote:
>
>> Summing up my thoughts and understanding about the threading model for
>> Chandler, quite possibly flawed at the moment...
>>
(Continue reading)

Kaitlin Duck Sherwood | 1 Jun 20:03 2004

OSAF Office Hours Wednesday 2 June, 11 AM PST, topic: Q&A with Mitch Kapor

This is a reminder that the next OSAF Office Hour will be tomorrow, 
Wednesday, 2 June at 11 AM Pacific time (18:00-19:00 GMT/UTC/Zulu).

Mitch Kapor, the founder and head of OSAF, is about to start appearing 
regularly in the "Design" slot of the roughly six-week Office Hours 
cycle.  For the first Mitch-centric Office Hour tomorrow, Mitch will 
answer general questions.

As always, our Office Hours are at irc.osafoundation.org, channel #chandler.

Please join us!

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev


Gmane