headers
Duane Searsmith | 20 Feb 2007 23:11

Cells and Threads

Hi --

Quick question.  I skimmed through the archives and didn't see any 
obvious answer to this question. Is cells thread safe?  If not, is there 
a particular pattern of use that one can follow to avoid problems in a 
multiprocess scenario?

I have an application that is multithreaded and I would like to explore 
how to use cells to drive the dataflow.  I've read through the examples 
and looked over the code a bit. Seems like I could get in trouble if 
multiple threads tried to change a cell with dependencies at the same time.

If this is a silly question and I have missed some obvious explanation 
somewhere, please just point me in the right direction.

Thanks,
-- Duane
Permalink | Reply | 0 comments |
headers
Ken Tilton | 21 Feb 2007 03:18
Picon

[Fwd: Re: Cells and Threads]


Duane Searsmith wrote:

> Hi --
>
> Quick question.  I skimmed through the archives and didn't see any 
> obvious answer to this question. Is cells thread safe?  If not, is 
> there a particular pattern of use that one can follow to avoid 
> problems in a multiprocess scenario?
>
> I have an application that is multithreaded and I would like to 
> explore how to use cells to drive the dataflow.  I've read through the 
> examples and looked over the code a bit. Seems like I could get in 
> trouble if multiple threads tried to change a cell with dependencies 
> at the same time.
>
> If this is a silly question and I have missed some obvious explanation 
> somewhere, please just point me in the right direction.

Not at all, good question, I have been waiting for it. :) I am afraid I 
have never programmed a threaded application, so I do not know what to 
say. Can threads enqueue on some resource (in the case of Cells, that 
would be the "data pulse", just a sequential counter of state changes 
that drives data integrity)? That would do the trick (I think!), but 
then does that defeat the whole point of threads? If each thread 
operates on a discrete subset of the application universe, well, I /did/ 
start to mess with allowing such a beast to have its own pulse -- then 
we just have to figure out the boundary (assuming one wants at least 
/some/ dependency to reach thru the wormhole, if you will).
(Continue reading)

Permalink | Reply | 501 comments |
headers
Attila Lendvai | 21 Feb 2007 10:25
Picon
Gravatar

Re: [Fwd: Re: Cells and Threads]

> > I have an application that is multithreaded and I would like to
> > explore how to use cells to drive the dataflow.  I've read through the
> > examples and looked over the code a bit. Seems like I could get in
> > trouble if multiple threads tried to change a cell with dependencies
> > at the same time.

i think a global read-write lock could work, which is acquired for
read whenever a cell is checked for validity and upgraded to writing
when the cell is invalid and needs recalculation. a write lock locks
out all other readers and writers, while readers can operate paralel.

interesting questions arise when the calculations themselves have to
acquire other locks in the application... then it's not trivial to
ensure the proper locking order everywhere that avoids random
deadlocks.

but at first i would just create a big-lock-of-the-world which is
acquired whenever anything is used that uses Cells in your app. but
this kills paralelism more and more as the Cells-using part of the app
is bigger and bigger.

hope i said something useful,

--

-- 
- attila

"- The truth is that I've been too considerate, and so became
unintentionally cruel...
 - I understand.
 - No, you don't understand! We don't speak the same language!"
(Continue reading)

Permalink | Reply | 502 comments |
headers
Duane Searsmith | 21 Feb 2007 13:38

Re: [Fwd: Re: Cells and Threads]

Thank you Ken and Attila for your feedback.  I think I have a fair 
understanding of the issues now.  I'm going to do some brainstorming 
over exactly how I would like the threads to interact with the Cells 
engine and then try some experiments with different locking strategies 
and see what happens. 

I'll report back on what I discover.

Best,
-- Duane

Attila Lendvai wrote:
>> > I have an application that is multithreaded and I would like to
>> > explore how to use cells to drive the dataflow.  I've read through the
>> > examples and looked over the code a bit. Seems like I could get in
>> > trouble if multiple threads tried to change a cell with dependencies
>> > at the same time.
>
> i think a global read-write lock could work, which is acquired for
> read whenever a cell is checked for validity and upgraded to writing
> when the cell is invalid and needs recalculation. a write lock locks
> out all other readers and writers, while readers can operate paralel.
>
> interesting questions arise when the calculations themselves have to
> acquire other locks in the application... then it's not trivial to
> ensure the proper locking order everywhere that avoids random
> deadlocks.
>
> but at first i would just create a big-lock-of-the-world which is
> acquired whenever anything is used that uses Cells in your app. but
(Continue reading)

Permalink | Reply | 1 comment |
headers
Ken Tilton | 21 Feb 2007 17:46
Picon

Re: [Fwd: Re: Cells and Threads]

Duane Searsmith wrote:

> Thank you Ken and Attila for your feedback.  I think I have a fair 
> understanding of the issues now.  I'm going to do some brainstorming 
> over exactly how I would like the threads to interact with the Cells 
> engine and then try some experiments with different locking strategies 
> and see what happens.
> I'll report back on what I discover. 

OK, don't be afraid to ask. And don't forget about lazy cells. :) If you 
use those, that creates other issues. I don't much.

One thing that might help (or hurt)  is that in Cells3 I started forcing 
the programmer to single-thread model perturbations:

   (with-integrity (:change ...) (setf <some-cell> ...)

....and there are multiple queues for follow-up work and propagation 
arising from any change, including a "client" queue for application 
work, and one can specify either or both a client queue sort function 
and client queue handler. So there might be some place to wire in a 
little thread awareness and let things keep moving despite activity by 
other threads. And with all that information around, there may also be a 
way for the programmer to indicate they are taking responsibility for 
data integrity so Just Let Me Do This. ie, They may know X is against 
the rules but X is quite safe so Just Do It.

kt
Permalink | Reply | 0 comments |
headers
Peter Denno | 21 Feb 2007 18:33
Favicon

cells and cells-gtk

Hi ken,

It has been a while (2006-06-07 to be exact) since I have updated the cells 
code in cells-gtk. Is there something in this new version, cells3, that would 
make an upgrade worthwhile? Bug fixes? 

--

-- 
Best regards,
  - Peter
Permalink | Reply | 1 comment |
headers
Ken Tilton | 21 Feb 2007 20:18
Picon

Re: cells and cells-gtk

[resending, Peter, to include the list]

Peter Denno wrote:

>Hi ken,
>
>It has been a while (2006-06-07 to be exact) since I have updated the cells 
>code in cells-gtk. Is there something in this new version, cells3, that would 
>make an upgrade worthwhile? Bug fixes? 
>
>  
>
Yes, some bug fixes, and a new concept: data integrity. Sounds
important, right? :) But I wrote an astonishing amount of reliable code
before getting serious about it, sooo... ah, but it was real-world apps
that forced me to get serious, so.... bottom line is that I think you
need to do it at some point or people will eventually hit the cases
where it matters, and I would not even call those edge cases. The new
scheme definitely made it easy to interface to Tk, which was needlessly
fussy about the order in which certain things happen, not good for a
declarative package like Cells that does things in whatever order it
happens to get to them.

The bad news is that Cells is not as transparent as it used to be,
because now if I want to SETF some cell in an observer I have to say:
(with-integrity (:change :id42) (setf <cell> 42)). The good news is the
data integrity and a formal model one can describe and understand and
plan for etc etc.

But it might be a big overhaul, especially because you too are talking
(Continue reading)

Permalink | Reply | 0 comments |

Gmane