Christopher Armstrong | 1 Mar 18:08 2008

Re: Storm and DBUtils

On Sat, Mar 1, 2008 at 11:58 AM, Sean Allen <sean@...> wrote:
>
>  On Feb 29, 2008, at 4:56 PM, Christopher Armstrong wrote:
>
>  > On Fri, Feb 29, 2008 at 4:29 PM, Sean Allen <sean@...>
>  > wrote:
>  >> Before I start in on this, I wanted to see if anyone had done any
>  >> work
>  >> on integrating DBUtils and Storm.
>  >>
>  >> If not, away I go, but if someone has, please point me in the proper
>  >> direction.
>  >
>  > I'm not sure if it makes sense to integrate the two; Storm provides
>  > many of the features already. All you need to do is make sure that
>  > Store objects are not shared between threads. Reconnection is done by
>  > Storm itself. Pooling is unnecessary. Managing thread-specific Store
>  > instances is something that's left up to you (or the framework), but
>  > we provide storm.zope to help with this (as well as zope transaction
>  > integration) for Zope users. It'd be great to add support for more
>  > thread-managing frameworks over time.
>
>  I'm particular interested in the persistent connection aspect of
>  DBUtils which I cant find any implementation of in Storm.
>
>  Am I wrong on that?

The latest release of Storm automatically reconnects when the
connection to the SQL server is lost. The statement that is being run
at the time the connection is lost will fail (with an exception type
(Continue reading)

Sean Allen | 1 Mar 18:30 2008

Re: Storm and DBUtils


On Mar 1, 2008, at 12:08 PM, Christopher Armstrong wrote:

> On Sat, Mar 1, 2008 at 11:58 AM, Sean Allen <sean@...>  
> wrote:
>>
>> On Feb 29, 2008, at 4:56 PM, Christopher Armstrong wrote:
>>
>>> On Fri, Feb 29, 2008 at 4:29 PM, Sean Allen <sean@...>
>>> wrote:
>>>> Before I start in on this, I wanted to see if anyone had done any
>>>> work
>>>> on integrating DBUtils and Storm.
>>>>
>>>> If not, away I go, but if someone has, please point me in the  
>>>> proper
>>>> direction.
>>>
>>> I'm not sure if it makes sense to integrate the two; Storm provides
>>> many of the features already. All you need to do is make sure that
>>> Store objects are not shared between threads. Reconnection is done  
>>> by
>>> Storm itself. Pooling is unnecessary. Managing thread-specific Store
>>> instances is something that's left up to you (or the framework), but
>>> we provide storm.zope to help with this (as well as zope transaction
>>> integration) for Zope users. It'd be great to add support for more
>>> thread-managing frameworks over time.
>>
>> I'm particular interested in the persistent connection aspect of
>> DBUtils which I cant find any implementation of in Storm.
(Continue reading)

Sean Allen | 1 Mar 17:58 2008

Re: Storm and DBUtils


On Feb 29, 2008, at 4:56 PM, Christopher Armstrong wrote:

> On Fri, Feb 29, 2008 at 4:29 PM, Sean Allen <sean@...>  
> wrote:
>> Before I start in on this, I wanted to see if anyone had done any  
>> work
>> on integrating DBUtils and Storm.
>>
>> If not, away I go, but if someone has, please point me in the proper
>> direction.
>
> I'm not sure if it makes sense to integrate the two; Storm provides
> many of the features already. All you need to do is make sure that
> Store objects are not shared between threads. Reconnection is done by
> Storm itself. Pooling is unnecessary. Managing thread-specific Store
> instances is something that's left up to you (or the framework), but
> we provide storm.zope to help with this (as well as zope transaction
> integration) for Zope users. It'd be great to add support for more
> thread-managing frameworks over time.

I'm particular interested in the persistent connection aspect of
DBUtils which I cant find any implementation of in Storm.

Am I wrong on that?

Christopher Armstrong | 1 Mar 18:33 2008

Re: Storm and DBUtils

On Sat, Mar 1, 2008 at 12:30 PM, Sean Allen <sean@...> wrote:
>  What I'm interested in is not closing out the connections period.
>  If I read the DBUtils code correctly ( and maybe i dont because i put
>  more than 30, less than 60 minutes
>  in on this ), database connection have to be explicitly closed via
>  Persistent db,
>  standard close calls etc wouldnt close the connection whereas
>  storm does close the connections which then have to be reopened.

No, Storm does not close the connections if you use it in the normal
way: have one Store per thread, and keep that Store around for the
lifetime of the thread.

--

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

Christopher Armstrong | 1 Mar 18:35 2008

Re: Storm and DBUtils

On Sat, Mar 1, 2008 at 12:33 PM, Christopher Armstrong
<radix@...> wrote:
> On Sat, Mar 1, 2008 at 12:30 PM, Sean Allen <sean@...> wrote:
>  >  What I'm interested in is not closing out the connections period.
>  >  If I read the DBUtils code correctly ( and maybe i dont because i put
>  >  more than 30, less than 60 minutes
>  >  in on this ), database connection have to be explicitly closed via
>  >  Persistent db,
>  >  standard close calls etc wouldnt close the connection whereas
>  >  storm does close the connections which then have to be reopened.
>
>  No, Storm does not close the connections if you use it in the normal
>  way: have one Store per thread, and keep that Store around for the
>  lifetime of the thread.

... Assuming you have a thread pooling system. Most threaded web
application servers these days do, so it should be fine.

--

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

Sean Allen | 1 Mar 19:33 2008

Re: Storm and DBUtils


On Mar 1, 2008, at 12:35 PM, Christopher Armstrong wrote:

> On Sat, Mar 1, 2008 at 12:33 PM, Christopher Armstrong
> <radix@...> wrote:
>> On Sat, Mar 1, 2008 at 12:30 PM, Sean Allen <sean@...>  
>> wrote:
>>> What I'm interested in is not closing out the connections period.
>>> If I read the DBUtils code correctly ( and maybe i dont because i  
>>> put
>>> more than 30, less than 60 minutes
>>> in on this ), database connection have to be explicitly closed via
>>> Persistent db,
>>> standard close calls etc wouldnt close the connection whereas
>>> storm does close the connections which then have to be reopened.
>>
>> No, Storm does not close the connections if you use it in the normal
>> way: have one Store per thread, and keep that Store around for the
>> lifetime of the thread.
>
> ... Assuming you have a thread pooling system. Most threaded web
> application servers these days do, so it should be fine.

Which requires a rewrite of our code.

Apache::DBI from perl does the following:

monkeypatches the connect and close methods in the DBI classes
so that a close doesnt really close and an open returns the
already open connection so you can do stuff like this:
(Continue reading)

James Henstridge | 2 Mar 03:11 2008
Picon

Re: Storm and DBUtils

On 01/03/2008, Sean Allen <sean@...> wrote:
> Before I start in on this, I wanted to see if anyone had done any work
>  on integrating DBUtils and Storm.
>
>  If not, away I go, but if someone has, please point me in the proper
>  direction.

What features from DBUtils are you after exactly?

Storm already has transaction aware disconnection handling (i.e. if
you handle DisconnectionError in the same way you'd handle a deadlock
or serialisation error, you should be fine).

And if you want to do per-thread or pooled connections, it is probably
better to do that on top of Storm's Store abstraction.  See the
storm.zope module for an example of managing per-thread stores.

James.

Lutz Horn | 3 Mar 13:14 2008

Like on DateTime using only date

Hi,

with at DateTime property I'd like to compare only using the date
part. But the like method of DateTime needs a datetime.datetime
instance, not a datetime.date instance. How can I formulate the like
condition to only select those records whose DateTime propery match a
datetime.date instance?

Regards

Lutz

Gustavo Niemeyer | 3 Mar 17:21 2008
Picon

Re: Like on DateTime using only date


Hey Lutz,

> with at DateTime property I'd like to compare only using the date
> part. But the like method of DateTime needs a datetime.datetime
> instance, not a datetime.date instance. How can I formulate the like
> condition to only select those records whose DateTime propery match a
> datetime.date instance?

I'm not sure I understand what's the problem you're facing.  The
DateTime indeed needs a datatime object.  We may extend it to support
automatic conversion from date objects, but it should be fairly simple
to make it work as it is now.

Can you provide some sample code that is failing, and that you're having
trouble to make it work correctly?

--

-- 
Gustavo Niemeyer
http://niemeyer.net

Lutz Horn | 3 Mar 17:42 2008

Re: Like on DateTime using only date

Hi Gustavo,

Am 03.03.2008 um 17:21 schrieb Gustavo Niemeyer:
>> with at DateTime property I'd like to compare only using the date
>> part. But the like method of DateTime needs a datetime.datetime
>> instance, not a datetime.date instance. How can I formulate the like
>> condition to only select those records whose DateTime propery match a
>> datetime.date instance?
>
> I'm not sure I understand what's the problem you're facing.  The
> DateTime indeed needs a datatime object.  We may extend it to support
> automatic conversion from date objects, but it should be fairly simple
> to make it work as it is now.
>
> Can you provide some sample code that is failing, and that you're  
> having
> trouble to make it work correctly?

I've the following table in MySQL:

mysql> describe screenings;
+----------+--------------+------+-----+---------+----------------+
| Field    | Type         | Null | Key | Default | Extra          |
+----------+--------------+------+-----+---------+----------------+
| id       | int(11)      |      | PRI | NULL    | auto_increment |
| when     | datetime     | YES  |     | NULL    |                |
| movie_id | int(11)      | YES  |     | NULL    |                |
+----------+--------------+------+-----+---------+----------------+

Using Strom, the corresponding model class is:
(Continue reading)


Gmane