initd.org problems

Lately we're experiencing problems with maya.initd.org. Its load and
memory usage are pretty high from time to time (she does a lot of
things) and that results in random crashes of the ssh daemon (and smtp
deamon). If you tried to access SVN or to ssh please, just be patient. I
am installing right now a new machine that will be located at our office
making maintenance much easier. It will be online ASAP (when we get the
public IP from the local telco).

Sorry for the problems,
federico

--

-- 
Federico Di Gregorio                         http://people.initd.org/fog
Debian GNU/Linux Developer                                fog@...
INIT.D Developer                                           fog@...
                   Ma chi sei?....-il trafficante di Nutella? -- Giorgia

_______________________________________________
Psycopg mailing list
Psycopg@...
http://lists.initd.org/mailman/listinfo/psycopg

Re: Improved transaction error handling

On 29/12/2007, Federico Di Gregorio <fog@...> wrote:
>
> Il giorno ven, 28/12/2007 alle 05.50 +0900, James Henstridge ha scritto:
> > Attached is a patch to improve the transaction error handling.
> >
> > It gets rid of the pq_set_critical() usage in Connection.commit() and
> > Connection.rollback(), and preserves the PGresult from commit/rollback
> > errors so that it can be used to raise an appropriate exception.
> >
> > This means that we can set an appropriate "pgcode" attribute on the
> > exceptions, allowing for better error handling.
>
> This would be very useful but I don't understand if you need all the
> other changes (I mean the _locked versions of the various functions) or
> that is a different feature. Can you explain, please?

Sorry for the delay in replying -- I was on holiday without any
internet access (or mobile signal).

I see the patch I sent as an intermediate step towards cleaning up
this part of Psycopg.  While it isn't as clean as it could be, it is
an improvement over what's there and provides a basis for further
improvements.

As it is now, some of the pqpath.[ch] functions are expected to be
called with the GIL held, while others expect to be called without the
GIL held and the connection's lock acquired.

In my patch, the functions named as *_locked() expect a locked
connection (without the GIL held), and the other ones expect to be

Re: Improved transaction error handling

On 31/12/2007, Karsten Hilbert <Karsten.Hilbert@...> wrote:
> On Fri, Dec 28, 2007 at 05:50:09AM +0900, James Henstridge wrote:
>
> > This means that we can set an appropriate "pgcode" attribute on the
> > exceptions, allowing for better error handling.
> That would be great !
>
> A related thing I have been working on is differentiating
> exceptions raised when failing to establish connections to
> the backend. I want to be able to differentiate faulty
> credentials from other errors in order to present meaningful
> messages to the user.
>
> Parsing the exception string for keywords works but only in
> very controlled circumstances. One needs to know beforehand
> which language to parse in. Even forcing lc_messages to 'C'
> in the database default doesn't make sure English error
> messages are returned. Apparently some messages are
> translated (generated ?) at the libpq level. Forcing libpq
> to C locale would mean not being able to set the proper
> locale on the entire application using libpq via psycopg2.

Yep, some exception strings definitely come from libpq.  The prime
example is the "could not connect to server" and "connection closed"
errors where there is no server to receive the message from ...

It might be worth filing a Postgres feature request for the library to
expose error codes for these connection-oriented errors.

> I'd be happy to contribute the (Python level) code once it

Re: Proposal: Adding "push" semantics to copy_from

Chris Cogdon ha scritto:
> Hi folks!
> 
> I'd like to get a new extension put into psycopg2 to allow "push"  
> semantics to "copy from" operations. Ie, rather than having  
> "copy_from" be passed a file handle ("pull" semantics), it returns a  
> object where I can generate my own data and "push" it into the  
> connection to the database (with flushing and closing operations, too).
> 
> If someone want's to go ahead and just do that, that's be totally  
> fantastic! In lieu of such unbounded generosity, I'd like to do the  
> work myself and offer it as a submission to psycopg2. So, if someone  
> has submission tips before I start work on that, I'd be grateful
> 

Some time ago I have written a PostgreSQL client implementation in Twisted:
http://hg.mperillo.ath.cx/twisted/pglib/

Actually the data to be copied to the server is handled using a "pull" 
semantic (data is read from an user supplied object every reactor cycle).

However this can be changed by allowing the user supplied object to 
return a deferred; the semantic is still "pull", but the user has more 
control on when to send the data.

 > [...]

Manlio Perillo

Re: Improved transaction error handling

On Wed, Jan 02, 2008 at 01:39:20PM +0900, James Henstridge wrote:

> There were some places where pq_abort() was being called as part of
> another method (e.g. setting charset, changing isolation level).  For
> these, I added the pq_abort_locked() call.  To simplify matters, I
> gave it the same calling conventions as the old pq_abort() call.  As a
> further improvement, it would be good for it to pass on the libpq
> error so that callers can raise appropriate exceptions.

Yes, please 

This may help in detecting connection-oriented errors, too.
Maybe a .libpqcode in addition to .pgcode on an exception ?

Thanks,
Karsten
--

-- 
GPG key ID E4071346  <at>  wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346

Re: Improved transaction error handling

On Wed, Jan 02, 2008 at 02:19:06PM +0900, James Henstridge wrote:

> Yep, some exception strings definitely come from libpq.  The prime
> example is the "could not connect to server" and "connection closed"
> errors where there is no server to receive the message from ...

Yes, those are the ones I am after the most, including
"fe_sendauth, no password supplied" and "password
authentication failed for user ...".

Karsten
--

-- 
GPG key ID E4071346  <at>  wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346

closing a connection from "outside" an application (for testing)

Hi all,

I have a mod_python web-app that connects to a database and every once
in a while has closed the connection when I try to get a cursor; now I
have code that tests for "conn.closed" and reopens if necessary before
returning the cursor.

My question:  is there some way to get the connection to close (on
purpose) in order to test the code that reopens the connection?

Thx!
W

Re: closing a connection from "outside" an application (for testing)

Webb Sprague wrote:
> Hi all,
> 
> I have a mod_python web-app that connects to a database and every once
> in a while has closed the connection when I try to get a cursor; now I
> have code that tests for "conn.closed" and reopens if necessary before
> returning the cursor.
> 
> My question:  is there some way to get the connection to close (on
> purpose) in order to test the code that reopens the connection?

Two methods I'm aware of:

1) Open a new connection to the database, get the process id of the
connection you want to die (select procpid from pg_stat_activity where
[...]), and kill that process using a stored procedure (or os.kill if your
running your tests as a unix user with relevant permissions).

2) Make psycopg connect to a tcp proxy rather than directly to PostgreSQL.
Kill the proxy and restart it.

Ideally you want to test both cases - the first will be what your app sees
when the database kills your connection and it will be terminated somewhat
nicely. The second case is what your app sees when the network connection dies.

--

-- 
Stuart Bishop <stuart@...>
http://www.stuartbishop.net/

Re: closing a connection from "outside" an application (for testing)

 > Two methods I'm aware of:
>
> 1) Open a new connection to the database, get the process id of the
> connection you want to die (select procpid from pg_stat_activity where
> [...]), and kill that process using a stored procedure (or os.kill if your
> running your tests as a unix user with relevant permissions).
>
> 2) Make psycopg connect to a tcp proxy rather than directly to PostgreSQL.
> Kill the proxy and restart it.
>
> Ideally you want to test both cases - the first will be what your app sees
> when the database kills your connection and it will be terminated somewhat
> nicely. The second case is what your app sees when the network connection dies.

Huh -- the things you learn....

Is there a way for psycopg to return the PID of the process it is
connected to, primarily for logging (and killing)?  I am still trying
to figure out how to stop a run of my program in mid-stream (besides a
sleep() or waiting on a signal...), but that will get me started.

Also -- if the program running psycopg2 dies, what happens to the
"idle in transaction" postgres process?

My apologies if these are easily findable in the docs.

Thanks!

Re: closing a connection from "outside" an application (for testing)

Il giorno gio, 03/01/2008 alle 21.26 -0800, Webb Sprague ha scritto:
> Is there a way for psycopg to return the PID of the process it is
> connected to, primarily for logging (and killing)?  I am still trying
> to figure out how to stop a run of my program in mid-stream (besides a
> sleep() or waiting on a signal...), but that will get me started.

No way, right now (unless you can do that using a query; I don't know.)

> Also -- if the program running psycopg2 dies, what happens to the
> "idle in transaction" postgres process?

The transaction is aborted and the process exits.

federico

--

-- 
Federico Di Gregorio                         http://people.initd.org/fog
Debian GNU/Linux Developer                                fog@...
INIT.D Developer                                           fog@...
                           Don't dream it. Be it. -- Dr. Frank'n'further

_______________________________________________
Psycopg mailing list
Psycopg@...
http://lists.initd.org/mailman/listinfo/psycopg