Zope & psycopg: ImportError: libpq.so.3: cannot open shared object file: No such file or directory

hello,

I am trying to use psycopg with Zope 2.6.1 on redhat 8.0 (psyche).

I followed the install psycopg instrucions:

I ran the psycopg configure file with (all in one line):

./configure --with-python=/usr/local/zopeservers/zope_dev02/bin/python
--with-zope=/usr/local/zopeservers/zope_dev02
--with-mxdatetime-includes=/usr/local/zopeservers/zope_dev02/lib/python2.1/site-packages/mx/DateTime/mxDateTime
--with-postgres-libraries=/usr/local/pgsql/lib --with-postgres-includes=/usr/local/pgsql/include

Then, I ran:
    make
    make install
    make install-zope

Everything seems to work fine so far. I have a ZPsycopgDA product in <$Zope_HOME>/lib/python/Products
and I also copied the file "psycopgmodule.so" to $ZOPE_HOME/lib/python2.1/lib-dynload/.
.
Now, when I start the zope server, I get the following ecxeption:

Traceback:
"""
[nico <at> localhost zope_dev02]$ ./start &
[1] 25865
[nico <at> localhost zope_dev02]$ ------
2003-07-01T12:24:41 INFO(0) zdaemon Started subprocess: pid 25868
------

Re: Zope & psycopg: ImportError: libpq.so.3: cannot open shared object file: No such file or directory

On Tue, Jul 01, 2003 at 12:43:27PM +0200, Nico Grubert wrote:
> hello,
> [...]
> Now, when I start the zope server, I get the following ecxeption:
> 
> Traceback:
> """
> [nico <at> localhost zope_dev02]$ ./start &
> [...]
> ImportError: libpq.so.3: cannot open shared object file: No such file or directory
> [...]
> 
> In the psycopg FAQ I found the following hint:
> 
> ** 1.4 What do I do if I get a "shared object not started" error?
> 
> Your dynamic linker does not find the PostgreSQL libraries. You have two
> options: reinstall POstgreSQL by telling the configure script to put the
> libraries in some standard place:
> 
>     ./configure --includedir=/usr/include/pgsql --libdir=/usr/lib
> 
> or just configure the dynamic linker search path either by modifying its
> configuration file (/etc/ld.so.conf) or by defining the LD_LIBRARY_PATH
> environment variable. (This works on GNU/Linux, on other systems your
> mileage may vary.)
> 
> 
> So, I add these 2 lines to the /etc/ld.so.conf file:
> /usr/local/pgsql/lib     

Format string types for bound variables

Hi

I am trying to auto build queries using bound variables. I need to be able to 
work out what format "type" to use in the format string from the postgres 
database type. Is there are list of these anywhere?

For example:

query_string = "insert into tbl values (%(field1)s, %(field2)d)"

works for a table with a text column and a int16 column but what is meant to 
work for datetime (%s ?) , float (%f ?) etc?

I may just be being stupid but I can't find a explanation for this anywhere.

Many thanks.

Richard

--

-- 
Jabber: RichardTaylor@...
PGPKey: http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA7DA9FD9
Key fingerprint = D051 A121 E7C3 485F 3C0E  1593 ED9E D868 A7DA 9FD9

Re: Format string types for bound variables

Il mar, 2003-07-08 alle 10:56, Richard Taylor ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi
> 
> I am trying to auto build queries using bound variables. I need to be able to 
> work out what format "type" to use in the format string from the postgres 
> database type. Is there are list of these anywhere?
> 
> For example:
> 
> query_string = "insert into tbl values (%(field1)s, %(field2)d)"
> 
> works for a table with a text column and a int16 column but what is meant to 
> work for datetime (%s ?) , float (%f ?) etc?

you can use %d for integers, %f for floats and %s for everything else
(because everything else will effectively be cast into a string and
quoted.)

hope this helps,
federico

--

-- 
Federico Di Gregorio
Debian GNU/Linux Developer                                fog@...
INIT.D Developer                                           fog@...
   God is real. Unless declared integer. -- Anonymous FORTRAN programmer

Re: Format string types for bound variables

On Tuesday 08 July 2003 10:00 am, Federico Di Gregorio wrote:
> Il mar, 2003-07-08 alle 10:56, Richard Taylor ha scritto:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hi
> >
> > I am trying to auto build queries using bound variables. I need to be
> > able to work out what format "type" to use in the format string from the
> > postgres database type. Is there are list of these anywhere?
> >
> > For example:
> >
> > query_string = "insert into tbl values (%(field1)s, %(field2)d)"
> >
> > works for a table with a text column and a int16 column but what is meant
> > to work for datetime (%s ?) , float (%f ?) etc?
>
> you can use %d for integers, %f for floats and %s for everything else
> (because everything else will effectively be cast into a string and
> quoted.)
>
> hope this helps,
> federico

Many thanks.

Richard

Re: Format string types for bound variables

Il mar, 2003-07-08 alle 11:04, Richard Taylor ha scritto:

> PS. You get the prize for the fastest mailing list response I have ever seen! 
> 5 minutes from subscribing to list to getting answer from the software 
> author. I think I need to go and sit down!

eheh. i get subscribe notices and on psycopg ML a new subscriber means
some kind of question (or bug report!) in the next 30m. so i was waiting
for your mail... :)

ciao,
federico

--

-- 
Federico Di Gregorio
Debian GNU/Linux Developer                                fog@...
INIT.D Developer                                           fog@...
  Nessuno dice che non si possa sognare in dettaglio, essere realistici
   e magari realizzarlo pure, il sogno.                        -- <dani>

_______________________________________________
Psycopg mailing list
Psycopg@...
http://lists.initd.org/mailman/listinfo/psycopg

Another (rare) MT issue w/ PGRES_FATAL_ERROR

        Hi all,

  On a highly stressed zope application (still the same: just using
  plain python, no ZPsycopgDA involved), we've recently encountered
  another hang of psycopg, where a connection is stuck into the locked
  state.

  Extracted from the logs:
pg [15307] _psyco_curs_execute(): Py_BEGIN_ALLOW_THREAD self:0xf10b860 
pg [15307] _psyco_curs_execute(): query = >SELECT ... < status=1
pg [15309] _psyco_curs_execute(): Py_BEGIN_ALLOW_THREAD self:0xf10b860 
pg [15309] _psyco_curs_execute(): query = >SELECT ... < status=1
pg [15309] _psyco_curs_execute(): Py_END_ALLOW_THREAD self:0xf10b860 
pg [15309] _psyco_curs_execute(): exit / restored status: 1 keeper: 0x8b46a70
[15309] cursor closed
pg [15309] _psyco_curs_execute(): Py_BEGIN_ALLOW_THREAD self:0xf0fc9e8 
pg [15309] _psyco_curs_execute(): query = >SELECT ... < status=1
pg [15309] _psyco_curs_execute(): Py_END_ALLOW_THREAD self:0xf0fc9e8 
pg [15309] _psyco_curs_execute(): exit / restored status: 1 keeper: 0x8b46a70
[15309] cursor closed
pg [15309] _psyco_curs_execute(): Py_BEGIN_ALLOW_THREAD self:0xf106870 
pg [15309] _psyco_curs_execute(): query = >SELECT ... < status=1
pg [15307] _psyco_curs_execute(): Py_END_ALLOW_THREAD self:0xf10b860 
pg [15307] _psyco_curs_execute(): PGRES_FATAL_ERROR
[15307] Couldn't evaluate expression SELECT ... Reason: psycopg.InterfaceError:already closed
pg [15309] _psyco_curs_execute(): Py_END_ALLOW_THREAD self:0xf106870 
pg [15309] _psyco_curs_execute(): exit / restored status: -1 keeper: 0x8b46a70

  Simply put, what happened here is that cursor.c:_psyco_curs_execute()

Dealing with embedded nulls in insert strings

Hi

We take text from third party sources and insert it into our database. 
Occasionally that data contains strange characters and we are trying to 
ensure that the sql inserts work regardless. We have particular concern about 
SQL injection attacks (although I can't make one work with the issue below).

We were doing all our own string quoting and not using the "bound variable" 
approach but this is not ideal because it is hard to trap every occurance of 
query building to ensure that the quoting machinery is being used. 

Moving over to the bound variable approach I have run in to a problem. The 
issue is strings that contain embedded NULLS ('\x00'). I notice that the 
quoting machinery does not attempt to detect illegal characters in input 
strings. 

What is the best way to achieve this?  Is it possible to register a type 
conversion that could do it or do I need to revert to my own quoting in every 
place that I use a string?

We would like to be able to simply remove the illegal characters from the 
string.

Many thanks.

Richard

--

-- 
QinetiQ                                  

Re: Dealing with embedded nulls in insert strings

Il mer, 2003-07-09 alle 15:13, Richard Taylor ha scritto:

> Moving over to the bound variable approach I have run in to a problem. The 
> issue is strings that contain embedded NULLS ('\x00'). I notice that the 
> quoting machinery does not attempt to detect illegal characters in input 
> strings. 

ouch. this is a bug. will be fixed. do you know of any other character
that is invalid in a string?

> What is the best way to achieve this?  Is it possible to register a type 
> conversion that could do it or do I need to revert to my own quoting in every 
> place that I use a string?

i am working at python->postgresql typecasters right now: will be
available in a future release of psycopg. 

--

-- 
Federico Di Gregorio
Debian GNU/Linux Developer                                fog@...
INIT.D Developer                                           fog@...
  We should forget about small efficiencies, say about 97% of the
   time: premature optimization is the root of all evil.    -- D.E.Knuth

_______________________________________________
Psycopg mailing list
Psycopg@...
http://lists.initd.org/mailman/listinfo/psycopg

Re: Another (rare) MT issue w/ PGRES_FATAL_ERROR

Il mer, 2003-07-09 alle 01:45, Sebastien Bigaret ha scritto:
[snip]
>   Simply put, what happened here is that cursor.c:_psyco_curs_execute()
>   returned after a PGRES_FATAL_ERROR and did not restore the keeper's
>   value, hence the second thread stored LOCKED as the current status and
>   restored LOCKED back...
[snip]
>   Back on the behaviour itself: it seems that this could be easily
> corrected by restoring the keeper's value before returning. After having
> read pgconn_set_critical() and pgconn_resolve_critical() I have the
> feeling that this does not directly impact the connection itself, so
> restoring the keeper's status would allow other cursors to behave as
> expected (if possible: if it's because of we ran out of memory, there's
> little one can do, sure ;).
> 
>   What do you think? I'm also including a sample test file which reveals
>   the problem quite quickly on my machine.

i think it is a real problem. you should not share the same cursor among
multiple threads, but the problem exists and i think it can happen even
in single threaded applications. 

anyway, the fix is just moving the if block above the status stuff (just
above the big comment talking about time-shifts :)

do you agree?

federico

p.s. thank you very much for your help with this MT stuff. :)