Jacob Rief | 9 Aug 2007 09:57
Picon
Picon

Solution for "array in a result query"

Hello,
in February there was a request about retrieving binary data from table rows containing array-data.
Jeroen T. Vermeulen replyed that array support will be integrated. I now worked on that same matter and I
found a solution, which is a bit tricky, but it works. I even wrote a template class to implement this feature.
If there is any interrest, I would like to publish this workarround on the wiki. Where shall I post this?
Jacob
--

-- 
GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS.
Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Rilson Nascimento | 10 Aug 2007 05:24
Picon

connection in TIME_WAIT with pqxx

Hi there,

I am using pqxx in a multi-threaded server that connects to a PostgreSQL database. I'm experiencing a problem in which my server leaves a bunch of TIME_WAIT socket connection with the PostgreSQL server when it is running.

When I run 'netstat -tcp' I see literally hundreds of connections from my server to PgSQL in TIME_WAIT state (see below), even after a short period of server activity:
tcp     0     0  localhost:38727       localhost:5432         TIME_WAIT
...

I'm using pqxx's lazyconnection. Actually there is only ONE client running in a loop sending transactions to the server (via tcp socket), which in turn connects to the PgSQL database via a lazyconnection (I tried with the usual connection object too).

I guess this is an effect of poor networking programming (socket programming) and/or poor pqxx programming.
Whatever, What should I do to realize this is not a problem related with misusing of pqxx? I mean, I want to be sure I am using pqxx in the right fashion to ensure this problem is not caused by pqxx.

Thanks for your help,

-Ron


_______________________________________________
Libpqxx-general mailing list
Libpqxx-general@...
http://gborg.postgresql.org/mailman/listinfo/libpqxx-general
Jeroen T. Vermeulen | 10 Aug 2007 06:53
Picon
Picon
Favicon

Re: Solution for "array in a result query"

On Thu, August 9, 2007 14:57, Jacob Rief wrote:

> in February there was a request about retrieving binary data from table
> rows containing array-data. Jeroen T. Vermeulen replyed that array support
> will be integrated. I now worked on that same matter and I found a
> solution, which is a bit tricky, but it works. I even wrote a template
> class to implement this feature.

Great!  I've been working on array support but never had time to finish it.

> If there is any interrest, I would like to publish this workarround on the
> wiki. Where shall I post this?

Easiest thing, I think, is to open a bug ticket for it on pqxx.org.

Jeroen
Jeroen T. Vermeulen | 10 Aug 2007 07:31
Picon
Picon
Favicon

Re: connection in TIME_WAIT with pqxx

On Fri, August 10, 2007 10:24, Rilson Nascimento wrote:

> I am using pqxx in a multi-threaded server that connects to a PostgreSQL
> database. I'm experiencing a problem in which my server leaves a bunch of
> TIME_WAIT socket connection with the PostgreSQL server when it is running.

If I remember correctly, those are closed sockets that the OS keeps around
for a while just in case more packets arrive for them.  So they should go
away eventually, completely independently of the coming and going of your
program.  That part is normal.

The suspicious part is that so many sockets were opened in the first
place.  You're not knowingly creating and closing lots of short-lived
connections?  Or even just a lot of long-lived connections?

The sockets could be by-products of retries (attempt to connect, fail,
close socket, try a new one) but on a localhost connection you generally
either fail or succeed consistently.  If a localhost connection fails, the
next attempt won't work either and the program just won't be able to
access the database.

> I'm using pqxx's lazyconnection. Actually there is only ONE client running
> in a loop sending transactions to the server (via tcp socket), which in
> turn connects to the PgSQL database via a lazyconnection (I tried with the
> usual connection object too).

The plain "connection" class provides the more useful information: if you
see the same behaviour there, that tells us this is not an obscure problem
in the lazy-connection logic, and that comes as a relief.

> I guess this is an effect of poor networking programming (socket
> programming) and/or poor pqxx programming.
> Whatever, What should I do to realize this is not a problem related with
> misusing of pqxx? I mean, I want to be sure I am using pqxx in the right
> fashion to ensure this problem is not caused by pqxx.

The safest thing as far as threading is concerned is to make sure that no
two threads access the same connection, or other objects belonging to the
same connection, simultaneously.  That's more strict than is really
necessary, but it's also relatively easy to maintain.

Jeroen
Rilson Nascimento | 10 Aug 2007 08:33
Picon

Re: connection in TIME_WAIT with pqxx

Hi Jeroen,

Many thanks for your fast and valuable reply. As I guessed in the beginning, it was poor network programming on my end. Fortunately, I found the problem just now after a long week of debugging. It ended up being what you said in the email: creating and closing lots of short-lived socket connections.

There is nothing wrong with pqxx, it is working smoothly. I can say now that I understand sockets way better ;-)

Thanks again, congratulations for the pqxx project, and keep up the good work!

kind regards,

-Rilson

On 8/10/07, Jeroen T. Vermeulen <jtv-qWit8jRvyhVmR6Xm/wNWPw@public.gmane.org> wrote:
On Fri, August 10, 2007 10:24, Rilson Nascimento wrote:

> I am using pqxx in a multi-threaded server that connects to a PostgreSQL
> database. I'm experiencing a problem in which my server leaves a bunch of
> TIME_WAIT socket connection with the PostgreSQL server when it is running.

If I remember correctly, those are closed sockets that the OS keeps around
for a while just in case more packets arrive for them.  So they should go
away eventually, completely independently of the coming and going of your
program.  That part is normal.

The suspicious part is that so many sockets were opened in the first
place.  You're not knowingly creating and closing lots of short-lived
connections?  Or even just a lot of long-lived connections?

The sockets could be by-products of retries (attempt to connect, fail,
close socket, try a new one) but on a localhost connection you generally
either fail or succeed consistently.  If a localhost connection fails, the
next attempt won't work either and the program just won't be able to
access the database.


> I'm using pqxx's lazyconnection. Actually there is only ONE client running
> in a loop sending transactions to the server (via tcp socket), which in
> turn connects to the PgSQL database via a lazyconnection (I tried with the
> usual connection object too).

The plain "connection" class provides the more useful information: if you
see the same behaviour there, that tells us this is not an obscure problem
in the lazy-connection logic, and that comes as a relief.


> I guess this is an effect of poor networking programming (socket
> programming) and/or poor pqxx programming.
> Whatever, What should I do to realize this is not a problem related with
> misusing of pqxx? I mean, I want to be sure I am using pqxx in the right
> fashion to ensure this problem is not caused by pqxx.

The safest thing as far as threading is concerned is to make sure that no
two threads access the same connection, or other objects belonging to the
same connection, simultaneously.  That's more strict than is really
necessary, but it's also relatively easy to maintain.


Jeroen



_______________________________________________
Libpqxx-general mailing list
Libpqxx-general@...
http://gborg.postgresql.org/mailman/listinfo/libpqxx-general
Rilson Nascimento | 10 Aug 2007 12:17
Picon

Re: connection in TIME_WAIT with pqxx

Hi Jeroen, list

My problem with sockets is definitely closed. Now my transactions are behaving strangely. I managed to open just one db connection per client, but now I cannot run multiple transactions per connection (it seems). Errors indicating that there is already a transaction open start to arise. Otherwise, if I change the code to create one db connection per transaction I start to see those TIME_WAIT (a lot of them!) connections with the database again. That's a perfect example of a dilemma.

Any thoughts or comment on this?

Thank you,

-Rilson



On 8/10/07, Rilson Nascimento < rilson.nascimento-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Hi Jeroen,

Many thanks for your fast and valuable reply. As I guessed in the beginning, it was poor network programming on my end. Fortunately, I found the problem just now after a long week of debugging. It ended up being what you said in the email: creating and closing lots of short-lived socket connections.

There is nothing wrong with pqxx, it is working smoothly. I can say now that I understand sockets way better ;-)

Thanks again, congratulations for the pqxx project, and keep up the good work!

kind regards,

-Rilson


On 8/10/07, Jeroen T. Vermeulen < jtv-qWit8jRvyhVmR6Xm/wNWPw@public.gmane.org> wrote:
On Fri, August 10, 2007 10:24, Rilson Nascimento wrote:

> I am using pqxx in a multi-threaded server that connects to a PostgreSQL
> database. I'm experiencing a problem in which my server leaves a bunch of
> TIME_WAIT socket connection with the PostgreSQL server when it is running.

If I remember correctly, those are closed sockets that the OS keeps around
for a while just in case more packets arrive for them.  So they should go
away eventually, completely independently of the coming and going of your
program.  That part is normal.

The suspicious part is that so many sockets were opened in the first
place.  You're not knowingly creating and closing lots of short-lived
connections?  Or even just a lot of long-lived connections?

The sockets could be by-products of retries (attempt to connect, fail,
close socket, try a new one) but on a localhost connection you generally
either fail or succeed consistently.  If a localhost connection fails, the
next attempt won't work either and the program just won't be able to
access the database.


> I'm using pqxx's lazyconnection. Actually there is only ONE client running
> in a loop sending transactions to the server (via tcp socket), which in
> turn connects to the PgSQL database via a lazyconnection (I tried with the
> usual connection object too).

The plain "connection" class provides the more useful information: if you
see the same behaviour there, that tells us this is not an obscure problem
in the lazy-connection logic, and that comes as a relief.


> I guess this is an effect of poor networking programming (socket
> programming) and/or poor pqxx programming.
> Whatever, What should I do to realize this is not a problem related with
> misusing of pqxx? I mean, I want to be sure I am using pqxx in the right
> fashion to ensure this problem is not caused by pqxx.

The safest thing as far as threading is concerned is to make sure that no
two threads access the same connection, or other objects belonging to the
same connection, simultaneously.  That's more strict than is really
necessary, but it's also relatively easy to maintain.


Jeroen




_______________________________________________
Libpqxx-general mailing list
Libpqxx-general@...
http://gborg.postgresql.org/mailman/listinfo/libpqxx-general
Jeroen T. Vermeulen | 10 Aug 2007 12:22
Picon
Picon
Favicon

Re: connection in TIME_WAIT with pqxx

On Fri, August 10, 2007 17:17, Rilson Nascimento wrote:

> My problem with sockets is definitely closed. Now my transactions are
> behaving strangely. I managed to open just one db connection per client,
> but
> now I cannot run multiple transactions per connection (it seems). Errors
> indicating that there is already a transaction open start to arise.
> Otherwise, if I change the code to create one db connection per
> transaction
> I start to see those TIME_WAIT (a lot of them!) connections with the
> database again. That's a perfect example of a dilemma.
>
> Any thoughts or comment on this?

As always, a connection can have only one transaction open at a time.  So
always make sure you destroy a transaction before you open a new one on
the same connection!

(And if you want your work to stay in the database, of course, you'll want
to commit the transaction first :-)

Jeroen
Jacob Rief | 10 Aug 2007 12:52
Picon
Picon

Re: Solution for "array in a result query"

Hello,

> Great!  I've been working on array support but never had time to finish
> it.
> 

I checked the HEAD of the repository-trunk, but did not find any evedence
for array support. Did you ever commit the work you have done so far?

> > If there is any interrest, I would like to publish this workarround on
> the
> > wiki. Where shall I post this?
> 
> Easiest thing, I think, is to open a bug ticket for it on pqxx.org.
> 

Basically it works like this: (I am afraid, this might by to much a
hack for some customers)

First I create a pure C function which does nothing else than copying the
raw array-data into a Postgres bytea-object. This function must be part of
a Postgres shared object, which has to be loaded into the server-side-part
of Postgres, ie. SPI-functions.

Next in the Postgres-shell you have to load this shared object, register
the function with Postgres and create a cast using this function:
# CREATE FUNCTION my_cast(int[]) RETURNS bytea AS 'mylib.so', 'my_cast' \
  LANGUAGE c IMMUTABLE STRICT;
# CREATE CAST(int[] AS bytea) WITH FUNCTION my_cast(int[]);

During a pqxx-query, one can ask for the array as
  std::ostringstream sql;
  sql<<"SELECT CAST(myarray AS bytea) FROM mytable WHERE 1";
  pqxx::result query = cur.exec(sql.str());
  (...snip...)
not the raw array can be used as
  pqxx::binarystring bytea(tuple[0]);
and by reintrerpret_cast'ing bytea.data() to my special class (which 
basically only recalculates the offsets), the real integer array can be 
extracted to an int*
This also works for float's and all other fixed-size types.

1. This solution has a big drawback. Since clients and servers can be
on different endian'ess hosts, array-data would have to be converted, 
using function such as htonl and ntohl. In my case I can skip this, because
I know that they run on the same hardware.

2. Many clients might not be willing to install a shared module to do
the conversion between int[] and bytea.

While my solution works fine for me, I am not sure, that this can be a
general approach. However, I did not find any other solution to transfer
array data, unless one is willing to perform a
binary-to-ascii-to-binray-conversion.

If there is a better concept, please let me know. If you think my solution
is of generic usage, I will work on my conversion-class to make it more
pqxx-like.

Jacob

--

-- 
GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS.
Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Denis Dzyubenko | 10 Aug 2007 14:38
Picon
Gravatar

problem with writing large floating point values to postgres

Hello,

I have a problem with writing values of type double in pqxx. I've
already created a ticket (#116), but received no feedback. Maybe the
ticket was unclear, so I'll try to explain my problem.

The problem is that when writing large double values to iostream, it
is displayed in exponential form, omitting last digits. So, when this
number in exponental form converted back to double, it gets rounded
and significant digits is lost.

For example, if I do this:

std::cout << 12345678;

I'll get '1.23457e+007' on screen, and as you see - last two digits
were omitted and in fact, the number is rounded to '12345700'.
This can be fixed by specifying the precision to iostream library:

std::cout.precision(41);
std::cout << 12345678;

And this time, I'll get on screen the number I expected '12345678'.

Since pqxx::to_string/from_string uses std::stringstream for
converting double values, we face the problem that when I try to write
large 'double' to postgres, the number that is actually written is the
rounded one.

Here is the test application that uses pqxx::to_string/from_string

#include <iostream>
#include <string>
#include <pqxx/pqxx>

int main(int, char**)
{
  std::stringstream ss;

  const double orig_value = 12345678;
  double value;

  // converting double to string and vice versa
  std::string str_value = pqxx::to_string(orig_value);
  pqxx::from_string(str_value, value);

  // using C-function
  printf("printf:\n \torig = %f \n\tnew  = %f\n", orig_value, value);

  // using iostream with default params
  std::cout << std::endl << "cout: " << std::endl
            << "\torig = " << orig_value << std::endl
            << "\tnew  = " << value << std::endl;

  // using iostream and manually specifying the floating value precision
  std::cout.precision(41);
  std::cout << std::endl << "cout with precision: " << std::endl
            << "\torig = " << orig_value << std::endl
            << "\tnew  = " << value << std::endl;
  return 0;
}

And its output:

printf:
        orig = 12345678.000000
        new  = 12345700.000000

cout:
        orig = 1.23457e+007
        new  = 1.23457e+007

cout with precision:
        orig = 12345678
        new  = 12345700

As you can see, after using to_string/from_string the number becomes incorrect.

This problem can be fixed by adding the following line to function
'to_string_fallback' in util.cxx

template<typename T> inline string to_string_fallback(T Obj)
{
  stringstream S;
#ifdef PQXX_HAVE_IMBUE
  S.imbue(locale("C"));
#endif
  S.precision(41); // ADD THIS LINE
  S << Obj;
  string R;
  S >> R;
  return R;
}

--

-- 
Denis.
xmpp:shad@...
Jeroen T. Vermeulen | 10 Aug 2007 15:14
Picon
Picon
Favicon

Re: Solution for "array in a result query"

On Fri, August 10, 2007 17:52, Jacob Rief wrote:

> I checked the HEAD of the repository-trunk, but did not find any evedence
> for array support. Did you ever commit the work you have done so far?

Nope.  Wasn't ready.

> First I create a pure C function which does nothing else than copying the
> raw array-data into a Postgres bytea-object. This function must be part of
> a Postgres shared object, which has to be loaded into the server-side-part
> of Postgres, ie. SPI-functions.

Ouch.  Exactly as you say: I can see that it makes sense as a solution for
your problem, but I don't think we can take this approach in libpqxx!

> While my solution works fine for me, I am not sure, that this can be a
> general approach. However, I did not find any other solution to transfer
> array data, unless one is willing to perform a
> binary-to-ascii-to-binray-conversion.

The conversion between text and binary (of the contents of the array)
isn't all that big a deal.  We do those conversions all the time anyway. 
The problem is that I need to add a parser that can unravel the text
representation of the array itself.  Which isn't _very_ hard, but consider
strings in multibyte encodings where a multibyte character may contain a
byte that looks just like an ASCII quote or string...  :-/

> If there is a better concept, please let me know. If you think my solution
> is of generic usage, I will work on my conversion-class to make it more
> pqxx-like.

For what you need, this is a very neat trick and I think it'd be worth
publishing somewhere as an independent add-on.  For libpqxx, I'm afraid
I'll have to plod on with text parsing.

On the bright side, there have been some noises about adding a bit of
support for that to libpq.  That'd save me the trouble.

Jeroen

Gmane