12 Feb 2008 21:07
Copy_from gets stuck with bigger input file
Dear list,
I am experiencing some problems with copy_from() when feeding it with
input files that are quite large (spoken in number of lines, not file
size). If a file exceeds a certain number of lines, copy_from simply
hangs and does not return. I am using psycopg2 2.0.6. PostgreSQL is
version 8.2.4.
In detail, I tried to feed the table as defined below:
create table test_2 (my_num int);
a) - created a file of 118k lines, each line is just "30000\n" (could
be any string)
- fed it into the DB via psycopg2
b) - added 1k lines, each line is just "30000\n" to the file before
- fed it to the DB via psycopg2
Although a) works fine for me, b) just stucks. To me it's not clear
what happens here, since neither CPU nor memory is wasted during this
time. SIGINT does not interrupt the copy_from() call, first SIGTERM
actually stops the computation. Looks like a memory story to me, or some
constraints of stdin. Apparently the number of lines is not the only
criteria, since copying tables with multiple columns leads to the same
problem with less rows.
Would be great if someone could help me out. Just for completeness, both
scenarios worked well using the cmd line client psql combined with COPY.
Here are the file sizes:
692K /tmp/input118k
(Continue reading)
RSS Feed