Connection via ODBC

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

Re: Connection via ODBC

camui wrote:
> Hi,
>
> I have posted about this problem in erlang-questions mailing lists.
> http://erlang.org/pipermail/erlang-questions/2008-July/036656.html
>

Well yes, and I posted an answer to this  question
http://erlang.org/pipermail/erlang-questions/2008-July/036994.html

Even though the archive seems to have lost the thread information somehow...

Sockets are used internally to communicate between erlang and c as
some odbc-drivers mess with standard fildescriptors rendering the
normal "erlang-port-program-communication" to malfunction. 
And this code is general to work with both ipv4 and ipv6 hence my
guess at what might be the problem.

Regards  Ingela Erlang/OTP, Ericsson
> ----------------
> "I have a problem with connection to the remote database via ODBC in 
> Erlang.
>
> 1> odbc:connect("DSN=pg", [{scrollable_cursors, off}]).
> {error,{{badmatch,{error,eafnosupport}},
> [{odbc,init,1},{gen_server,init_it,6},{proc_lib,init_p,5}]}}
>
> The database is located on another machine in the local network.
> I have tried to connect to this database in the OpenOffice Base (using 
> ODBC)
> and there are no problems. And I had not any troubles with connection in
> Erlang on the machine where database is located."
> ----------------
>
> I am not sure it is a bug, but I have trouble with connection only in 
> erlang.
>
> PS: Sorry for my english
>
> Andrey.
> ------------------------------------------------------------------------
>
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs <at> erlang.org
> http://www.erlang.org/mailman/listinfo/erlang-bugs

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

Re: Connection via ODBC

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

Re: erts_port[].drv_ptr == 0, when erts_port[].status not free

Hi Paul,

Thanks for the bug report. The attached patch should fix the problem. 
This fix will be included in R12B-4.

How to apply the patch:
   $ gtar -zxf otp_src_R12B-3.tar.gz
   $ gpatch -ZNp0 < OTP-7464.patch
   patching file otp_src_R12B-3/erts/emulator/beam/io.c
   $ # Build and install as usual

BR,
Rickard Green, Erlang/OTP, Ericsson AB.

Raimo Niskanen wrote:
> Thank you for your bug report.
> 
> We will look into your problem when the concerned developers
> comes in after their vacation.
> 
> Can you give us host OS and Erlang release too?
> 
> 
> 
> On Tue, Jul 01, 2008 at 07:24:05PM -0500, Paul Fisher wrote:
>> We have a system where we run lots of linked-in driver ports that get
>> created/used/closed frequently and sometimes very quickly.  Today when
>> several open_port/2, port_command/2 and port_close/1 cycles happened
>> rapid succession, a SIGSEGV occurrect in erl_bif_ddl.c:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 1125235040 (LWP 12087)]
>> 0x0000000000449712 in erl_ddll_try_unload_2 (p=0x2aaaab11fc90,
>>     name_term=659339, options=46912503328425) at beam/erl_bif_ddll.c:592
>>
>> The emulator was run on a Q6600 (quad-core, 2.4Ghz), and started with +A
>> 8,
>> and the linked-in driver executes the bulk of its work with
>> driver_async().
>> There were continuously 8 driver cycles running for 5-10 seconds before
>> the
>> segfault occurred.
>>
>> ???(gdb) where
>> #0  0x0000000000449712 in erl_ddll_try_unload_2 (p=0x2aaaab11fc90,
>>     name_term=659339, options=46912503328425) at beam/erl_bif_ddll.c:592
>> #1  0x000000000052337f in process_main () at beam/beam_emu.c:2073
>> #2  0x000000000049c213 in sched_thread_func (vesdp=0x2ae18cb74f98)
>>     at beam/erl_process.c:741
>> #3  0x00000000005b6818 in thr_wrapper (vtwd=0x7fff1eb77de0)
>>     at common/ethread.c:474
>> #4  0x00002ae18c530f1a in start_thread () from /lib/libpthread.so.0
>> #5  0x00002ae18c8135d2 in clone () from /lib/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> So the code at the point of the SIGSEGV  <at>  erl_bif_ddll.c:592 says:
>>
>>         for (j = 0; j < erts_max_ports; j++) {
>> =>          if (!(erts_port[j].status &  FREE_PORT_FLAGS)
>>                 && erts_port[j].drv_ptr->handle == dh) {
>>
>> It appears that the code assumes that if the erts_port array entry being
>> evaluated during the search has a valid (non-zero) drv_ptr value, if the
>> entry is not marked as free.  At the time of the crash, this is clearly
>> not
>> the case:
>>
>> (gdb) p j
>> $8 = 896
>>
>> (gdb) p erts_port[j]
>> $7 = {sched = {next = 0x0, prev = 0x0, taskq = 0x0, exe_taskq = 0x0},
>>   timeout_task = {counter = 0}, refc = {counter = 2}, lock = 0x81b3c8,
>>   xports = 0x0, id = 14343, connected = 0, caller = 0, data = 0, bp =
>> 0x0,
>>   nlinks = 0x0, monitors = 0x0, bytes_in = 0, bytes_out = 0, ptimer =
>> 0x0,
>>   tracer_proc = 18446744073709551611, trace_flags = 0, ioq = {size = 0,
>>     v_start = 0x0, v_end = 0x0, v_head = 0x0, v_tail = 0x0, v_small = {{
>>         iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {
>>         iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {
>>         iov_base = 0x0, iov_len = 0}}, b_start = 0x0, b_end = 0x0,
>>     b_head = 0x0, b_tail = 0x0, b_small = {0x0, 0x0, 0x0, 0x0, 0x0}},
>>   dist_entry = 0x0, name = 0x0, drv_ptr = 0x0, drv_data = 0, suspended =
>> 0x0,
>>   linebuf = 0x0, status = 4096, control_flags = 0, reg = 0x0,
>>   port_data_lock = 0x0}
>>
>> (gdb) p erts_port[j].drv_ptr
>> $6 = (ErlDrvEntry *) 0x0
>>
>>
>> So the real questions are: 1) is whether the assumption built into this
>> code is correct; and 2) if so, how did we get in the position of
>> violating
>> it.  I'd appreciate some insight into what could be going on here, and
>> where I should can start looking.
>>
>>
>> -- 
>> paul
>>
>> _______________________________________________
>> erlang-bugs mailing list
>> erlang-bugs <at> erlang.org
>> http://www.erlang.org/mailman/listinfo/erlang-bugs
> 

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

dbg:tracer() aborts on binary match

Hi,

I'm writing a small parser for reading log lines.  When it didn't do  
what I expected I tried to use dbg:tracer() to see where it was going  
wrong and it aborted.  Here is the cut down minimal test case.

I'm expecting it to match and then try and call parse_headers with Hdr  
added to the Hdrs list, but instead the VM aborts.  Am I doing  
something wrong/strange?

Many thanks, Jon.

%% Abort when running dbg:tracer().
%%
%% 1> binbug:bug().
%% size_object: matchstate term not allowedAbort trap

-module(binbug).
-export([bug/0,
          parse_headers/4]).

parse_headers(<<"\\r\\n", T/binary>>, Hdr, Hdrs, Toks) ->
     parse_headers(T, <<>>, [Hdr | Hdrs], Toks).

bug() ->
     dbg:tracer(),
     dbg:p(all, call),
     dbg:tpl(?MODULE, []),
     parse_headers(<<"\\r\\n]">>, <<"hdr1">>, [], []).
_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

Re: dbg:tracer() aborts on binary match

<div><div dir="ltr">There is wrong escaping off topic. It should be:<br><br>parse_headers(&lt;&lt;"\r\n", T/binary&gt;&gt;, Hdr, Hdrs, Toks) -&gt;<br>
 &nbsp; &nbsp; parse_headers(T, &lt;&lt;&gt;&gt;, [Hdr | Hdrs], Toks).<br><br>
bug() -&gt;<br>
 &nbsp; &nbsp; dbg:tracer(),<br>
 &nbsp; &nbsp; dbg:p(all, call),<br>
 &nbsp; &nbsp; dbg:tpl(?MODULE, []),<br>
 &nbsp; &nbsp; parse_headers(&lt;&lt;"\r\n]"&gt;&gt;, &lt;&lt;"hdr1"&gt;&gt;, [], []).<br><br><br><div class="gmail_quote">On Tue, Aug 5, 2008 at 12:07 AM, Jon Meredith <span dir="ltr">&lt;<a href="mailto:jmeredith <at> thehive.com">jmeredith <at> thehive.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">Hi,<br><br>
I'm writing a small parser for reading log lines. &nbsp;When it didn't do<br>
what I expected I tried to use dbg:tracer() to see where it was going<br>
wrong and it aborted. &nbsp;Here is the cut down minimal test case.<br><br>
I'm expecting it to match and then try and call parse_headers with Hdr<br>
added to the Hdrs list, but instead the VM aborts. &nbsp;Am I doing<br>
something wrong/strange?<br><br>
Many thanks, Jon.<br><br>
%% Abort when running dbg:tracer().<br>
%%<br>
%% 1&gt; binbug:bug().<br>
%% size_object: matchstate term not allowedAbort trap<br><br>
-module(binbug).<br>
-export([bug/0,<br>
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;parse_headers/4]).<br><br>
parse_headers(&lt;&lt;"\\r\\n", T/binary&gt;&gt;, Hdr, Hdrs, Toks) -&gt;<br>
 &nbsp; &nbsp; parse_headers(T, &lt;&lt;&gt;&gt;, [Hdr | Hdrs], Toks).<br><br>
bug() -&gt;<br>
 &nbsp; &nbsp; dbg:tracer(),<br>
 &nbsp; &nbsp; dbg:p(all, call),<br>
 &nbsp; &nbsp; dbg:tpl(?MODULE, []),<br>
 &nbsp; &nbsp; parse_headers(&lt;&lt;"\\r\\n]"&gt;&gt;, &lt;&lt;"hdr1"&gt;&gt;, [], []).<br>
_______________________________________________<br>
erlang-questions mailing list<br><a href="mailto:erlang-questions <at> erlang.org">erlang-questions <at> erlang.org</a><br><a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote>
</div>
<br><br clear="all"><br>-- <br>--Hynek (Pichi) Vychodil<br>
</div></div>

Re: dbg:tracer() aborts on binary match

Jon Meredith <jmeredith <at> thehive.com> writes:

> Hi,
> 
> I'm writing a small parser for reading log lines.  When it didn't do  
> what I expected I tried to use dbg:tracer() to see where it was going  
> wrong and it aborted.  Here is the cut down minimal test case.

Thanks!

A correction for this bug will be included in R12B-4. I have attached
a patch in case you need a correction before that.

/Bjorn

--

-- 
Björn Gustavsson, Erlang/OTP, Ericsson AB

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

out of memory and crash when using linebuf io

Hi,

I knew that the new regexp library ("re") in Erlang/OTP R12B-3 is
fast. But I wanted to know just how fast "erlgrep" could be compared
to the well-known unix grep which has been around since 1973. My input
file was quite big (about 980 Mb) and in the first attempt it took
some 30 minutes for my io:get_line-based program to scan through the
input compared to 8 minutes for grep. However, I quickly discovered
that io was the bottleneck rather than the regexps. Also to my
surprise, the beam process grew until it had a resident size of 950 Mb
(according to top). Not quite what I had hoped for. I wonder what
would have happened on my machine with 2 Gb RAM if the file had been
three times the size?

I then tried the well-known "open_port({fd,0,1}, [eof,binary,line])"
approach, and ended up in a program that quickly grew until there was
no memory left (>1.2 Gb before top stopped working) and then
*completely* locking up the computer for some 20 minutes before I
finally managed to take beam down by force.

I believe the later issue is the same that was discussed a about a
year ago by Ulf Wiger and others and the suggestion then was to add
some kind of flow control to line oriented port io. So I hereby humbly
submit a patch to add some very very basic flow control. I am sure
that more skilled designers can easily improve it to also handle
io:get_line() on large files without beam growth. I am also sure there
is a corner case or two which I have not thought of.

This is arguably not a bug since piping big chunks of data into beam
was not what the system originally was built for, but I am still
hoping to get away with sending it to erlang-bugs and getting it in
since:

a) It has been experienced by others as well that it is easy to crash
beam if you have big piped input files
b) Currently there is no *fast* way to read data from stdin without
risking beam growth/crash - this is a serious limitation
c) I have provided a patch (which also seems to keep the memory
consumption at more normal levels)

*** erts/emulator/beam/io.c.old 2008-08-01 00:29:16.000000000 +0200
--- erts/emulator/beam/io.c     2008-08-04 22:26:00.000000000 +0200
*************** static void missing_drv_callback(Port *p
*** 2437,2447 ****
--- 2437,2472 ----
  void
  erts_port_ready_input(Port *p, ErlDrvEvent hndl)
  {
+     Process *rp = NULL;
+     ErtsProcLocks rp_locks = ERTS_PROC_LOCK_MSGQ;
+
      ERTS_SMP_CHK_NO_PROC_LOCKS;
      ERTS_SMP_LC_ASSERT(erts_lc_is_port_locked(p));

      ASSERT((p->status & ERTS_PORT_SFLGS_DEAD) == 0);

+     /* When using linebuf_io there is a risk of ending up with an ever growing
+      * message queue in the connected process since there is no flow control
+      * and we usually end up with many small messages instead of one large.
+
+      * The obvious disadvantage with the simple flow control below is that a
+      * message queue which has grown for other reasons than io may
temporarily
+      * stop io messages to the controlling process. On the other hand if we
+      * have a message queue of 4*ERL_MESSAGE_BUF_SZ then it is not helping us
+      * to fill it up with even more messages.
+
+      * With this patch at least we will not run out of memory and crash, and
+      * this makes it easier to analyse the problem... */
+
+     if (p->status & ERTS_PORT_SFLG_LINEBUF_IO) {
+         rp = erts_pid2proc(NULL, 0, p->connected, rp_locks);
+         if (rp) {
+            int mq_too_long = rp->msg.len > 4*ERL_MESSAGE_BUF_SZ;
+            erts_smp_proc_unlock(rp, rp_locks);
+            if (mq_too_long) return;
+         }
+     }
+
      if (!p->drv_ptr->ready_input)
        missing_drv_callback(p, hndl, DO_READ);
      else {

BR /Fredrik

PS. So how did the new re library match up to grep with the above
patch? Quite well I would say. Almost two minutes faster than grep
(using the mmap option to grep only made a few seconds difference).
Even though I knew re was fast from previous experiments I was quite
pleasantly surprised. It is still using more memory than grep, for
sure, but still, I am willing to make that trade-off in this case.

$ time otp_src_R12B-3/bin/erl -smp disable -user logreader <
/tmp/messages.4 > output2.txt

real    6m16.410s
user    6m12.910s
sys     0m3.220s

$ time grep -E '(Mar|Apr) *[0-9]+ *0(1|2):' < /tmp/messages.4 > output3.txt

real    8m17.259s
user    8m15.170s
sys     0m1.640s

$ diff output2.txt output3.txt
$

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

Re: out of memory and crash when using linebuf io

Fredrik Svahn skrev:

> PS. So how did the new re library match up to grep with the above
> patch? Quite well I would say. Almost two minutes faster than grep
> (using the mmap option to grep only made a few seconds difference).
> Even though I knew re was fast from previous experiments I was quite
> pleasantly surprised. It is still using more memory than grep, for
> sure, but still, I am willing to make that trade-off in this case.

Pretty impressive payoff for 11 extra lines of code.  (:

I think that in practice, it should usually be a result of very
poor design to have the receiving process also handle large
amounts of messages from other sources.

BR,
Ulf W
_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs

Re: out of memory and crash when using linebuf io

Greetings,

It is a good thing to correct the problem with having no flow control
for line oriented port io.

It would also be (IMHO) a good thing to add the possibility to use
standard io with file:read/2 . For best effect this could be done like
in io:read/3 .

bengt

On Tue, 2008-08-05 at 10:34 +0200, Fredrik Svahn wrote:
> Hi,
> 
> I knew that the new regexp library ("re") in Erlang/OTP R12B-3 is
> fast. But I wanted to know just how fast "erlgrep" could be compared
> to the well-known unix grep which has been around since 1973. My input
> file was quite big (about 980 Mb) and in the first attempt it took
> some 30 minutes for my io:get_line-based program to scan through the
> input compared to 8 minutes for grep. However, I quickly discovered
> that io was the bottleneck rather than the regexps. Also to my
> surprise, the beam process grew until it had a resident size of 950 Mb
> (according to top). Not quite what I had hoped for. I wonder what
> would have happened on my machine with 2 Gb RAM if the file had been
> three times the size?
> 
> I then tried the well-known "open_port({fd,0,1}, [eof,binary,line])"
> approach, and ended up in a program that quickly grew until there was
> no memory left (>1.2 Gb before top stopped working) and then
> *completely* locking up the computer for some 20 minutes before I
> finally managed to take beam down by force.
> 
> I believe the later issue is the same that was discussed a about a
> year ago by Ulf Wiger and others and the suggestion then was to add
> some kind of flow control to line oriented port io. So I hereby humbly
> submit a patch to add some very very basic flow control. I am sure
> that more skilled designers can easily improve it to also handle
> io:get_line() on large files without beam growth. I am also sure there
> is a corner case or two which I have not thought of.
> 
> This is arguably not a bug since piping big chunks of data into beam
> was not what the system originally was built for, but I am still
> hoping to get away with sending it to erlang-bugs and getting it in
> since:
> 
> a) It has been experienced by others as well that it is easy to crash
> beam if you have big piped input files
> b) Currently there is no *fast* way to read data from stdin without
> risking beam growth/crash - this is a serious limitation
> c) I have provided a patch (which also seems to keep the memory
> consumption at more normal levels)
> 
> 
> *** erts/emulator/beam/io.c.old 2008-08-01 00:29:16.000000000 +0200
> --- erts/emulator/beam/io.c     2008-08-04 22:26:00.000000000 +0200
> *************** static void missing_drv_callback(Port *p
> *** 2437,2447 ****
> --- 2437,2472 ----
>   void
>   erts_port_ready_input(Port *p, ErlDrvEvent hndl)
>   {
> +     Process *rp = NULL;
> +     ErtsProcLocks rp_locks = ERTS_PROC_LOCK_MSGQ;
> +
>       ERTS_SMP_CHK_NO_PROC_LOCKS;
>       ERTS_SMP_LC_ASSERT(erts_lc_is_port_locked(p));
> 
>       ASSERT((p->status & ERTS_PORT_SFLGS_DEAD) == 0);
> 
> +     /* When using linebuf_io there is a risk of ending up with an ever growing
> +      * message queue in the connected process since there is no flow control
> +      * and we usually end up with many small messages instead of one large.
> +
> +      * The obvious disadvantage with the simple flow control below is that a
> +      * message queue which has grown for other reasons than io may
> temporarily
> +      * stop io messages to the controlling process. On the other hand if we
> +      * have a message queue of 4*ERL_MESSAGE_BUF_SZ then it is not helping us
> +      * to fill it up with even more messages.
> +
> +      * With this patch at least we will not run out of memory and crash, and
> +      * this makes it easier to analyse the problem... */
> +
> +     if (p->status & ERTS_PORT_SFLG_LINEBUF_IO) {
> +         rp = erts_pid2proc(NULL, 0, p->connected, rp_locks);
> +         if (rp) {
> +            int mq_too_long = rp->msg.len > 4*ERL_MESSAGE_BUF_SZ;
> +            erts_smp_proc_unlock(rp, rp_locks);
> +            if (mq_too_long) return;
> +         }
> +     }
> +
>       if (!p->drv_ptr->ready_input)
>         missing_drv_callback(p, hndl, DO_READ);
>       else {
> 
> 
> BR /Fredrik
> 
> PS. So how did the new re library match up to grep with the above
> patch? Quite well I would say. Almost two minutes faster than grep
> (using the mmap option to grep only made a few seconds difference).
> Even though I knew re was fast from previous experiments I was quite
> pleasantly surprised. It is still using more memory than grep, for
> sure, but still, I am willing to make that trade-off in this case.
> 
> 
> $ time otp_src_R12B-3/bin/erl -smp disable -user logreader <
> /tmp/messages.4 > output2.txt
> 
> real    6m16.410s
> user    6m12.910s
> sys     0m3.220s
> 
> $ time grep -E '(Mar|Apr) *[0-9]+ *0(1|2):' < /tmp/messages.4 > output3.txt
> 
> real    8m17.259s
> user    8m15.170s
> sys     0m1.640s
> 
> $ diff output2.txt output3.txt
> $
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs <at> erlang.org
> http://www.erlang.org/mailman/listinfo/erlang-bugs

_______________________________________________
erlang-bugs mailing list
erlang-bugs <at> erlang.org
http://www.erlang.org/mailman/listinfo/erlang-bugs