Ivan Maidanski | 12 Apr 09:33 2014
Picon

[Gc] Fwd[2]: Re: Stack overflows, Ex: bdwgc SIGSEGV when stack size is exeeded


-------- Forwarded message --------
From: oops <at> posteo.de
To: Ivan Maidanski <ivmai <at> mail.ru>
Date: Thu, 10 Apr 2014 18:11:15 +0200
Subject: Re: Fwd: [Gc] Stack overflows, Ex: bdwgc SIGSEGV when stack size is exeeded


Am 10.04.2014 15:23 schrieb Ivan Maidanski:
> -------- Forwarded message --------
> From: Ingo Albrecht <prom <at> opendylan.org>
> To: bdwgc <at> lists.opendylan.org
> Date: Thu, 10 Apr 2014, 01:18 +04:00
> Subject: [Gc] Stack overflows, Ex: bdwgc SIGSEGV when stack size is
> exeeded
>
> On 04/09/2014 01:12 PM, Ivan Maidanski wrote:
> >
> > I am wondering if there is by any chances a way for a soft(er)
> landing
> > if the stack space is exhausted? I believe the cause to be in the
> stack
> > size, since setting a ulimit -s changes the point in time when the
> > program crashes. The program is an interpreter and is given a
> recursive
> > definition - so it is expected that something bad happens; I am
> just
> > trying to figure out what my options are to not crash to hard or at
> > least print a diagnostic message or something.
> >
> > I'd apprechiate any comments you may have.
> >
>
> It might be possible to use protocols from libraries like this one:
>
> http://libsigsegv.sourceforge.net/
>
> AFAIK, this library has been created for CLISP.
>
> I've had some good experience with it in (non-GC) C++ projects.
>
> Currently, it probably collides with BDWGC signal handling or might
> be dependent on a specific initialization order. But it might be
> possible to figure out how to use it with BDWGC portably.
>
> Greetings
> prom
>
> _______________________________________________
> bdwgc mailing list
> bdwgc <at> lists.opendylan.org
> https://lists.opendylan.org/mailman/listinfo/bdwgc

Thanks. Much appreciated and I'll look at the things you suggested.
Perhaps this is an idea for a future development. Being able to set a
max stack size and have the GC return gracefully if it tries to exceed
that limit. But I should also mention that the GC is very useful! Also
thanks for the mailing list, I was not aware of it.

_______________________________________________
bdwgc mailing list
bdwgc@...
https://lists.opendylan.org/mailman/listinfo/bdwgc
Ivan Maidanski | 9 Apr 13:12 2014
Picon

[Gc] Fwd: bdwgc SIGSEGV when stack size is exeeded

Forwarding to mailing list

-------- Forwarded message --------
From: oops <at> posteo.de
To: <ivmai <at> mail.ru>
Date: Tue, 08 Apr 2014, 01:07 +04:00
Subject: bdwgc SIGSEGV when stack size is exeeded

Hi Ivan,

I am looking at  https://github.com/ivmai/bdwgc and the documentation
explicitly states that

"GC_malloc may return 0 if it is unable to acquire sufficient space
from the operating system. This is the most probable consequence of
running out of space. Other possible consequences are that a function
call will fail due to lack of stack space, or that the collector will
fail in other ways because it cannot maintain its internal data
structures, or that a crucial system process will fail and take down the
machine."

I am wondering if there is by any chances a way for a soft(er) landing
if the stack space is exhausted? I believe the cause to be in the stack
size, since setting a ulimit -s changes the point in time when the
program crashes. The program is an interpreter and is given a recursive
definition - so it is expected that something bad happens; I am just
trying to figure out what my options are to not crash to hard or at
least print a diagnostic message or something.

I'd apprechiate  any comments you may have.

For you convenience here is a stack trace, if you want to test this
your self I can send instructions but to build the interpreter and the
command that get into this state.

Much thanks.
oops.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b8586e in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:336
336         BZERO((/* no volatile */ void *)dummy, sizeof(dummy));
(gdb) bt
#0  0x00007ffff7b8586e in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:336
#1  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338
#2  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338
#3  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338#4  0x00007ffff7b85886 in GC_clear_stack_inner
(arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338
#5  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338
#6  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338
#7  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
---Type <return> to continue, or q <return> to quit---
     at misc.c:338
#8  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=limit <at> entry=0x7fffff7feda0 <Address 0x7fffff7feda0 out of
bounds>)
     at misc.c:338
#9  0x00007ffff7b85886 in GC_clear_stack_inner (arg=arg <at> entry=0x0,
     limit=0x7fffff7feda0 <Address 0x7fffff7feda0 out of bounds>) at
misc.c:338
#10 0x00007ffff7b858ee in GC_clear_stack (arg=arg <at> entry=0x0) at
misc.c:384
#11 0x00007ffff7b821ca in GC_generic_malloc_many (lb=lb <at> entry=96,
k=k <at> entry=1,
     result=result <at> entry=0x7ffff7dab438 <first_thread+312>) at
mallocx.c:363
#12 0x00007ffff7b8a179 in GC_malloc (bytes=87) at
thread_local_alloc.c:175
#13 0x00007ffff7b7ce4a in GC_debug_malloc (lb=48,
     s=0x4412d0 "/home/ruebenko/gitTest/splendini/uexLib/utils.c", i=6)
     at dbg_mlc.c:516

_______________________________________________
bdwgc mailing list
bdwgc@...
https://lists.opendylan.org/mailman/listinfo/bdwgc
Noah Lavine | 18 Mar 16:10 2014
Picon

Re: [Gc] Memory accounting in libgc

Hello,

This reminds me of a related but distinct issue - region-based allocation. Hypothetically, you could use regions to implement a limited form of this accounting, if you could ask the question, "what is the size of this memory region?"

It certainly wouldn't do as much as custodians, but it would serve for the simple purpose of a sandbox. It would also have another nice effect, because you could allocate all of the sandbox's objects together and get memory locality when running in the sandbox.

I don't know if this is the right solution, but I am curious what other people think.

Best,
Noah


On Sun, Mar 9, 2014 at 6:48 AM, Andy Wingo <wingo-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org> wrote:
Hi,

I was thinking this morning about memory accounting in libgc.  There are
a couple of use cases for this.

One is simply asking "how much memory does this object use?"

Another case is the common "implement an IRC bot that people can use to
evaluate expressions in a sandbox, but with a limit on the amount of
time and space those programs can use".  Let's assume that we've already
arranged that the program can't write to files or do other things we
wouldn't like it to do, and also that we have given it a time limit
through some means.

Both of these cases should be handled by GC, since GC is what can see
the whole object graph in a program.  The rest of this mail describes
what Racket does in their GC, and how we might be able to do it in
libgc.

Racket has a type of objects called "custodians".  Custodians nest.
There is one root custodian for the whole program, and creating a new
custodian adds a new kid of the current custodian.  There is a separate
form to make a custodian current.  New threads are created "within" the
current custodian.  (Racket threads are not pthreads, but this concept
generalizes to pthreads.)  There is also a form to "enter" a new
custodian, effectively partitioning a thread's stack into a range owned
by the old custodian, and a range owned by the new custodian.

If there are multiple custodians in a program -- something that isn't
always the case -- then after GC runs, their GC calls
BTC_do_accounting() here:

  http://git.racket-lang.org/plt/blob/c27930f9399564497208eb31a44f7091d02ab7be:/racket/src/racket/gc2/mem_account.c#l417

This does a mark of the program's objects, but in a different mode (the
"accounting" flag is set).  Custodians and threads have a separate GC
"kind", in BDW-GC terms.  When marking a custodian or a thread in
accounting mode, the mark procedure only recurses into sub-objects if
the custodian is "current".  First, all the program static roots are
marked, excluding stacks, and memory use from those roots is attributed
to the root custodian.  Then BTC_do_accounting traverses the custodian
tree, in pre-order, for each custodian setting it "current" and running
its mark procedure, and marking stack segments for any threads created
by that custodian.  Any visited unmarked object is accounted to the
current custodian (i.e., the size of the object is added to the current
custodian's size counter).  Then the tree is traversed in post-order,
and child memory use is also accounted to parents.  This is O(n) in live
heap size and doesn't require any additional memory.

Finally if any custodian is over its limit, an after-GC hook sets that
custodian as scheduled for shutdown, which in Racket will close
resources associated with the custodian (for example, files opened when
the custodian was current), arrange to kill threads spawned by the
custodian, arrange to longjmp() out of any entered custodian, etc.

How does this affect libgc?

First of all, it gives an answer to the question of "how much memory
does an object use" -- simply stop the world, mark the heap in two parts
(the first time ignoring the object in question, the second time
starting from the object), and subtract the live heap size of the former
from the latter.  Libgc could do this without too much problem, it seems
to me, on objects of any kind.  It would be a little extra code but it
could be useful.  Or not?  Dunno.

On the question of limiting memory usage -- clearly, arranging to free
memory is out of libgc's purview, as that's an application issue.  But
allowing an application to impose a memory limitation on a part of the
program does need libgc help, and it would be most usefully done via
something like a custodian tree.  Custodians could be implemented by a
new GC kind, so that they get a mark procedure; but then we would need a
hook to be able to do accounting after GC is done, and really it seems
this is best done in libgc somehow (even if its implementation is
layered on top of gc kinds, mark procedures, and such).

Accounting a thread's stack to multiple owners is tricky, but doable --
we could do it with a new version of GC_call_with_gc_active or so.

There is also the issue of objects reachable by custodians that are
"peers" -- where one does not dominate the other.  In that case the
object would be charged randomly to one or the other.  That's probably
OK.

Also, you would probably want to maintain a general set of per-custodian
data -- file descriptors, for example.  One would also want to maintain
an estimate of non-GC allocations per-custodian (for example for
third-party image processing libraries), but I don't know how that could
be done; perhaps it is only possible if you register a separate GC kind
for those objects, perhaps with a mark procedure.

Anyway, just some thoughts.  I don't know when I could implement this,
but I thought I'd put this out there in case someone had any ideas about
this, or pointers to literature.  Cheers.

Regards,

Andy
--
http://wingolog.org/


_______________________________________________
bdwgc mailing list
bdwgc@...
https://lists.opendylan.org/mailman/listinfo/bdwgc
Boehm, Hans | 18 Feb 19:41 2014
Picon

Web site / mailing list changes

The GC web site has moved to http://www.hboehm.info/gc/ .  This is currently an almost exact copy of the one
on hpl.hp.com.  But the latter will not be maintained, and is expected to disappear soon.  (We'll try to keep
some sort of forwarding link in place as long as possible.)  If you see any issues, please send mail to
boehm@...  If you control any links to the bdwgc site it would be helpful to
update them.

Gc and gc-announce mailing lists will also be moving.  That is still a work in progress.  Stay tuned and expect hiccups.

Hans
Bostjan Vilfan | 4 Feb 19:57 2014
Picon

a question on the overhead of Boehm-gc

Hello,
I'm planning to use Boehm-gc in a project, and I have done some preliminary testing in the course of which I stumbled on a question that I don't know how to answer. I wonder if I can get some comments.

Attached is a small C program that simply allocates memory in chunks of 5000 bytes, and after each 10 such allocations the used memory is printed out (heap size - free bytes). When used memory decreases, indicating garbage collection, that fact is noted. Now the question. According to the previous description each printout corresponds to an increase of used memory by 50000 bytes; however, the actual printout indicates an increase by more than 80000 bytes (the program was tested on Debian 7). This seems like a lot of overhead. Could someone, please, explain.
Regards,
bv
Attachment (loop.c): text/x-csrc, 2634 bytes
_______________________________________________
Gc mailing list
Gc@...
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
Jim Pryor | 30 Jan 20:37 2014
Picon

When disappearing links are collected before their targets

I'm just starting to work with the Boehm GC library, and am trying to
add weakref functionality to a project that already uses the main
functions of the library (including finalizers).

I think I've got a basic understanding of how
GC_general_register_disappearing_link and nearby functions work.
However, I wasn't able to tell from the comments or other documentation
what is the answer to this question:

If I create a disappearing link L to another object O, but then L ends
up being collected before O does, when the time comes around for O to be
collected, we obviously don't in that case want L to still be cleared.
So do I have to manually create a finalizer for L to
GC_unregister_disappearing_link it before it gets collected? Or does the
library take care of this automatically.

Looking at the source (finalizer.c): I haven't fully digested everything
here yet, but superficially at least it looks like the answer to my
question is that the library takes care of this automatically, when it
invokes the function GC_remove_dangling_disappearing_links.

I'm writing the list for two reasons: (1) to invite others to correct me
if I've misunderstood, and (2) to make this question and answer
available to others who might search for it. (I wasn't able to find any
explicit discussion of this in my own searches.) Perhaps (3) it'd be
good to add a sentence or two to the comments in gc.h (or finalize.c)
declaring explicitly that this is the designed behavior, and that one
doesn't need to manually unregister weakrefs on one's own, just before
they're collected.

--

-- 
dubiousjim@...
Will Schmidt | 28 Jan 22:22 2014
Picon

test_stack on powerpc (power7)

Hi All, 
  I've been looking at the test_stack test case failure as seen on
ppc64 / power7 based systems.    I don't have a fix, but believe I
understand where the problem is occurring.  

The simplest case I've been able to duplicate is with three threads.
As I've added debug to the code, the problem gets harder to nail down
precisely, but this is what seems to be happening.

In the failure scenario:
  The list appears OK during run_one_test() before and after
AO_stack_pop() is called.  The thread is holding two entries in the t[i]
array, and the list still looks OK. The list is damaged after the
AO_stack_push() call is made.

Within AO_stack_push(),
[src/atomic_ops_stack.c:AO_stack_pop_explicit_aux_require()]
The malfunction seems to be triggered while one of the threads is
between the "first=AO_load(list);" and the
"AO_compare_and_swap_release(list,first,next);".  Either one or both of
the other threads will have removed and replaced multiple elements, such
that the compare and swap of list,first,next will pass the check, but
the list entries, particularly the next pointer at first, has changed. 

This is referenced in the comment at that location:
  /* Thus its next link cannot have changed out from under us, and we   */
  /* removed exactly one entry and preserved the rest of the list.      */
  /* Note that it is quite possible that an additional entry was        */
  /* inserted and removed while we were running; this is OK since the   */
  /* part of the list following first must have remained unchanged, and */
  /* first must again have been at the head of the list when the        */
  /* compare_and_swap succeeded.                                        */

which seem to be untrue in this case.

The powerpc AO_* functions seem to be OK.  We'd prefer the gcc atomic
builtins be used (http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html),
(thats what they are there for), but I don't think that change 
would help in this case.

My recommendation is that the test be rewritten to handle the case where
first->next has changed underneath the current thread.
 A shorter term fix would probably be to disable the test_stack test for
power7 and newer processors, until it can be fixed. 

Thanks, 
-Will
Juan Wajnerman | 28 Jan 14:12 2014
Picon

GC and Coroutines

I'm trying to use the GC together with this coroutine library: http://xmailserver.org/pcl.html
But when the GC tries to free some memory running inside a coroutine it crashes.
I guess the problem happens because other coroutine stacks are inaccessible during coroutine execution but I don't know how to fix this.
Does anyone have some tips or tell me at least how could I debug this issue?

Thanks,
- Juan

_______________________________________________
Gc mailing list
Gc@...
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
Andreas Zwinkau | 15 Jan 10:31 2014

Debugging Help (X10i on OctoPOS)

I could use some help/hints for debugging a GC problem.

First some context: We use a modified X10 compiler (X10i) with a custom
backend (libfirm). Compilation target is a custom operating system
(OctoPOS), which in my case is running on top of Linux. We ported bdwgc
(7.3alpha3) to use the corresponding OctoPOS API to stop the world etc.
It works, except of the problem below.

For a longer running application we sometimes observe some memory
corruption. (We mostly observe a deadlock, which might be symptom of the
same bug or another bug.)

Using GC_DEBUG I sometimes get:
GC_debug_free: found previously deallocated (?) object at e811eee0 in or
near object at e811eee8(<smashed>, appr. sz = 133)

Occasionally it crashes on the following assert in malloc.c:270, which
apparently checks that an address from the freelist is within the
correct heap or NULL.

        GC_ASSERT(0 == obj_link(op)
                  || ((word)obj_link(op)
                        <= (word)GC_greatest_plausible_heap_addr
                     && (word)obj_link(op)
                        >= (word)GC_least_plausible_heap_addr));

Using gdb, I can see that "obj_link(op)" points to a vtable, which is
not heap allocated but static data.

The vtable-pointer is at offset 0 in our objects. Object initialization
follows the usual flow of "allocate;set vptr;call constructor". Hence my
current theory: Either allocate does not remove the object from the
freelist correctly or "set vptr" is performed on unallocated/freed memory.

Any tips how to proceed?

I foolishly closed the gdb session, so currently I am rerunning the app
to trigger the assertion again. :-/

cheers
--

-- 
Andreas Zwinkau

 Karlsruhe Institute of Technology (KIT)
 Phone:  +4972160848351
 Email:  zwinkau@...
 Web:    http://pp.info.uni-karlsruhe.de/personhp/andreas_zwinkau.php

_______________________________________________
Gc mailing list
Gc@...
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
Ivan Maidanski | 8 Jan 12:03 2014
Picon

Re: [bdwgc] X10 7 2 (#32)

Hi David,

I'll review the patch within a week or 2.
Thank you.

Regards,
Ivan

Tue, 7 Jan 2014, 13:56 -08:00 from David Grove <notifications <at> github.com>:

Hi,

These are some accumulated small patches we've been maintaining as part of the X10 project (http://http://x10-lang.org/). It would be great if we could get some/all of these merged back into the mainline. They enable BDWGC to be used on a variety of HPC Systems (IBM BlueGene/P and BlueGene/Q, Fujitsu's FX10, and on Power/Linux systems with IBM Parallel Environment).

I'd be happy to discuss them with you if things are unclear or there are better ways to accomplish.

thanks,

--dave

You can merge this Pull Request by running

git pull https://github.com/dgrove-oss/bdwgc x10_7_2

Or view, comment on, or merge it at:

  https://github.com/ivmai/bdwgc/pull/32

Commit Summary

  • Add config option to use STGRTMIN-based signals for thread suspend/resume
  • ChangeLog entry for enableRTSignals.patch.txt
  • Increase size of thread table.
  • Use HEURISTIC2 to find stackbottom on BlueGene systems
  • include powerpc*linux in hosts where GC_LINUX_THREADS is defined
  • Add check for Fujitsu compiler (used on K-Computer/FX10).
  • ChangeLog entries for rest of X10 patches

File Changes

Patch Links:


Reply to this email directly or view it on GitHub.


_______________________________________________
Gc mailing list
Gc@...
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
Ivan Maidanski | 28 Dec 22:20 2013
Picon

Re: Boehm GC problem

Hi Vadim,

It is better to send questions to the ML (gc <at> linux.hpl.com). Probably someone on the list knows the answer.

I have tried your sample but it worked for me (on cygwin), the output:
1000
1

1001
1

Just for reference, I used the recent BDWGC snapshot and following commands to build the sample:
gcc -I include -c extra/gc.c
g++ -I include test_app.cpp gc.o

[Translated] Fri, 27 Dec 2013, 16:51 +04:00 from Vadim <grutargo <at> gmail.com>:
Hi.
I have an issue with Boehm GC, probably you could help me.
I found the solution http://www.drdobbs.com/the-boehm-collector-for-c-and-c/184401632?pgno=6 how to use the collector for the std types but it does not work for me.

The code:

#include "app.h"
I replaced it to:
#include <iostream>
#include <stdlib.h>
#include "gc_cpp.h"


using namespace std;

long allocated = 0;

template<class Super>
class Boehmable : public Super, public virtual gc_cleanup
{
public:
Boehmable() : Super(), gc_cleanup() { allocated++; }
virtual ~Boehmable() { allocated--; }
};

class MyClass
{
public:
MyClass() {};
~MyClass() {};
};

int main(int argc, char **argv)
{
for (int i = 0; i < 1000; i++)
{
new Boehmable<MyClass>();
}

cout << allocated << endl;  // 1000
GC_gcollect();
cout << allocated << endl << endl;  // 1 - It works

for (int i = 0; i < 1000; i++)
{
new Boehmable<string>();
}

cout << allocated << endl;  // 1001
GC_gcollect();
cout << allocated << endl << endl;  // 1001 - Does not work!

system("pause");

return 0;
}

Does not work for list, vector, etc. types neither.
Regards,
Ivan
_______________________________________________
Gc mailing list
Gc@...
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/

Gmane