Re: Setlist maintenance

Am 31.10.10 20:52, schrieb Alistair Crooks:
> Our static setlists enable us to cross-build easily on other
> platforms, but can sometimes be a double-edged sword - they are
> difficult to maintain, can be unintuitive in what has to be added, and
> can be prone to problems with definitions and duplication in other
> sets.  I doubt anyone has properly managed the syspkg names in ages,
> if at all.
> 
> We've also seen another instance lately where the compat libs raise
> other problems for everybody's education.
> 
> Some time ago I wrote a utility that is designed to be added to the
> *.mk infrastructure, which uses the Makefile definitions to produce
> output for what the set lists should look like.
> 
> I've put a tar file of the gensetlist utility at:
> 
> 	http://www.netbsd.org/~agc/gensetlist-20101031.tar.gz
> 
> I'm not proposing it be added to NetBSD, merely hoping that its
> presence could make set list maintenance easier.

In the past, having a vehicle like this would have helped me a lot.  I
add new stuff to NetBSD, but I don't have all arches at hand.  So in
consequence I fail to update some of the many lists correctly.

I would really favour if we could a mechanism like this in.

> 
> If anyone reading this finds it useful:

dlopen()/dlclose() and memory reclamation

Hi, consider the following situation:

App->libA->libB

Application uses dlopen() to load libA, which depends on libB, so libB
is automatically loaded.

When application has finished with libA, it calls dlclose(). Assuming
no other applications are using libA and libB, is memory used by those
libraries reclaimed, i.e. is it marked free for usage by other
processes, or does it permanently stay loaded in memory for the
duration of application?

Re: dlopen()/dlclose() and memory reclamation

On Wed, Nov 03, 2010 at 09:35:26AM +0000, Sad Clouds wrote:
> When application has finished with libA, it calls dlclose(). Assuming
> no other applications are using libA and libB, is memory used by those
> libraries reclaimed, i.e. is it marked free for usage by other
> processes, or does it permanently stay loaded in memory for the
> duration of application?

What do you mean with "memory used by those libraries"? The dynamic
linker will unmap the libraries after calling the destructor functions.
It doesn't itself interacts with malloc in anyway.

Joerg

Re: dlopen()/dlclose() and memory reclamation

On Wed, 3 Nov 2010 15:01:30 +0100
Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> On Wed, Nov 03, 2010 at 09:35:26AM +0000, Sad Clouds wrote:
> > When application has finished with libA, it calls dlclose().
> > Assuming no other applications are using libA and libB, is memory
> > used by those libraries reclaimed, i.e. is it marked free for usage
> > by other processes, or does it permanently stay loaded in memory
> > for the duration of application?
> 
> What do you mean with "memory used by those libraries"? The dynamic
> linker will unmap the libraries after calling the destructor
> functions. It doesn't itself interacts with malloc in anyway.
> 
> Joerg

OK, what I mean is this:

As far as I understand, if an application is linked to a set of
shared libraries, those libraries are loaded on application's start up,
or whenever the application references symbols in those libraries. Once
loaded, they remain in memory until application exits.

What I'm trying to understand is how ld.so handles complex dependencies
when application calls dlopen() and then dlclose(). For example:

Application -> libA -> libB

Application calls dlopen() for libA, however libA depends on libB,
hence ld.so will automatically load libB. This way ld.so automatically

Re: dlopen()/dlclose() and memory reclamation

On Wed, Nov 03, 2010 at 03:20:33PM +0000, Sad Clouds wrote:
> As far as I understand, if an application is linked to a set of
> shared libraries, those libraries are loaded on application's start up,
> or whenever the application references symbols in those libraries. Once
> loaded, they remain in memory until application exits.

The ELF object has a list of DT_NEEDED entries. Those are loaded at
start time including dependencies and remain loaded. You can use
"readelf -d" to show them.

> 2. Application calls dlclose() on libA. For each library that libA
> depends on, ld.so recursively checks their dependencies and if no other
> process is using each of those libraries, those libraries get unmapped.
> Finally, ld.so unmaps libA.

This is what happens. Shared objects use a reference counting garbage
collector.

Joerg

Re: dlopen()/dlclose() and memory reclamation

On Wed, 3 Nov 2010 17:31:56 +0100
Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> This is what happens. Shared objects use a reference counting garbage
> collector.
> 
> Joerg

OK thanks, that makes sense.

Re: Replacing usr.bin/grep with the version from FreeBSD

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> any objection to integrate the current BSD grep version from FreeBSD?
> It is supposed to be working for almost all cases as GNU grep
> replacement. It is slower in some cases (which might be fixed by TRE),
> but it should generally be in a much better shape than the current code
> we have.

Please do.

Thanks.

--

-- 
Mindaugas

Improved implementation of *env(3) functions in "libc"

	Hello,

the current implementation of setenv(3), putenv(3), unsetenv(3) and
getenv(3) is a bit messy. This mostly because we try hard to keep
the environment array and the array with the allocat memory
block in sync.

I've attached a new implementation which remembers the memory blocks
in a seperate tree as suggested by Enami Tsugutomo. The code passes
the ATF regression tests including "t_getenv" which currently fails
as our implementation of getenv(3) is slightly broken.

I would appreciate a code review and other comments.

	Kind regards

--

-- 
Matthias Scheler                                  http://zhadum.org.uk/

Index: lib/libc/compat/stdlib/compat_unsetenv.c
===================================================================
RCS file: /cvsroot/src/lib/libc/compat/stdlib/compat_unsetenv.c,v
retrieving revision 1.2
diff -u -r1.2 compat_unsetenv.c
--- lib/libc/compat/stdlib/compat_unsetenv.c	23 Sep 2010 17:30:49 -0000	1.2
+++ lib/libc/compat/stdlib/compat_unsetenv.c	13 Nov 2010 21:32:44 -0000
 <at>  <at>  -54,16 +54,12  <at>  <at> 
 __weak_alias(unsetenv,_unsetenv)

Re: Improved implementation of *env(3) functions in "libc"

On Mon, Nov 15, 2010 at 09:40:55AM +0900, enami tsugutomo wrote:
> Matthias Scheler <tron <at> zhadum.org.uk> writes:
> 
> > I've attached a new implementation which remembers the memory blocks
> > in a seperate tree
> 
> Now the scrubing part can be written like mark and sweep gc, ...

As "environ" is unsorted it would be expensive to look for the memory
referenced by every tree node.

> ... and can be called also when we detect that entire environ vector is
> replaced.

I was considering to add a special case which deletes all nodes in
tree when environ[0] == NULL. But it didn't considere it very
important.

	Kind regards

--

-- 
Matthias Scheler                                  http://zhadum.org.uk/

Re: Improved implementation of *env(3) functions in "libc"

On Mon, Nov 15, 2010 at 08:53:27AM +0000, Matthias Scheler wrote:
> > > Now the scrubing part can be written like mark and sweep gc, ...
[...]
> I think I understand now what you mean:
> 1.) Iterate over "environ" and mark all RB tree nodes whose memory block
>     is still referenced.
> 2.) Iterate over the RB tree and remove all nodes which aren't marked.

I've committed environment scrubbing using that approach to
NetBSD-current yesterday.

	Kind regards

--

-- 
Matthias Scheler                                  http://zhadum.org.uk/