Martin Buchholz | 1 Jul 01:39 2009
Picon

Re: Review request for 5049299

[+libc-alpha]

I just filed glibc bug
http://sources.redhat.com/bugzilla/show_bug.cgi?id=10353
Methods for deleting all file descriptors greater than given integer

In the JDK...
We close all non-relevant file descriptors in the child instead of relying on
having the FD_CLOEXEC bit set properly for every fd in the parent.

The technique of choice appears to be to read /proc/self/fd and
close all the fds found therein. 
That's even portable between linux and solaris!

Martin

On Tue, Jun 30, 2009 at 02:48, Michael McMahon <Michael.McMahon-xsfywfwIY+M@public.gmane.org> wrote:
Roland McGrath wrote:
But...posix_spawn doesn't give you any way to delete *all* file descriptors
and if you try to collect them before spawning, there is a race in a
multithreaded program.
   

This is something you should never want to do.  There is notoriously no
good way to do it on some systems, where getdtablesize() can return
RLIM_INFINITY and is not proper to use in a loop.

Use FD_CLOEXEC when you open the descriptor in the first place.

 
We also need to chdir() before calling exec. And while some descriptors
already have FD_CLOEXEC set, it's not easy to guarantee this for all of them.

- Michael.

Martin Buchholz | 1 Jul 01:47 2009
Picon

posix_spawn should use vfork() in more cases than presently

I just filed glibc bug

posix_spawn should use vfork() in more cases than presently
http://sources.redhat.com/bugzilla/show_bug.cgi?id=10354

glibc posix_spawn uses vfork() in some cases, fork() in others.
Currently it is rather conservative in this regard.
For example, if there are any file actions, vfork() is avoided.
This restriction can be lifted, I think,
especially for the common case of closing file descriptors.
Martin

On Mon, Jun 29, 2009 at 19:28, Roland McGrath <roland <at> redhat.com> wrote:

> (Aside: I also wonder why glibc's implementation of posix_spawn avoids
> using vfork if there are file actions specified)

Hmm, I'm not sure about that.  I also have no idea why you aren't asking
these questions on the libc-alpha mailing list.

Martin Buchholz | 1 Jul 01:58 2009
Picon

Re: Review request for 5049299

Roland,

You write "It's all the same "clone" code in the kernel."
and that may be true, but I'm thinking about the glibc wrappers around them.
In particular, it seems to me that vfork and clone(CLONE_VM|CLONE_VFORK)
should have similar assembly wrappers around them.  But the
vfork code uses SAVE_PID/RESTORE_PID which frobs only the pid, not the tid,
while clone used RESET_PID which frobs both the pid and tid.
It seems to me they should share common infrastructure.

Martin

On Mon, Jun 29, 2009 at 19:28, Roland McGrath <roland-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Linux clone has avoided vfork's bad press, and has occasionally been
> described as "elegant".  For a while I believed that clone() was the only
> system call that created new processes, and that vfork() was just an
> inflexible special case of clone(), and on ia64 that appears to be true,
> but on x86 clone(), fork() and vfork() are separate system calls,
> and glibc has different gobs of assembly code around each.

Internally it's true.  On some machines there are separate vfork and/or
fork calls, some not.  If you are worried about what the semantics mean,
it is immaterial which system call number you used and how many arguments
you passed.  It's all the same "clone" code in the kernel.


Thanks,
Roland

Christos Zoulas | 1 Jul 02:18 2009

Re: Review request for 5049299

On Jun 30,  4:39pm, martinrb@... (Martin Buchholz) wrote:
-- Subject: Re: Review request for 5049299

| In the JDK...
| We close all non-relevant file descriptors in the child instead of relying
| on
| having the FD_CLOEXEC bit set properly for every fd in the parent.
| 
| The technique of choice appears to be to read /proc/self/fd and
| close all the fds found therein.
| That's even portable between linux and solaris!

Well, on solaris it is best to use closefrom(3C). I think some BSDs have it
too, AIX, IRIX. So perhaps detect if it is there and use that instead (
or fcntl(fd, F_CLOSEM) if it exists.

christos

Andrew John Hughes | 1 Jul 02:23 2009
Picon

Re: timsort

2009/6/30 Martin Buchholz <martinrb@...>:
>
>
> On Tue, Jun 30, 2009 at 09:24, Andrew John Hughes
> <gnu_andrew@...> wrote:
>>
>> 2009/6/30 Martin Buchholz <martinrb@...>:
>>
>> > (There is the deeper governance issue of who gets to make
>> > such decisions.  I would like most of such decisions to be made
>> > by the "group" of engineers who do the work.  For
>> > collections/concurrency,
>> > such a group has worked informally for many years.)
>> >
>>
>> OpenJDK is (or at least should now be) a community-driven open source
>> project.  And so, the community as a whole should be making such
>> decisions, not just those who happen to be employed by Sun.
>
> Right.  There is a problem when different sets of contributors have
> different
> objectives for things like
> compatibility, portability, stability, benchmark performance....
>

Sure, you're right.  My point was just that the decisions you mention
should be made in the open by discussing the issues on the mailing
lists (as I believe we are doing).  I still expect those involved to
come down to the group of interested engineers as you mentioned, who
would be able to obtain a suitable consensus on how to move forward.
I would just hope that these days such a group would not be
constrained by the companies the engineers choose to work for.  I
think we're starting to get there and doing so has advantages for all,
whether at Sun, Google, Red Hat or elsewhere.

> It might be that a significant contributor (like Sun or IcedTea) would
> maintain
> a separate set of patches essentially forever, since they would not be
> acceptable to the greater community.  Oh, I guess that's already happened,
> eh?

In some cases, yes.  IcedTea has a hefty pile of patches, but we're
trying to push most of them upstream.  There are obviously ones that
are too specific though and which it wouldn't make sense to send
upstream, and I'm sure the same is true at Google and Sun.  Of course,
all of ours are there for all to see :) It would be nice if others did
the same too, a public Google-maintained OpenJDK forest seems a nice
idea to me.

>
> There's also the issue of the size of the project.  Is OpenJDK a "project",
> or is it an aggregator of projects maintained by various third parties,
> like a Linux distro?  Given how the code base has grown, at least a large
> number of components should be treated in the latter way.
> It is already the case, I believe, that the JAX* code is essentially
> imported
> unchanged from upstream maintainers.
>

It's certainly not on the scale of a distro, nor is it simply an
aggregator.  It does depend on external code, just like pretty much
any significant project.  Anything else would be recreating the wheel
and thus bad software engineering.  There is also a heck of a lot of
new code being created here, which isn't true of distros (or shouldn't
be in any case!!!!)

The jdk, langtools and hotspot repos are developed locally.  JAXWS,
JAXP and CORBA are all imports (from Glassfish I think).  I think in
the long run it would be better if these were handled as such (as has
been suggested on one or other of these lists), rather than
maintaining the numerous local copies.  I do know there is a plan to
pull from upstream before JDK7.  I don't know if patches ever go the
opposite direction though.

In turn, IcedTea is aggregating OpenJDK with other stuff too of course
-- like a plugin, webstart functionality, additional platform support,
etc.

> Martin
>
>

--

-- 
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

Roland McGrath | 1 Jul 02:10 2009
Picon

Re: Review request for 5049299

> You write "It's all the same "clone" code in the kernel."
> and that may be true, but I'm thinking about the glibc wrappers around them.

I only made that point in response to your contrasts between "a dedicated
vfork syscall number" and "a clone call passed CLONE_VFORK | CLONE_VM |
SIGCHLD", for which the kernel's semantics are completely identical.

> In particular, it seems to me that vfork and clone(CLONE_VM|CLONE_VFORK)
> should have similar assembly wrappers around them.  But the
> vfork code uses SAVE_PID/RESTORE_PID which frobs only the pid, not the tid,
> while clone used RESET_PID which frobs both the pid and tid.
> It seems to me they should share common infrastructure.

That makes sense to me, but I have not looked closely at that logic.

Thanks,
Roland

Jean-Christophe Collet | 1 Jul 13:23 2009
Picon

Re: hg: jdk7/tl/jdk: 6811297: Add more logging to HTTP protocol handler

Yes, it would be nice and there are actually already 2 RFEs covering this:
4640805: Protocol and content handlers SPI
6249864: make it easier to write custom url handlers

However this is no small task, specially considering the compatibility 
and security constraints.
So far resources (or lack of thereof) have prevented us of working on it.

Jim Andreou wrote:
> Hi,
>
> A quick note: Wouldn't it be nice if clients could add URL handling 
> interceptors and monitor incoming/outgoing data themselves? For any 
> URL protocol they would be interested into, not http in particular. 
> Recently I looked into the existing URL protocol handling architecture 
> (e.g. [1], which promises a new era for protocol handlers, but it 
> seems that the existing support is cumbersome and  restricting - 
> sorry, I can't elaborate right now) but it seems to make any such 
> scenario impossible. 
>
> Any comments?
>
> Thanks,
>
> Dimitris Andreou
>
> [1] http://java.sun.com/developer/onlineTraining/protocolhandlers/
>
> 2009/6/25 <jean-christophe.collet@... 
> <mailto:jean-christophe.collet@...>>
>
>     Changeset: 70c0a927e21a
>     Author:    jccollet
>     Date:      2009-06-25 18:56 +0200
>     URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/70c0a927e21a
>
>     6811297: Add more logging to HTTP protocol handler
>     Summary: Added extra logging to HttpURLConnection and HttpClient.
>     Added a capture tool.
>     Reviewed-by: chegar
>
>     ! make/sun/net/FILES_java.gmk
>     + src/share/classes/sun/net/www/http/HttpCapture.java
>     + src/share/classes/sun/net/www/http/HttpCaptureInputStream.java
>     + src/share/classes/sun/net/www/http/HttpCaptureOutputStream.java
>     ! src/share/classes/sun/net/www/http/HttpClient.java
>     + src/share/classes/sun/net/www/protocol/http/HttpLogFormatter.java
>     ! src/share/classes/sun/net/www/protocol/http/HttpURLConnection.java
>
>

Doug Lea | 1 Jul 14:27 2009

Re: Faster HashMap implementation

While I'm posting in-progress stuff, here are some other notes on hash
maps:

First, some observations about usage (repeating some
already posted):

Via some dated but probably still valid dynamic measurments:
    Ops: Approx 84% read, 15% add/modify, 1% remove
    Sizes: Peak at 0, 1, 2, then mostly flat distribution across all others
Static guesses from frequencies of code forms, mainly via google code search
   Key types
     50-60% Strings
     10-20% Numbers -- mainly Integer, then Long,  then others
     10-20% Identity-based (no override of equals/hashCode)
     10-30% eveything else
   Reads might be around 70% get, 30% traverse?
   get: At least 80% successful (no null checks on get)
   Iterators: Probably less than 20% Entry, others about half key vs value
   Puts: typically add, not modify

The tables below (from MapMicrobenchmark) show that cache misses
completely dominate performance for large maps. You'd like to both
push out the sizes for which they come into play, and lessen the
number of indirections (thus misses) even when they do. HashMapV2
mostly does the latter. Of course, for cache misses to dominate, you
also need good key hashing and collision strategies to cope with
possibly poor hashCode()'s.

Times are approximately nanosecs per element/op.  Here are runs on an
Intel x86 linux 2.6.24-23 with jdk6 -server.
Absolute times vary on other platforms but show the same patterns (*).

HashMap:
Type/Size:      36     144     576    2304    9216   36864  147456  589824
BigDecimal      40      37      43      47      57     174     267     317
BigInteger      39      34      38      39      48     187     247     288
String          37      34      38      42      54     149     227     253
Double          41      36      39      44      49      80     219     250
Float           40      38      41      44      48      95     216     267
Long            34      29      33      34      43      74     194     242
Integer         38      33      36      39      46      70     199     233
Object          36      33      35      39      47      64     215     275

average         38      34      37      41      49     111     223     265

HashMapV2:
Type/Size:      36     144     576    2304    9216   36864  147456  589824
BigDecimal      34      34      40      44      58     146     255     286
BigInteger      28      25      29      34      44     151     220     243
String          34      29      34      40      59     136     208     238
Double          37      32      32      44      46      55     180     192
Float           34      37      38      42      42      67     165     246
Long            26      21      22      29      32      40     123     147
Integer         29      25      30      37      46      60     137     157
Object          34      31      33      41      49      64     196     250

average         32      29      32      38      47      89     185     219

Here are some more details listed across some operation categories
(via updated MapCheck) using as keys, English words (Strings) from a
big dictionary.  Times are scaled differently than above, and these
runs are from AMD-x86/solaris, also typical of others:

                         HashMap  :     V2
Access Present         :    213  :    200
Add    Absent          :    265  :    245
Modify Present         :    231  :    199
Remove Present         :    111  :    101
Search Absent          :    119  :     93
Traverse access entry  :    173  :    137
Traverse access key/val:    147  :     87

(*) At least on Solaris jdk6u14, adding the -XX:+AggressiveOpts option
also seems to enable the specialized Integer version of HashMap, which
improves performance on some tests but worsens on others. On all
platforms, the -XX:+AggressiveOpts switch seems to lead to more
run-to-run variation.

Here's synosis of MapMicrobenchmark pasted from javadoc:

  * A micro-benchmark with key types and operation mixes roughly
  * corresponding to some real programs.
  *
  * The main results are a table of approximate nanoseconds per
  * element-operation (averaged across get, put etc) for each type,
  * across a range of map sizes.
  *
  * The program includes a bunch of microbenchmarking safeguards that
  * might underestimate typical performance. For example, by using many
  * different key types and exercising them in warmups it disables most
  * dynamic type specialization.  Some test classes, like Float and
  * BigDecimal are included not because they are commonly used as keys,
  * but because they can be problematic for some map implementations.
  *
  * By default, it creates and inserts in order dense numerical keys
  * and searches for keys in scrambled order. Use "r" as second arg to
  * instead use random numerical values, and "s" as third arg to search
  * in insertion order.

Alex Yakovlev | 1 Jul 14:50 2009
Picon

Re: Faster HashMap implementation

Doug,

Checking key reference equality before hashcode looks like a big win.

Have you considered using balancing tree instead of bit trie?
This could decrease look depth from number of hash bits to log2(tree size).

Also, there could be a sence in using adjacent table cell for key/value if it's
not used. Maybe also without a hashcode check for less cache misses.
In practice this gives ~4-5% performance for get(). In theory, when
70% of gets are resolved at first try, 23% on second, assuming that
adjacent cell is empty in 50% cases, and in best case (referential equality)
we'll have 1 cache miss instead of 3, thus .23*.5*(2/3) = 7.67%
in worst case (2 cache misses instead of 4 with hashcode check) +5.75%

On Wed, Jul 1, 2009 at 2:09 AM, Doug Lea<dl@...> wrote:
>
> Inspired by the combination of Alex's efforts and ongoing
> work on concurrent caches, I put together a version
> of HashMap (called HashMapV2 for now) with a snapshot at
>  http://gee.cs.oswego.edu/dl/wwwtmp/HashMapV2.java
> (This retains the openjdk GPL header etc.)

Michael McMahon | 1 Jul 14:55 2009
Picon

Re: Review request for 5049299

Can we decide on a plan for this issue? I'd like to have
it resolved in good time for M5.

Architecturally, the solution for Solaris is clear I think:
posix_spawn() of helper program which exec()'s the final target.

So, can we recap on what the situation is for Linux?

Is it vfork() plus exec()?
But ,if (as the man page suggests) vfork() is just a special
case of clone(), then are we sure there still isn't too much
"stuff happening" between clone()/vfork() and exec() ?

Maybe, if this question isn't resolved (or resolvable) soon
then should we go ahead with a Solaris only fix?

- Michael.


Gmane