.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
Hi, I have a problem running Java applications in a 64-bit JVM (with HotSpot compiler enabled) in a 64-bit UML instance.
Sometimes I can start and run the application 5 times in a row, just to experience 5 subsequent crashes. And, even if the application starts up ok, it might crash after some hours.
So, maybe 50% of the time, the Java application crashes soon (~8 seconds) after startup, almost always when running one of the "CompilerThread"s. The majority (~95%) of error log contains lines like these:
Current thread (0x00000000402fb800): JavaThread "CompilerThread1" daemon [_thread_in_native, id=13200, stack(0x00000000453e7000,0x00000000454e8000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x000000003f400010
Many times (but not always) the problem occurs when compiling this method:
Current CompileTask:
C2:155 outline.plugin.ber.AsnTermMap.blow(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Loutline/plugin/ber/AsnTerm;)V (254 bytes)
The problem has been reproduced for Outline (own developed tool), jconsole and Java2D.
Find attached examples of the JVM error logs.
Information HW and OS
Intel(R) Xeon(R) CPU E5540 <at> 2.53GHz
16 CPUs
cache size : 8192 KB
cpu cores : 4
Host OS: Red Hat Enterprise Linux Server release 5.4 (Tikanga), 64-bits
Host kernel: 2.6.18-164.el5
UML kernel: 2.6.37
Findings
After many hours of trying to analyze this problem I have found out 6 things:
1) no, it is not only my Java application that has the problem. Running e g "jconsole" and “Java2D” shows the same behaviour.
2) other (native) 64-bit applications (e g Wireshark) seems to work fine
The problem does not occur:
3) when running the 64-bit JVM directly on the host (not using the UML)
4) when running a 32-bit JVM on a 32-bit UML
5) if the JVM is running in interpreted mode (using the flag "-Djava.compiler=NONE")
6) always - which to me indicates that there might be some kind of conflict of memory addresses or thread execution
My conclusion from the findings is that it is the 64-bit UML and 64-bit Java HotSpot Compiler that doesn't work well together, at least not always.
Trying some workarounds
I have tried to exclude specific methods (like the "blow" method mentined above) from HotSpot compilation, but the result is only that the crash happens when compiling another method.
The only way to avoid crashes (found so far 
is to turn the HotSpot compiler off completely.
What if I change the JVM from the Oracle jdk1.6.0_23 I am using to something else? Without success I have tried:
1) IBMs JVM (J9) -> "segmentation error"
2) JRockit (jrockit-jdk1.6.0_20-R28.1.0-4.0.1-linux-x64) -> "Illegal memory access"
3) Older Sun JVM (Java HotSpot(TM) 64-Bit Server VM (10.0-b19) for linux-amd64 JRE (1.6.0_04-b12)) -> "CompilerThread0" SEGV_ACCERR
What about disk access? I have tried to change the location of the application and the JVM (local UML access fetched from network), but unfortunately I really can't see any difference at all.
Maybe it's a garbage collection problem? Apart from the default GC I have tried a few other types, but no change in the behavior.
Memory problem?
Since the problem doesn't occur every time, there must some other mechanism around which affects my environment?
Most error logs says something which I think is related to memory, so I have made many attempts trying to increase different memory areas.
I have noticed that the PermGen area was at some times filled to 99% at the time of the crash, when running with default memory allocation.
With my current memory setting
-Xmx1024m -Xms1024m -XX:NewSize=256m -XX:MaxPermSize=128m -XX:PermSize=64m
the 99% problem isn't seen anymore, but the crash frequency is about the same as before.
What to do now?
Our current workaround is to run the application without the HotSpot Compiler (which saves us from crashes), but the performance of the application makes it next to unusable.
Any suggestions for tracking down this problem is most welcome!
Best Regards,
Patrik Nyman
RSS Feed