Mailing list for the Jakarta Hivemind services microkernel

Kyrill Alyoshin (JIRA | 6 Dec 2005 00:30

Picon

Picon

[jira] Created: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader
----------------------------------------------------------------------------------------

         Key: HIVEMIND-161
         URL: http://issues.apache.org/jira/browse/HIVEMIND-161
     Project: HiveMind
        Type: Bug
  Components: framework  
    Versions: 1.1    
 Environment: Tomcat 5.5.12, J2SE 1.6 Mustang build 62, Win XP SP2
    Reporter: Kyrill Alyoshin
    Priority: Critical

The following prevents the web application classloader from being garbage collected on hot redeploys of
web applications. 

Please note that the issue is present on J2SE 1.5 as well. I use 1.6 only because it has far better support for
JVMTI and as such the issue was easy to discover with YourKit profiler. Here it is:

1. org.apache.hivemind.service.impl.ThreadEventNotifierImpl puts EventListenerList ("list"
variable) on its private ThreadLocal "_storage" variable in addThreadCleanupListener method.

2. No where in that class does the actual list gets removed from ThreadLocal.

Thus, the WebappClassLoader cannot be garbage collected because one of the object that it loaded (i.e.
EventListenerList) is strongly referenced by live threads in the container, which leads to huge memory
leaks on hot-redeploys.

--

-- 
This message is automatically generated by JIRA.

(Continue reading)

Kyrill Alyoshin (JIRA | 6 Dec 2005 00:32

Picon

Picon

[jira] Updated: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

     [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=all ]

Kyrill Alyoshin updated HIVEMIND-161:
-------------------------------------

    Attachment: hivemind-threadlocals.pdf

Here is a snapshot from YourKit taken on a classloader of a fully unloaded application.

> ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader
> ----------------------------------------------------------------------------------------
>
>          Key: HIVEMIND-161
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-161
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Tomcat 5.5.12, J2SE 1.6 Mustang build 62, Win XP SP2
>     Reporter: Kyrill Alyoshin
>     Priority: Critical
>  Attachments: hivemind-threadlocals.pdf
>
> The following prevents the web application classloader from being garbage collected on hot redeploys of
web applications. 
> Please note that the issue is present on J2SE 1.5 as well. I use 1.6 only because it has far better support for
JVMTI and as such the issue was easy to discover with YourKit profiler. Here it is:
> 1. org.apache.hivemind.service.impl.ThreadEventNotifierImpl puts EventListenerList ("list"
variable) on its private ThreadLocal "_storage" variable in addThreadCleanupListener method.
> 2. No where in that class does the actual list gets removed from ThreadLocal.

(Continue reading)

James Carman (JIRA | 6 Dec 2005 03:43

Picon

Picon

[jira] Commented: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

    [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=comments#action_12359375 ] 

James Carman commented on HIVEMIND-161:
---------------------------------------

I was wondering what was causing these memory leaks.  I was playing around with Tapestry 4.0 over the weekend
and after a few redeploys in IntelliJ IDEA, it would run out of memory!  So, do you think just clearing the
thread-local variable in fireThreadCleanup() should do it?

> ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader
> ----------------------------------------------------------------------------------------
>
>          Key: HIVEMIND-161
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-161
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Tomcat 5.5.12, J2SE 1.6 Mustang build 62, Win XP SP2
>     Reporter: Kyrill Alyoshin
>     Priority: Critical
>  Attachments: hivemind-threadlocals.pdf
>
> The following prevents the web application classloader from being garbage collected on hot redeploys of
web applications. 
> Please note that the issue is present on J2SE 1.5 as well. I use 1.6 only because it has far better support for
JVMTI and as such the issue was easy to discover with YourKit profiler. Here it is:
> 1. org.apache.hivemind.service.impl.ThreadEventNotifierImpl puts EventListenerList ("list"
variable) on its private ThreadLocal "_storage" variable in addThreadCleanupListener method.
> 2. No where in that class does the actual list gets removed from ThreadLocal.

(Continue reading)

James Carman (JIRA | 6 Dec 2005 03:53

Picon

Picon

[jira] Commented: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

    [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=comments#action_12359376 ] 

James Carman commented on HIVEMIND-161:
---------------------------------------

My fix breaks the testConcur() method which does the following:

public void testConcur()
    {
        ThreadEventNotifier n = new ThreadEventNotifierImpl();

        Listener l1 = new Listener();
        ConcurMod l2 = new ConcurMod(n);
        Listener l3 = new Listener();

        n.addThreadCleanupListener(l1);
        n.addThreadCleanupListener(l2);
        n.addThreadCleanupListener(l3);

        n.fireThreadCleanup();

        assertEquals(true, l1._cleanup);
        assertEquals(true, l2._cleanup);
        assertEquals(true, l3._cleanup);

        l1._cleanup = false;
        l2._cleanup = false;
        l3._cleanup = false;

        n.fireThreadCleanup();

(Continue reading)

Kyrill Alyoshin (JIRA | 6 Dec 2005 04:57

Picon

Picon

[jira] Commented: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

    [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=comments#action_12359377 ] 

Kyrill Alyoshin commented on HIVEMIND-161:
------------------------------------------

It'd be nice to know what your fix was, Carman. 

Anyway, the point is that there is a private ThreadLocal variable  (in particular  "_storage" variable in
ThreadEventNotifierImpl) that is never "cleaned up" in that object. (There is simply nothing that does
"_storage.set(null);" in that class); therefore, this Thread (i.e. an object loaded by System
classloader) will always contain strong references to objects stored in "_storage" and, as a result,
WebappClassLoader will never be garbage collected along with everything it points to. 

Howard said he's gonna look into it in a couple of days once he settles in his new home on the west coast. 

> ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader
> ----------------------------------------------------------------------------------------
>
>          Key: HIVEMIND-161
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-161
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Tomcat 5.5.12, J2SE 1.6 Mustang build 62, Win XP SP2
>     Reporter: Kyrill Alyoshin
>     Priority: Critical
>  Attachments: hivemind-threadlocals.pdf
>
> The following prevents the web application classloader from being garbage collected on hot redeploys of

(Continue reading)

James Carman (JIRA | 6 Dec 2005 12:35

Picon

Picon

[jira] Commented: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

    [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=comments#action_12359407 ] 

James Carman commented on HIVEMIND-161:
---------------------------------------

Here's what I have done in the code (comment is on the line I added):

public void fireThreadCleanup()
    {
        // Here's where we need the CursorableLinkedList since listeners
        // are free to unregister as listeners from threadDidCleanup() and
        // we need to avoid concurrent modification errors.

		EventListenerList list = (EventListenerList) _storage.get();

        if (list == null)
            return;
        _storage.set( null ); // Allow garbage collection (HIVEMIND-161)
        Iterator i = list.getListeners();

        while (i.hasNext())
        {
            ThreadCleanupListener listener = (ThreadCleanupListener) i.next();

			// Each listener may decide to remove itself; that's OK,
			// EventListenerList handles that kind of concurrent modification
			// well.
			
            listener.threadDidCleanup();
        }

(Continue reading)

James Carman (JIRA | 6 Dec 2005 12:35

Picon

Picon

[jira] Commented: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

    [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=comments#action_12359408 ] 

James Carman commented on HIVEMIND-161:
---------------------------------------

Oh, I could probably move the _storage.set( null ) down after the while loop.  

> ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader
> ----------------------------------------------------------------------------------------
>
>          Key: HIVEMIND-161
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-161
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Tomcat 5.5.12, J2SE 1.6 Mustang build 62, Win XP SP2
>     Reporter: Kyrill Alyoshin
>     Priority: Critical
>  Attachments: hivemind-threadlocals.pdf
>
> The following prevents the web application classloader from being garbage collected on hot redeploys of
web applications. 
> Please note that the issue is present on J2SE 1.5 as well. I use 1.6 only because it has far better support for
JVMTI and as such the issue was easy to discover with YourKit profiler. Here it is:
> 1. org.apache.hivemind.service.impl.ThreadEventNotifierImpl puts EventListenerList ("list"
variable) on its private ThreadLocal "_storage" variable in addThreadCleanupListener method.
> 2. No where in that class does the actual list gets removed from ThreadLocal.
> Thus, the WebappClassLoader cannot be garbage collected because one of the object that it loaded (i.e.
EventListenerList) is strongly referenced by live threads in the container, which leads to huge memory

(Continue reading)

Howard M. Lewis Ship (JIRA | 7 Dec 2005 04:04

Picon

Picon

[jira] Commented: (HIVEMIND-162) Performance bottleneck with threaded services

    [ http://issues.apache.org/jira/browse/HIVEMIND-162?page=comments#action_12359523 ] 

Howard M. Lewis Ship commented on HIVEMIND-162:
-----------------------------------------------

I believe we can make that method non-synchronized.  What I'm not sure about is why having that one method
synchronized is causing such a large performance hit.

> Performance bottleneck with threaded services
> ---------------------------------------------
>
>          Key: HIVEMIND-162
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-162
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Linux 2.6.14.2, JBoss 4.0.3, Tomcat 5.5
> Quad Xeon 3.2GHz, 1GB RAM
>     Reporter: Jeff Lubetkin

>
> Note: This may be better classified as a Hivemind issue, but it's affecting Tapestry throughput so I'm
putting it here.
> We've been running some perf tests using the Grinder (with 20 threads) generating as much load as possible
on a single non-trivial page.  The page doesn't touch many of our biz logic services, but does have some
complex componentry to render.
> We were seeing performance ramp just fine until we reached about 200TPS, using only 50% CPU.  No matter how
many clients we threw at it, we couldn't get it any higher.   A thread dump showed that most threads were
bottlenecked on a synchronized method in HiveMind

(Continue reading)

Jeff Lubetkin (JIRA | 7 Dec 2005 04:46

Picon

Picon

[jira] Commented: (HIVEMIND-162) Performance bottleneck with threaded services

    [ http://issues.apache.org/jira/browse/HIVEMIND-162?page=comments#action_12359531 ] 

Jeff Lubetkin commented on HIVEMIND-162:
----------------------------------------

I'm kinda curious about that myself.  If by " I believe we can make that method non-synchronized" you mean
that simply removing the synchronized keyword will be fine, I'd be happy to rebuild hivemind with that
change and throw it on our test box.  If you have other changes in mind, I'd be happy to pre-flight them.

> Performance bottleneck with threaded services
> ---------------------------------------------
>
>          Key: HIVEMIND-162
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-162
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Linux 2.6.14.2, JBoss 4.0.3, Tomcat 5.5
> Quad Xeon 3.2GHz, 1GB RAM
>     Reporter: Jeff Lubetkin

>
> Note: This may be better classified as a Hivemind issue, but it's affecting Tapestry throughput so I'm
putting it here.
> We've been running some perf tests using the Grinder (with 20 threads) generating as much load as possible
on a single non-trivial page.  The page doesn't touch many of our biz logic services, but does have some
complex componentry to render.
> We were seeing performance ramp just fine until we reached about 200TPS, using only 50% CPU.  No matter how
many clients we threw at it, we couldn't get it any higher.   A thread dump showed that most threads were

(Continue reading)

Kyrill Alyoshin (JIRA | 7 Dec 2005 06:40

Picon

Picon

[jira] Commented: (HIVEMIND-161) ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader

    [ http://issues.apache.org/jira/browse/HIVEMIND-161?page=comments#action_12359544 ] 

Kyrill Alyoshin commented on HIVEMIND-161:
------------------------------------------

Yes, moving it below the while loop should cleanup the ThreadLocal. I wonder if Tapestry 4 is set up to call
fireThreadCleanup() when ApplicationServlet stops, since only then the solution will be complete.

> ThreadLocal object is never removed in ThreadEventNotifierImpl and holds the classloader
> ----------------------------------------------------------------------------------------
>
>          Key: HIVEMIND-161
>          URL: http://issues.apache.org/jira/browse/HIVEMIND-161
>      Project: HiveMind
>         Type: Bug
>   Components: framework
>     Versions: 1.1
>  Environment: Tomcat 5.5.12, J2SE 1.6 Mustang build 62, Win XP SP2
>     Reporter: Kyrill Alyoshin
>     Priority: Critical
>  Attachments: hivemind-threadlocals.pdf
>
> The following prevents the web application classloader from being garbage collected on hot redeploys of
web applications. 
> Please note that the issue is present on J2SE 1.5 as well. I use 1.6 only because it has far better support for
JVMTI and as such the issue was easy to discover with YourKit profiler. Here it is:
> 1. org.apache.hivemind.service.impl.ThreadEventNotifierImpl puts EventListenerList ("list"
variable) on its private ThreadLocal "_storage" variable in addThreadCleanupListener method.
> 2. No where in that class does the actual list gets removed from ThreadLocal.
> Thus, the WebappClassLoader cannot be garbage collected because one of the object that it loaded (i.e.

(Continue reading)

Group

gmane.comp.jakarta.hivemind.devel.

Project Web Page

Mailing list for the Jakarta Hivemind services microkernel

Search Archive

Page

1 | 2 | 3 | 4 | 5 | 6

Articles in period: 57

December 2005

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane

Recent entries

Shutting down the dev list (18 Apr 2009)
The Attic and Beyond (17 Apr 2009)
[ann] HiveMind retirement (16 Apr 2009)
[DRAFT] HiveMind Board Report - March 2009 (4 Mar 2009)
Version 1.X Road Map (6 Jan 2009)
HiveMind 1.2 (22 Dec 2008)
HiveMind 1.2 (12 Dec 2008)
[jira] Updated: (HIVEMIND-116) add destroy-method attribute to the con (27 Nov 2008)
[jira] Created: (HIVEMIND-213) MessageFormatter should use ClassLoader (26 Nov 2008)
[DISCUSS] [REPORT] HiveMind November 2008 (11 Nov 2008)
ApacheCon live video streaming available; keynotes and Apache 101 are (4 Nov 2008)
[DRAFT] Board Report - August 2008 (18 Aug 2008)
[jira] Created: (HIVEMIND-212) Pluggable constructor/property injectio (25 Jun 2008)
Explicit service injection by service interface (25 Jun 2008)
Replace commons-logging with slf4j? (25 Jun 2008)
JIRA summary (19 May 2008)
Recent check-in... (16 May 2008)
Reasons for using hivemind (15 May 2008)
HiveMind for Applications (13 May 2008)
[REPORT] Apache HiveMind - May 2008 (13 May 2008)
[DRAFT] Board Report - May 2008 (12 May 2008)

Archives

8	April 2009
7	March 2009
8	January 2009
5	December 2008
22	November 2008
1	August 2008
11	June 2008
68	May 2008
26	February 2008
3	January 2008
9	November 2007
6	October 2007
4	September 2007
2	August 2007
3	July 2007
4	June 2007
8	May 2007
35	April 2007
5	March 2007
68	February 2007
67	January 2007
7	December 2006
49	November 2006
19	October 2006
21	September 2006
61	August 2006
86	July 2006
86	June 2006
94	May 2006
66	April 2006
74	March 2006
36	February 2006
68	January 2006
57	December 2005
17	November 2005
63	October 2005
65	September 2005
86	August 2005
83	July 2005
108	June 2005
220	May 2005
105	April 2005
102	March 2005
160	February 2005
76	January 2005
36	December 2004
70	November 2004
94	October 2004
142	September 2004
185	August 2004
160	July 2004
122	June 2004
31	May 2004
45	April 2004

Design

Zawodny | Right Menu | Left Menu

Your Own Design

Paste the URL of your CSS below. Download CSS template