Re: Grinder REST Console Service and Remote Agents
2013-03-24 12:32:26 GMT
The strace showing the SIGPIPE is further evidence that the problem does not lie within the worker process. The socket its trying to write to has been closed.
Does the other end (the console) log anything?
On 12/03/13 14:03, Ouray Viney wrote:
Hi Phil,Thank you for sticking with this, greatly appreciated.Some more details about the test that is being run by Grinder may help.- We are using Grinder to run some Java client side api’s.- There are over 128 jars that get loaded during the first run.- Profiling the system during the initial run shows that the system is pinned. I thought that this might be a contributor, but one of 9 agents is functioning correctly. This particular system is actually a little slower than the other (less bogomips).- The test that Grinder instruments makes use of a local CUPs server- The test that Grinder instruments makes use of local beans for the purpose of mailing and shipping requests being sent to a backend web service (REST).- I profiled the Grinder agent and WorkerProcessor using visualvm (bundled with Java 1.6), not much to say other than the CPU profile once the test started was flat at 98%.The current workaround is to start the test the first time. Expect to see the issue, stop the test. Start a second run and things *should* work as expected, i.e. no comms. issue from the WorkerProcessor to the ConsoleCheers,OurayOn Tue, Mar 12, 2013 at 8:19 AM, Philip Aston <philipa-O5WfVfzUwx8@public.gmane.org> wrote:Something is tearing down a connection, or otherwise blocking communication, on a TCP connection that was previously fine. I suspect the former, since lsof says the connection no longer exists. The trace below is just reporting that the worker could not send data to the console, and doesn't provide anything else of use.
The worker creates the connection when it starts up. It sends a few bytes of data to the console as part of establishing the connection. Then it regularly sends a report message to the console.
The worker process relies on the TCP connection being resilient. It doesn't try to re-connect.
Maybe a tcpdump would help, maybe not.
On 11/03/13 17:28, Ouray Viney wrote:
Hi Philip,No problem at all. Totally appreciate that you are busy .I have noticed that when this problem occurs, I see the following exception in the agent output. Yet, if I re-run this a few times, it eventually goes a way. The bottom line is the communication from the WorkerProcessor to the Console is unreliable for some reason. Please keep in mind I am not blaming The Grinder for this.Environment details:=================- when the problem is reproducible, lsof reports no connection to the console- no firewalls are blocking any of the Grinder console ports 6372/6373- no clear signs of any issues with ulimits for the account that owns the WorkerProcessor process- the exception trace below means that something happened, I need to review the source code I don't know what condition is causing this code to fail.2013-03-11 12:22:22,216 INFO 3582299-001-0: Report to console failednet.grinder.communication.CommunicationException: Exception whilst sending messageat net.grinder.communication.AbstractSender.send(AbstractSender.java:57) ~[grinder-core-3.11.jar:na]at net.grinder.communication.QueuedSenderDecorator.flush(QueuedSenderDecorator.java:60) ~[grinder-core-3.11.jar:na]at net.grinder.engine.process.GrinderProcess.sendStatusMessage(GrinderProcess.java:638) [grinder-core-3.11.jar:na]at net.grinder.engine.process.GrinderProcess.access$1100(GrinderProcess.java:110) [grinder-core-3.11.jar:na]at net.grinder.engine.process.GrinderProcess$ReportToConsoleTimerTask.run(GrinderProcess.java:615) ~[grinder-core-3.11.jar:na]at net.grinder.engine.process.GrinderProcess.run(GrinderProcess.java:465) [grinder-core-3.11.jar:na]at net.grinder.engine.process.WorkerProcessEntryPoint.run(WorkerProcessEntryPoint.java:86) [grinder-core-3.11.jar:na]at net.grinder.engine.process.WorkerProcessEntryPoint.main(WorkerProcessEntryPoint.java:59) [grinder-core-3.11.jar:na]Caused by: java.net.SocketException: Broken pipeat java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_07]at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_07]at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_07]at java.io.BufferedOutputStream.flushBuffer(Unknown Source) ~[na:1.6.0_07]at java.io.BufferedOutputStream.flush(Unknown Source) ~[na:1.6.0_07]at java.io.ObjectOutputStream$BlockDataOutputStream.flush(Unknown Source) ~[na:1.6.0_07]at java.io.ObjectOutputStream.flush(Unknown Source) ~[na:1.6.0_07]at net.grinder.communication.AbstractSender.writeMessageToStream(AbstractSender.java:90) ~[grinder-core-3.11.jar:na]at net.grinder.communication.StreamSender.writeMessage(StreamSender.java:70) ~[grinder-core-3.11.jar:na]at net.grinder.communication.AbstractSender.send(AbstractSender.java:53) ~[grinder-core-3.11.jar:na]... 7 common frames omittedKind Rgds,OurayOn Wed, Mar 6, 2013 at 2:57 AM, Philip Aston <philipa-O5WfVfzUwx8@public.gmane.org> wrote:
Sorry for the delay Ouray.
1) A thread dump of the worker process might be useful if it shows the worker process trying to establish a connection. But its a long shot.
What does netstat tell you? Or better, lsof -i tcp -a -p <worker process pid>?
2) If there's no summary table, the worker process isn't shutting down correctly. Can you provide the logs from a worker process?
On 26/02/13 16:31, Ouray Viney wrote:
Hi Philip,I wanted to revisit this thread. I am back on this problem and would like to review your recommendations for debugging the problem.As far as I can diagnose, the problem appears to:======================================1) The Worker Processor (child of the Grinder Agent JVM process) spawns correctly. What it fails to do is communicate its tests statistics back to the Console.2) When a given test run completes, the log has no summary table of tests results.Suggestions on debugging so far:==========================1) This problem appears to be related to the Worker Processors ability to communicate to the Console - no tests stats visible in the Console GUI - no stats available when probing the REST console service for stats. Thread dump? I don't see what the thread dump will tell me about the lack of tests statistics making their way back to the console. IMHO, tcpdump would perhaps be more useful here? Open to suggestions.2) What trigger would cause the Work Processor to not correctly create a summary table at the end of the log. The log is populated with all the test scripts data checks etc., the data log is correctly populated. I have no idea where to start here. I am not afraid to review the code, but before I invest the time I thought it would be worth asking for some recommendations first.Let me know if you require any further details.Kind Rgds,OurayOn Thu, Feb 7, 2013 at 11:41 AM, Ouray Viney <ouray <at> viney.ca> wrote:Hi Philip,I have doubts on that. Here is why. I ran the problematic script, observed the issues. Change "grinder.script = problematic.py" to "grinder.script = console.py". Ran and saw absolutely no issues - Console and worker processors worked as expected.I have this same setup with the problematic script working in my Ubuntu VM - no issues.I will take your advice and generate a dump of the WorkerProcessor JVM to see what is is showing. As well, I guess it might be worth doing a tcpdump of the WorkProcessor to Console traffic on port 6372.Thank you for your time and advice, greatly appreciated.Kind Rgds,OurayOn Thu, Feb 7, 2013 at 10:44 AM, Philip Aston <philipa <at> mail.com> wrote:Might be a firewall thing?
The agent reports back to the console over the connection it creates when it starts up. The worker processes create their own connections.
On 07/02/13 07:36, Ouray Viney wrote:
Hi Philip,My comments inline
On 2013-02-07, at 2:18, Philip Aston <philipa-O5WfVfzUwx8@public.gmane.org> wrote:
If the worker process isn't logging any tests (I assume its data file is similarly empty), then they'll be no statistics to report back to the console, however it should at least have reported "worker X (m/n threads)".
Los are fine at this point. Exactly as they should be.
What's at the start of the worker process log, does it successfully load an instrumenter?
I will check this tomorrow, but assume yes since the logs are populated correctly.
It smells like the worker threads aren't running correctly. Does the worker process log contain any errors? Are the worker threads forcibly terminated when the worker process is stopped - perhaps they're blocked in the Java code under test? Maybe take a thread dump of the worker process?
No clear sign of anything blocking. No related errors. Console can see the agents, just not the worker bees .
On 07/02/13 07:09, Ouray Viney wrote:
Hi All,I have nailed this issue down to a particular script. This same script works fine in my dev. environment. However, when I try to run the same script in a our perf. environment it causes the communication between the Console and WorkProcessors. This script is not using the http recording model, rather a Java based test, instrumenting the Java fat client API. Evening though the Console doesn't report any stats. The Processes tab doesn't show the running threads, only the Agent process. If I simply swap out the script for a sample script, things also run as expected. This is what leads me to believe that the issue is being caused by the script (which used to work fine).If I run a sample script with the same grinder.properties, the sample scripts behaves correctly i.e. the Console is showing stats from the WorkerProcessors.I am getting close to having exhausted all possible things to try.Any help would be appreciatedOuray VineyOn Thu, Feb 7, 2013 at 1:19 AM, Ouray Viney <ouray <at> viney.ca> wrote:Hi All,OK, so after a bunch of testing/debugging I believe this particular issue that I described has been isolated to a particular environment. I am not able to reproduce the issues in my dev Ubuntu VM.If I can figure out what the issues is in my environment, I'll post the details. What was interesting in all my testing was the fact that a sample script from the examples dir ran fine and gave the desired behavior in the Console.Sorry for the verbose chatter, got a little frustrated today with this one =).Kind Rgds,OurayOn Wed, Feb 6, 2013 at 3:06 PM, Ouray Viney <ouray <at> viney.ca> wrote:Another interesting fact is that the WorkProcessor log doesn't contain the summary table at the end of the log as it usually does:
2013-02-06 14:51:48,445 INFO 1128450-001-0 : elapsed time is 610319 ms2013-02-06 14:51:48,445 INFO 1128450-001-0 : Final statistics for this process:2013-02-06 14:51:48,457 INFO 1128450-001-0 :Tests Errors Mean Test Test Time TPSTime (ms) StandardDeviation(ms)Totals 0 0 - 0.00 0.00Tests resulting in error only contribute to the Errors column.Statistics for individual tests can be found in the data file, including(possibly incomplete) statistics for erroneous tests. Composite testsare marked with () and not included in the totals.On Wed, Feb 6, 2013 at 2:09 PM, Ouray Viney <ouray <at> viney.ca> wrote:
Hi All,What might be causing the WorkProcessors (child proc to the Agent) when running on a remote system to not correctly report back to the Console Service. I am seeing the agents, but when I start the WorkProcessors the Console doesn't see them when I request status on the agents. The call returns the status of the agents, but doesn't show any relevant information about the running WorkerProcessors. I did validate the a child process did spawn.Is there a particular Console setting/grinder.property that I need to set to allow the WorkProcessors (when running from a remote Grinder Agent) to communicate with the Grinder Console.Environment details:==================- separate host for the console service- 9 separate servers hosting two running agents- When my test starts, two WorkerProcessors are invoked.Other Details:============- I can see the WorkerProcess logs shows the test is running correctly.- When I query the Console for--
------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________ grinder-use mailing list grinder-use@... https://lists.sourceforge.net/lists/listinfo/grinder-use