headers
Wawrzyniec Niewodniczanski | 5 Aug 2009 10:36
Favicon

URLs from Apache log

Hi,

I  think that one of the obvious usage of tool like Grinder would be 
having a script reading data from a production apache (other httpd 
server) logs to create a same same set of request on test box(s). Maybe 
I'm a bit blind but I couldn't find such example in your script gallery.

Anyway it's not so hard to write such thing beside I have some 
additional issues. The file is rather big (apache log 14 GB and cut 
version - only URL - 1.2 GB), so it's not ideal to read such files in 
one move. So, I decided to make it incremental, but I wanted to make 
additional step and results of each chunk of data to be save as a 
separate test. I wrote the following script, which 'nearly works' and 
falls with following error message:

8/4/09 7:08:17 PM (thread 0 run 0 test 1): Aborted run due to Java 
exception calling TestRunner
Java exception calling TestRunner
        File "/home/grinder/projects/b_stress_test.py", line 22, in __call__
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
[...]

The tail of out log file indicates that the first test has been finished:

[...]
  Tests resulting in error only contribute to the Errors column.         
  Statistics for individual tests can be found in the data file, including
  (possibly incomplete) statistics for erroneous tests. Composite tests  
  are marked with () and not included in the totals.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Tuvell_Walter | 5 Aug 2009 15:53

What does it mean when Grinder threads die?

During multi-day tests, I sometimes see Grinder threads die.  What does
it mean?

For example, yesterday I started a test with grinder.threads=12.  This
morning, after running overnight, only 1 thread ("Grinder thread 5") was
still alive, as shown both by the Grinder GUI ("Running 1/12 threads"),
as well as the attached jstack dump (which shows nothing else
suspicious, insofar as I can tell).  The one live thread was happily
going about its business (sending/receiving tests to the server), and no
problems were posted in the Grinder GUI or in the logfile.

In fact during this test, there were 4 Agent machines.  Besides the one
just discussed, another one had 10 threads still running (2 died), while
the other 2 had all 12 threads running.  And the Console machine seemed
just fine, too.

What's happening?

 <<deadThreads.txt>> 


2009-08-05 08:21:12
Full thread dump IcedTea 64-Bit Server VM (1.7.0-b21 mixed mode):

"Attach Listener" daemon prio=10 tid=0x000000000097c400 nid=0x2de4 waiting on condition [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"SocketTimeout" daemon prio=10 tid=0x0000000002516400 nid=0x2273 waiting on condition [0x0000000040a7d000..0x0000000040a7db90]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
(Continue reading)

Permalink | Reply | 6 comments |
headers
Shayne | 5 Aug 2009 18:27
Picon

Re: URLs from Apache log

I don't see anything is wrong with your script.  So, I think it maybe a networking error or url formatting error.  Here are 2 things I would try:
1) Make a very simple test script that does a GET on a hand typed URL from your site.  This will prove that grinder is communicating with your site.
2) Try a simple print statement to verify the URLs are being formatted correctly:

for l in imageslist:
     print myurl+l
     response = request.GET(l)



On Wed, Aug 5, 2009 at 2:36 AM, Wawrzyniec Niewodniczanski <wawrzek.niewodniczanski-8JvEc4dgHbZBDgjK7y7TUQ@public.gmane.org> wrote:
Hi,

I  think that one of the obvious usage of tool like Grinder would be
having a script reading data from a production apache (other httpd
server) logs to create a same same set of request on test box(s). Maybe
I'm a bit blind but I couldn't find such example in your script gallery.

Anyway it's not so hard to write such thing beside I have some
additional issues. The file is rather big (apache log 14 GB and cut
version - only URL - 1.2 GB), so it's not ideal to read such files in
one move. So, I decided to make it incremental, but I wanted to make
additional step and results of each chunk of data to be save as a
separate test. I wrote the following script, which 'nearly works' and
falls with following error message:

8/4/09 7:08:17 PM (thread 0 run 0 test 1): Aborted run due to Java
exception calling TestRunner
Java exception calling TestRunner
       File "/home/grinder/projects/b_stress_test.py", line 22, in __call__
Caused by: java.net.NoRouteToHostException: No route to host
       at java.net.PlainSocketImpl.socketConnect(Native Method)
[...]

The tail of out log file indicates that the first test has been finished:

[...]
 Tests resulting in error only contribute to the Errors column.
 Statistics for individual tests can be found in the data file, including
 (possibly incomplete) statistics for erroneous tests. Composite tests
 are marked with () and not included in the totals.
8/4/09 7:08:18 PM (process dc01gen-01.dqs.lhr1.booking.com-0): finished


What wrong with my script?

Regards,
Wawrzek

from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair

#
class TestRunner:
       def __call__(self):
               myurl='http://test-static.booking.com'
               request= HTTPRequest(url=myurl)
               chunk=0
               f = open('/home/grinder/sourcedata/bo_access.reqs', 'r')
               while 1:
                       imageslist=[w.rstrip('\n') for w in
f.readlines(2**24)]
                       if not imageslist:
                               break
                       else:
                               chunk = chunk+1
                               print ("chunk=%05d  size=%d"
%(chunk,len(imageslist)))
                               request= Test(chunk, 'Stress test %03d'
% chunk).wrap(request)
                               for l in imageslist:
                                       response = request.GET(l)

               f.close()

       def __init__(self):
               connectionDefaults =
HTTPPluginControl.getConnectionDefaults()
               httpUtilities = HTTPPluginControl.getHTTPUtilities()
               log = grinder.logger.output

               connectionDefaults.defaultHeaders = (
NVPair('Accept-Language', 'en-gb,en;q=0.5'), NVPair('Host',
'static.booking.com'), )

--
Dr Wawrzyniec Niewodniczański
Unix System Administrator at Booking.com,  Cambridge Office
tel: +44 02086 128 028 email: wawrzek.niewodniczanski-8JvEc4dgHbZBDgjK7y7TUQ@public.gmane.org
PhD in Quantum Chemistry, MSc in Material (Molecular) Science



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use-5NWGOfrQmnd4wTydcyPnfg@public.gmane.orgceforge.net
https://lists.sourceforge.net/lists/listinfo/grinder-use

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use
Permalink | Reply | 0 comments |
headers
Ashwini Nambiar | 7 Aug 2009 00:40
Favicon

Grinder for swing app

Is it possible to use Grinder to performance test the back end of  a swing application? The client-server communication is via http and uses serialized java objects?

Thanks,
Ashwini 
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use
Permalink | Reply | 503 comments |
headers
Trilok Khairnar | 7 Aug 2009 15:03
Picon

Re: Grinder for swing app

Short answer is yes. Alternatively, you can use a Java aware proxy
like Rabbit in the middle to handle the serialization,
deserialization.
Phil's answer in the following snippet from the archives should help.

-Trilok

<snippet>
---------- Forwarded message ----------
From: Philip Aston <paston <at> bea.com>
Date: Tue, Mar 6, 2007 at 4:14 AM
Subject: Re: [Grinder-use] Java thick-client applications serialised objects
To: grinder-use <grinder-use <at> lists.sourceforge.net>

Terence Walsh wrote:
>
>     Using "The Grinder 3" now, and have a project that is java human
>     machine interface hooked up to a server but passing data through
>     serialisation of objects over HTTPS.
>
>     What is the best approach to this.
>
>     My thoughts are the following:
>
>     Read through the development code and take out the references
>     (methods) where the connection's are occuring reimplement them in
>     jython and hit the server that way by calling the local methods.
>     Obviously i dont care about the interface etc so id need to just
>     call the methods relating to comms with the server.
>
>     Anyone have any better ideas / suggestions ?
>

Well, it sounds like you have a custom request/reply protocol built on
HTTP. Presumably you can do something like:

requestParameter = ... # create parameter using Java classes
requestBytes = serialise(requestParameter)

response = myRequest.POST(url, requestBytes) # Using HTTPRequest
responseObject = deserialise(response.data)

where serialise() and deserialise() would use Java ObjectOutputStream,
ObjectInputStream to do Java serialisation to and from byte arrays.

- Phil

</snippet>

On Fri, Aug 7, 2009 at 4:10 AM, Ashwini
Nambiar<ashwini.nambiar <at> pearson.com> wrote:
> Is it possible to use Grinder to performance test the back end of  a swing
> application? The client-server communication is via http and uses serialized
> java objects?
>
> Thanks,
> Ashwini
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> grinder-use mailing list
> grinder-use <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/grinder-use
>
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/grinder-use
Permalink | Reply | 503 comments |
headers
Paul de Lucena | 11 Aug 2009 19:51
Picon

Grinder/JSON and Character Encoding

I am attempting to post a JSON string in UTF-8 and running into character encoding errors that I believe indicate missing packages in the Jython that ships with The Grinder. I use JYSON to create the the JSON string:


from com.xhaus.jyson import JysonCodec as json
...
postData = '{"forumId":"'+pk+'","since":null,"bodies":false,"headers":true,"linkedEntryId":null,"inCourseEntryId":"'+self.dbPk+'"}'
jsonData = json.dumps(postData)
encodedData = jsonData.encode('utf-8')

The last line ends up throwing:

8/11/09 1:38:01 PM (thread 0 run 0 test 404): Aborted run due to Jython exception: LookupError: no codec search functions registered: can't find encoding [calling TestRunner]
LookupError: no codec search functions registered: can't find encoding

The code itself is from the examples on the JYSON website, but it is standard python - I've tried spelling the encoding a few different ways since there are a few aliases in the documentation, but it has made no difference. Using jsonData.encode('ascii') works, but the page expects an encoded string.

Any idea how to fix the issue? What packages to add or what flags to set where? I've already replaced the Jython jar with a full download (same file size as the pre-existing jar). Any assistance is much appreciated.

Thanks in advance.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use
Permalink | Reply | 2 comments |
headers
Ashwini Nambiar | 11 Aug 2009 23:17
Favicon

NullPointerException

On starting TCP proxy and then setting up the browser to use the proxy, get following error on loading a site in the browser:

8/11/09 2:13:15 PM (tcpproxy): Initialising as an HTTP/HTTPS proxy with the     
parameters:                                                                     
   Request filters:    HTTPRequestFilter                                        
   Response filters:   HTTPResponseFilter                                       
   Local address:      localhost:8001                                           
8/11/09 2:13:16 PM (tcpproxy): Engine initialised, listening on port 8001       
Exception in thread "Filter thread for ...snip snip...:80->localhost:53811" java.lang.NullPointerException
    at net.grinder.plugin.http.tcpproxyfilter.ConnectionHandlerImplementation$Request.expectingResponseBody(ConnectionHandlerImplementation.java:396)
    at net.grinder.plugin.http.tcpproxyfilter.ConnectionHandlerImplementation.handleResponse(ConnectionHandlerImplementation.java:273)
    at net.grinder.plugin.http.tcpproxyfilter.ConnectionCache.response(ConnectionCache.java:111)
    at net.grinder.plugin.http.tcpproxyfilter.HTTPResponseFilter.handle(HTTPResponseFilter.java:71)
    at net.grinder.tools.tcpproxy.CompositeFilter.handle(CompositeFilter.java:62)
    at net.grinder.tools.tcpproxy.AbstractTCPProxyEngine$OutputStreamFilterTee.handle(AbstractTCPProxyEngine.java:547)
    at net.grinder.tools.tcpproxy.AbstractTCPProxyEngine$FilteredStreamThread.interruptibleRun(AbstractTCPProxyEngine.java:416)
    at net.grinder.util.thread.InterruptibleRunnableAdapter.run(InterruptibleRunnableAdapter.java:59)
    at java.lang.Thread.run(Thread.java:613)


Any ideas?

Thanks,


Ashwini Nambiar
Software Engineer
School Systems
Pearson
http://www.pearson.com
(916) 288 1749





 
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use
Permalink | Reply | 501 comments |
headers
Juergen Weber | 12 Aug 2009 10:02
Picon

Re set Console Data in 3.2


Hi,

it looks like in 3.2 you can no longer reset the console data/statistics. Is
that true or is there a better work-around than restarting the console?

Thanks,
Juergen
--

-- 
View this message in context: http://www.nabble.com/Reset-Console-Data-in-3.2-tp24931665p24931665.html
Sent from the Grinder - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
Permalink | Reply | 11 comments |
headers
Philip Aston | 12 Aug 2009 11:01
Picon
Gravatar

Re: Re set Console Data in 3.2

Juergen Weber wrote:
> Hi,
>
> it looks like in 3.2 you can no longer reset the console data/statistics. Is
> that true or is there a better work-around than restarting the console?
>
> Thanks,
> Juergen

Nothing's changed here. Perhaps you should set "Reset Console with
Worker Processes" in the console options.

- Phil

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
Permalink | Reply | 10 comments |
headers
Philip Aston | 12 Aug 2009 10:34
Picon
Gravatar

Re: NullPointerException

This looks like the TCP Proxy is receiving a response before a request
has been sent!

Can you repeat this without "-http" (so the TCP Proxy will just echo the
requets and response streams) and post the resulting output.

Thanks,

- Phil

Ashwini Nambiar wrote:
> On starting TCP proxy and then setting up the browser to use the
> proxy, get following error on loading a site in the browser:
>
> 8/11/09 2:13:15 PM (tcpproxy): Initialising as an HTTP/HTTPS proxy
> with the     
> parameters:
>                                                                     
>    Request filters:    HTTPRequestFilter
>                                        
>    Response filters:   HTTPResponseFilter
>                                       
>    Local address:      localhost:8001
>                                           
> 8/11/09 2:13:16 PM (tcpproxy): Engine initialised, listening on port
> 8001       
> Exception in thread "Filter thread for ...snip
> snip...:80->localhost:53811" java.lang.NullPointerException
>     at
> net.grinder.plugin.http.tcpproxyfilter.ConnectionHandlerImplementation$Request.expectingResponseBody(ConnectionHandlerImplementation.java:396)
>     at
> net.grinder.plugin.http.tcpproxyfilter.ConnectionHandlerImplementation.handleResponse(ConnectionHandlerImplementation.java:273)
>     at
> net.grinder.plugin.http.tcpproxyfilter.ConnectionCache.response(ConnectionCache.java:111)
>     at
> net.grinder.plugin.http.tcpproxyfilter.HTTPResponseFilter.handle(HTTPResponseFilter.java:71)
>     at
> net.grinder.tools.tcpproxy.CompositeFilter.handle(CompositeFilter.java:62)
>     at
> net.grinder.tools.tcpproxy.AbstractTCPProxyEngine$OutputStreamFilterTee.handle(AbstractTCPProxyEngine.java:547)
>     at
> net.grinder.tools.tcpproxy.AbstractTCPProxyEngine$FilteredStreamThread.interruptibleRun(AbstractTCPProxyEngine.java:416)
>     at
> net.grinder.util.thread.InterruptibleRunnableAdapter.run(InterruptibleRunnableAdapter.java:59)
>     at java.lang.Thread.run(Thread.java:613)
>
>
> Any ideas?
>
> Thanks,
> *
> **/
> /Ashwini Nambiar
> *Software Engineer
> School Systems
> Pearson
> http://www.pearson.com
> (916) 288 1749
> /
>
>
> /

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
Permalink | Reply | 501 comments |

Gmane