I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message -------- Subject: Date: From: To: CC:

Re: Passing time factor to tests run under jtreg

Tue, 15 Nov 2011 22:45:03 +0000

Alan Bateman <Alan.Bateman-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

gary Adams <Gary.Adams-veTT2BtV2gBXrIkS9f7CXA@public.gmane.org>

core-libs-dev-0nJqbsLSQw0FDOXUYO6UHQ@public.gmane.org

Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: > I've been scanning a number of the slow machine test > bugs that are reported and wanted to check to see if > anyone has looked into time dependencies in the regression > tests previously. From what I've been able to learn so far > individual bugs can use the "timeout" parameter to indicate to > the test harness an expected time to run. > > The test harness has command line arguments where it can > filter out tests that take too long (timelimit) or can apply a > multiplier to > to the timeout when conditions are known to slow down the process > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > I see that there are some wrappers that can be applied around running > a particular test to allow processing before main(). Could this mechanism > be exploited so the harness command line options could be made known > to the time dependent tests as command line arguments or as system > properties? > > My thought is the current timeout granularity is too large and only > applies > to the full test execution. If a test knew that a timeoutFactor was to > be applied, > it could internally adjust the time dependent delays appropriately. e.g. > not every sleep(), await(), join() with timeouts would need the > timeoutFactor > applied. > > Before any test could be updated the information would need to be > available > from the test context. > > Any feedback/pointers appreciated! > > > See > timeoutFactorArg > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > runOtherJVM() > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > maxTimeoutValue > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > >

The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds a -timefactor:2

option I think, you could have the test/Makefile detect a slower system and adjust the timefactor on the

command line there.  Or is that what you are suggesting?

I know it's a bit crude to use Makefiles as an interface for testing, but there is a big advantage to

JPRT, Nightly testing, and developers being able to run the tests the exact same way.

-kto

On Nov 16, 2011, at 5:08 AM, Gary Adams wrote:

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message -------- Subject: Date: From: To: CC:

Re: Passing time factor to tests run under jtreg

Tue, 15 Nov 2011 22:45:03 +0000

Alan Bateman <Alan.Bateman-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

gary Adams <Gary.Adams-veTT2BtV2gBXrIkS9f7CXA@public.gmane.org>

core-libs-dev-0nJqbsLSQw0FDOXUYO6UHQ@public.gmane.org

Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: > I've been scanning a number of the slow machine test > bugs that are reported and wanted to check to see if > anyone has looked into time dependencies in the regression > tests previously. From what I've been able to learn so far > individual bugs can use the "timeout" parameter to indicate to > the test harness an expected time to run. > > The test harness has command line arguments where it can > filter out tests that take too long (timelimit) or can apply a > multiplier to > to the timeout when conditions are known to slow down the process > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > I see that there are some wrappers that can be applied around running > a particular test to allow processing before main(). Could this mechanism > be exploited so the harness command line options could be made known > to the time dependent tests as command line arguments or as system > properties? > > My thought is the current timeout granularity is too large and only > applies > to the full test execution. If a test knew that a timeoutFactor was to > be applied, > it could internally adjust the time dependent delays appropriately. e.g. > not every sleep(), await(), join() with timeouts would need the > timeoutFactor > applied. > > Before any test could be updated the information would need to be > available > from the test context. > > Any feedback/pointers appreciated! > > > See > timeoutFactorArg > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > runOtherJVM() > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > maxTimeoutValue > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > >

I'm not concerned how the timefactor is set.
It could be from makefiles or could be from
developer running jtreg from the command line.

What I'm talking about is the actual test code
being informed that the timefactor is being applied.
e.g. communicated from the test harness to the test case

A typical test using timeouts internally has some expectation
how long particular tasks will take. It might  use 5 seconds for
threads to be set up and then 1 second delay while each
thread is shut down.

An overall timefactor setting will still catch the runaway test,
but in some cases the test will terminate prematurely because
individual steps were not alloted sufficient time.



On 11/16/11 11:21 AM, Kelly O'Hair wrote:

The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds a -timefactor:2

option I think, you could have the test/Makefile detect a slower system and adjust the timefactor on the

command line there.  Or is that what you are suggesting?

I know it's a bit crude to use Makefiles as an interface for testing, but there is a big advantage to

JPRT, Nightly testing, and developers being able to run the tests the exact same way.

-kto

On Nov 16, 2011, at 5:08 AM, Gary Adams wrote:

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message -------- Subject: Date: From: To: CC:

Re: Passing time factor to tests run under jtreg

Tue, 15 Nov 2011 22:45:03 +0000

Alan Bateman <Alan.Bateman-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

gary Adams <Gary.Adams-veTT2BtV2gBXrIkS9f7CXA@public.gmane.org>

core-libs-dev-0nJqbsLSQw0FDOXUYO6UHQ@public.gmane.org

Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: > I've been scanning a number of the slow machine test > bugs that are reported and wanted to check to see if > anyone has looked into time dependencies in the regression > tests previously. From what I've been able to learn so far > individual bugs can use the "timeout" parameter to indicate to > the test harness an expected time to run. > > The test harness has command line arguments where it can > filter out tests that take too long (timelimit) or can apply a > multiplier to > to the timeout when conditions are known to slow down the process > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > I see that there are some wrappers that can be applied around running > a particular test to allow processing before main(). Could this mechanism > be exploited so the harness command line options could be made known > to the time dependent tests as command line arguments or as system > properties? > > My thought is the current timeout granularity is too large and only > applies > to the full test execution. If a test knew that a timeoutFactor was to > be applied, > it could internally adjust the time dependent delays appropriately. e.g. > not every sleep(), await(), join() with timeouts would need the > timeoutFactor > applied. > > Before any test could be updated the information would need to be > available > from the test context. > > Any feedback/pointers appreciated! > > > See > timeoutFactorArg > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > runOtherJVM() > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > maxTimeoutValue > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > >

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message -------- Subject: Date: From: To: CC:

Re: Passing time factor to tests run under jtreg

Tue, 15 Nov 2011 22:45:03 +0000

Alan Bateman <Alan.Bateman-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

gary Adams <Gary.Adams-veTT2BtV2gBXrIkS9f7CXA@public.gmane.org>

core-libs-dev-0nJqbsLSQw0FDOXUYO6UHQ@public.gmane.org

Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: > I've been scanning a number of the slow machine test > bugs that are reported and wanted to check to see if > anyone has looked into time dependencies in the regression > tests previously. From what I've been able to learn so far > individual bugs can use the "timeout" parameter to indicate to > the test harness an expected time to run. > > The test harness has command line arguments where it can > filter out tests that take too long (timelimit) or can apply a > multiplier to > to the timeout when conditions are known to slow down the process > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > I see that there are some wrappers that can be applied around running > a particular test to allow processing before main(). Could this mechanism > be exploited so the harness command line options could be made known > to the time dependent tests as command line arguments or as system > properties? > > My thought is the current timeout granularity is too large and only > applies > to the full test execution. If a test knew that a timeoutFactor was to > be applied, > it could internally adjust the time dependent delays appropriately. e.g. > not every sleep(), await(), join() with timeouts would need the > timeoutFactor > applied. > > Before any test could be updated the information would need to be > available > from the test context. > > Any feedback/pointers appreciated! > > > See > timeoutFactorArg > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > runOtherJVM() > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > maxTimeoutValue > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > >

Hi Jon,
   I don't think it is as simple as designating a length of time
a test is allowed to run. In the scenario, I mentioned below
a test needed 5 seconds to perform it's main task and 1 second to
clean up. Internally both operations were manged with separate
timeouts.

  The jtreg harness allows for timefactor=2 to allow for 2*(5+1)
or 12 seconds to complete, but the test as written really needs
(2*5) + (2*1) seconds to run correctly. e.g. it has to allow for 2 seconds
in the cleanup on the slower machine.

  If we can provide the (6 seconds * 2 timefactor) to the test it
could divide up the time between subtasks, but that may be a fairly
complicated computation over a large number of tests currently using
hard coded timeouts.

Gary
======= With respect, my sense is that this is somewhat misdirected micromanagement. It's not the time factor that the test should know, but the actual allotted time. Given a certain period of time, a test could breakdown the allotted time into intervals for each step of its processing. Would that work for what you have in mind? -- Jon On 11/16/11 8:08 AM, Gary Adams wrote:

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message -------- Subject: Date: From: To: CC:

Re: Passing time factor to tests run under jtreg

Tue, 15 Nov 2011 22:45:03 +0000

Alan Bateman <Alan.Bateman-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

gary Adams <Gary.Adams-veTT2BtV2gBXrIkS9f7CXA@public.gmane.org>

core-libs-dev-0nJqbsLSQw0FDOXUYO6UHQ@public.gmane.org

Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: > I've been scanning a number of the slow machine test > bugs that are reported and wanted to check to see if > anyone has looked into time dependencies in the regression > tests previously. From what I've been able to learn so far > individual bugs can use the "timeout" parameter to indicate to > the test harness an expected time to run. > > The test harness has command line arguments where it can > filter out tests that take too long (timelimit) or can apply a > multiplier to > to the timeout when conditions are known to slow down the process > (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp > > I see that there are some wrappers that can be applied around running > a particular test to allow processing before main(). Could this mechanism > be exploited so the harness command line options could be made known > to the time dependent tests as command line arguments or as system > properties? > > My thought is the current timeout granularity is too large and only > applies > to the full test execution. If a test knew that a timeoutFactor was to > be applied, > it could internally adjust the time dependent delays appropriately. e.g. > not every sleep(), await(), join() with timeouts would need the > timeoutFactor > applied. > > Before any test could be updated the information would need to be > available > from the test context. > > Any feedback/pointers appreciated! > > > See > timeoutFactorArg > jtreg/src/share/classes/com/sun/javatest/regtest/Main.java > runOtherJVM() > jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java > maxTimeoutValue > > jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java > >