Nathan Jones | 18 Dec 01:02 2014

[Cucumber] Detect unreliable tests

I would like to be able to detect unreliable Selenium tests that are being executed in my Cucumber test suite. I realise the "rerun" output format can be used to avoid unreliable tests causing a test run to fail (see [1] and [2]) but I would like to take this a step further so that unreliable tests can be identified in the report.

My proposal is to add a new "--retries" argument to cucumber which causes failing scenarios to be immediately retried. The json report would then include a "retries" property with each step result, for example:

{
  "result": {
    "duration": 233027,
    "status": "passed",
    "retries": 2
  },
  "match": {
    "location": "features/step_definitions/calulator_steps.rb:1",
    "arguments": [
      {
        "val": "70",
        "offset": 15
      }
    ]
  },
  "line": 8,
  "name": "I have entered 70 into the calculator",
  "keyword": "And "
}

A non-zero retries property on a step result indicates the test is unreliable.

What do you think about this proposal? Does something like this already exist? Is there value in me implementing this?

 - Nathan

[1]: http://blog.crowdint.com/2011/08/22/auto-retry-failed-cucumber-tests.html
[2]: http://blog.mattheworiordan.com/post/42587188567/automatically-retrying-failing-non-deterministic


------------  Our Legal Fees At Work -----------------------------------
This communication and its contents are confidential and may contain protected intellectual property or other information protected from disclosure. If you are not the intended recipient you may not read, copy, distribute or use this information. If you have received this communication or its contents in error, please notify Findly immediately by replying to this message and then delete this communication and its contents from your system.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Kirill Zhukov | 17 Dec 20:57 2014
Picon

[Cucumber] How to measure time running of each step?

I need to measure how much time it takes to run each step in scenarios. How would I do that in Cucumber? I didn't find Before and After Step hooks, which I would use...
Thanks.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Celina | 17 Dec 08:18 2014
Picon

[Cucumber] TypeError: Cannot call method 'id' of undefined

Hello,
I am a newbie to cucumber.js+protractor.  I am struggling to make my first test work. 

 my world.js looks like:
        var protractor = require('protractor');
var webdriver = require('selenium-webdriver');
var driver = new webdriver.Builder().usingServer('http://localhost:4444/wd/hub').
withCapabilities(webdriver.Capabilities.firefox()).build();

driver.manage().timeouts().setScriptTimeout(100000);
  

module.exports.World = function World(callback) {
this.browser = protractor.wrapDriver(driver);
this.by=protractor.by;
 
callback();
};

then in steps.js:
..................
element(by.id('username')).sendKeys("admin");
...............

When I ran it, the error is:
TypeError: Cannot call method 'id' of undefined

Could anyone kindly give me some suggestion?

Thanks a lot!

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Niranjala Senaratne | 16 Dec 16:00 2014
Picon

[Cucumber] Can anyone give me some basic cucumber stats like market share, following etc

I have been searching the web to find some statistics around Cucumber. 

I had some very basic questions like 
1. What is the current following for cucumber in the bdd world 
2. What is the current market share 
3. Whats going to be new next 5 years down 
4. In addition to the forums what else do you have in terms of support

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Biswajit Samal | 16 Dec 11:59 2014
Picon

[Cucumber] Unable to Close Browser when using Shared Driver

Respected All,

I am facing an issue, I have followed several blogs but unable to fix this issue. I can close browser with out using SharedDriver.java, But when I want to use Page object model then I have to use SharedDriver.Java where the problem comes.

There is only one browser is opening but closing the browser is not working.

Below is the snippets of my code. Please rectify me :-

1. LoginStepDef :-

public class LoginStepDef {
public WebDriver driver;
public LoginStepDef(SharedDriver driver){
this.driver = driver;
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
}
<at> Given("^the login page is open \"(.*?)\"$")
public void the_login_page_is_open(String page) throws Throwable {
   // Write code here that turns the phrase above into concrete actions
driver.get(page);
  
}

<at> When("^I fill in \"(.*?)\" with \"(.*?)\"$")
public void i_fill_in_with(String field, String value) throws Throwable {
   // Write code here that turns the phrase above into concrete actions
driver.findElement(By.id(field)).sendKeys(value);   
}
<at> When("^I click on \"(.*?)\" button$")
public void i_click_on_button(String SignInButton) throws Throwable {
   // Write code here that turns the phrase above into concrete actions
driver.findElement(By.id(SignInButton)).click(); 
}

<at> Then("^I should see the result$")
public void i_should_see_the_result() throws Throwable {
   // Write code here that turns the phrase above into concrete actions
Assert.assertEquals(driver.getTitle(), "Sign in - Google Accounts");
}


<at> When("^I am in the when step$")
public void i_am_in_the_when_step() throws Throwable {
   // Write code here that turns the phrase above into concrete actions
System.out.println("I am in the When for SampleStepDef");
}


2. RunCukesByCompositionTest:-

<at> CucumberOptions(monochrome = true ,format = {"pretty", "html:target/cucumber", "json:target/cucumber.json"})
public class RunCukesByCompositionTest extends RunCukesByCompositionBase{
<at> Test(groups = "examples-testng",description = "Example of using TestNGCucumberRunner to invoke Cucumber")
public void runCukes() {
new TestNGCucumberRunner(getClass()).runCukes();
}

}

3. RunCukesTest:-

<at> CucumberOptions(features = "Search",format = {"pretty", "html:target/cucumber", "json:target/cucumber.json"})
public class RunCukesTest extends AbstractTestNGCucumberTests{

}


4. SharedDriver :-

public class SharedDriver extends EventFiringWebDriver{
private static final WebDriver REAL_DRIVER = new FirefoxDriver();
private static final Thread CLOSE_THREAD = new Thread() {
<at> Override
public void run() {
REAL_DRIVER.close();
}
};

static {
Runtime.getRuntime().addShutdownHook(CLOSE_THREAD);
}

public SharedDriver() {
super(REAL_DRIVER);
// REAL_DRIVER.close();
// TODO Auto-generated constructor stub
}

<at> Override
public void close() {
if (Thread.currentThread() != CLOSE_THREAD) {
throw new UnsupportedOperationException(
"You shouldn't close this WebDriver. It's shared and will close when the JVM exits.");
}
super.close();
}

<at> Before
public void deleteAllCookies() {
manage().deleteAllCookies();
}

<at> After
public void embedScreenshot(Scenario scenario) {
try {
byte[] screenshot = getScreenshotAs(OutputType.BYTES);
scenario.embed(screenshot, "image/png");
} catch (WebDriverException somePlatformsDontSupportScreenshots) {
System.err
.println(somePlatformsDontSupportScreenshots.getMessage());
}
}




--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Rakesh | 16 Dec 11:09 2014
Picon

[Cucumber] Re: Cross platform mobile testing



On Tue Dec 16 2014 at 10:08:34 AM Rakesh Patel <patel.rakesh-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Hi,

when it comes to Android, my plan is to cucumber-jvm with a glue layer using espresso.

I am against using appium as it will encourage end to end tests which are flaky as hell right now on Android. Not to mention the disadvantages of maintenance and using cucumber in this way *wrong*.

However, what about iOS testing? Does it have the equivalent of Espresso/ActivityInstrimentationTestCase2? Can it only be achieved with third party libraries like Appium/Calabash?

Regards

Rakesh

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
A R K Satyanarayana Raju | 13 Dec 12:29 2014
Picon

[Cucumber] I am getting Cucumber-JVM's --format option is deprecated. error

Hi,

I am very new to cucumber. When i am trying to execute my feature file i am getting error like,

WARNING: Cucumber-JVM's --format option is deprecated. Please use --plugin instead.
Feature: Proof of concept that my framework worksException in thread "main" java.lang.NoSuchMethodError: gherkin.formatter.model.Scenario.getId()Ljava/lang/String;
at cucumber.runtime.ScenarioImpl.<init>(ScenarioImpl.java:28)
at cucumber.runtime.Runtime.buildBackendWorlds(Runtime.java:144)
at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:38)
at cucumber.runtime.model.CucumberFeature.run(CucumberFeature.java:163)
at cucumber.runtime.Runtime.run(Runtime.java:120)
at cucumber.api.cli.Main.run(Main.java:36)
at cucumber.api.cli.Main.main(Main.java:18)

My feature file is:
Feature: Proof of concept that my framework works

  Scenario: My first test
    Given this is my first step
    When this is my second step
    Then this is my final step

I am using JDK1.8.

Can any one help me...

Thanks,
Raju

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
ziffification | 12 Dec 22:04 2014
Picon

[Cucumber] Newbie question starting off cunfused

Hi All


was recommended to CUCUMBER by a colleague , who knows I have some web and rest test scripts written in python and selenium.

Now these tests arent really atomic at the moment so I will try and get them that way I.E. just test the JSON output from a web service instead of iterating through all and 
testing if valid json is found. 

other scripts ...the ones using selenium do things like fire up a browser IE/CHROME/FIREFOX depending on command line params and then proceed to operate on the web interface of a device (running the web server)
and then do active things like cause the website to download device logs , reboot the system or upgrade the system.

So from downloading and installing ruby and cucumber as well as trying a demo I found at 

http://www.agileforall.com/2011/08/getting-started-with-ruby-cucumber-and-capybara-on-windows/

I see that in the demo 


there is a 
C:\Users\robertme\code\cucumber\test\features\step_definitions\demo_steps.rb 

When /^I search Google for "(.*)"$/ do |query|
  visit 'http://www.google.com/advanced_search?hl=en'
  fill_in 'as_q', :with => query
  click_button 'Search'
end

Then /^there should be a result for "(.*)"$/ do |expected_result|
  results = all('cite').map { |el| el.text }
  results.should include expected_result
end

and a 
C:\Users\robertme\code\cucumber\test\features\demo.features  
file

Feature: Demo
  Make sure Cucumber and Capybara are wired up properly
  
  Scenario: Search Google
    When I search Google for "cucumber"
    Then there should be a result for "cukes.info/"


and a 

C:\Users\robertme\code\cucumber\test\features\support\env.rb
require 'rubygems'
require 'capybara'
require 'capybara/dsl'
require 'rspec'

Capybara.run_server = false
Capybara.default_driver = :selenium
Capybara.default_selector = :css

module Helpers
  def without_resynchronize
    page.driver.options[:resynchronize] = false
    yield
    page.driver.options[:resynchronize] = true
  end
end

World(Capybara::DSL, Helpers)


so how would I integrate my python test code in or should i just re-write it as above in ruby?

sorry for the amophous questions I am just starting out and a little confused 



--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Rajeshwari Sudhakar | 11 Dec 17:59 2014
Picon

[Cucumber] Initialization error appears when parametrizing

Initialization error appears : 


error started when trying to parameterize in the feature file, but now it continues and am not able to do anything


cucumber.runtime.CucumberException: Failed to instantiate public cucumber.runtime.java.JavaBackend(cucumber.runtime.io.ResourceLoader) with [cucumber.runtime.io.MultiLoader <at> 1df82230]
at cucumber.runtime.Reflections.newInstance(Reflections.java:44)
at cucumber.runtime.Reflections.instantiateSubclasses(Reflections.java:30)
at cucumber.runtime.Runtime.loadBackends(Runtime.java:97)
at cucumber.runtime.Runtime.<init>(Runtime.java:64)
at cucumber.api.junit.Cucumber.createRuntime(Cucumber.java:78)
at cucumber.api.junit.Cucumber.<init>(Cucumber.java:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:29)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:21)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.<init>(JUnit4TestReference.java:33)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestClassReference.<init>(JUnit4TestClassReference.java:25)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createTest(JUnit4TestLoader.java:48)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.loadTests(JUnit4TestLoader.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:444)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at cucumber.runtime.Reflections.newInstance(Reflections.java:41)
... 23 more
Caused by: cucumber.runtime.CucumberException: Failed to instantiate public cucumber.runtime.java.guice.impl.GuiceFactory() throws java.io.IOException with []
at cucumber.runtime.Reflections.newInstance(Reflections.java:44)
at cucumber.runtime.Reflections.instantiateSubclasses(Reflections.java:30)
at cucumber.runtime.Reflections.instantiateExactlyOneSubclass(Reflections.java:16)
at cucumber.runtime.java.JavaBackend.loadObjectFactory(JavaBackend.java:65)
at cucumber.runtime.java.JavaBackend.<init>(JavaBackend.java:44)
... 28 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at cucumber.runtime.Reflections.newInstance(Reflections.java:41)
... 32 more
Caused by: java.lang.NoClassDefFoundError: com/google/inject/Scope
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$100(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at cucumber.runtime.java.guice.impl.InjectorSourceFactory.createDefaultScenarioModuleInjectorSource(InjectorSourceFactory.java:31)
at cucumber.runtime.java.guice.impl.InjectorSourceFactory.create(InjectorSourceFactory.java:24)
at cucumber.runtime.java.guice.impl.GuiceFactory.<init>(GuiceFactory.java:22)
... 37 more
Caused by: java.lang.ClassNotFoundException: com.google.inject.Scope
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 52 more





My pom.xml is as follows:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.proj.selenium</groupId>
  <artifactId>POM_Cucumber_TEST</artifactId>
  <version>0.0.1-SNAPSHOT</version>


<dependencies>
       

<!--
cucumber - junit
-->

<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.0</version>
</dependency>



<!--
cucumber - core
-->

<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-core</artifactId>
<version>1.2.0</version>
</dependency>



<!--
cucumber - java
-->

<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-java</artifactId>
<version>1.2.0</version>
</dependency>




<!--
cucumber-picocontainer
-->
<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-picocontainer</artifactId>
<version>1.2.0</version>
</dependency>



<!--
selenium-java 
-->

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>2.44.0</version>
</dependency>



<!--
 selenium-ie-driver
-->

 <dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-ie-driver</artifactId>
<version>2.44.0</version>
</dependency>



<!--
 gherkin


<dependency>
<groupId>info.cukes</groupId>
<artifactId>gherkin</artifactId>
<version>2.12.2</version>
</dependency>
-->


<!--
  cucumber-guice
-->

<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-guice</artifactId>
<version>1.2.0</version>
</dependency>





    </dependencies>


</project>

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Klaus Bayrhammer | 11 Dec 15:19 2014
Picon

[Cucumber] [JVM] Cucumber-Report-DB

Hi there,

At Porsche Informatik we have been using Cucumber-JVM extensively to specify acceptance-tests, but the more tests we added the longer it took our test-suites to complete. In addition it got more difficult to identify the steps where we would benefit from performance-optimizations, so we had problems getting feedback from our tests in acceptable time.

To tackle this problem we came up with the idea of a "report-database". We wanted to store JSON-reports from test-runs in this database so we could query and aggregate all test-results for specified versions of our products. 

We have been using this "Cucumber-Report-DB" for over a year and it made the identification of flaky, unstable or slow steps a lot easier. So we thought projects with large acceptance-test-suites could benefit from using something like this. We definitely did!

The setup of the "report-db" just requires a mongodb and java. The acceptance-tests just have to include a formatter, which publishes the results of test-runs to the database. Additionally we have extracted the view per test-run in a separate HTML-Reporter which can be used independently (without the report-db) just by adding the formatter.

You can find the source on github https://github.com/porscheinformatik/cucumber-report-db/wiki and the we have hosted a sample app with sample data on heroku https://cucumber-report-db.herokuapp.com/

Any feedback is welcome, as always.

Greetings
Klaus

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Ismael Gorissen | 11 Dec 15:59 2014
Picon

[Cucumber] [Cucumber-JS] Cucumber process timed out ...

Hi,

As the subject says, I always get this message in the console

Cucumber process timed out after waiting 60 seconds for the node.js event loop to empty.  There may be a resource leak.  Have all resources like database connections and network connections been closed properly?

And I can't find where is the problem in my code. You can find my code in this gist.

Thanks.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Gmane