Discussion of Python testing tools ()

headers

Nagappan Alagappan | 23 May 2013 00:17

[Ann] Cobra 3.5 - Windows GUI test automation tool

Hello,

New features:

* ooldtp python client

* Support setting text on combo box

* Added simple command line options

* Support state.editable in hasstate

* Handle valuepattern in click API

* Support ToolBar type on click

* Write to log file if environment variable is set (set LDTP_LOG_FILE=c:\ldtp.log)

* Support control type Table, DataItem in Tree implementation

* Added scrollbar as supported type

New API:

* MouseMove

* setcellvalue

* guitimeout

* oneup

* onedown

* oneleft

* oneright

* scrollup

* scrolldown

* scrollright

* scrollleft

Bugs fixed:

* Fix to support taskbar with consistent index

* istextstateenabled API

* Fallback to object state enabled if value pattern is not available

* Fix to support InvokePattern on Open button

* Use width, height if provided while capturing screenshot

* Work around for copying text to clip board

* QT 5.0.2 specific changes

* Check errno attribute to support cygwin environment

* Fix keyboard APIs with new supported key controls (+, -, :, ;, ~, `, arrow up, down, right, left)

* Don't grab focus if type is tab item

Java client:

* Fixed selectRow arguments

* Fixed compilation issues

Python client:

* Fix optional argument issue in doesrowexist

C# client:

* Added new APIs (scrollup, scrolldown, scrollleft, scrollright, oneup, onedown, oneleft, oneright)

Ruby/Perl client: No changes

Credit:

Nagappan Alagappan <nagappan <at> gmail.com>

John Yingjun Li <yjli <at> vmware.com>

Helen Wu <liyuan.wu <at> oracle.com>

Eyas Kopty <e.kopty <at> yahoo.com>

VMware colleagues

Please spread the word and also share your feedback with us (email me: nagappan <at> gmail.com).

About LDTP:

Cross Platform GUI test automation tool Linux version is LDTP, Windows version is Cobra and Mac version is PyATOM.

* Linux version is known to work on GNOME / KDE (QT >= 4.8) / Java Swing / LibreOffice / Mozilla application on all major Linux distribution

* Windows version is known to work on application written in .NET / C++ / Java / QT on Windows XP SP3 / Vista SP2 / Windows 7 SP1 / Windows 8.

* Mac version is known to work on OS X Snow Leopard /Lion/Mountain Lion. Where ever PyATOM runs, LDTP should work on it.

Tests can be written in: Python/Ruby/Perl/Java/C#/Clojure/VB.NET/PowerShell

Download source: https://github.com/ldtp/cobra

Download binary (Windows XP / Vista / Windows 7 / Windows 8): https://code.google.com/p/cobra-winldtp/downloads/list

System requirement: .NET 3.5, refer README.txt after installation

Documentation references:

For detailed information on LDTP framework and latest updates visit

http://ldtp.freedesktop.org

For information on various APIs in LDTP including those added for this release can be got from http://ldtp.freedesktop.org/user-doc/index.html

Java doc - http://ldtp.freedesktop.org/javadoc/

Report bugs - http://ldtp.freedesktop.org/wiki/Bugs

To subscribe to LDTP mailing lists, visit http://ldtp.freedesktop.org/wiki/Mailing_20list

Thanks

Nagappan

Cross platform GUI testing

Linux Desktop (GUI Application) Testing Project - http://ldtp.freedesktop.org
Cobra - Windows GUI Automation tool - https://github.com/ldtp/cobra

ATOMac - Mac GUI Automation tool - https://github.com/pyatom/pyatom

http://nagappanal.blogspot.com

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

John Papanastasiou | 20 May 2013 12:52

Error with Checking Calls to Multiple Mocks

Hi,

I believe I have found a problem with mock.call_args_list. My understanding is that for a given mock, if the function you are testing makes multiple calls to a function on that mock, the call_args_list is set to a list of the successive argument lists for each call.

The problem I have is that the call_args_list for one mock is being replaced with the call_args_list for a second mock that is being used in the same function under test.

To demonstrate this, I have written a script which you can just run.

I have an event object which has a property which may be set to a list of extra event objects. Event objects have a property called handler which itself has a function trigger which called with a context (a dict with a key 'event' set to the event instance).

The function i am trying to test triggers the handler on the top level event, then goes through each event in the list triggering their handlers. To test this function, I am making assertions about the calls to the respective trigger functions.

The attached code demonstrates that the call_args_list for the top_level event (the one being called first in the function) is set to the call_args_list for the extra event (the one being called second).

The TestCase includes two unit tests which test the calls to the top_level event trigger and the nested event trigger independently and these pass. The third unit test, test_top_level_and_extra prints out some information about the mocks involved and the call_args_lists. This output demonstrates the error I am seeing. I've pasted a sample output below.

Is this a defect in the mock.call_args_list or have I misunderstood the documentation?

Any help you can give is greatly appreciated.

Regards,

John.

Sample Output:

$ ./nested_calls.py

Mock: top_level: <mock.MagicMock object at 0x100567590>

Call Args, top_level.handler.trigger: [(({'event': <mock.Mock object at 0x10059a950>},), {})]

Mock: nested: <mock.Mock object at 0x10059a950>

Call Args, nested.handler.trigger: [(({'event': <mock.Mock object at 0x10059a950>},), {})]

======================================================================

FAIL: test_top_level_and_extra (__main__.TestNestedCalls)

----------------------------------------------------------------------

Traceback (most recent call last):

File "./nested_calls.py", line 53, in test_top_level_and_extra

top_level.handler.trigger.assert_called_once_with(ctx)

AssertionError: Expected: (({'event': <mock.MagicMock object at 0x100567590>},), {})

Called with: (({'event': <mock.Mock object at 0x10059a950>},), {})

----------------------------------------------------------------------

Ran 3 tests in 0.059s

FAILED (failures=1)

Attachment (nested_calls.py): application/octet-stream, 1609 bytes

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

Kevin Tran | 17 May 2013 01:46

Making mock objects that call the original function

The mock objects in the Jasmine testing framework have a `andCallthrough()` method.  This allows you have a
mock that when called, calls the original function.  An example:

    describe("A spy, when configured to call through", function() {
      var foo, bar, fetchedBar;

      beforeEach(function() {
        foo = {
          setBar: function(value) {
            bar = value;
          },
          getBar: function() {
            return bar;
          }
        };

        spyOn(foo, 'getBar').andCallThrough();

        foo.setBar(123);
        fetchedBar = foo.getBar();
      });

      it("tracks that the spy was called", function() {
        expect(foo.getBar).toHaveBeenCalled();
      });

      it("should not effect other functions", function() {
        expect(bar).toEqual(123);
      });

      it("when called returns the requested value", function() {
        expect(fetchedBar).toEqual(123);
      });
    });

The only way I know how to do this in the Python mock library (http://www.voidspace.org.uk/python/mock/)
is by setting the original function manually:

    with patch('my_project.forms.create_user') as mock_create_user:
        mock_create_user.side_effect = create_user
        client.post('/signup/', data)
        self.assertEqual(1, mock_create_user.call_count)
        self.assertEqual(1, User.objects.count())

Is there a better built-in way to do this?

headers

Elizabeth Lin | 15 May 2013 00:14

dynamically retrieving fixtures which depend on generated parameters from the comand-line

Here's the simplified example code:

https://gist.github.com/elizabethlin/7754f0324b8f7b1c7388

this gets generates the following tests:

test_example.py:31: TestExample.test_one[std] FAILED

test_example.py:35: TestExample.test_two[std] FAILED

test_example.py:39: TestExample.test_three[std] FAILED

test_example.py:31: TestExample.test_one[virt] FAILED

test_example.py:35: TestExample.test_two[virt] FAILED

when really I want

test_example.py:31: TestExample.test_one[std] FAILED

test_example.py:35: TestExample.test_two[std] FAILED

test_example.py:39: TestExample.test_three[std] FAILED

test_example.py:31: TestExample.test_one[virt-typeA] FAILED

test_example.py:35: TestExample.test_two[virt-typeA] FAILED

" [virt-typeB]

Can someone help me out? I imagine that pytest_generate_tests gets called at an earlier step, before the fixtures are executed, and at that time, it can't detect that we need the parametrization for generated fix to happen. There's probably other reasons this won't work, but I'm having trouble decifering the pytest code, and the flow of it.

Thanks,

Liz

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

Nagappan Alagappan | 12 May 2013 04:35

Announce : LDTP 3.5.0 - Linux GUI test automation tool

Hello,

New API:

* inserttext, objtimeout, guitimeout, getcellsize, getcellvalue, getobjectnameatcoords, getcombovalue, getaccesskey in Python client

* doubleClick, doubleClickRow, onWindowCreate, getCellSize, getComboValue, appUnderTest, getAccessKey in Java client

* getcellsize, getcellvalue in Ruby client

* GetCellSize, GetComboValue, AppUnderTest, GetAccessKey, MouseRightClick, DoubleClick, DoubleClickRow, RightClick in C# client

New control type:

* POPUP MENU for Ubuntu environment

Bugs fixed:

Ruby client:

* Fixed optional arguments to imagecapture

* Check window_name parameter, if empty then use <at> window_name passed in constructor

Python client:

* Fixed optional argument APIs to work on both Windows and Linux

* imagecapture x, y offset, height and width parameters are disregarded if window parameter is provided - Bug#685548

* Return unicode string all the time on gettextvalue

* Fix partial match argument in selectrow, compatible with Windows

* Patch by ebass to support Python 2.6

* Added Errno 101 as we see in ebass Ubuntu 10.04 environment

Core LDTP2

* Include label type on gettextvalue

* Don't include separators in the list

Perl client:

* Added perl client

Credit:

* Sawyer X for the Perl interface

* ebass (IRC nick name)

* Marek Rosa <marek.j.rosa <at> gmail.com>

* Thanks to all others who have reported bugs through forum / email / in-person / IRC

About LDTP:

Cross Platform GUI Automation tool Linux version is LDTP, Windows version

is Cobra and Mac version is PyATOM.

* Linux version is known to work on GNOME / KDE (QT >= 4.8) / Java Swing / LibreOffice / Mozilla application on all major Linux distribution.

* Windows version is known to work on application written in .NET / C++ / Java / QT on Windows XP SP3 / Windows 7 / Windows 8 development version.

* Mac GUI testing is known to work on OS X Snow Leopard/Lion/Mountain Lion. Where ever PyATOM runs, LDTP should work on it.

Download source: https://github.com/ldtp/ldtp2

Download binary (RPM / DEB):

http://download.opensuse.org/repositories/home:/anagappan:/ldtp2:/

Documentation references:

For detailed information on LDTP framework and latest updates visit http://ldtp.freedesktop.org

For information on various APIs in LDTP including those added for this release can be got from http://ldtp.freedesktop.org/user-doc/index.html

Java doc - http://ldtp.freedesktop.org/javadoc/

Report bugs - http://ldtp.freedesktop.org/wiki/Bugs

To subscribe to LDTP mailing lists, visit http://ldtp.freedesktop.org/wiki/Mailing_20list

IRC Channel - #ldtp on irc.freenode.net

How can you help: Spread the news and send back your feedback to us

Thanks

Nagappan

Cross platform GUI testing

Linux Desktop (GUI Application) Testing Project - http://ldtp.freedesktop.org
Cobra - Windows GUI Automation tool - https://github.com/ldtp/cobra

ATOMac - Mac GUI Automation tool - https://github.com/pyatom/pyatom

http://nagappanal.blogspot.com

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

holger krekel | 4 May 2013 12:40

speeding up tox runs / devpi-server

Hi all,

for those of you using tox on your laptop or on some Jenkins server.
You might want to consider using it together with a new project i just
released, a super-fast and auto-updating pypi.python.org caching proxy.  See

    http://pypi.python.org/pypi/devpi-server

for how to set it up.  It usually cuts down tox setup time 
by a factor of 10 for me.  If you have a devpi-server running
at port 3141, you can invoke tox without changing your tox.ini files:

    tox -i ALL=http://localhost:3141/ext/pypi/simple/ ...

which will make tox configure pip-installation to use your devpi cache serve.r

cheers,
holger

headers

Dan Wandschneider | 2 May 2013 03:22

Nose: marking a class with attrib doesn't skip the class (only the methods)

All-

I'm using Nose to discover, run and report on our collection of unit tests, all of which are written using python unittest classes and methods. Many of the tests have circumstances in which they should not be run, for instance, if certain components fail to build, or on certain platforms. We've been decorating the tests using the attrib module to define these limitations. Sometimes I've decorated a class, other times a specific test within the class.

I've noticed that if I decorate a class using attrib, Nose still runs through the class, though it runs none of the test methods within it. It looks like attrib operates on methods and functions rather than classes. In particular, this means that setUpClass class methods are still run. Clearly, this is surprising, and it can actually cause problems in our circumstances.

Is this expected/intended? What is the rationale for not allowing classes to be ignored, and instead ignored at the method level?

Thanks-

Dan W.

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

Macie Korte | 1 May 2013 19:57

Mock speaker in NYC area?

I'm helping to starting a new user's group in the NYC area. We want our first meetup to be a talk + Q&A on mock. Does anyone have any interest in giving a talk or know how I could find someone who might? The backing company is willing to fork over some money.

Sorry if this isn't the right place to ask. I didn't have any better ideas of where else to start.

- Macie

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

holger krekel | 30 Apr 2013 16:39

pytest-2.3.5: bug fixes and little improvements

pytest-2.3.5: bug fixes and little improvements
===========================================================================

pytest-2.3.5 is a maintenance release with many bug fixes and little
improvements.  See the changelog below for details.  No backward
compatibility issues are foreseen and all plugins which worked with the
prior version are expected to work unmodified.   Speaking of which, a
few interesting new plugins saw the light last month:

- pytest-instafail: show failure information while tests are running
- pytest-qt: testing of GUI applications written with QT/Pyside
- pytest-xprocess: managing external processes across test runs
- pytest-random: randomize test ordering

And several others like pytest-django saw maintenance releases.
For a more complete list, check out 
https://pypi.python.org/pypi?%3Aaction=search&term=pytest&submit=search.

For general information see:

     http://pytest.org/

To install or upgrade pytest:

    pip install -U pytest # or
    easy_install -U pytest

Particular thanks to Floris, Ronny, Benjamin and the many bug reporters
and fix providers.

may the fixtures be with you,
holger krekel

Changes between 2.3.4 and 2.3.5
-----------------------------------

- never consider a fixture function for test function collection

- allow re-running of test items / helps to fix pytest-reruntests plugin
  and also help to keep less fixture/resource references alive

- put captured stdout/stderr into junitxml output even for passing tests
  (thanks Adam Goucher)

- Issue 265 - integrate nose setup/teardown with setupstate
  so it doesnt try to teardown if it did not setup

- issue 271 - dont write junitxml on slave nodes

- Issue 274 - dont try to show full doctest example
  when doctest does not know the example location

- issue 280 - disable assertion rewriting on buggy CPython 2.6.0

- inject "getfixture()" helper to retrieve fixtures from doctests,
  thanks Andreas Zeidler

- issue 259 - when assertion rewriting, be consistent with the default
  source encoding of ASCII on Python 2

- issue 251 - report a skip instead of ignoring classes with init

- issue250 unicode/str mixes in parametrization names and values now works

- issue257, assertion-triggered compilation of source ending in a
  comment line doesn't blow up in python2.5 (fixed through py>=1.4.13.dev6)

- fix --genscript option to generate standalone scripts that also
  work with python3.3 (importer ordering)

- issue171 - in assertion rewriting, show the repr of some
  global variables

- fix option help for "-k"

- move long description of distribution into README.rst

- improve docstring for metafunc.parametrize()

- fix bug where using capsys with pytest.set_trace() in a test
  function would break when looking at capsys.readouterr()

- allow to specify prefixes starting with "_" when 
  customizing python_functions test discovery. (thanks Graham Horler)

- improve PYTEST_DEBUG tracing output by puting
  extra data on a new lines with additional indent

- ensure OutcomeExceptions like skip/fail have initialized exception attributes

- issue 260 - don't use nose special setup on plain unittest cases

- fix issue134 - print the collect errors that prevent running specified test items

- fix issue266 - accept unicode in MarkEvaluator expressions

headers

Balthazar Rouberol | 30 Apr 2013 14:38

Return value of a read-only attribute only settable via 'return_value' kwarg

Hello all,

I wanted to mock an class read only property (which only does some I/O with mongoDB) by setting its return_value in an entire unittest.TestCase scope.

What I first tried was defining a patch in the classSetUp classmethod, then call its .start() method, and its .stop() method in tearDownClass.

<at> classmethod
def setUpClass(cls):
    # patch the Dataminer.blacklisted_domains so that it does not
    # communicate with the database
    cls.black_patch = mock.patch.object(
       Dataminer,
       'blacklisted_domains',
       new_callable=mock.PropertyMock
    )

cls.black_patch.return_value = ['some', 'manual', 'test', 'value']

cls.black_patch.start()

<at> classmethod
def tearDownClass(cls):
cls.black_patch.stop()

However, when I called the Dataminer.blacklisted_domains property later in a test, it returned <MagicMock name='blacklisted_domains()' id='34773392'>, and not the value I defined.

What worked for me was passing the return_value as a kwarg to patch.object:

<at> classmethod
    def setUpClass(cls):
        # patch the Dataminer.blacklisted_domains so that it does not
        # communicate with the database
        cls.black_patch = mock.patch.object(
            Dataminer,
            'blacklisted_domains',
            new_callable=mock.PropertyMock,

**{'return_value': ['some', 'manual', 'test', 'value']}

        )
        cls.black_patch.start()

    <at> classmethod
    def tearDownClass(cls):
        cls.black_patch.stop()

With this method, the return value of the blacklisted_domains property of my Dataminer instance was what I defined in the classSetUp.

Could anyone walk me through the differences between these two methods, as it's not really clear to me why the first would fail when the second would work..

Thank you very much!
--

Balthazar Rouberol

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

headers

Dan Wandschneider | 29 Apr 2013 22:10

Re: testing-in-python Digest, Vol 75, Issue 16

I don't know if there already an option to do this, but I think it would probably be easier to use a post-test nose plugin than to rewrite a parser.

On Mon, Apr 29, 2013 at 12:05 PM, <testing-in-python-request <at> lists.idyll.org> wrote:

Send testing-in-python mailing list submissions to
testing-in-python <at> lists.idyll.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.idyll.org/listinfo/testing-in-python
or, via email, send a message with subject or body 'help' to
testing-in-python-request <at> lists.idyll.org

You can reach the person managing the list at
testing-in-python-owner <at> lists.idyll.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of testing-in-python digest..."

Today's Topics:

1. Rolling up test results (medford bridges)
2. Re: Rolling up test results (John Wong)

---------- Forwarded message ----------
From: medford bridges <medford.bridges <at> gmail.com>
To: testing-in-python <at> lists.idyll.org
Cc:
Date: Mon, 29 Apr 2013 12:23:55 -0400
Subject: [TIP] Rolling up test results
Apologies in advance if this is a too opt-asked, frequently answered question so I'll keep it really simple.

Given this output from a couple of tests in a couple of files, run by nosetests:

...

Ran 8 tests in 0.077s

FAILED (failures=4)

Is there a option or other tool set to produce something more like this, either instead or in addition to basic output?

...

SimpleTest1 2/2/0/4 (pass/fail/error/total tests)

SimpleTest2 2/2/0/4 (pass/fail/error/total tests)

Summary 4/4/0/8 (pass/fail/error/total tests)

In particular, I'd like to be able to track total tests run, pass/fail percentages, and more without diving into output parsing.

It would be useful to get a count of tests in a test case file or test suite collection, even if some were skipped for various reasons.

Thanks.

---------- Forwarded message ----------
From: John Wong <gokoproject <at> gmail.com>
To: medford bridges <medford.bridges <at> gmail.com>
Cc: "testing-in-python <at> lists.idyll.org" <testing-in-python <at> lists.idyll.org>
Date: Mon, 29 Apr 2013 13:18:35 -0400
Subject: Re: [TIP] Rolling up test results
I think nose has the ability to customize the output. Haven't dived in too much.
But I do know nose-progressive can output pretty nicely.

https://pypi.python.org/pypi/nose-progressive/

In particular the option --progressive-advisories you can check it out.

Search in http://www.verious.com/code/svetlyak40wt/nose-progressive/

BY default it gives you the nice text. Another way I can think of is the XML report.

John

On Mon, Apr 29, 2013 at 12:23 PM, medford bridges <medford.bridges <at> gmail.com> wrote:
Apologies in advance if this is a too opt-asked, frequently answered question so I'll keep it really simple.

Given this output from a couple of tests in a couple of files, run by nosetests:

...

Ran 8 tests in 0.077s

FAILED (failures=4)

Is there a option or other tool set to produce something more like this, either instead or in addition to basic output?

...

SimpleTest1 2/2/0/4 (pass/fail/error/total tests)

SimpleTest2 2/2/0/4 (pass/fail/error/total tests)

Summary 4/4/0/8 (pass/fail/error/total tests)

In particular, I'd like to be able to track total tests run, pass/fail percentages, and more without diving into output parsing.

It would be useful to get a count of tests in a test case file or test suite collection, even if some were skipped for various reasons.

Thanks.

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

_______________________________________________
testing-in-python mailing list
testing-in-python <at> lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python

18	May 2013
37	April 2013
33	March 2013
74	February 2013
65	January 2013
50	December 2012
39	November 2012
55	October 2012
15	September 2012
54	August 2012
79	July 2012
26	June 2012
34	May 2012
52	April 2012
77	March 2012
21	February 2012
99	January 2012
96	December 2011
77	November 2011
78	October 2011
38	September 2011
90	August 2011
115	July 2011
65	June 2011
27	May 2011
55	April 2011
152	March 2011
82	February 2011
83	January 2011
38	December 2010
111	November 2010
91	October 2010
99	September 2010
138	August 2010
97	July 2010
23	June 2010
38	May 2010
66	April 2010
209	March 2010
155	February 2010
67	January 2010
75	December 2009
53	November 2009
60	October 2009
134	September 2009
85	August 2009
97	July 2009
16	June 2009
100	May 2009
597	April 2009
113	March 2009
40	February 2009
11	January 2009
14	December 2008
57	November 2008
6	October 2008
108	September 2008
84	August 2008
122	July 2008
53	June 2008
11	May 2008
45	April 2008
15	March 2008
37	February 2008
92	January 2008
32	December 2007
68	November 2007
11	October 2007
17	September 2007
15	August 2007
25	July 2007
13	June 2007
36	May 2007
29	April 2007
194	March 2007

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31