buildbot, an automated build tool, development and release announcement list

db.schedulers.classifyChanges relies on integrity check

Today I reconfigured buildbot (renamed schedulers) while there where some changes waiting for the treeStableTimer., there are some changes awaiting in the database.

Now the reconfiguration does not succeed as an internal error occurs during the creation of the scheduler:

The stacktrace in the buildbot.log contains an error “(InternalError) current transaction is aborted, commands ignored until end of transaction block”

The last buildbot source in this stacktrace is at File "/home/buildmaster/buildbot/buildbot-latest-mr-gui/master/buildbot/db/schedulers.py", line 45

The statement that follows looks good:

UPDATE scheduler_changes SET important=%(important)s WHERE scheduler_changes.objectid = %(objectid_1)s AND scheduler_changes.changeid = %(wc_changeid)s'

{'wc_changeid': 934, 'important': 1, 'objectid_1': 39}

The error occurs in db.schedulers.classifyChanges. This method relies on the integrity check of the database. When an insert does not succeed then an update is performed.

Looking in the postgres logging I see a lot of unsuccessful Inserts + successful update couples.

This is what I see in the postgress logging…

2012-05-23 13:33:20 CEST STATEMENT: INSERT INTO scheduler_changes (objectid, changeid, important) VALUES (55, 934, 1)

2012-05-23 13:33:20 CEST ERROR: duplicate key value violates unique constraint "scheduler_changes_unique"

2012-05-23 13:33:20 CEST STATEMENT: INSERT INTO scheduler_changes (objectid, changeid, important) VALUES (44, 934, 1)

2012-05-23 13:33:20 CEST ERROR: current transaction is aborted, commands ignored until end of transaction block

2012-05-23 13:33:20 CEST STATEMENT: UPDATE scheduler_changes SET important=1 WHERE scheduler_changes.objectid = 44 AND scheduler_changes.changeid = 934

2012-05-23 13:33:20 CEST ERROR: duplicate key value violates unique constraint "scheduler_changes_unique"

I doubt if the ‘UPDATE’ is really causing the error as this update can never cause an duplicate key error. I think that an Insert goes wrong but that the error is seen during the update.

Did anyone see this before??

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Richard Offer | 21 May 20:11

Current State of glob/wildcards in File Upload/Download steps

I've found some old messages fron 2007 about FileUpload and Download and globbing - but nothing since then.

Is there a suggested solution for handling globbing with both Upload and Download ?

The reason being that while my build number is generated by Buildbot - the software itself sets its version number - and I don't want that logic to move since we can track it in SVN.

Since its only the upload/download steps that need to deal with the specifics of filenames, would be useful to handle the case.

richard.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Jürgen Schulz-Brüssel | 16 May 17:05

using **kwargs with MasterShellCommand


Hi,

I am using Buildbot 0.8.6p1 with Twisted 12.0.0. I have a problem in 
using **kwargs with the MasterShellCommand.

I am trying the following:

class NewClass(MasterShellCommand):

     def __init__(self, **kwargs):
       self.arg1 = kwargs.get('arg1')
       self.arg2 = kwargs.get('arg2')
       print self.arg1
       print self.arg2
       MasterShellCommand.__init__(self, "")

     def start(self):
       self.command=[self.arg1 + 'somestring',
                           WithProperties("build-%s.tar.gz", "revision"),
                           self.arg2]
       return MasterShellCommand.start(self)

     def finished(self, result):
         ...

it is called

NewClass(arg1='path', arg2='string')

Everything is just fine when I am running

buildbot checkconfig
The print statements deliver the expected outputs. When I am running the 
buildstep the logfile shows

exceptions.TypeError: unsupported operand type(s) for +: 'NoneType' and 
'str'

So for some reason self.arg1 and self.arg2 are a None object while I am 
awaiting a string. Can anybody pleas help me with this?
Additionally there is an unexpected problem using WithProperties. If I 
am using the shown command an exception is thrown:

File
"/usr/lib/python2.6/site-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/process/buildstep.py",
line 551, in _startStep_3
     if self.start() == SKIPPED:
   File "/think/buildbot/master/master.cfg", line 158, in start
     return MasterShellCommand.start(self)
   File
"/usr/lib/python2.6/site-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/steps/master.py",
line 99, in start
     stdio_log.addHeader(" ".join(command) + "\n\n")
exceptions.TypeError: sequence item 5: expected string, instance found

This occurs only if I am using theWithProperties as shown above.

Best
Juergen

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

headers

Eric Zhu | 16 May 10:16

A question about multiple changes

Hi there,

I am using buildbot as my CI tool. I have a question about multiple changes. Hope anyone can help me. Thanks.

I use git repository to manage my code and use git_buildbot.py to send changes to buildbot. Recently I find if multiple changes are sent, buildbot only buids the first and the last changes.

For example, there are changes 1, 2, 3, 4. I am sure all of them are sent to buildbot from the log. But buildbot only builds change 1 and 4. Change 2 and 3 are displayed on the waterfall view, but not built. I am wondering if it's normal. I haven't found anything in documentation. Is there anybody able to help me figure out? Thanks.

Cheers,

Eric

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Harry Borkhuis | 11 May 16:55

Trigger step

I am looking at adding multirepo functionality to the triggerstep, but I am not sure about the sourcestamp attribute.

The Triggerstep can be configured with a SourceStamp dictionairy. Is this only used to pass repository, branch and revision or is there other information passed to the step.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Dustin J. Mitchell | 9 May 06:42

multirepo has landed!

I just merged Harry's multirepo3 branch, which brings in the last of
the multirepo implementation.  This merge alone was 80 commits.  There
are doubtless still bugs and minor oversights, but nothing we can't
handle.

This has been a HUGE project.  Harry's been at it for quite a while
now, and his dedication and attention to detail show in the results.

  Thank you, Harry -- and Benoît, too! -- for a substantial
contribution to Buildbot.

This puts us on the final approach to 0.8.7, so shut down your
approved electronic devices, check that your seat-backs and tray
tables in the full upright and locked position, and find some time in
your schedule to test out the master branch.  We're going to need all
the help we can get to shake out the last few bugs, and avoid a
0.8.7p1!  As things settle out a bit more, I'll make some tarballs for
download and announce them here, for easy pip-installing.

Seriously, QA has been the biggest pain-point for releasing Buildbot:
most of the testing happens *after* a release.  Halp!

Dustin

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Richard Offer | 6 May 21:03

"patch" command not being found on Windows

I'm trying to get the Try scheduler to work , but my Windows build slave is having a problem finding the patch executable

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): File "c:\BOT\win32\bin\lib\site-packages\buildbot_slave-0.8.6p1-py2.7.egg\buildslave\runprocess.py", line 201, in processEnded self.command.finished(sig, rc) File "c:\BOT\win32\bin\lib\site-packages\buildbot_slave-0.8.6p1-py2.7.egg\buildslave\runprocess.py", line 687, in finished d.callback(rc) File "c:\BOT\win32\bin\lib\site-packages\twisted-12.0.0-py2.7-win32.egg\twisted\internet\defer.py", line 368, in callback self._startRunCallbacks(result) File "c:\BOT\win32\bin\lib\site-packages\twisted-12.0.0-py2.7-win32.egg\twisted\internet\defer.py", line 464, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "c:\BOT\win32\bin\lib\site-packages\twisted-12.0.0-py2.7-win32.egg\twisted\internet\defer.py", line 551, in _runCallbacks current.result = callback(current.result, *args, **kw) File "c:\BOT\win32\bin\lib\site-packages\buildbot_slave-0.8.6p1-py2.7.egg\buildslave\commands\base.py", line 567, in doPatch utils.getCommand("patch.exe"), File "c:\BOT\win32\bin\lib\site-packages\buildbot_slave-0.8.6p1-py2.7.egg\buildslave\commands\utils.py", line 25, in getCommand raise RuntimeError("Couldn't find executable for '%s'" % name) exceptions.RuntimeError: Couldn't find executable for 'patch' exceptions.RuntimeError: Couldn't find executable for 'patch' ]

I've installed it (GNUwin32 version of patch) into C:\Windows\System32\ (as patch.exe) to avoid all issues with PATH. It will quite happily run from the cmd prompt that I use to manually start the build slave.

(I am running everything as the Administrator)

Is buildbot or twisted caching the previous search results ? (I installed the patch command after everything else was up and running)

richard

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Todd Cooper | 4 May 19:19

column headings showing 'building' when not building

Sometimes my column headings show that a build is 'building' even when it is done. Has anyone seen that before? Should I submit a bug?

(see attached where only builders 1 and 5 are actively building when 6, 9 & 10 are finished)

I am running 0.8.5 on linux .. I know I should upgrade and will be doing that sometime soon.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

headers

Richard Offer | 4 May 19:08

Using Properties numerically


I need/want to be able to use a property value numerically, i.e.

        set_properties = { 
            'SAUSAGEBOT_BUILDID' : 40000 + int(
buildbot.process.properties.WithProperties("%(buildnumber:-0000)s") ), 
        },

Obviously that doesn't work or anything close to that…

Can I get a basic string or int type out of WithProperties() ?

[ View from the 1 week buildbot newbie ]

It seems to be something that we can use - but I've really had a hard time getting it to work with the existing
build code.

Our existing build code (using bamboo) is all environment variable driven, trying to get Properties in
sync with environment variables has proved to be magic - and not the good type…

I think the complexity has been trying to provide support for both individual and system builds (the system
build triggers all the individual builds) - and I want to be able to use a single top level build-id for all
generated products (split across each of the builders) - and that id is not the revision version.

The "from the 1week newbie" take away is that the documentation and examples around properties and their
interaction with the system might be better explained… I never could get SetPropertiesFromEnv to work
(probably because I wanted it on the slave not master).

Is there a way to configure the initial buildnumber on a one off basis ? I don't want it to start from 0 when I
roll this into production - I don't what to confuse people with a lower build number than the system I'm replacing…

The overall build system is spread across 11 Builders on 5 different build slaves (I expect to add some
duplicates for the compile part before production) using triggers to provide "1 button" builds.
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

headers

Catherine Knutson | 3 May 18:00

Re: create-master error


Do these look new enough and compatible with each other?

Thanks,
Catherine

$ pip freeze
SQLAlchemy==0.8.0b1
Twisted==12.0.0
buildbot==0.8.6p1
decorator==3.3.3
ez-setup==0.9
jinja2==2.6
mock==1.0.0alpha1
pyopenssl==0.13
pywin32==217
sqlalchemy-migrate==0.7.1
tempita==0.5.1
zope.interface==3.3.0

-----Original Message-----
From: djmitche@...
[mailto:djmitche@...] On Behalf Of Dustin J. Mitchell
Sent: Tuesday, May 01, 2012 11:06 PM
To: Catherine Knutson
Subject: Re: [Buildbot-devel] create-master error

Try running 'pip freeze' to give information on the versions of each that are installed.

Dustin

On Tue, May 1, 2012 at 9:21 AM, Catherine Knutson <CKnutson@...> wrote:
>
> More info... I am trying to install the master on a Windows XP box.
>
> Catherine
> ________________________________________
> From: djmitche@...
[djmitche@...] On Behalf Of Dustin J. 
> Mitchell [dustin <at> v.igoro.us]
> Sent: Monday, April 30, 2012 12:36 PM
> To: Catherine Knutson
> Cc: buildbot-devel@...
> Subject: Re: [Buildbot-devel] create-master error
>
> On Mon, Apr 30, 2012 at 12:29 PM, Catherine Knutson 
> <CKnutson@...> wrote:
>> exceptions.ImportError: cannot import name exceptions
>
> It looks like your version of sqlalchemy is too old?
>
> Dustin
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

headers

Todd Cooper | 1 May 16:34

Sending email depending on completion of one builder

I don't see how to program the emailer to send email to one set of people on completion of one builder and another set of people with another builder.

Anyone have any ideas?

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

28	May 2012
87	April 2012
117	March 2012
64	February 2012
36	January 2012
36	December 2011
81	November 2011
77	October 2011
127	September 2011
57	August 2011
103	July 2011
133	June 2011
87	May 2011
83	April 2011
105	March 2011
98	February 2011
160	January 2011
91	December 2010
120	November 2010
82	October 2010
77	September 2010
125	August 2010
118	July 2010
149	June 2010
134	May 2010
147	April 2010
148	March 2010
183	February 2010
236	January 2010
133	December 2009
37	November 2009
37	October 2009
77	September 2009
91	August 2009
235	July 2009
81	June 2009
115	May 2009
93	April 2009
131	March 2009
171	February 2009
117	January 2009
104	December 2008
85	November 2008
72	October 2008
130	September 2008
192	August 2008
148	July 2008
99	June 2008
73	May 2008
121	April 2008
81	March 2008
95	February 2008
134	January 2008
60	December 2007
149	November 2007
66	October 2007
53	September 2007
78	August 2007
107	July 2007
60	June 2007
89	May 2007
89	April 2007
74	March 2007
74	February 2007
78	January 2007
48	December 2006
83	November 2006
90	October 2006
101	September 2006
87	August 2006
36	July 2006
79	June 2006
85	May 2006
129	April 2006
83	March 2006
20	February 2006
68	January 2006
14	December 2005
105	November 2005
63	October 2005
22	September 2005
21	August 2005
59	July 2005
64	June 2005
62	May 2005
86	April 2005
8	March 2005
3	February 2005

Mon	Tue	Wed	Thu	Fri	Sat	Sun

	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31