Dustin J. Mitchell | 1 Apr 2012 03:04
Picon
Gravatar

Re: buildbot master performance on Windows since 0.8.3p1

On Fri, Mar 30, 2012 at 11:46 AM, Philippe McLean
<philippe.mclean@...> wrote:
> Hi all,
>
> I was wondering what experience and observations people have had regarding
> the performance of the buildbot master on Windows in recent versions of
> buildbot,
> when generating views.
>
> - is it fast, slow, has it improved, dropped...

Very little has changed recently in the web status, so I don't expect
there's been any change here.

I'm curious why you're asking?

Dustin

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
Philippe McLean | 1 Apr 2012 03:07
Picon

Re: buildbot master performance on Windows since 0.8.3p1

Sometimes the master takes a very long time (seconds) to render pages, so I was wondering about performance in general. Some of the discussions yesterday might indicate why (sequential access to pickles?)


Will resetting the master help performance? (delete all log files, start from scratch with a fresh directory and the same master.cfg.)

On Sat, Mar 31, 2012 at 6:04 PM, Dustin J. Mitchell <dustin <at> v.igoro.us> wrote:
On Fri, Mar 30, 2012 at 11:46 AM, Philippe McLean
<philippe.mclean-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi all,
>
> I was wondering what experience and observations people have had regarding
> the performance of the buildbot master on Windows in recent versions of
> buildbot,
> when generating views.
>
> - is it fast, slow, has it improved, dropped...

Very little has changed recently in the web status, so I don't expect
there's been any change here.

I'm curious why you're asking?

Dustin

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel
Dustin J. Mitchell | 1 Apr 2012 03:28
Picon
Gravatar

Re: Google Summer of Code proposal for Javascript Frontend

On Wed, Mar 28, 2012 at 1:01 PM, Naveen Kumar <naveen.iitm90 <at> gmail.com> wrote:
> This is first draft of my proposal. Please give your feedback

As I mentioned in IRC, this looks good.

> 1. Javasciprt frameworks:
> Possible options:
>         1. jQuery
>         2. Dojo
>         3. YUI

This is a good preliminary list.  I suspect that part of the summer
will be building "test" implementations in a few frameworks so that
you can make a judgement based on experience.

> 2. User interface design:
> This is the best time to implement a better looking design. I am not
> very good with web design. I am going to need help with that part. My
> idea of implementation is, first implement a general template which
> will be used in all the pages then implement rest of the features on
> it.

As Tom said, I think that the UI design is a separate task from the
implementation of basic usability.  We don't have anyone within the
project who has a great strength here, either, so this may be
something to leave unfinished, in the sense that the UI at the end of
the summer is perhaps a bit clunky, but all of the support is there
for someone else to improve it.

I suspect that building that support will be most of a summers' work,
in fact -- and for the project such work is actually *far* more
important than getting any particular pages implemented at all.  If
you finished the summer with an aweomse buildbot.js library written
that integrates tightly with one or more external JS libraries like
jQuery and provides easy, reliable access to the Buildbot API -- but
only actually implemented the build page with it -- I would count that
a resounding success.  On the other hand, if you produce working
JavaScript code for a half-dozen user-visible pages, but each is
implemented in an ad-hoc fashion and not particularly flexible or
maintainable, that would be somewhat disappointing.

> As I understand JsonStatusResource class serves the /json requests,
> and is used in WebStatus class in baseweb.py. Hence it is synchronous.
> Any python coding work will be mostly in status_json.py unless I have
> to interact with databases. I am not aware of code outside of this
> module.

And you could coordinate with other developers on the project to make
the needed changes.

As a note, depending on the timeline, I may have a significant chunk
of the *new* API finished, and it would be great to begin your
JavaScript work against that.  In particular, it will include support
for dynamically updating pages as state changes.

> Schedule:
> Google Summer of Code program is spanned over 12 weeks(excluding
> community bonding period).
> 2 weeks: familizing with code, deciding upon things(javascript
> framework, features to implement).
> 5 -6 weeks: implement existing features i.e. builders, buildslaves,
> build details, waterfall. As directed in other thread of devel list
> tests also will be written along with the code.
> 2-3 weeks: Survey for extra features and implement them.
> 2 weeks: Wrap up. Merge missing code(if any).

One thing to keep in mind is that there is potentially a long time
between finishing some code and seeing it merged into the upstream
codebase.  I'd like to see a bit more detail here about what can
happen in overlapping chunks: after you've submitted a patch for one
thing and are waiting for review, what will you work on?  I also like
to see concrete artifacts described in a schedule: as of week three I
will produce ___.  This gives the you and your mentor measurable goals
to aim for, and helps the rest of the development community know what
to expect over the course of the summer.

Perhaps more importantly, I think that your project will include a
good bit of experimentation.  For example, an effective technique for
selecting the best JS library for the purpose might be to implement a
"fake" buildmaster in Django (since you're comfortable with it), and
then build a simple JS frontend on top of it using each of the
possible libraries.  Then you, and the development community as a
whole, could get some real-world experience with the libraries and
make an informed decision.  Even though most of that code would not
end up in Buildbot, this would be an excellent contribution to the
project.

Even once that is done, you may do several iterations of writing a
low-level library, then building a page on top of it, only to find
that the library could be better designed.

> I have fairly good experience in python. I know twisted basics. I
> mostly work on django for web development. I used javascipt/jQuery in
> web development projects. I have little experience with UI design.
> Though I follow some opensource projects never really contributed to
> any of them. This is the first time I am working for an open source
> project. We use git in our college for versioning. I am comfortable
> with it.

Welcome!  I think that this is a strong application, and demonstrates
that you've done your homework.  I look forward to seeing the final
proposal! (of course, another draft is welcome too)

Dustin

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/buildbot-devel
Dustin J. Mitchell | 1 Apr 2012 03:42
Picon
Gravatar

Re: buildbot master performance on Windows since 0.8.3p1

On Sat, Mar 31, 2012 at 8:07 PM, Philippe McLean
<philippe.mclean@...> wrote:
> Sometimes the master takes a very long time (seconds) to render pages, so I
> was wondering about performance in general. Some of the discussions
> yesterday might indicate why (sequential access to pickles?)

Yes, that's generally behind it..

> Will resetting the master help performance? (delete all log files, start
> from scratch with a fresh directory and the same master.cfg.)

Yes, and also set your buildHorizon to a small value so that builds
are aggressively expired.

Dustin

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
Philippe McLean | 1 Apr 2012 03:44
Picon

Re: buildbot master performance on Windows since 0.8.3p1

I didn't know about the buildHorizon... I will try that first. 
Thank you
 
Yes, and also set your buildHorizon to a small value so that builds
are aggressively expired.

Dustin

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel@...
https://lists.sourceforge.net/lists/listinfo/buildbot-devel
Tom Prince | 1 Apr 2012 06:38

Re: buildbot master performance on Windows since 0.8.3p1

On Sat, 31 Mar 2012 20:42:40 -0500, "Dustin J. Mitchell" <dustin <at> v.igoro.us> wrote:
> Yes, and also set your buildHorizon to a small value so that builds
> are aggressively expired.

Also, setting buildCacheSize can have a significant impact.

Another thing I have noticed, is that the grid displays can be *very*
slow, if you are dispaying builds from multiple projects. grid shows the
latest build for the most recent sourcestamps. And if some builders
don't have *any* builds for some sourcestamps, builtbot needs to read
*all* of the buidls for that builder to learn there are no interesting
ones. (This is true at least for 0.8.2, but I don't think there have
been any relevant changes here).

  Tom

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
Dustin J. Mitchell | 1 Apr 2012 18:18
Picon
Gravatar

Re: Behavior expected from SVN master side source step

On Sat, Mar 31, 2012 at 1:34 PM, Saurabh Kumar
<saurabhgeek92@...> wrote:
> I discussed this issue with Tom, and both of us have not used SVN much,
> so this new behavior may miss out on corner cases, or may not be what
> users expect from repourl, especially if they need branch information
> from the source stamp ignored. Any feedback from SVN users would be helpful.

I would very much like to hear back from people here, too!

In https://github.com/buildbot/buildbot/pull/371 Callek suggested
using WithProperties or Interpolate, rather than doing magic string
substitutions.  I *think* that this can cover all of the necessary
cases, and more, and for all of the VC's, not just SVN.

However, this comes at the cost of backward compatibility for users of
the *master-side* classes.  I suspect that's a rather small bunch of
folks, since those have only been available in 0.8.5 and 0.8.6, so if
we provided sufficient assistance for upgrading, I think we could get
away with the compatibility break.

Thoughts?

Dustin

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
Jarod McBride | 2 Apr 2012 20:38
Picon
Favicon

Notification on SVN Polling error

Greetings all,

I was wondering if there is a way for BuildBot to notify or throw an error on the waterfall when the SVN poller
is giving an error.  We had an instance where one of our repos changed and caused the buildbot to stop
connecting but we didn't realize it until much later.  Is there a way for BuildBot to be more proactive on
these kinds of errors?

Thanks,

Jarod

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
Charles Lepple | 3 Apr 2012 14:32
Picon

Re: Behavior expected from SVN master side source step

On Apr 1, 2012, at 12:18 PM, Dustin J. Mitchell wrote:

> On Sat, Mar 31, 2012 at 1:34 PM, Saurabh Kumar
<saurabhgeek92@...> wrote:
>> I discussed this issue with Tom, and both of us have not used SVN much,
>> so this new behavior may miss out on corner cases, or may not be what
>> users expect from repourl, especially if they need branch information
>> from the source stamp ignored. Any feedback from SVN users would be helpful.
> 
> I would very much like to hear back from people here, too!
> 
> In https://github.com/buildbot/buildbot/pull/371 Callek suggested
> using WithProperties or Interpolate, rather than doing magic string
> substitutions.  I *think* that this can cover all of the necessary
> cases, and more, and for all of the VC's, not just SVN.
> 
> However, this comes at the cost of backward compatibility for users of
> the *master-side* classes.  I suspect that's a rather small bunch of
> folks, since those have only been available in 0.8.5 and 0.8.6, so if
> we provided sufficient assistance for upgrading, I think we could get
> away with the compatibility break.

I'll be honest, when I upgraded to 0.8.5, I didn't see much of an advantage of moving to the master-side
classes for SVN ("if it ain't broke..."), especially since the code I'm building is eventually moving
from Subversion to Git. So I'm probably not part of the target audience, but I do think that it might be good
to standardize on something like WithProperties.

For what it's worth, I usually end up including a custom split_file_branches() function for the change
source that returns the text 'trunk' instead of None for the default branch, so I might not be seeing all of
the potential problems that might arise from using WithProperties. Also, this means we have to
explicitly specify 'branches/foo' instead of just 'foo' in the force build page for NUT - not sure how
other people are handling that in their Subversion+Buildbot setup, but it Works For Me (tm).

--

-- 
Charles Lepple
clepple <at> gmail

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
Saurabh Kumar | 3 Apr 2012 16:00
Picon
Gravatar

GSoC 2012 proposal for Implementation of a new Master-Slave Protocol.

Hello,

I would be glad to receive comments on this draft of my proposal.

------------------------------------------------------------------------------------

Name: Saurabh Kumar
e-mail: saurabhgeek92@...
irc nick: sa1 on freenode.net

Project Proposal:

Implementing a new master-slave protocol:

While the exact nature of the protocol would evolve over the project
timeline, the protocol should have the following features:
- It would be a message-based protocol, rather than a stream or rpc
based protocol.
- It would be a thin abstraction layer over other protocols, so that
multiple backend protocols can be added.
- The data to be transferred should be serialized through JSON. I also
considered Protobuf and Apache Thrift, but I believe that there is no
need for a binary protocol, which would add complexity.
- The layer should be thin enough so that backend implementations are
not limited by the abstraction over them.
- The backends I am looking at adding are amp, 0mq, amqp. The
abstraction layer should support easy addition of backends, so that
the user has a choice.
- The connections should optionally be on a secure protocol, such as
SSH or TLS. This seems to be the least effort way to add security till
backends add their own security features. There would be implications
on what it means for the backend since the backends and the tunnel may
handle reconnections, etc differently.
- The protocol itself should be scalable to multiple masters, to a
large number of slaves, etc. and should gracefully handle the removal
of masters or slaves due to disconnection or other problems. It is to
be seen how  e.g 0mq's brokerless design(or any other fitting design)
can be ported to or will be compatible with the higher level
abstraction.
-Backward compatibility would somehow have to be maintained, so that
current masters work with old slaves, or current slaves with old
masters. At the very least, the user should be able to choose between
the present protocol and the new one.

A schedule for the above would ideally be:
1) Understanding the code, coming up with network models and
implementing toy models of proposed protocols can be done in the
period before coding starts. This will also be the period that I will
have to learn and compensate for any missing experience, including the
nature of asynchronous programming, Twisted, the mq libraries, and
with development processes, testing etc in general.
2) Ideally when the coding period starts, it should be apparent about
what network models can be implemented across backends, and the nature
of the abstraction would be clear. Hopefully 2-3 weeks would be enough
to implement the abstraction layer. I may choose to begin with
existing work, such as [1]
3) The rest of the period would be dedicated to writing backends,
starting with amp or 0mq and proceeding further as time allows, while
making required changes to the abstraction layer.
4) Testing and Documentation would be worked upon simultaneously.

If I am not able to complete the project to my or the organizations
satisfaction within the deadline, I would be happy to keep working on
it. I would be happy to do other work or related work after the
application period too.

Experience:
I am new to open source development, though I have been observing it
from outside for some time. I am familiar with python but am new to
twisted, mq libraries, and asynchronous systems. I have worked with
simple distributed applications before. I hope to learn them as part
of my summer work. I am used to simple workflows on git. I am a second
year student of IIT Kharagpur studying Mathematics and Computing, and
have tried diverse unrelated projects while trying to choose an area
of interest. These include a versioning filesystem, available on my
github page[2], and on implementing AI for the robocup project[3]. I
am new to opensource so I am not affiliated to any project yet, so I
will try to maintain a instance to test projects some of the hackers
from my local LUG[4] hack on, such as git.git. I will probably also
need hosting help.

[1]: https://github.com/djmitche/buildbot/commits/mq
[2]: https://github.com/sa1
[3]: http://www.robocup.org/
[4]: https://github.com/klug

--Saurabh

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev

Gmane