Unified bcfg2.spec file?

Hello list,
all the talk about the spec file last week made me wonder why we
currently have two of them (misc/bcfg2.spec and redhat/bcfg2.spec.in)
and who is using which one.

Them educationated folks on IRC clarified for me that the latter file
can be used to build with "make rpmdist" from git (or previously, from
svn) directly (at least as long as you retain the .git/ directory,
from which the version info is fetched). The misc/ file can be used
with the open build service, rpmbuild, and mock, and it seems to have
received a little more love recently (the redhat one does not build
the bcfg2-web package, f.e.). But then again, the misc/ version still
uses the debian init-scripts and /etc/defaults directory (and so does
the, most likely derived, fedora package, and some of my own
packages).

On IRC, everybody seemed to be using the file in misc/. Is anybody on
this list using the redhat/ flavor?

On top of the above differences, at least the Fedora project (Fabian
Affolter et al) and myself (openSUSE, someday maybe even SLES again)
roll our own, so we have all these versions:
    https://github.com/Bcfg2/bcfg2/blob/master/misc/bcfg2.spec
    https://github.com/Bcfg2/bcfg2/blob/master/redhat/bcfg2.spec.in
    http://pkgs.fedoraproject.org/gitweb/?p=bcfg2.git;a=blob;f=bcfg2.spec;hb=HEAD
    https://build.opensuse.org/package/view_file?file=bcfg.spec&package=bcfg2-stable&project=home%3Am4z&rev=97acdd17c95e23e1cb99cd4bfcddf2f7
(that's one out of five-and-a-half different ones I currently have to
build.)

In other words, we have a zoo of different files that I'd like to either

Making the bcfg2.spec mock-friendly

In an effort to make the .../misc/bcfg2.spec buildable in mock
(https://fedorahosted.org/mock/) I think we need to remove the part of
the specfile that runs the 'rpm' command, which doesn't work under
mock.

To address this issue, I went back to see why it was added in the
first place.  It appears to have been added to address a requirement
difference between RHEL and SUSE:
https://github.com/Bcfg2/bcfg2/commit/f5bb417f750cc4b39d8b6cc6cae31e5c95800c67
(to address ticket 445
http://trac.mcs.anl.gov/projects/bcfg2/ticket/445).

Now, I'm looking at the OpenSUSE repos, I only see python-lxml
packages anyway, so would it be ok to revert to just requiring
python-lxml?  I don't have any installed SUSE systems so I'm not sure
if older versions of OpenSUSE or the commercial distro still require a
simple 'lxml' package.  If so, I think we should simply use the RPM
spec macro language and distro variables to detect that platform and
require lxml, rather than executing the rpm command.

Any comments?

--

-- 
Jonathan Billings <jsbillin <at> umich.edu>
College of Engineering - CAEN - Unix and Linux Support

Bcfg2 1.2.1 released

A new release for Bcfg2 1.2.1 is now available at:

ftp://ftp.mcs.anl.gov/pub/bcfg

Bcfg2 1.2.1 is primarily a bugfix release, with over 2 dozen reported
bugs and dozens more unreported bugs fixed. The only significant
change from 1.2.0 is that regular expression support in Rules is now
optional, and off by default; see
http://docs.bcfg2.org/server/plugins/generators/rules.html#using-regular-expressions-in-rules
for more details.

A full list of bug fixes and cool new features is available at
https://trac.mcs.anl.gov/projects/bcfg2/wiki/ReleaseAnnouncement/1.2.1

--

-- 
Chris St. Pierre

Re: Bcfg2 server rearchitecture

On 01/23/2012 05:59 AM, Chris St. Pierre wrote:
> On Sun, Jan 22, 2012 at 7:20 PM, Gordon Messmer<yinyang <at> eburg.com>  wrote:
>> If you don't mind questions: How long does it take to copy that data from
>> disk to tmpfs?  How long does it take to start up on tmpfs?  Did you try a
>> readahead process before moving to tmpfs?
>
> I'm not sure, since that only happens on _server_ restart, not on
> _service_ restart; the latter is far more common.  The bottleneck in
> the initial checkout is, of course, the SVN server.
>
> Our startup times dropped to 10-15 minutes when we moved to tmpfs.  We
> eventually got rid of Ohai altogether -- that much probe data doesn't
> even remotely scale currently -- and server startup times are down to
> around 4-5 minutes.

OK, but let us acknowledge that you reduced the bcfg2-server startup 
time from 45 minutes to 10-15, while adding some unknown amount of time 
required to mount a tmpfs and check out your data (which presumably came 
off of a disk) as well as adding a big chunk of fixed memory dedicated 
to the bcfg2-server process in the form of an in-memory filesystem.

> No, we didn't try readahead.  We liked tmpfs to speed up the new
> Packages stuff anyway, so went with it as a general solution.

That's unfortunate.  tmpfs is essentially fixed cached data.  Readahead 
should get you the same benefits without the cost of a permanent chunk 
of memory.

>> Maybe you mean that you'll keep all of
>> the XML files in tmpfs and parse them for each request.  That doesn't really

Proposal: Bcfg2 Branching Model

http://trac.mcs.anl.gov/projects/bcfg2/wiki/DevelopmentBranchingModel

I've created a wiki page with a proposed branching model and request
your feedback. The goal of this wiki is to document how branches in
the Bcfg2 code repo will be managed. My hope is that it meets the
following stories:

 * As a user, I want to easily clone Bcfg2 and have confidence that it
will work in my environment.

 * As a developer, I want clear guidelines on how to contribute code
and work with others on features and fixes.

 * As QA or release manager, I want to change the CI server's job
configurations as little as possible.

--
Raul Cuza

Bcfg2 server rearchitecture

As hinted at earlier this week, I've been working on writing up a plan
for overhauling the Bcfg2 server architecture to make it more
scalable.  I'm finally done with the first draft, and it's long.
Really long.  I've put it up on Github so we can work on it
collaboratively:

https://github.com/stpierre/bcfg2-rearchitecture/

If you're interested in this kind of stuff and get a few spare
moments, please read it over, critique it, and, ideally, fork and
improve it.

Thanks!

--

-- 
Chris St. Pierre

Bcfg2 server architecture -- backend storage

I'm working on a writeup of what I see as the path forward to
rearchitect the server portion of Bcfg2 to make it more scalable --
or, really, scalable at all -- according to some discussions I've had
with Narayan and Sol, plus my own half-baked ideas.  I need some input
on what the best way to store the data is going to be as we move to a
more distributed architecture.

My thought is that we can split up the server into a few basic
components which, with proper asynchronous message passing, could be
run on a single machine or on multiple machines to let people scale
where their bottlenecks are.  The components are:

1.  A "master" process that handles SSL, authentication, and request
routing.  This would be fairly lightweight; ideally, something like a
WSGI app under Nginx/Apache, a Mongrel2 app, or a Twisted app.  (All
of those have benefits and drawbacks that I will be exploring further
as I feel this process out.)  It would probably also handle the
initial portion of the client-server discussion -- asserting profiles,
getting probes, receiving probe data, and getting decision lists.
2.  A configuration render process that does all the actual heavy
lifting of rendering templates, generating package lists, and so on.
This is most people's bottleneck nowadays, so I figure most people
would want to run multiple renderers.
3.  Some sort of document/content store to hold the data that is today
stored on the filesystem.  This is my sticking point.

The plain old boring filesystem actually works really great for this,
for a number of reasons:

1.  Really simple retrieval of documents stored in a tree hierarchy.

Metadata group parsing improvements

This is another BEP - a Bcfg2 Enhancement Proposal.

I'd like to rewrite the groups.xml Metadata parser to implement
several new features:

1.  Nested group tags.

Currently, Bcfg2 does not support nesting group tags in groups.xml,
although it's supported everywhere else.  This reduces consistency,
and also makes it difficult to specify certain types of
configurations.  You can implement this sort of logic in Bundler, but
you cannot push out Cfg specific files to compositions of groups.
This would be implemented as, e.g.:

<Group name="foo">
  <Group name="bar">
    <Group name="baz"/>
    <Bundle name="quux"/>
  </Group>
  <Bundle name="xyzzy"/>
</Group>

2.  Client tags.

Client tags are supported in most other structural files -- anything
that uses the StructFile object for its internal modeling.  This
includes Bundler, info.xml, Decisions, Properties, and plenty of
others.  Adding Client tags to groups.xml will increase consistency.
It does not add any features that cannot already be obtained with
GroupPatterns, but it can reduce the complexity of both groups.xml and

Packages plugin acting wonky

Re: bcfg2-server and NAT

bcfg2-server and NAT