ARK APIs

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
ark-dev mailing list
ark-dev <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ark-dev

Re: Arusha (ARK) on Wikipedia

(Sorry to be slow in replying.)

Daniel Clark writes:

> There is some movement towards getting a good comparison of
> Configuration Management software up on Wikipedia; ...

Excellent; what you've put up so far is accurate.

> BTW, what's happening with Arusha these days? Is it still in use at
> any sites?

I continue to use it daily.  Either no-one else is using it, or
they are _very_ contented (and have nothing to say .  I have a
few scraps to check in, but haven't scared up the cycles to mess
with it.

Will

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Arusha (ARK) on Wikipedia

There is some movement towards getting a good comparison of
Configuration Management software up on Wikipedia; so someone with
more intimate knowledge of ark/arusha than I may want to take a stab
at adding more to these articles:

http://en.wikipedia.org/wiki/Arusha_Project
http://en.wikipedia.org/wiki/Comparison_of_open_source_configuration_management_software

BTW, what's happening with Arusha these days? Is it still in use at
any sites? I've been working on Bcfg2 lately, but I think Arusha has a
lot of great ideas (my main problem with it when I tried setting it up
years ago was that the bring-up was sort of complex, and I was much
more shy back then than I am now 

Cheers,
--

-- 
Daniel Clark # http://dclark.us # http://opensysadmin.com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: squishing install and deploy dirs together

Hi, Jim et al. -- Taking your questions/comment out of order:

> 3.) Am I out of my mind?

No 

> 1.) In this context, are there reasons to keep install and deploy 
> distinct?  If so, what?

Probably not (i.e. your analysis is sound).

The install/deploy/reveal thing is *not* supposed to be an all-knowing
everyone-should-do-it-this-way thing.  It's just one way.  The more
common "just stick it in /usr/local" -- make install prefix=/usr/local
-- should also be possible.  Or what you want to do.

For the install/deploy/reveal thing, the steps (and what they get you)
go like this:

* You tell the software at build time that it is in the deploy place; i.e.
  make install prefix=$deploy_dir [whatever that is]

  Because the deploy_dir is traditionally version-specific, you can have
  many versions live at once.

* The business of copying over what you've built to the install area
  (again version-specifically) is so that you have a single golden copy
  per type of machine with which you can do clever things.

* We now deploy the golden cop{y,ies} onto individual machines.  If an
  individual machine has a big disk (they all do now, but they didn't
  when ARK started...), you can deploy by taking a copy.  Or you can
  deploy by symlinking.  Or whatever you like, machine by machine.

  Another thing you can customize is what set of versions of the
  software each machine sees.  So, for example, the Team A machines
  might have an old version; but the Team B machines would have all
  versions.

* Finally, revealing: This is creating a symlink or something in the
  users' PATH so that they can run whatever it is.  My experience is
  that asking users to meddle with their individual PATHs constantly is
  just a recipe for disaster.  So, for example, we just say "put
  /our/bin near the front of your PATH" and declare victory.  Revealing
  puts the right links to the right versions in /our/bin.  Again, it can
  be varied machine by machine (if you're crazy enough to do that).

If you don't want any of that stuff, don't do it that way.

> 2.) Do you have guidance for me in terms of the general approach to 
> making them be the same?  (i.e, Where do I hack?)  Is this going to be a 
> big job or a small one?  Has anyone done something similar that would be 
> useful as a starting point?

I am not aware of someone doing something similar.  I hope it would be a
pretty small job.

Assuming you're sucking on the Sidai methods (i.e. in
$ARK/sidai/package/{GNU,ALL}.xml), I'd proceed by...

* cutting out unnecessary methods -- so, for instance, if you want
'reveal-bits' to depend on 'install' rather than 'deploy', just change
the dependency in the 'reveal-bits' method.

* simply change the method shell scripts to do what you want

I'd do them in place (for simplicity).  Afterwards, you can move your
changed files out of the way, re-'cvs co' the sidai stuff, then use the
diffs (which I would expect to be modest) to fashion some .xml for your
own team that would then override the sidai stuff and do what you want.

Hope this helps,

Will

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: gcc, specs, and rethinking the sidai -R thing

Responding to Jim Rowan... sorry to be so slow.

> I'd love to see the -R stuff go away as well.  It is a major detractor.
> 
> However, your proposal causes the behavior of ark installed programs
> to be dependant on the configuration of the machine (OS) itself --

[Recap: "my" [vague] proposal was: lose the -R stuff, have some extra
ARK gunk to "manage" the ld.config game (where to find .so files).]

Jim, I agree with your misgivings.  After all, something must've
pushed me into this -R thing in the first place 

I wonder if some sort of hybrid scheme would make sense?  I observe
that not all .so files are created equal...

* things like libgcc_s.so -- every program needs it; carefully managed
  by the GCC lads... Why not just put this in your ld.config...

    /our/.-ark-deploy/gcc--4.0.2/lib
    /our/.-ark-deploy/gcc--3.4.3/lib
    /our/.-ark-deploy/gcc--2.95.3/lib

  ... and declare victory?  If your program picks up the version from
  a different version than what you might've expected, it will almost
  certainly be OK.

* things like libz.so -- very stable interfaces; likely only upgraded
  for security reasons; ubiquitous use; versioning done responsibly.
  Again, having these picked up from a well-controlled ld.config will
  probably be OK.

* things like the Berkeley DB libraries -- you do _not_, on pain of
  death, want a program picking up a version other than the one you
  explicitly intended! -- a horrible horrible death will certainly
  follow.  -R all the way.

Of course, judgement calls are being made in all of that.  So (if you
decided to try to distinguish between those cases), it would a
delicate bit of ARK scripting to keep it all together.

Meanwhile, I wildly support that you should be able to do the kinds of
things you're trying to do ("you don't need anything but a raw box").

(Let me know if I've forgotten something, e.g. from earlier in the
thread.)

Will

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click

Re: patch: sidai/package/GNU.xml

Jim Rowan writes:

> I think you're missing the point; ...

No, just floundering helplessly as usual 

> And finally, I figured out I should check this outside of ark.
> export CONFIG_SHELL=/bin/bash
> ./configure
> 
> FAILS!  This is an autoconf/python configure issue... it doesn't work
> like it should. :(

Well figured-out.  This would suggest that the Sidai-ish way to encode
your knowledge is not to set <CONFIG-SHELL> for a host (because it's
not a platform-ish thing); rather, in $ARK/sidai/package/python.xml
[assuming you're using that as one of your prototypes], record...

<configure>
  <!-- setting config_shell breaks things as at version 2.x.y -->
  <param name="CONFIG_SHELL"></param>
</configure>

I guess I'll add that in, unless someone yells.

Will

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php

Re: patch: sidai/package/GNU.xml

Jim Rowan wrote:

> + ./configure --prefix=/our/.-ark-deploy/python--2.4.2 --disable-nls
> LDFLAGS= -L/our/.-ark-deploy/python--2.4.2/lib
> -Wl,-R/our/.-ark-deploy/python--2.4.2/lib CXXFLAGS=-O2
> -I/our/.-ark-deploy/python--2.4.2/include CFLAGS=-O2 -fstrict-aliasing
> -I/our/.-ark-deploy/python--2.4.2/include CPPFLAGS=
> -I/our/.-ark-deploy/python--2.4.2/include
> CXX=/pkg/gnu_compilers/gcc-3.2.3/bin/c++ CC=/our/bin/gcc
> CONFIG_SHELL=/bin/ksh

Hmmm... Well, at least we know it got set.  Doing a little test on
this sol10 box:

  partain <at> slimy% ksh
  $ test -e ~/foo
  $ type test
  test is a shell builtin
  $ test -e
  ksh: test: argument expected
  $ test -e /dev/ptmx
  $ echo $?
  0
  $ 

In summary: it makes no sense at all   Unless you have some other
program called 'test' in your PATH and it misbehaves in just the right
way... I doubt it.

What if you change your <CONFIG-SHELL> setting for solaris to /bin/bash?
I mean, if you've *got* a /bin/bash, I would expect GNU stuff to be
very bash-friendly.

An unsatisfactory answer... I hate things like this 

Will

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php

Re: patch: sidai/package/GNU.xml

Jim Rowan writes:

> RCS file: /cvsroot/ark/sidai/package/GNU.xml,v
> retrieving revision 1.36
> diff -u -r1.36 GNU.xml
> --- GNU.xml     11 Apr 2005 11:29:09 -0000      1.36
> +++ GNU.xml     9 Nov 2005 23:39:59 -0000
>  <at>  <at>  -121,7 +121,7  <at>  <at> 
>      eval "$pre_config_eval"
>  fi
>  -CONFIG_SHELL="$CONFIG_SHELL" \
> +CONFIGURE="$CONFIG_SHELL" \
>  CC="$cc"               \
>  CXX="$cxx"             \
>  CPPFLAGS="$cppflags"   \

Hmm... I don't buy this.  It simply says "Don't set CONFIG_SHELL"
(which is probably not right for all GNU-built programs, which is what
that prototype is for), and instead sets the CONFIGURE environment
variable... which is never used in a configure script.  (I checked the
Python one and one or two others.)

You will note that the <host> prototypes
sidai/host/{linux,solaris}.xml have settings for <CONFIG-SHELL>, to
either /bin/bash or /bin/ksh, respectively.

Did you try setting <CONFIG-SHELL> for your relevant host (the one
doing the building), either with the sidai prototype, or one of your
own (or just directly in your <host>.xml)?

If setting CONFIG-SHELL correctly doesn't work, then we have a problem


Will

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php

Re: welcome to Arusha!

Hi, Jim (and others) -- Arusha land is pretty quiet, but still
here...  On your various remarks:

> 1.) A sidai suggestion: sidai/package/rsync.xml has a phrase in the
> <patch> section that names a bunch of rsync versions.  I think the
> sense of the test should be reversed, so it tests *for* [
> "$PKG_VERSION" = "2.5.5" ] and defaults to : ok # no worries, mate.
> (The "doesn't need a patch" list is already out of date, 2.6.6 is out
> and the build fails because of this...)

*Yes*.  There are many instances of equally egregious code.  Reason:
the shortest path to success is to add '-o <new-version>' and move on.
But, this being "collaborative sysadmin", I guess I'll need to do
better 

> 2.) Mostly a statement: the 'how do I migrate from the "try-ark" mode
> to a real site' instructions are .... lacking.  I got a bit frustrated
> trying to discover in which dark corner the magic config file that was
> causing certain behavior lived... turned out I had a mixture of old
> and new happening, and didn't know that, because of the level of magic
> involved.  At that early phase of maturity, this needs to be smoother.
> (I know -- send patches...  :)

Urgh.  I _do_ tend to test that stuff before releases (which are now
about a year apart -- CVS updates happen, of course...), but of course
I never see it with newbie eyes.  All advice welcome...

> 3.) Newbie Q: what happens if a machine is down when a deploy/reveal
> command is issued?  Is there anything akin to "self-healing state
> management" in here?

Er, not really.  There may be some command-line gunk (--down-hosts=?)
to advise the tool to steer clear; I'd have to look it up.  You can
also change a host <status> to "pending", which means it will get put
into tables and things (i.e. its a "player") but won't get ssh'd to.
I often do this if I know a machine's going to be down for a few days.

The _right_ solution is something like a client cron job that does
'ark package reveal ALL' every 15 mins.  This would need some slight
extra mechanism (which I've never needed) for someone to say "this one
is good to go", after which the cron job would get it.  The machinery
exists... 

   ark package good-to-go synopsys-vcs--2005.06

... which does little more than 'touch' some sentinel; then change
the deploy method a bit to look for the sentinel.

> 4.) Anyone done any windows integration?    Thoughts in that direction?

Um... sorry.

> 5.) Are there any utilities/commands to report on the ark-state info?

No; there should be.  Got an idea for one? -- maybe we could write it.

> 6.) A sidai question: I built a version of gcc to go into /our; used
> the sidai template with just a couple of tiny tweaks.  It got
> installed/deployed in a somewhat unexpected manner -- in the deploy
> tree I have a whole tree of subdirectories, and (most of) the leaves
> are symlinks into the install tree.  A couple of things are directly
> stored in the deploy tree.  All the other apps I've done so far have
> the top of the deploy tree just symlinked to the top of the respective
> install tree. Is this expected, and if so -- what is the logic behind
> doing it this way; and what directive triggered the different install?

The idea was: for a few bits of a GCC build (libgcc.so for example),
you really don't want to be making NFS trips to get it.  So each
(per-client) deployment has a copy of that.  On the other hand,
for swathes of a GCC distribution, one shared copy is fine.  So
symlinks to those bits.  The guilty party is ($ARK/sidai/package/gcc.xml):

<deployment-spec><table><entry name="*">
 <at> TYPE <at> =manifest
^lib/lib.*\.so copy * * 755
.              link
</entry></table></deployment-spec>

If you just wanted to copy blindly, you'd change (override) that with

<deployment-spec><table><entry name="*">
 <at> TYPE <at> =copy
</entry></table></deployment-spec>

for example.

Will

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php

squishing install and deploy dirs together

Hi,

I'm considering a model where the install, deploy and reveal dirs are 
all in AFS.  As I understand it, most of the arguments for having 
distinct install and deploy dirs disappear in this context.  (I can't 
think of any good reasons to have separate dirs, except that you have to 
avoid trampling on existing deployed code that is being used.)  Thus, 
I'm thinking that the ideal configuration is one where you install into 
the deploy dir and just leave it there -- instead of copying it back to 
install, etc.

I haven't gotten as far as looking into exactly how one might achieve 
this.  Before I do, I have have some questions:

1.) In this context, are there reasons to keep install and deploy 
distinct?  If so, what?

2.) Do you have guidance for me in terms of the general approach to 
making them be the same?  (i.e, Where do I hack?)  Is this going to be a 
big job or a small one?  Has anyone done something similar that would be 
useful as a starting point?

3.) Am I out of my mind?

Jim

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Questions from the start..

> I'm just starting to play with Ark in order to see if it addresses
> some problems I've got.

Hi, Michael; happy to see what we can do...

> When following the "try-ark" doc, I came up with these questions:
> 1) Does the test-dir area have to not be /tmp?

You should be able to put it anywhere.  /tmp is sometimes slightly
magical (esp across reboots .

> 2) Does the ark-src-dir have to be a hard path and not an
>     environmental variable (as in ark_profile)?

A path, yes. I'd recommend an absolute path.

> 3) Is "solo-host" the "controller"

I'm not sure what you mean by "the controller".  The idea of the try-ark
experiment is that you use ARK on a single machine to build a few
packages in a non-special directory (e.g. under your home directory).

Once the experiment is over, you can throw it away.  Then you will write
an ARK description that fits your site and what you are trying to do.
ARK isn't a product that you download and it magically does fixed tasks
for you; rather, it is a "language" with which you can describe the
sysadmin tasks/situations at your site that you care about.  (Happily,
you can build upon what others have described -- the "sidai" team is
nothing but a big pile of descriptions that you can build upon.  That's
what the try-ark experiment does.)

> It seems like the activity in this product has dwindled.

Which may be a good or bad thing, depending on your point of view 

The core ARK engine (which processes the "language" mentioned above)
hasn't changed much in five years or so.  I personally consider this a
Very Good Thing.

I use the ARK stuff every day and plan to keep doing so; I don't know
about anyone else.  I usually release a full set of my bits about once a
year and, yes, I'm running a little behind schedule this year 

> Does anyone have an alternative to this one?

Depending on what you are trying to do, the answer is likely to be 'yes'.

Will

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642