Brian Luckau | 18 Jul 23:36 2012
Picon

ARK APIs

Hi,

I have a couple of questions --

1.  What API(s) does ARK have for other software to interface with it?
2. Can it manage a chroot environment, LXC container, or any other method of setting up an image that will eventually be applied as the OS for a Linux host


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
ark-dev mailing list
ark-dev <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ark-dev
Will Partain | 5 Mar 13:48 2007
Picon
Picon

Re: Arusha (ARK) on Wikipedia

(Sorry to be slow in replying.)

Daniel Clark writes:

> There is some movement towards getting a good comparison of
> Configuration Management software up on Wikipedia; ...

Excellent; what you've put up so far is accurate.

> BTW, what's happening with Arusha these days? Is it still in use at
> any sites?

I continue to use it daily.  Either no-one else is using it, or
they are _very_ contented (and have nothing to say :-).  I have a
few scraps to check in, but haven't scared up the cycles to mess
with it.

Will

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
Daniel Clark | 28 Feb 20:05 2007
Picon

Arusha (ARK) on Wikipedia

There is some movement towards getting a good comparison of
Configuration Management software up on Wikipedia; so someone with
more intimate knowledge of ark/arusha than I may want to take a stab
at adding more to these articles:

http://en.wikipedia.org/wiki/Arusha_Project
http://en.wikipedia.org/wiki/Comparison_of_open_source_configuration_management_software

BTW, what's happening with Arusha these days? Is it still in use at
any sites? I've been working on Bcfg2 lately, but I think Arusha has a
lot of great ideas (my main problem with it when I tried setting it up
years ago was that the bring-up was sort of complex, and I was much
more shy back then than I am now :-)

Cheers,
--

-- 
Daniel Clark # http://dclark.us # http://opensysadmin.com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
Will Partain | 8 Aug 14:40 2006
Picon
Picon

Re: squishing install and deploy dirs together

Hi, Jim et al. -- Taking your questions/comment out of order:

> 3.) Am I out of my mind?

No :-)

> 1.) In this context, are there reasons to keep install and deploy 
> distinct?  If so, what?

Probably not (i.e. your analysis is sound).

The install/deploy/reveal thing is *not* supposed to be an all-knowing
everyone-should-do-it-this-way thing.  It's just one way.  The more
common "just stick it in /usr/local" -- make install prefix=/usr/local
-- should also be possible.  Or what you want to do.

For the install/deploy/reveal thing, the steps (and what they get you)
go like this:

* You tell the software at build time that it is in the deploy place; i.e.
  make install prefix=$deploy_dir [whatever that is]

  Because the deploy_dir is traditionally version-specific, you can have
  many versions live at once.

* The business of copying over what you've built to the install area
  (again version-specifically) is so that you have a single golden copy
  per type of machine with which you can do clever things.

* We now deploy the golden cop{y,ies} onto individual machines.  If an
  individual machine has a big disk (they all do now, but they didn't
  when ARK started...), you can deploy by taking a copy.  Or you can
  deploy by symlinking.  Or whatever you like, machine by machine.

  Another thing you can customize is what set of versions of the
  software each machine sees.  So, for example, the Team A machines
  might have an old version; but the Team B machines would have all
  versions.

* Finally, revealing: This is creating a symlink or something in the
  users' PATH so that they can run whatever it is.  My experience is
  that asking users to meddle with their individual PATHs constantly is
  just a recipe for disaster.  So, for example, we just say "put
  /our/bin near the front of your PATH" and declare victory.  Revealing
  puts the right links to the right versions in /our/bin.  Again, it can
  be varied machine by machine (if you're crazy enough to do that).

If you don't want any of that stuff, don't do it that way.

> 2.) Do you have guidance for me in terms of the general approach to 
> making them be the same?  (i.e, Where do I hack?)  Is this going to be a 
> big job or a small one?  Has anyone done something similar that would be 
> useful as a starting point?

I am not aware of someone doing something similar.  I hope it would be a
pretty small job.

Assuming you're sucking on the Sidai methods (i.e. in
$ARK/sidai/package/{GNU,ALL}.xml), I'd proceed by...

* cutting out unnecessary methods -- so, for instance, if you want
'reveal-bits' to depend on 'install' rather than 'deploy', just change
the dependency in the 'reveal-bits' method.

* simply change the method shell scripts to do what you want

I'd do them in place (for simplicity).  Afterwards, you can move your
changed files out of the way, re-'cvs co' the sidai stuff, then use the
diffs (which I would expect to be modest) to fashion some .xml for your
own team that would then override the sidai stuff and do what you want.

Hope this helps,

Will

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Jim Rowan | 7 Aug 21:00 2006

squishing install and deploy dirs together


Hi,

I'm considering a model where the install, deploy and reveal dirs are 
all in AFS.  As I understand it, most of the arguments for having 
distinct install and deploy dirs disappear in this context.  (I can't 
think of any good reasons to have separate dirs, except that you have to 
avoid trampling on existing deployed code that is being used.)  Thus, 
I'm thinking that the ideal configuration is one where you install into 
the deploy dir and just leave it there -- instead of copying it back to 
install, etc.

I haven't gotten as far as looking into exactly how one might achieve 
this.  Before I do, I have have some questions:

1.) In this context, are there reasons to keep install and deploy 
distinct?  If so, what?

2.) Do you have guidance for me in terms of the general approach to 
making them be the same?  (i.e, Where do I hack?)  Is this going to be a 
big job or a small one?  Has anyone done something similar that would be 
useful as a starting point?

3.) Am I out of my mind?

Jim

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Will Partain | 5 May 16:35 2006
Picon
Picon

Re: Questions from the start..

> I'm just starting to play with Ark in order to see if it addresses
> some problems I've got.

Hi, Michael; happy to see what we can do...

> When following the "try-ark" doc, I came up with these questions:
> 1) Does the test-dir area have to not be /tmp?

You should be able to put it anywhere.  /tmp is sometimes slightly
magical (esp across reboots :-).

> 2) Does the ark-src-dir have to be a hard path and not an
>     environmental variable (as in ark_profile)?

A path, yes. I'd recommend an absolute path.

> 3) Is "solo-host" the "controller"

I'm not sure what you mean by "the controller".  The idea of the try-ark
experiment is that you use ARK on a single machine to build a few
packages in a non-special directory (e.g. under your home directory).

Once the experiment is over, you can throw it away.  Then you will write
an ARK description that fits your site and what you are trying to do.
ARK isn't a product that you download and it magically does fixed tasks
for you; rather, it is a "language" with which you can describe the
sysadmin tasks/situations at your site that you care about.  (Happily,
you can build upon what others have described -- the "sidai" team is
nothing but a big pile of descriptions that you can build upon.  That's
what the try-ark experiment does.)

> It seems like the activity in this product has dwindled.

Which may be a good or bad thing, depending on your point of view :-)

The core ARK engine (which processes the "language" mentioned above)
hasn't changed much in five years or so.  I personally consider this a
Very Good Thing.

I use the ARK stuff every day and plan to keep doing so; I don't know
about anyone else.  I usually release a full set of my bits about once a
year and, yes, I'm running a little behind schedule this year :-(

> Does anyone have an alternative to this one?

Depending on what you are trying to do, the answer is likely to be 'yes'.

Will

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Michael Tiernan | 4 May 23:47 2006
Picon
Picon

Questions from the start..

I'm just starting to play with Ark in order to see if it addresses
some problems I've got.

When following the "try-ark" doc, I came up with these questions:
1) Does the test-dir area have to not be /tmp?
2) Does the ark-src-dir have to be a hard path and not an
    environmental variable (as in ark_profile)?
3) Is "solo-host" the "controller"

It seems like the activity in this product has dwindled.
Does anyone have an alternative to this one?
--
    << MCT >>   Michael C Tiernan.
    Is God a performance artist?
    EGO hack vivo quod ago accido.

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0709&bid&3057&dat1642
Will Partain | 2 Dec 22:55 2005
Picon
Picon

Re: gcc, specs, and rethinking the sidai -R thing

Responding to Jim Rowan... sorry to be so slow.

> I'd love to see the -R stuff go away as well.  It is a major detractor.
> 
> However, your proposal causes the behavior of ark installed programs
> to be dependant on the configuration of the machine (OS) itself --

[Recap: "my" [vague] proposal was: lose the -R stuff, have some extra
ARK gunk to "manage" the ld.config game (where to find .so files).]

Jim, I agree with your misgivings.  After all, something must've
pushed me into this -R thing in the first place :-)

I wonder if some sort of hybrid scheme would make sense?  I observe
that not all .so files are created equal...

* things like libgcc_s.so -- every program needs it; carefully managed
  by the GCC lads... Why not just put this in your ld.config...

    /our/.-ark-deploy/gcc--4.0.2/lib
    /our/.-ark-deploy/gcc--3.4.3/lib
    /our/.-ark-deploy/gcc--2.95.3/lib

  ... and declare victory?  If your program picks up the version from
  a different version than what you might've expected, it will almost
  certainly be OK.

* things like libz.so -- very stable interfaces; likely only upgraded
  for security reasons; ubiquitous use; versioning done responsibly.
  Again, having these picked up from a well-controlled ld.config will
  probably be OK.

* things like the Berkeley DB libraries -- you do _not_, on pain of
  death, want a program picking up a version other than the one you
  explicitly intended! -- a horrible horrible death will certainly
  follow.  -R all the way.

Of course, judgement calls are being made in all of that.  So (if you
decided to try to distinguish between those cases), it would a
delicate bit of ARK scripting to keep it all together.

Meanwhile, I wildly support that you should be able to do the kinds of
things you're trying to do ("you don't need anything but a raw box").

(Let me know if I've forgotten something, e.g. from earlier in the
thread.)

Will

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
Will Partain | 21 Nov 20:51 2005
Picon
Picon

gcc, specs, and rethinking the sidai -R thing

Following on from some of Jim Rowan's recent experience (and mine),
I'm rethinking some of the Sidai way of dealing with shared
libraries...

Context ... The old traditional way was: build a library, throw it
into /usr/local/lib, put /usr/local/lib in your LD_LIBRARY_PATH, and
off you go.  Good: easy; bad: /usr/local/lib becomes a mess and (if
you're not careful) a bit of a no-go area ("Don't delete anything, you
have no idea what program relies on that.")

Reaction, the Sidai way: Libraries get built, put into (something
like) /our/.-ark-deploy/<package>--<version>/lib, and everything that
needs those libraries gets linked with "-R
/our/.-ark-deploy/<package>--<version>/lib".  Good: you're being
Really Explicit about what you want; bad: it's a bit of a pain, and
it's hard to replace a library with a simple improvement (e.g. one
security fix).

You've also got to get GCC to cooperate... it links a shared library
libgcc_s.so into lots of things.  So, in building GCC, we 'sed' the
so-called 'specs' file to produce one that inserts a "-R
/our/.-ark-deploy/gcc--4.0.2/lib" (for example) into every link
invocation.

Ugly, but it used to work :-( Recently, Jim reported that it didn't
for him; when I just built gcc 4.0.2, it doesn't even *have* a
deployed specs file!  (I think the specs must get wired into the 'gcc'
binary, or something.)

Now, there's another mechanism for this kind of thing, namely setting
up ld.config (or equivalent).  You tell the runtime linker "here are
some places to look for shared libraries", and then it does the rest.
So, for example, I did [on a Sun]...

  sudo crle -l /usr/lib:/our/.-ark-deploy/gcc--4.0.2/lib

... and my libgcc_s problems went away forever :-)

Now, it's quite easy to play this ld.config game so that it's just as
ill-disciplined as the "throw it all in /usr/local/include" game.  But
surely there is a way to gain most of the discipline of the "I'll tell
you _exactly_ what I want" -R'ing scheme.  Something like... each
package providing libraries has a field...

    <ld-config-spec>
      <param name="prefix">!sidai_prefix</param>
    <string>$prefix/lib</string>
    </ld-config-spec>

... and some small program comes through and collects these, and sets
ld.config appropriately.

I haven't done all this yet, but am seriously considering it.  Love to
see all that -R stuff go up in smoke.  Thoughts?

Will

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
Register for a JBoss Training Course.  Free Certification Exam
for All Training Attendees Through End of 2005. For more info visit:
http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
Jim Rowan | 17 Nov 23:27 2005

python; configure; ark...

Still fighting against this python configure. :(  Following is long and 
complicated, but I would appreciate whatever ideas anyone has!

My earlier discovery that unsetting CONFIG_SHELL allowed me to run 
configure to completion.  However, I have now run into something else 
and I am baffled by the results of my debugging efforts.

The "problem" is that when I ask ark to configure the package, I get an 
error from configure:

...
+ ./configure --prefix=/our/.-ark-deploy/python--2.4.2 --disable-nls
LDFLAGS= -L/our/.-ark-deploy/python--2.4.2/lib 
-Wl,-R/our/.-ark-deploy/python--2.4.2/lib CXXFLAGS=-O2 
-I/our/.-ark-deploy/python--2.4.2/include CFLAGS=-O2 -fstrict-aliasing 
-I/our/.-ark-deploy/python--2.4.2/include CPPFLAGS= 
-I/our/.-ark-deploy/python--2.4.2/include CXX=/our/bin/g++ 
CC=/our/bin/gcc CONFIG_SHELL=
checking MACHDEP... sunos5
checking EXTRAPLATDIR...
checking for --without-gcc... no
checking for --with-cxx=<compiler>... no
checking for c++... /our/bin/g++
checking for C++ compiler default output file name... a.out
checking whether the C++ compiler works... configure: error: cannot run 
C++ compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details.

config.log tells us a bit more:

...
configure:1833: result: a.out
configure:1838: checking whether the C++ compiler works
configure:1844: ./a.out
ld.so.1: ./a.out: fatal: libgcc_s.so.1: version `GCC_3.3' not found 
(required by file /our/.-ark-deploy/gcc--3.4.4/lib/libstdc++.so.6)
configure:1847: $? = 137
configure:1854: error: cannot run C++ compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details.
...

And this would point fairly clearly to a runtime load library path kind 
of problem...  except that I have very conflicting/confusing other info:

I've tried to recreate this failure by running it in various ways by 
hand, and I cannot recreate it.

0.) If I run configure by hand, (with no args) it works just fine.

1.) I hacked the configure script to save a copy of the a.out that it 
built when run under ark.  I can run a.out just fine.  I don't have 
LD_LIBRARY_PATH set in my environment.

2.) I tracked down the python code in arkbase/ark/think.py that runs the 
shell code that is built up, and uncommented the debug statements near 
line 1800:
        elif lang == 'sh':
            #print 'pre=',`self.preambleSh()`
            #print 'exp=',`self.expandParams(lang)`
            #print 'cod=',`string.strip(code)`
Ran the ark package configure again; saved the resulting statements to a 
file, messed with it to remove/adjust the quoting, and ended up with a 
file (foo) that I can run with the command "sh foo".  This does 
everything that I think ark is doing (certainly the output looks very 
similar).  *It* also runs to completion, and doesn't have trouble with 
the C++ compiler/loader issue.  foo is attached.

This latter one is the most perplexing.  I don't understand how the 
"environment" when run via "ark package configure" might be different 
from what "foo" does -- but it clearly is different enough that it has 
this problem....

Got any ideas?

(I may or may not have a problem with the compiler and libstdc++, but 
even if I do, why does this stuff succeed when run by hand and fail when 
run by ark.?)

(I think I really need to understand this unable-to-recreate-failure, so 
that I can successfully debug any compiler problem I have.)

ARK_PROFILE='/usr2/jmr/.ark_profile';export
ARK_PROFILE
VERBOSE='-v'
PRETEND=''
KEEP_GOING=''
HARD_UNDO=''
PATH='/our/bin:/usr/local/bin:/usr/ccs/bin:/usr/bin:/bin:/usr/sbin:/sbin:/pkg/qct/bin:/usr/X/bin';export
PATH # don't leave PATH to chance!

# useful shell functions:
sidaiPickProgram () {
    for p in "$ <at> " ; do
	case "$p" in
	  /*|./*|../*)
	    p_w1=`echo $p | sed -e 's/ .*//'`
	    if [ -x "$p_w1" ] ; then
	       echo $p
	       return
	    fi
	    ;;

	  *) # relative path, so don't even try
            echo $p
	    return
	    ;;
	esac
   done
   echo "No executable program among: $*" 2>& 1
   exit 1
}

sidaiClearPkgState () {
   tag=$1
   PKG_STATE_DIR="$PKGS_STATE_DIR/$PKG_ID"
   if [ -d $PKG_STATE_DIR ] ; then
       cd $PKG_STATE_DIR
       f1=$tag
       f2=$f1.xml
       f1f=$f1-FORCED
       f2f=$f2-FORCED
       for i in $f1 $f2 $f1f $f2f ; do
          if [ -f $i ]; then /bin/rm $i ; fi
       done
   fi
}

umask 022 # so we don't pick up something random from ssh/rsh
PKG_BUILD_DIR='/prj/qct/sysadmin/aus-it/arik/ark-builds/python--2.4.2'
PKG_DEPLOY_DIRS='/our/.-ark-deploy/python--2.4.2'

set -e
set -u
set -x

CONFIG_SHELL=''
PKG_PROTO_HOST='sparc-solaris9'
dep_iflags='-I/our/.-ark-deploy/python--2.4.2/include'
dep_lflags='-L/our/.-ark-deploy/python--2.4.2/lib -Wl,-R/our/.-ark-deploy/python--2.4.2/lib'
pre_config_eval='/bin/rm -f Iflags Lflags; echo "$dep_iflags" > Iflags; echo "$dep_lflags" > Lflags'
extra_configure_args='--disable-nls'
FIND='/usr/bin/find'
XARGS='/usr/bin/xargs'
cc='/our/bin/gcc'
cxx='/our/bin/g++'
ccboot='/pkg/gnu_compilers/gcc-3.2.3/bin/gcc'
cxxboot='/pkg/gnu_compilers/gcc-3.2.3/bin/c++'
cppflags=' -I/our/.-ark-deploy/python--2.4.2/include'
cflags='-O2 -fstrict-aliasing -I/our/.-ark-deploy/python--2.4.2/include'
cxxflags='-O2 -I/our/.-ark-deploy/python--2.4.2/include'
ldflags=' -L/our/.-ark-deploy/python--2.4.2/lib -Wl,-R/our/.-ark-deploy/python--2.4.2/lib'
cppflagsboot=' -I/our/.-ark-deploy/python--2.4.2/include'
cflagsboot='-O2 -fstrict-aliasing -I/our/.-ark-deploy/python--2.4.2/include'
cxxflagsboot='-O2 -I/our/.-ark-deploy/python--2.4.2/include'
ldflagsboot=' -L/our/.-ark-deploy/python--2.4.2/lib -Wl,-R/our/.-ark-deploy/python--2.4.2/lib'
in_dir='.'
CONFIGURE='./configure'
configure_args=''
prefix='/our/.-ark-deploy/python--2.4.2'

cd $PKG_BUILD_DIR/$PKG_PROTO_HOST/$in_dir

# better safe than sorry...
$FIND . \( -name 'config*.cache' -o -name autom4te.cache -o -name config.log -o -name config.status -o
-name conftest.c \) -print | $XARGS /bin/rm -rf

# don't want any user prefs to trip us up:
if [ "x${CONFIG_SITE:-}" != 'x' ] ; then unset CONFIG_SITE ; fi

cc=`sidaiPickProgram  "$cc"  "$ccboot"`
if [ "x$cc" = "x$ccboot" ] ; then
    cppflags="$cppflagsboot"
    cflags="$cflagsboot"
    ldflags="$ldflagsboot"
fi

cxx=`sidaiPickProgram "$cxx" "$cxxboot"`
if [ "x$cxx" = "x$cxxboot" ] ; then
    cppflags="$cppflagsboot"
    cxxflags="$cxxflagsboot"
    ldflags="$ldflagsboot"
fi

if [ "x${pre_config_eval:-}" != "x" ] ; then
    # this let's you get weird stuff in the environment, for example
    eval "$pre_config_eval"
fi

CONFIG_SHELL="$CONFIG_SHELL" \
CC="$cc"		\
CXX="$cxx"		\
CPPFLAGS="$cppflags"	\
CFLAGS="$cflags"	\
CXXFLAGS="$cxxflags"	\
LDFLAGS="$ldflags"	\
$CONFIGURE --prefix=$prefix $configure_args $extra_configure_args

Will Partain | 14 Nov 16:39 2005
Picon
Picon

Re: patch: sidai/package/GNU.xml

Jim Rowan writes:

> I think you're missing the point; ...

No, just floundering helplessly as usual :-)

> And finally, I figured out I should check this outside of ark.
> export CONFIG_SHELL=/bin/bash
> ./configure
> 
> FAILS!  This is an autoconf/python configure issue... it doesn't work
> like it should. :(

Well figured-out.  This would suggest that the Sidai-ish way to encode
your knowledge is not to set <CONFIG-SHELL> for a host (because it's
not a platform-ish thing); rather, in $ARK/sidai/package/python.xml
[assuming you're using that as one of your prototypes], record...

<configure>
  <!-- setting config_shell breaks things as at version 2.x.y -->
  <param name="CONFIG_SHELL"></param>
</configure>

I guess I'll add that in, unless someone yells.

Will

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php

Gmane