headers
Willemse, Menno | 2 Oct 2006 18:07
Picon

Distributing private key information at install time

Hello World,

Here's a problem that I'm sure some of the larger sites have solved already, and that just isn't irritating
enough for me to solve it here and now due to a lack of circular tuits. Since there is an infrastructure angle
to this, I decided to post it here.

Whenever I reinstall a machine, I set up secure shell on the box, and every time it will run ssh-keygen to make
host keys. And then SSH gets all annoyed at me because the host key has changed and I have to remove it from
known_hosts to assure SSH that it's allright really. Not just me, actually. Everybody has to do that for
each name under which they ever accessed the box. So what you really need is a way to restore the original
host keys, so that all the ssh clients will have the correct information already.

This always gets me thinking: Is there a cryptographically sound way to restore the key information to the
client from a file on the install server? You can't get it from an NFS server. My first instinct is to create
an install image with a known private key, and to trust that key to fetch the real key at install time. But
this is obviously vulnerable to the install image's key being stolen. It needs to be on an NFS server as I use
NIM for AIX. For Linux, it's FAI.

Now I know that the only truly secure method is to carry the key over to the machine in question on a floppy and
then burn the floppy, but I'm willing to settle for "Reasonably Secure", as long as the whole method is
hands-off. I may not have that many Unix boxes now, but we may get a lot of Linux workstations in the future.

What is everybody else doing?

Cheers,
   Menno Willemse

--

-- 
Menno Willemse - John Guest IT Department
Tel: 01895-449233 ext 290 Email: menno.willemse <at> johnguest.co.uk
(Continue reading)

Permalink | Reply | 3 comments |
headers
Daniel Hagerty | 2 Oct 2006 20:20

Distributing private key information at install time

 > Whenever I reinstall a machine, I set up secure shell on the box,
 > and every time it will run ssh-keygen to make host keys. And then
 > SSH gets all annoyed at me because the host key has changed and I
 > have to remove it from known_hosts to assure SSH that it's allright
 > really. Not just me, actually. Everybody has to do that for each
 > name under which they ever accessed the box. So what you really
 > need is a way to restore the original host keys, so that all the
 > ssh clients will have the correct information already.

    Well, there's a different problem that you have some chance of
running into: what do you do if you're regenerating a key because a
machine was compromised?  You really do want to burn the old key at
that point.

    You want to centralize the ssh_known_hosts file for your site to
the /etc version.  Ssh provides some tools for doing this, although
what they give you isn't quite a 100% solution.  Take a look at
ssh-keyscan.
Permalink | Reply | 0 comments |
headers
Wesley Craig | 3 Oct 2006 02:41
Picon

Re: state machines

On 22 Sep 2006, at 03:28, Willemse, Menno wrote:
> Without the occasional image backup, as far as I can tell, you are  
> stuck replaying the whole history since time began. If you simply  
> restore a backup, you won't have the latest changes. To get the  
> whole thing, you need to do both.

In our environment, we're making the change with the tool that will  
also be used to restore, so a restore includes the latest changes.

:wes
Permalink | Reply | 0 comments |
headers
Wesley Craig | 3 Oct 2006 02:53
Picon

Re: Distributing private key information at install time

On 02 Oct 2006, at 12:07, Willemse, Menno wrote:
> This always gets me thinking: Is there a cryptographically sound  
> way to restore the key information to the client from a file on the  
> install server?

We give admins boot CDs which contain an individualized key which  
expires.  The admin can burn this CD with a short expiration many  
times, if they happen to be installing a large number of machines.   
Typically they carry a CD with a longer term key with them, for  
troubleshooting.  The key on the CD identifies the admin to the  
centralized server(s), so they are allowed to retrieve the ssh key  
files (among other things) that they're responsible for.  Down side  
of the system is that CDs might be stolen.

:wes
Permalink | Reply | 0 comments |
headers
Daniel Clark | 4 Oct 2006 02:34
Picon
Favicon
Gravatar

Re: Distributing private key information at install time

On 10/2/06, Willemse, Menno <Menno.Willemse <at> johnguest.co.uk> wrote:
> Hello World,
>
> Here's a problem that I'm sure some of the larger sites have solved already, and that just isn't irritating
enough for me to solve it here and now due to a lack of circular tuits. Since there is an infrastructure angle
to this, I decided to post it here.
>
> Whenever I reinstall a machine, I set up secure shell on the box, and every time it will run ssh-keygen to
make host keys. And then SSH gets all annoyed at me because the host key has changed and I have to remove it
from known_hosts to assure SSH that it's allright really. Not just me, actually. Everybody has to do that
for each name under which they ever accessed the box. So what you really need is a way to restore the original
host keys, so that all the ssh clients will have the correct information already.
>
> This always gets me thinking: Is there a cryptographically sound way to restore the key information to the
client from a file on the install server? You can't get it from an NFS server. My first instinct is to create
an install image with a known private key, and to trust that key to fetch the real key at install time. But
this is obviously vulnerable to the install image's key being stolen. It needs to be on an NFS server as I use
NIM for AIX. For Linux, it's FAI.
>
> Now I know that the only truly secure method is to carry the key over to the machine in question on a floppy and
then burn the floppy, but I'm willing to settle for "Reasonably Secure", as long as the whole method is
hands-off. I may not have that many Unix boxes now, but we may get a lot of Linux workstations in the future.
>
> What is everybody else doing?

This is one one of the main plugins system example cases for the Bcfg2
(http://www.bcfg2.org) configuration management system.

"[This problem] makes the ideal case for the plugin interface, since
(a) all admins know the problem well, and (b) can easily envision a
(Continue reading)

Permalink | Reply | 0 comments |
headers
Willemse, Menno | 4 Oct 2006 12:05
Picon

Distributing private key information at install time

Hello World,

Thanks to all those who responded. The prevailing wisdom seems to be that you use a boot CD with a temporary
key to do the re-install, or that you somehow keep the old keys on the system where they won't be erased by the
reinstall. A different hard disk was suggested, but a USB key, floppy or a CD would probably work just as
well. All of these methods are of course sensitive to the media being stolen, but that's something we'll
just have to live with.

I think I'll set things up so that the install image has a key in that allows you to get the proper key from the
install server. The installation image will only be NFS-exported to the machines that need it, as long as
they need it. This will lead to exposure while the machine downloads its install images, but so be it.

I'll also have a good look at ssh-keyscan and centralising the known_hosts file. That may be another way
around this problem: after a reinstall, scan the box' host key and have all other machines pull down the
file on a regular basis.

Cheers,
Menno 

--

-- 
Menno Willemse - John Guest IT Department
Tel: 01895-449233 ext 290 Email: menno.willemse <at> johnguest.co.uk
There is no Cabal.

Internet communications are not secure and therefore John Guest companies do not accept legal
responsibility for the contents of this message.  Any views or opinions presented are solely those of the
author and do not necessarily represent those of John Guest companies.
Permalink | Reply | 1 comment |
headers
Andrew Cowie | 5 Oct 2006 00:22
Favicon
Gravatar

Re: Training Infrastructure Teams

On Fri, 2006-08-18 at 17:42 -0400, devick <at> lanier.com wrote:
> The attached file is an introduction to the training process

Sorry I haven't had a chance to respond to this earlier: your document,
misleadingly named trainingintro.html but titled with the provocative
title "The Craft" is excellent.

Hiring people with technical skills is easy. Finding someone who has a
sense of verve and who will embrace the wider responsibilities and
obligations of their job - that's very difficult.

I have often commented in my keynotes that Systems Administration, as
such, is not a profession. If more people pursue their work with the
diligence and spirit that is implied here, then I think we will have
taken a long leap forward to changing this.

++

This is a wonderful document. I would love to use it in my own work as a
resource document. Are you willing to let others use it?

Your posting it here and describing it in terms of infrastructures.org
implies that this is the case. If you would be able to put it under an
open licence [creative commons or GFDL] then I for one would be willing
to offer comments and improvements.

AfC
Sydney

--

--
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brendan Strejcek | 6 Oct 2006 16:40
Favicon

Re: Distributing private key information at install time

On 10/4/06, Willemse, Menno <Menno.Willemse <at> johnguest.co.uk> wrote:

> The prevailing wisdom seems to be that you use a boot CD with
> a temporary key to do the re-install

This seems unnecessarily manual to me. I don't think it would scale
very well. How many machines are you managing?

> or that you somehow keep the old keys on the system where they
> won't be erased by the reinstall.
> ...
> centralising the known_hosts file. That may be another way around
> this problem: after a reinstall, scan the box' host key and have all
> other machines pull down the file on a regular basis.

Are you using any sort of centralized configuration management system?
How do you propagate configuration changes to live machines?

An ssh key pair is really no different from any other bit of
configuration data, and is easy to centralize. Without more
information about your particular infrastructure though, it is hard to
offer more concrete suggestions.

Best,
Brendan

--
http://praksys.blogspot.com
Permalink | Reply | 0 comments |
headers
Brendan Strejcek | 27 Oct 2006 20:15
Favicon

Re: Distributing private key information at install time

(Wesley sent me a message off-list, but said it was fine to repost to the list.)

On 10/6/06, Wesley Craig <wes <at> umich.edu> wrote:

> On 06 Oct 2006, at 10:40, Brendan Strejcek wrote:
>
> > Are you using any sort of centralized configuration management system?
> > How do you propagate configuration changes to live machines?
>
> The question is how do you deal with a machine whose hardware you are
> replacing.  It's all fine to store this data centrally, but how do
> you bootstrap security from a naked machine?

Hooks can be added to any automated installation procedure. The
security of this initial bootstrap will depend upon the particular
installation method. For example, a custom install CD could easily
include various security bits, such as server public keys. Another
example would be a PXE boot on an isolated network where a server
decides what credentials to issue to a new machine based on ethernet
address.

Best,
Brendan

--
http://praksys.blogspot.com
Permalink | Reply | 0 comments |
headers
Brendan Strejcek | 27 Oct 2006 20:27
Favicon

Re: Distributing private key information at install time

(Sorry in advance if I got some of the attribution wrong, the quotes
and forwardings made it a bit complicated.)

<Menno.Willemse <at> johnguest.co.uk> on 04 October 2006 wrote:

> That's secure, but it involves walking up to the machine, which I
> don't want to do. They are in locked rooms or on remote locations.
> Maybe the best solution is just to leave a CD, HD partition or
> floppy in the machine with its crypto-information.

I use a trick to get around this. If I have to install a fresh
operating system on a truly blank machine, I need to physically be at
the console. In that case, however, the machine was usually just
delivered and has not been deployed yet. Once a machine has been
installed, it has the ability to remotely reinstall itself with
several supported operating systems. I use alternative kernels and the
boot loader to accomplish this. Thus, once a machine has been
installed physically once (in most cases) it can be reinstalled
remotely.

> All security
> flies out of the window as soon as someone  can touch the machine
> anyway.

That is not totally true. They may be able to take over the machine,
but they will not be able to compromise confidentiality (that is, you
can encrypt data on the disk).

> If you want to do hands-off, unattended installs, I suppose there
> just isn't a way that's 100% secure.
(Continue reading)

Permalink | Reply | 0 comments |

Gmane