Harald Kubota | 1 Oct 02:11 2006
Picon
Picon

Re: Network based storage - NBD/AoE/iSCSI other?

Ian P. Christian wrote:
> I'm currently investigating setting up few Xen servers.  I've noticed
> that currently live migration of Xen's require you to use a network
> based storage system.
>   

There is no such requirement. FC is certainly ok, as is anything which 
can be shared and
directly connected (e.g. Firewire drives with a bridge from Oxford) and 
of course
the old way of using (parallel) SCSI drives (which have their own problems).

Using network is likely the cheapest and most convenient one unless you 
happen to have
FC already.

> iSCSI looked like a good bet, but the fact it's using TCP/IP seems like
> a crazy overhead to use when I'm most likely going to be hosting the
> storage on the same layer 2 network as the servers.
>   

The overhead is not that large and it's nice to be using a standard 
which will continue to
exist for a while. Should you need performance, you can replace the 
iSCSI client (the server
with the storage) with a faster box without changing anything at the 
iSCSI initiator side (the Xen
running machine).

> AoE solves that above concern but I'm confused as to what would happen
(Continue reading)

Gregory Gee | 1 Oct 04:56 2006
Picon

Re: Xen and Dapper

Z24 wrote:
> I had some problems with Dapper Drake but never dom0 freezing; I had a
> broken dd, some missing or broken packages, a missing hvmloader binary
> in Xen 3.0 binaries and a hvmloader makefile issue in Xen 3.0.2
> sources; the hvmloader problems were solved compiling Xen testing and
> all problems concerning WinXP domU were related to missing sdl
> packages or domU configuration.
>
> I was running Xen on a AMD X2 with 2 GB RAM, 1 GB for dom0 and 512 MB
> for WinXP domU. I wrote all what I did step by step here:
> http://computingart.freehostia.com/index.cgi/linux/Z24.xen3-ubuntu-amd64.html
>
> Where are you trying to install WinXP? To a disk partition, to an
> image file? Maybe the partition is already mounted in dom0?
>
>   
  Your instructions look really great and will hopefully trying it out 
soon.  I have a few questions related to the XP install.

1. If I use Ubuntu-server, how do you install XP since there is no 
window that will popup, since X11 isn't installed on DOM0?

2. How much RAM does DOM0 really need?  If all it is going to do is be 
host for all the DOMU, what is the require minimum?

Thanks,
Greg
Hiamal Llanos | 1 Oct 05:44 2006
Picon

No network on DomU

Hi,

I have problems getting xenbr0 up in dom0 and which means that I don't have any network interface in domU.
I tested different solutions presented on the list but none of those worked for me :-(

The system I'm running is Debian Etch and xen 3.0.2.
On other systems running xen 2.0.7 and 3.0.0 on Debian and Ubuntu everytihng works fine.

Installed packages:
libc6-xen
linux-image-2.6.17-2-xen-686
linux-modules-2.6.17-2-xen-686
xen-hypervisor-3.0-unstable-1-i
xen-linux-system-2.6.17-2-xen-6
xen-utils-3.0-unstable-1
xen-utils-common

$ cat /boot/grub/menu.lst
...
title           Xen 3.0-unstable-1-i386 / Debian GNU/Linux, kernel 2.6.17-2-xen-686
root            (hd0,0)
kernel          /boot/xen-3.0-unstable-1-i386.gz dom0_mem=65536
module          /boot/vmlinuz-2.6.17-2-xen-686 root=/dev/hda1 ro console=tty0
module          /boot/initrd.img-2.6.17-2-xen-686
savedefault
...

$ cat /etc/xen/sarge
name = "sarge"
kernel = "/boot/vmlinuz-2.6.17-2-xen-686"
ramdisk = "/boot/initrd.img-2.6.17-2-xen-686"
root = "/dev/hda1 ro"
memory = 68
disk = [ 'phy:/dev/vg/sarge,hda1,w','phy:/dev/vg/sargeswap,hda2,w' ]
dhcp ="dhcp"
extra = "3"

$ grep -v -e '^#' -e '^$' /etc/xen/xend-config.sxp
(logfile /var/log/xen/xend.log)
(loglevel DEBUG)
(xend-http-server yes)
(xend-relocation-server yes)
(xend-port            8000)
(xend-relocation-port 8002)
(xend-address localhost)
(xend-relocation-address localhost)
(network-script network-dummy)
(network-script 'network-bridge netdev=eth0')
(network-script 'network-bridge bridge=xenbr0')
(vif-script vif-bridge)
(dom0-min-mem 64)
(dom0-cpus 0)

$ brctl show
bridge name     bridge id               STP enabled     interfaces


I'm stuck now. I don't know what I'm doing wrong or what's missing.
Does somebody know what I'm missing or what the problem is?



Thanks,
Hiamal

_______________________________________________
Xen-users mailing list
Xen-users <at> lists.xensource.com
http://lists.xensource.com/xen-users
Luke Crawford | 1 Oct 06:38 2006

Re: Network based storage - NBD/AoE/iSCSI other?

On Sun, 1 Oct 2006, Harald Kubota wrote:
> Ian P. Christian wrote:
>> I'm currently investigating setting up few Xen servers.  I've noticed
>> that currently live migration of Xen's require you to use a network
>> based storage system.
>
> Using network is likely the cheapest and most convenient one unless you 
> happen to have
> FC already.

If you are the boss, used 1G FC is quite a bit cheaper (and faster) than 
used 1G Ethernet.

However, most bosses refuse to use used stuff;  and some people think that 
commodity Ethernet will scale faster than commodity FC, so it's better to 
just run Ethernet everywhere.   (these people may be right; my point still 
stands that 1G fibre channel, bought used, gives you better storage 
performance per dollar than 1G Ethernet)
Tom Mornini | 1 Oct 08:30 2006

Re: Network based storage - NBD/AoE/iSCSI other?

On Sep 30, 2006, at 9:38 PM, Luke Crawford wrote:

> On Sun, 1 Oct 2006, Harald Kubota wrote:
>> Ian P. Christian wrote:
>>> I'm currently investigating setting up few Xen servers.  I've  
>>> noticed
>>> that currently live migration of Xen's require you to use a network
>>> based storage system.
>>
>> Using network is likely the cheapest and most convenient one  
>> unless you happen to have
>> FC already.
>
> If you are the boss, used 1G FC is quite a bit cheaper (and faster)  
> than used 1G Ethernet.

Faster, probably (I'm certainly not arguing), but the big storage  
vendors have recently said that 4Gb fiber will be the top speed for  
years to come.

Cheaper? Are you talking about buying used FC disks as well? Because  
FC disks -vs- SATA disk is no comparison in terms of $/GB. It's my  
understanding that *most* FC solutions require FC disks...

> However, most bosses refuse to use used stuff;  and some people  
> think that commodity Ethernet will scale faster than commodity FC,  
> so it's better to just run Ethernet everywhere.   (these people may  
> be right; my point still stands that 1G fibre channel, bought used,  
> gives you better storage performance per dollar than 1G Ethernet)

Performance, yes, but how about capacity? And just how much faster is  
it?

Here's an article suggesting 4Gb will dominate the FC market until 2010.

http://www.internetnews.com/storage/article.php/3627306

Something tells me commodity 10GE will be cheaper by then.

--

-- 
-- Tom Mornini
John Smith | 1 Oct 09:48 2006
Picon

Re: No network on DomU

Hiamal Llanos wrote:
> Hi,
> 
> I have problems getting xenbr0 up in dom0 and which means that I don't
> have any network interface in domU.
> I tested different solutions presented on the list but none of those
> worked for me :-(
> 
> The system I'm running is Debian Etch and xen 3.0.2.
> On other systems running xen 2.0.7 and 3.0.0 on Debian and Ubuntu
> everytihng works fine.
> 
> Installed packages:
> libc6-xen
> linux-image-2.6.17-2-xen-686
> linux-modules-2.6.17-2-xen-686
> xen-hypervisor-3.0-unstable-1-i
> xen-linux-system-2.6.17-2-xen-6
> xen-utils-3.0-unstable-1
> xen-utils-common
> 
> $ cat /boot/grub/menu.lst
> ...
> title           Xen 3.0-unstable-1-i386 / Debian GNU/Linux, kernel
> 2.6.17-2-xen-686
> root            (hd0,0)
> kernel          /boot/xen-3.0-unstable-1-i386.gz dom0_mem=65536
> module          /boot/vmlinuz-2.6.17-2-xen-686 root=/dev/hda1 ro
> console=tty0
> module          /boot/initrd.img-2.6.17-2-xen-686
> savedefault
> ...
> 
> $ cat /etc/xen/sarge
> name = "sarge"
> kernel = "/boot/vmlinuz-2.6.17-2-xen-686"
> ramdisk = "/boot/initrd.img-2.6.17-2-xen-686"
> root = "/dev/hda1 ro"
> memory = 68
> disk = [ 'phy:/dev/vg/sarge,hda1,w','phy:/dev/vg/sargeswap,hda2,w' ]
> dhcp ="dhcp"
> extra = "3"
> 
> $ grep -v -e '^#' -e '^$' /etc/xen/xend-config.sxp
> (logfile /var/log/xen/xend.log)
> (loglevel DEBUG)
> (xend-http-server yes)
> (xend-relocation-server yes)
> (xend-port            8000)
> (xend-relocation-port 8002)
> (xend-address localhost)
> (xend-relocation-address localhost)
> (network-script network-dummy)
> (network-script 'network-bridge netdev=eth0')
> (network-script 'network-bridge bridge=xenbr0')
> (vif-script vif-bridge)
> (dom0-min-mem 64)
> (dom0-cpus 0)
> 
> $ brctl show
> bridge name     bridge id               STP enabled     interfaces
> 
> 
> I'm stuck now. I don't know what I'm doing wrong or what's missing.
> Does somebody know what I'm missing or what the problem is?
> 
> 
> 
> Thanks,
> Hiamal
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users <at> lists.xensource.com
> http://lists.xensource.com/xen-users

Hi Hiamal,

	try putting netloop in your /etc/modules and restarting. If
it still does not work, run /etc/xen/scripts/network-<your flavour>
manually.

	Hop this helps.

Sincerely,

Jan.
Luke Crawford | 1 Oct 10:06 2006

Re: Network based storage - NBD/AoE/iSCSI other?

On Sat, 30 Sep 2006, Tom Mornini wrote:
> On Sep 30, 2006, at 9:38 PM, Luke Crawford wrote:
>> If you are the boss, used 1G FC is quite a bit cheaper (and faster) than 
>> used 1G Ethernet.
>
> Faster, probably (I'm certainly not arguing), but the big storage vendors 
> have recently said that 4Gb fiber will be the top speed for years to come.

Yes, I think I mentioned that this equation may change when 10G Ethernet 
becomes affordable.  I would be suprised if FC was still the best choice 5 
years from now; however, even if I go with 1G ethernet now, I'll still 
have to buy all new equipment when the 10G stuff comes out, so I might as 
well get the most performance for my dollar now.

(now, I predict that SAS, and not 10G ethernet will be the best solution 
5 years from now, I would be using SAS now if I could buy affordable 
components from different vendors and reasonably expect them to work 
together as I can with FC.  Of course, this just my prediction, and it is 
worth exactly what you paid for it.)

> Cheaper? Are you talking about buying used FC disks as well? Because FC disks 
> -vs- SATA disk is no comparison in terms of $/GB. It's my understanding that 
> *most* FC solutions require FC disks...

You can get 12-bay SATA -> FC arrays for around $1K.  If you know where to 
get cheaper SATA-> gigabit Ethernet arrays, I'd like to know about it.

http://cgi.ebay.com/EMC-AX100-Fibre-Channel-SATA-Drive-Array_W0QQitemZ270035121613QQihZ017QQcategoryZ80219QQssPageNameZWDVWQQrdZ1QQcmdZViewItem?hash=item270035121613

I think 7.3Krpm SATA drives are not up to snuff for virtual hosting (at 
least not on my systems-  these disks are quite shared and heavily used; 
when I was using SATA, I had disk i/o latency issues with only 10 
dns/mail/internal infrastructure servers.)  I imagine a write-back cache 
of some sort (that most high-end redundant NAS units have-  or simply 
mounting all your disks async) would solve this problem, but it is rather 
expensive to do that properly.  Most SATA nas units are just a single PC, 
so if they enable write-back caching and the box panics or your new admin 
pulls the power plug, you have issues.

but really, if you can get away with IDE disk, you can probably get away 
with NFS over 100Mbps, which is cheaper and easier than FC.

>> However, most bosses refuse to use used stuff;  and some people think that 
>> commodity Ethernet will scale faster than commodity FC, so it's better to 
>> just run Ethernet everywhere.   (these people may be right; my point still 
>> stands that 1G fibre channel, bought used, gives you better storage 
>> performance per dollar than 1G Ethernet)
>
> Performance, yes, but how about capacity? And just how much faster is it?

For me, capacity is a minor issue compared to latency under heavy 
concurrent access.  I think IOPS/sec is where SCSI (and SCSI over FC) is 
really where the FC/SCSI disks show their worth.

(and yes, I usually use used disks;  I mirror them and run a SMART monitor 
on them, so the reduced reliability isn't a huge deal.  I would *not* 
recommend using raid5 with used disks-  well, I don't recommend raid5 in 
general, except for a substitute for a stripe that is less of a pain-in 
the ass to rebuild, simply because raid5 performance drops precipitously 
during a rebuild;  your array is essentially down for a day if you are 
running it near capacity.)

Like I said, if you are just going for capacity, use IDE over NFS on a 
10/100 network.  a 1000 network might be worth it if most of your stuff 
is sequential (as IDE comes pretty darn close and sometimes beats SCSI for 
sequential access) but in my environment, there really is no such thing as 
sequential access.

My main point was that compared to a 1GB Ethernet 'dedicated to storage' 
network, a 1GB FC network is cheaper and faster;   I believe this to be 
true even if your end disks are SATA.  (but like I said, I might be wrong 
on that part;  I'm not really familiar with pricing for network-attached 
IDE arrays; I can't afford gigabit Ethernet equipment of a quality I'd 
like to maintain, and I use 10 or 15K scsi/FC for everything that 
matters anyhow.)
Hiamal Llanos | 1 Oct 11:07 2006
Picon

Re: No network on DomU

Hi!

On 10/1/06, John Smith <netman1 <at> home.nl> wrote:
Hiamal Llanos wrote:
> Hi,
>
> I have problems getting xenbr0 up in dom0 and which means that I don't
> have any network interface in domU.
> I tested different solutions presented on the list but none of those
> worked for me :-(

...

Hi Hiamal,

        try putting netloop in your /etc/modules and restarting. If
it still does not work, run /etc/xen/scripts/network-<your flavour>
manually.

Thanks John, that got me a bit further.

Now I have vif0.0 but no xenbr0. To get it working correctly I have to do the following, after a reboot:

$ sudo /etc/init.d/xendomains stop
$ sudo /etc/init.d/xend stop
$ sudo /etc/xen/scripts/network-bridge stop
$ sudo /etc/xen/scripts/network-bridge start vif0.0
$ sudo /etc/init.d/xend start
$ sudo /etc/init.d/xendomains start

So now I have peth0, vif0.0 and xenbr0
And addind vif = [''] to /etc/xen/sarge gawe my back eth0 on domU.

But...wy dosn't xenbr0 get createt on system boot? What I'm missing?
Must I change something in /etc/xen/xend- config.sxp?:

(logfile /var/log/xen/xend.log)
(loglevel DEBUG)
(xend-http-server yes)
(xend-relocation-server yes)
(xend-port            8000)
(xend-relocation-port 8002)
(xend-address localhost)
(xend-relocation-address localhost)
(network-script network-dummy)
(network-script 'network-bridge netdev=eth0')
(network-script 'network-bridge bridge=xenbr0')
(vif-script vif-bridge)
(dom0-min-mem 64)
(dom0-cpus 0)



Thanks
Hiamal
 
_______________________________________________
Xen-users mailing list
Xen-users <at> lists.xensource.com
http://lists.xensource.com/xen-users
John Smith | 1 Oct 11:28 2006
Picon

Re: No network on DomU

Hiamal Llanos wrote:
> Hi!
> 
> On 10/1/06, *John Smith* <netman1 <at> home.nl <mailto:netman1 <at> home.nl>> wrote:
> 
>     Hiamal Llanos wrote:
>     > Hi,
>     >
>     > I have problems getting xenbr0 up in dom0 and which means that I don't
>     > have any network interface in domU.
>     > I tested different solutions presented on the list but none of those
>     > worked for me :-(
> 
> 
> ...
> 
>     Hi Hiamal,
> 
>             try putting netloop in your /etc/modules and restarting. If
>     it still does not work, run /etc/xen/scripts/network-<your flavour>
>     manually.
> 
> 
> Thanks John, that got me a bit further.
> 
> Now I have vif0.0 but no xenbr0. To get it working correctly I have to
> do the following, after a reboot:
> 
> $ sudo /etc/init.d/xendomains stop
> $ sudo /etc/init.d/xend stop
> $ sudo /etc/xen/scripts/network-bridge stop
> $ sudo /etc/xen/scripts/network-bridge start vif0.0
> $ sudo /etc/init.d/xend start
> $ sudo /etc/init.d/xendomains start
> 
> So now I have peth0, vif0.0 and xenbr0
> And addind vif = [''] to /etc/xen/sarge gawe my back eth0 on domU.
> 
> But...wy dosn't xenbr0 get createt on system boot? What I'm missing?
> Must I change something in /etc/xen/xend- config.sxp?:
> 
> (logfile /var/log/xen/xend.log)
> (loglevel DEBUG)
> (xend-http-server yes)
> (xend-relocation-server yes)
> (xend-port            8000)
> (xend-relocation-port 8002)
> (xend-address localhost)
> (xend-relocation-address localhost)
> (network-script network-dummy)
> (network-script 'network-bridge netdev=eth0')
> (network-script 'network-bridge bridge=xenbr0')
> (vif-script vif-bridge)
> (dom0-min-mem 64)
> (dom0-cpus 0)
> 
> 
> 
> Thanks
> Hiamal
>  **
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users <at> lists.xensource.com
> http://lists.xensource.com/xen-users

Hiamal,

   I think those network-script entries in your config are
conflicting: as far as I know you can only have one.

   Here is my xen-config.sxp;

(xend-relocation-server no)
(xend-relocation-hosts-allow '')
(network-script 'network-bridge netdev=eth0')
(vif-script vif-bridge)
(dom0-min-mem 196)
(dom0-cpus 0)

Sincerely,

Jan.
Peter Fern | 1 Oct 13:29 2006
Picon

Re: Xen and Dapper

Gregory Gee wrote: 
>  Your instructions look really great and will hopefully trying it out
> soon.  I have a few questions related to the XP install.
>
> 1. If I use Ubuntu-server, how do you install XP since there is no
> window that will popup, since X11 isn't installed on DOM0?

VNC

>
> 2. How much RAM does DOM0 really need?  If all it is going to do is be
> host for all the DOMU, what is the require minimum?
>

This depends on a number of factors, in particular disk I/O, if your
domUs are doing a lot of I/O, you'll need more RAM in dom0.  That said,
less than 256MB is probably not a great idea...

Gmane