Bogdan-Stefan Rotariu | 1 Dec 16:49 2011
Picon

Re: Heavy Disk IO from a single VM can block the other VMs on the same host

On Nov 29, 2011, at 18:13, Hubert Krause
<hubert.krause@...> (by way of HubertKrause
<hubert.krause@...>) (by way of HubertKrause
<hubert.krause@...>) wrote:

> Hello,
> 
> my environment is a Debian squeeze host with a few debian squeeze
> guests. The private and root filesystems of the guest are locatet on
> the same raid device (raid5) 

maybe offtopic, maybe not, but stop using raid5 for VM deployment, use raid10, raid1, raid0 -- with lvm and snapshots

raid5 will always be slow on io, as it has checksums because "recalculation and redistribution of parity
data on a per-write basis"
Kirill Korotaev | 1 Dec 18:27 2011

Re: Heavy Disk IO from a single VM can block the other VMs on the same host

That's most likely due to a single file system used for containers - journal becomes a bottleneck.
fsync forces journal flushes and other workloads begin to wait for journal... In reality workload looks
like this are typical for
heavy loaded databases or mail systems only.

How to improve:
- increase journal size
- split file systems, i.e. run each container from it's own file system

Thanks,
Kirill

On Nov 29, 2011, at 20:13 , Hubert Krause wrote:

> Hello,
> 
> my environment is a Debian squeeze host with a few debian squeeze
> guests. The private and root filesystems of the guest are locatet on
> the same raid device (raid5) in an luksCrypt Container in an LVM
> container on an ext4 partition with nodelalloc as mountoption. If I run
> the tool stress:
> 
> stress --io 5 --hdd 5 --timeout 60s (which means fork 5 threads doing
> read/write access and 5 threads doing constantly fsync) the
> responsivness of the other VMs is very bad. That means, Isolation for
> IO operations is not given. I've tried to reduce the impact of the
> VM with 'vzctl set VID --ioprio=0'. There was only a
> minor effect, my application on the other VM where still not
> responsive.
> 
(Continue reading)

Stephen Balukoff | 1 Dec 23:03 2011
Picon

Re: Is there a stable OpenVZ kernel, and which should be fit for production

We're also seeing a big increase in instability since moving to the
RHEL 6 kernels.  Specifically, our typical platform consists of a
Supermicro motherboard with dual 12-core AMD procs (ie. 24 in one
system);   The most frustrating part is that the symptom we're seeing
is highly intermittent (sometimes it takes 10 minutes to trigger,
sometimes several days), and doesn't result in a kernel panic or dump
per se.  Instead what we're seeing is an unresponsive system (still
responding to ping, but all services on the box are unresponsive),
with this scrolling by on the console:

BUG: soft lockup - CPU#22 stuck for 67s! [node:585441]
BUG: soft lockup - CPU#23 stuck for 68s! [node:585419]

(multiple times per second, repeating all the different process
numbers and many different processes running within containers).

We're going to file a bug report on this, of course, but wondered if
there was anything else we can do here to get any other information
which can help the devs to come up with the cause and hopefully fix
for the above?  (Again, we're not getting a panic, and we're not able
to do anything on the console.)

Thanks,
Stephen

--

-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
(Continue reading)

Stephen Balukoff | 1 Dec 23:06 2011
Picon

Re: Is there a stable OpenVZ kernel, and which should be fit for production

Oh!  And for what it's worth, we're seeing this on both the latest
stable RHEL6 kernel, as well as the latest testing RHEL6 kernel
available in the repositories for download.  (That is, 042stab39.11
and 042stab044.1 respectively).

Stephen

On Thu, Dec 1, 2011 at 2:03 PM, Stephen Balukoff <sbalukoff@...> wrote:
> We're also seeing a big increase in instability since moving to the
> RHEL 6 kernels.  Specifically, our typical platform consists of a
> Supermicro motherboard with dual 12-core AMD procs (ie. 24 in one
> system);   The most frustrating part is that the symptom we're seeing
> is highly intermittent (sometimes it takes 10 minutes to trigger,
> sometimes several days), and doesn't result in a kernel panic or dump
> per se.  Instead what we're seeing is an unresponsive system (still
> responding to ping, but all services on the box are unresponsive),
> with this scrolling by on the console:
>
> BUG: soft lockup - CPU#22 stuck for 67s! [node:585441]
> BUG: soft lockup - CPU#23 stuck for 68s! [node:585419]
>
> (multiple times per second, repeating all the different process
> numbers and many different processes running within containers).
>
> We're going to file a bug report on this, of course, but wondered if
> there was anything else we can do here to get any other information
> which can help the devs to come up with the cause and hopefully fix
> for the above?  (Again, we're not getting a panic, and we're not able
> to do anything on the console.)
>
(Continue reading)

Stephen Balukoff | 2 Dec 01:39 2011
Picon

Re: Is there a stable OpenVZ kernel, and which should be fit for production

Ok, y'all:

We managed to get a call trace.  I've opened the following bug on this
issue:  http://bugzilla.openvz.org/show_bug.cgi?id=2110

Anything else I can do or provide to get traction on getting a
developer to look at this?  (This is a complete show-stopper for our
Scientific Linux 6.1 OpenVZ roll-out.)

Stephen

On Thu, Dec 1, 2011 at 2:06 PM, Stephen Balukoff <sbalukoff@...> wrote:
> Oh!  And for what it's worth, we're seeing this on both the latest
> stable RHEL6 kernel, as well as the latest testing RHEL6 kernel
> available in the repositories for download.  (That is, 042stab39.11
> and 042stab044.1 respectively).
>
> Stephen
>
> On Thu, Dec 1, 2011 at 2:03 PM, Stephen Balukoff
<sbalukoff@...> wrote:
>> We're also seeing a big increase in instability since moving to the
>> RHEL 6 kernels.  Specifically, our typical platform consists of a
>> Supermicro motherboard with dual 12-core AMD procs (ie. 24 in one
>> system);   The most frustrating part is that the symptom we're seeing
>> is highly intermittent (sometimes it takes 10 minutes to trigger,
>> sometimes several days), and doesn't result in a kernel panic or dump
>> per se.  Instead what we're seeing is an unresponsive system (still
>> responding to ping, but all services on the box are unresponsive),
>> with this scrolling by on the console:
(Continue reading)

tim Doyle | 2 Dec 19:18 2011

Re: Heavy Disk IO from a single VM can block the other VMs on the same host

You can use vzctl --ioprio to set relative disk I/O priorities:
http://wiki.openvz.org/I/O_priorities_for_VE

-Tim

-- 
Timothy Doyle
CEO
Quantact Hosting Solutions, Inc.
tim@...
http://www.quantact.com

On 12/01/2011 09:27 AM, Kirill Korotaev wrote:
> That's most likely due to a single file system used for containers - journal becomes a bottleneck.
> fsync forces journal flushes and other workloads begin to wait for journal... In reality workload looks
like this are typical for
> heavy loaded databases or mail systems only.
>
> How to improve:
> - increase journal size
> - split file systems, i.e. run each container from it's own file system
>
> Thanks,
> Kirill
>
>
> On Nov 29, 2011, at 20:13 , Hubert Krause wrote:
>
>> Hello,
>>
(Continue reading)

Daniel Bauer | 5 Dec 08:55 2011
Picon

Re: OpenVZ Container with a PCI ISDN Card - SOLVED

From: "Adeel Nazir" <adeel_n@...>
> In theory, you should be able to. from the man pages of vzctl:
>
>       --devnodes device:r|w|rw|none
>           Give the container an access (r - read only, w - write only, 
> rw -
> read/write, none - no access) to a device  designated  by  the 
> special
>           file /dev/device. Device file is created in a container by 
> vzctl.

it really takes a long time, but now it's solved. You're absolutly 
right, my mistake was, to look for the PCI Card without any drivers 
installed in the HN. After loading the correct modules, I moved the 
access to the asterisk in the VPS.

Thanks
Daniel

>       --devices b|c:major:minor|all:[r|w|rw|none]
>           Give  the  container  an  access to a block or character 
> device
> designated by its major and minor numbers. Device file have to be 
> created
>           manually.
>
>
>
>
> ----- Original Message ----
(Continue reading)

U.Mutlu | 6 Dec 01:07 2011

NFQUEUE in VE

I need to use, in a VE, an app that uses libnetfilter_queue (ie. the NFQUEUE target of iptables).
Which module do I need to specify in vz.cfg (IPTABLES="...") ?

I tried the following modules

find /lib/modules/ -iname "*queu*" -ls
   /lib/modules/2.6.32-5-openvz-amd64/kernel/drivers/md/dm-queue-length.ko
   /lib/modules/2.6.32-5-openvz-amd64/kernel/net/ipv6/netfilter/ip6_queue.ko
   /lib/modules/2.6.32-5-openvz-amd64/kernel/net/netfilter/nfnetlink_queue.ko
   /lib/modules/2.6.32-5-openvz-amd64/kernel/net/netfilter/xt_NFQUEUE.ko
   /lib/modules/2.6.32-5-openvz-amd64/kernel/net/ipv4/netfilter/ip_queue.k

but vzctl gives such errors/warnings, and the app cannot access the NFQUEUE queue:
   Warning: Unknown iptable module: nfnetlink_queue, skipped

The same app on the HN works fine.
So, how can I use NFQUEUE on the VE ?
Hubert Krause | 6 Dec 18:18 2011

Re: Heavy Disk IO from a single VM can block the other VMs on the same host

Hello Kirill,

Am Thu, 1 Dec 2011 21:27:49 +0400
schrieb Kirill Korotaev <dev@...>:

> That's most likely due to a single file system used for containers -
> journal becomes a bottleneck. fsync forces journal flushes and other
> workloads begin to wait for journal... In reality workload looks like
> this are typical for heavy loaded databases or mail systems only.
> 
> How to improve:
> - increase journal size
> - split file systems, i.e. run each container from it's own file
> system

I've created another lv with an ext4 filesystem with maximum
journal-size and mounted this filesystem
under /var/lib/vz/private/≤VID>. I will call this vm as VM-sep. All
other vhosts where kept inside the volume as before. Than I
start stressing the VM-sep and tested the impact to the other VMs. It
was exactly the same as if I run all VMs on the same partition.

There was indeed a difference, when I stress the Host itself. If I do
filesystem stress in the same Partition (/var/lib/vz) The perfomance of
VM is much worse (similar to stress in a VM, a little better) than if I
would stress in a completly different Partition (/var/tmp in my case)

To get some Numbers: (not very sientific, but good for a measure)

Throughput of a Webserver in a VM called VM-web in KB/s:
(Continue reading)

Enrico Weigelt | 7 Dec 05:23 2011
Picon

Re: OS/app in the OpenVZ container

* Kir Kolyshkin <kir@...> wrote:

> It's pretty simple. What OpenVZ kernel does it creates a container and
> runs /sbin/init inside it. What goes next is up to that particular
> /sbin/init. If you need to run an app let /sbin/init run just it, or let
> this app be /sbin/init (but bear in mind there will be no child reaper
> as in usual linux system so you might need to take some extra care about
> zombie processes).

ACK.

The probably easiest way is to take some minimal OS image, so you also
get a shell, libc, and essential setups (mounting /proc and /sys), etc.
You'll most likely also need some little setup stuff to let your app 
communicate with the outside world (eg. routing, filesystem access, etc).

Doing this all completely on your own takes a lot of more efforts,
but makes a good learning project :)

Some time ago (left it aside due lack of time), I've been working on
an kind of nano-distro for such use-cases, built with my Briegel
buildsystem (it crosscompiles tiny images which only include the
really essential stuff) . If you're really interested, I'd dig it
out again.

cu
--

-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

(Continue reading)


Gmane