Luiz Capitulino | 26 Jan 20:14 2015

kernel-rt rcuc lock contention problem


We're running some measurements with cyclictest running inside a
KVM guest where we could observe spinlock contention among rcuc

Basically, we have a 16-CPU NUMA machine very well setup for RT.
This machine and the guest run the RT kernel. As our test-case
requires an application in the guest taking 100% of the CPU, the
RT priority configuration that gives the best latency is this one:

 263  FF   3  [rcuc/15]
  13  FF   3  [rcub/1]
  12  FF   3  [rcub/0]
 265  FF   2  [ksoftirqd/15]
3181  FF   1  qemu-kvm

In this configuration, the rcuc can preempt the guest's vcpu
thread. This shouldn't be a problem, except for the fact that
we're seeing that in some cases the rcuc/15 thread spends 10us
or more spinning in this spinlock (note that IRQs are disabled
during this period):

	if (cpu_needs_another_gp(rsp, rdp)) {
		raw_spin_lock(&rcu_get_root(rsp)->lock); /* irqs disabled. */
(Continue reading)

Josh Cartwright | 24 Jan 03:03 2015

3.14-rt ARM performance regression?

Hey folks-

We've recently undertaken an upgrade of our kernel from 3.2-rt to
3.14-rt, and have run into a performance regression on our ARM boards.
We're still in the process of trying to isolate what we can, but
hopefully someone's already run into this and has a solution or might
have some useful debugging ideas.

The first test we did was to run cyclictest[1] for comparison:

   # Total: 312028761 312028761 624057522
   # Min Latencies: 00010 00011
   # Avg Latencies: 00018 00020
   # Max Latencies: 00062 00066 00066
   # Histogram Overflows: 00000 00000 00000

   # Total: 304735655 304735657 609471312
   # Min Latencies: 00013 00013
   # Avg Latencies: 00023 00024
   # Max Latencies: 00086 00083 00086
   # Histogram Overflows: 00000 00000 00000

As you can see, we're seeing a 30%-40% degradation not just max latencies, but
also the minimum/maximum latencies.  The above numbers are with the system
under a network throughput load (iperf), but changing the load seems to have
little impact (and in fact, we see a general slowdown even when otherwise

(Continue reading)

Dorian VEGARA | 22 Jan 20:39 2015

Cannot boot RTLinux

Hello everyone,

I'm trying to boot RTLinux but I'm encountering problems to load it.
I patched successfully the Linux kernel (version 3.14.25) with the
RTLinux one (3.14.25 too).

Then, I typed this command :
make defconfig (in the linux-patched-repertory). It chose the x86_64 defconfig.

After it, I typed make && make modules_install. The x86-only bzImage
has been created.
The modules have been installed properly. The last message is "DEPMOD

The GRUB2 configuration is OK. But when I boot RTLinux, it crashes.
A "Warning CPU" message appears :

Could anyone tell me what is wrong please ?
Here is a PowerPoint document to explain in details the bug and what
exactly I did :

Thanks a lot !
Best regards,
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo <at>
More majordomo info at
(Continue reading)

Bogdan Purcareata | 22 Jan 10:39 2015

[PATCH] KVM: PPC: Convert openpic lock to raw_spinlock

This patch enables running intensive I/O workloads, e.g. netperf, in a guest
deployed on a RT host. It also enable guests to be SMP.

The openpic spinlock becomes a sleeping mutex on a RT system. This no longer
guarantees that EPR is atomic with exception delivery. The guest VCPU thread
fails due to a BUG_ON(preemptible()) when running netperf.

In order to make the kvmppc_mpic_set_epr() call safe on RT from non-atomic
context, convert the openpic lock to a raw_spinlock. A similar approach can
be seen for x86 platforms in the following commit [1].

Here are some comparative cyclitest measurements run inside a high priority RT
guest run on a RT host. The guest has 1 VCPU and the test has been run for 15
minutes. The guest runs ~750 hackbench processes as background stress.

                  spinlock  raw_spinlock
Min latency (us)  4         4
Avg latency (us)  15        19
Max latency (us)  70        62

Due to the introduction of the raw_spinlock, guests with a high number of VCPUs
may induce great latencies on the underlying RT Linux system (e.g. cyclictest
reports latencies of ~15ms for guests with 24 VCPUs). This can be further
aggravated by sending a lot of external interrupts to the guest. A malicious app
can abuse this scenario, causing a DoS of the host Linux. Until the KVM openpic
code is refactored to use finer lock granularity, impose a limitation on the
number of VCPUs a guest can have when running on a PREEMPT_RT_FULL system with
KVM_MPIC emulation.

Sent against v3.14-rt branch of
(Continue reading)

Choksing Taweponsomkiat | 22 Jan 00:30 2015

RT patch for kernel 3.15 in doubt?

I have a need for an RT patch for kernel 3.15 (because Baytrail SoC support starts there). I have heard that
the RT development effort is plagued by funding issues. Should I not keep my hopes up for the 3.15 release?

Choksing Taweponsomkiat

To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo <at>
More majordomo info at

Gustavo Bittencourt | 20 Jan 21:02 2015

[PATCH] rtmutex: enable deadlock detection in ww_mutex_lock functions

The functions ww_mutex_lock_interruptible and ww_mutex_lock should return -EDEADLK when faced with
a deadlock. To do so, the paramenter detect_deadlock in rt_mutex_slowlock must be TRUE.
This patch corrects potential deadlocks when running PREEMPT_RT with nouveau driver.

Kernel version: 3.14.25-rt22

Signed-off-by: Gustavo Bittencourt <gbitten <at>>
 kernel/locking/rtmutex.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 6c40660..3f6ef91 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
 <at>  <at>  -1965,7 +1965,7  <at>  <at>  __ww_mutex_lock_interruptible(struct ww_mutex *lock, struct ww_acquire_ctx *ww_c

 	mutex_acquire(&lock->base.dep_map, 0, 0, _RET_IP_);
-	ret = rt_mutex_slowlock(&lock->base.lock, TASK_INTERRUPTIBLE, NULL, 0, ww_ctx);
+	ret = rt_mutex_slowlock(&lock->base.lock, TASK_INTERRUPTIBLE, NULL, 1, ww_ctx);
 	if (ret)
 		mutex_release(&lock->base.dep_map, 1, _RET_IP_);
 	else if (!ret && ww_ctx->acquired > 1)
 <at>  <at>  -1984,7 +1984,7  <at>  <at>  __ww_mutex_lock(struct ww_mutex *lock, struct ww_acquire_ctx *ww_ctx)

 	mutex_acquire_nest(&lock->base.dep_map, 0, 0, &ww_ctx->dep_map,
-	ret = rt_mutex_slowlock(&lock->base.lock, TASK_UNINTERRUPTIBLE, NULL, 0, ww_ctx);
+	ret = rt_mutex_slowlock(&lock->base.lock, TASK_UNINTERRUPTIBLE, NULL, 1, ww_ctx);
(Continue reading)

Jing Shao | 16 Jan 23:14 2015

RT Patch to use for Kernel 3.2.0

Dear sir,

I am trying to patch a Linux 3.2.0 kernel with PREEMPT_RT patch, I tried the patch-3.2.64-rt94.patch.gz,
but the Linux kernel fails to compile afterwards, is this the right patch to use?

For your info, the ouput of "uname -r" is 3.2.0. The kernel comes in as part of the SDK for a TI Sitara processor.

Your help are greatly appreciated.


Jing Shao
Senior Software Engineer

Power System Solutions
3001 Summit Avenue Ste. 400
Plano, Texas 75074
469-331-9844 x8881
jing.shao <at>

Confidential Information:This message is sent to the intended recipient and may contain privileged or
confidential information. If you received this transmission in error, please notify the sender with a
replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert:
Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does
not accept liability for any errors or omissions.
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo <at>
More majordomo info at

(Continue reading)

Clark Williams | 9 Jan 15:42 2015

Re: Help regarding Cyclictest.

On Fri, 9 Jan 2015 09:37:07 +0530
Ipsit Kumar <kumaripsit1 <at>> wrote:

> Thanks for the reply.
> I think I forgot to mention about the kernel I am running. It is Linux
> Kernel 3.3 with preemption enabled.
> Could this be a problem ? And do you think this kind of behavior is
> expected if I use a RT Patch.

Large spikes are not unusual running on a stock kernel. If you are
trying to prevent these spikes, you will need to run an RT kernel
Clark Williams | 6 Jan 22:26 2015

Re: Help regarding Cyclictest.

On Fri, 2 Jan 2015 15:02:11 +0530
Ipsit Kumar <kumaripsit1 <at>> wrote:

> Hi sir,
> Good Afternoon.
> Thank you for your reply.
> I am facing some issues with cyclictest.
> Basically i am using this tool on ARM platform, specifically on TCI6614 evm.
> I am seeing strange spike in the cyclictest result.
> *root # ./cyclictest -t1 -p80 -i 10000 -l 10000# /dev/cpu_dma_latency set
> to 0uspolicy: fifo: loadavg: 0.70 0.48 0.61 1/434 15283           T: 0
> (15226) P:80 I:10000 C:  10000 Min:      5 Act:    9 Avg:   17 Max:    5797*
> When I am plotting the Histogram, I see the latency to go beyond 3ms once.
> I am unable to figure out what is the reason behind this.
> Checked with the kernel configuration and found that the CPU Frequency
> scaling is Disabled.
> No power management features are enabled.
> It would be a great help, if you could point out something I am doing
> wrong.
> Thank you.
(Continue reading)

Vegara Dorian | 6 Jan 17:27 2015

How can I disable VHCI support ?


I'm trying to patch my Linux with RT Linux 3.14.25. The installation has
been successfully done, but problems appeared when I typed this command :

" make modules_install ".

The error message is : Module 'hci_vhci' has devname (vhci) but lacks
major and minor information. Ignoring.

How can I fix this bug ? I don't want Bluetooth. So I think I can disable

Thanks a lot,
Best regards


To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo <at>
More majordomo info at

Gustavo Bittencourt | 5 Jan 23:47 2015

RT is freezing

Hi everybody

I compiled the 3.14.25-rt22, but my system freezes when I start Unity 
and some programs like Chrome or Thunderbird. The problem happens only 
when PREEMPT_RT_FULL=y. No log is generated. I would like to find the 
root of this problem, but I don't know how. Do you have any suggestion?

Best regard,
Gustavo Bittencourt
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo <at>
More majordomo info at