Uli Brueggemann | 1 Feb 16:06 2008
Picon

2.6.24-rt1 SSttrraannggee keyboard behaviour

Hi,

I'm trying to use 2.6.24-rt1 in combination with BruteFIR (audio). So
far everything looks fine except:

After starting BruteFIR (which is doing its job) after a while the
keyboard responds like with double clicks. TTeerrrriibbllee. I cannot
enter a command anymore. The running BruteFIR itself still is running
stable.
 The system without starting BruteFIR keeps stable. BruteFIR itself is
working fine with older kernels. Also with e.g. 2.6.24-rc3-zen3. So I
guess the realtime patch is causing this.
BTW Brutefir is using realtime priorities 1 to 4 and it also uses SCHED_FIFO.

Any idea?

Uli
Clark Williams | 1 Feb 17:20 2008
Picon

Re: 2.6.24-rt1 SSttrraannggee keyboard behaviour


Uli Brueggemann wrote:
> Hi,
> 
> I'm trying to use 2.6.24-rt1 in combination with BruteFIR (audio). So
> far everything looks fine except:
> 
> After starting BruteFIR (which is doing its job) after a while the
> keyboard responds like with double clicks. TTeerrrriibbllee. I cannot
> enter a command anymore. The running BruteFIR itself still is running
> stable.
>  The system without starting BruteFIR keeps stable. BruteFIR itself is
> working fine with older kernels. Also with e.g. 2.6.24-rc3-zen3. So I
> guess the realtime patch is causing this.
> BTW Brutefir is using realtime priorities 1 to 4 and it also uses SCHED_FIFO.
> 
> Any idea?
> 
> Uli

Uli,

I've seen this behavior before; on my system in addition to the doubled keystrokes,
cursor animations were jerky.

Are you running on a 64-bit kernel, is your clocksource hpet and is vsyscall64 on?

$ uname -m
x86_64
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
(Continue reading)

Uli Brueggemann | 3 Feb 17:45 2008
Picon

Re: 2.6.24-rt1 SSttrraannggee keyboard behaviour

>
> I've seen this behavior before; on my system in addition to the doubled keystrokes,
> cursor animations were jerky.
>
> Are you running on a 64-bit kernel, is your clocksource hpet and is vsyscall64 on?
>
> $ uname -m
> x86_64
> $ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> hpet
> $ cat /proc/sys/kernel/vsyscall64
> 1
>
> If so, try putting this into your /etc/sysctl.conf and rebooting:
>
> # vsyscall64 off
> kernel.vsyscall64 = 0
>

Kernel is i686 (32bit), clocksource is tsc and vsyscall64 does not exist.
Only console, no X.

In the meantime I have further tried to detect a possible reason and I
guess I kow it now. But I do not know how to overcome:

The system is booted from USB memorystick. The keyboard is a USB
keyboard. And there is an IMON VFD display connected by USB. To use it
I run the lcdproc package.
It seems that the LCDd daemon (with imon_pad driver) is the reason. It
causes problems with the keyboard (as described in the topic) and also
(Continue reading)

Daniel Walker | 5 Feb 03:51 2008

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1

On Mon, Feb 04, 2008 at 03:35:13PM -0800, Max Krasnyanskiy wrote:
> This is just an FYI. As part of the "Isolated CPU extensions" thread Daniel suggest for me
> to check out latest RT kernels. So I did or at least tried to and immediately spotted a couple
> of issues.
>
> The machine I'm running it on is:
> 	HP xw9300, Dual Opteron, NUMA
>
> It looks like with -rt kernel IRQ affinity masks are ignored on that 
> system. ie I write 1 to lets say /proc/irq/23/smp_affinity but the 
> interrupts keep coming to CPU1. Vanilla 2.6.24 does not have that issue.

I tried this, and it works according to /proc/interrupts .. Are you
looking at the interrupt threads affinity?

> Also the first thing I tried was to bring CPU1 off-line. Thats the fastest 
> way to get irqs, soft-irqs, timers, etc of a CPU. But the box hung 
> completely. It also managed to mess up my ext3 filesystem to the point 
> where it required manual fsck (have not see that for a couple of
> years now). I tried the same thing (ie echo 0 > 
> /sys/devices/cpu/cpu1/online) from the console. It hang again with the 
> message that looked something like:
> 	CPU1 is now off-line
> 	Thread IRQ-23 is on CPU1 ...

I get the following when I tried it,

BUG: sleeping function called from invalid context bash(5126) at
kernel/rtmutex.c:638
in_atomic():1 [00000001], irqs_disabled():1
(Continue reading)

Gregory Haskins | 5 Feb 04:27 2008
Picon

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1

Hi Daniel,

  See inline...

>>> On Mon, Feb 4, 2008 at  9:51 PM, in message
<20080205025144.GA31774 <at> dwalker1.mvista.com>, Daniel Walker
<dwalker <at> dwalker1.mvista.com> wrote: 
> On Mon, Feb 04, 2008 at 03:35:13PM -0800, Max Krasnyanskiy wrote:
>> This is just an FYI. As part of the "Isolated CPU extensions" thread Daniel 
> suggest for me
>> to check out latest RT kernels. So I did or at least tried to and 
> immediately spotted a couple
>> of issues.
>>
>> The machine I'm running it on is:
>> 	HP xw9300, Dual Opteron, NUMA
>>
>> It looks like with -rt kernel IRQ affinity masks are ignored on that 
>> system. ie I write 1 to lets say /proc/irq/23/smp_affinity but the 
>> interrupts keep coming to CPU1. Vanilla 2.6.24 does not have that issue.
> 
> I tried this, and it works according to /proc/interrupts .. Are you
> looking at the interrupt threads affinity?
> 
>> Also the first thing I tried was to bring CPU1 off-line. Thats the fastest 
>> way to get irqs, soft-irqs, timers, etc of a CPU. But the box hung 
>> completely. It also managed to mess up my ext3 filesystem to the point 
>> where it required manual fsck (have not see that for a couple of
>> years now). I tried the same thing (ie echo 0 > 
>> /sys/devices/cpu/cpu1/online) from the console. It hang again with the 
(Continue reading)

Max Krasnyansky | 5 Feb 05:21 2008

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1


Daniel Walker wrote:
> On Mon, Feb 04, 2008 at 03:35:13PM -0800, Max Krasnyanskiy wrote:
>> This is just an FYI. As part of the "Isolated CPU extensions" thread Daniel suggest for me
>> to check out latest RT kernels. So I did or at least tried to and immediately spotted a couple
>> of issues.
>>
>> The machine I'm running it on is:
>> 	HP xw9300, Dual Opteron, NUMA
>>
>> It looks like with -rt kernel IRQ affinity masks are ignored on that 
>> system. ie I write 1 to lets say /proc/irq/23/smp_affinity but the 
>> interrupts keep coming to CPU1. Vanilla 2.6.24 does not have that issue.
> 
> I tried this, and it works according to /proc/interrupts .. Are you
> looking at the interrupt threads affinity ?
Nope. I'm looking at the /proc/interrupts. ie The interrupt count keeps incrementing for cpu1 even
though affinity mask is set to 1.

IRQ thread affinity was btw set to 3 which is probably wrong.
To clarify, by default after reboot:
	- IRQ affinity set 3, IRQ thread affinity set to 3
	- User writes 1 into /proc/irq/N/smp_affinity
	- IRQ affinity is now set to 1, IRQ thread affinity is still set to 3

It'd still work I guess but does not seem right. Ideally IRQ thread affinity should have change as well.
We could of course just have some user-space tool that adjusts both.

Looks like Greg already replied to the cpu hotplug issue. For me it did not oops. Just got stuck probably
because it could not move an IRQ due to broken IRQ affinity logic.
(Continue reading)

Gregory Haskins | 5 Feb 06:02 2008
Picon

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1

>>> On Mon, Feb 4, 2008 at  9:51 PM, in message
<20080205025144.GA31774 <at> dwalker1.mvista.com>, Daniel Walker
<dwalker <at> dwalker1.mvista.com> wrote: 
> I get the following when I tried it,
> 
> BUG: sleeping function called from invalid context bash(5126) at
> kernel/rtmutex.c:638
> in_atomic():1 [00000001], irqs_disabled():1

Hi Daniel,
  Can you try this patch and let me know if it fixes your problem?

-----------------------

use rcu for root-domain kfree

Signed-off-by: Gregory Haskins <ghaskins <at> novell.com>

diff --git a/kernel/sched.c b/kernel/sched.c
index e6ad493..77e86c1 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
 <at>  <at>  -339,6 +339,7  <at>  <at>  struct root_domain {
        atomic_t refcount;
        cpumask_t span;
        cpumask_t online;
+       struct rcu_head rcu;

        /*
         * The "RT overload" flag: it gets set if a CPU has more than
(Continue reading)

Gregory Haskins | 5 Feb 15:00 2008
Picon

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1

>>> On Mon, Feb 4, 2008 at  9:51 PM, in message
<20080205025144.GA31774 <at> dwalker1.mvista.com>, Daniel Walker
<dwalker <at> dwalker1.mvista.com> wrote: 
> On Mon, Feb 04, 2008 at 03:35:13PM -0800, Max Krasnyanskiy wrote:

[snip]

>>
>> Also the first thing I tried was to bring CPU1 off-line. Thats the fastest 
>> way to get irqs, soft-irqs, timers, etc of a CPU. But the box hung 
>> completely.

After applying my earlier submitted patch, I was able to reproduce the hang you mentioned.  I poked around in
sysrq and it looked like a deadlock on a rt_mutex, so I turned on lockdep and it found:

=======================================================
[ INFO: possible circular locking dependency detected ]
[ 2.6.24-rt1-rt #3
-------------------------------------------------------
bash/4604 is trying to acquire lock:
 (events){--..}, at: [<ffffffff802537b6>] cleanup_workqueue_thread+0x16/0x80

but task is already holding lock:
 (workqueue_mutex){--..}, at: [<ffffffff80254615>] workqueue_cpu_callback+0xe5/0x140

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #5 (workqueue_mutex){--..}:
(Continue reading)

Daniel Walker | 5 Feb 17:59 2008

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1

On Mon, Feb 04, 2008 at 10:02:12PM -0700, Gregory Haskins wrote:
> >>> On Mon, Feb 4, 2008 at  9:51 PM, in message
> <20080205025144.GA31774 <at> dwalker1.mvista.com>, Daniel Walker
> <dwalker <at> dwalker1.mvista.com> wrote: 
> > I get the following when I tried it,
> > 
> > BUG: sleeping function called from invalid context bash(5126) at
> > kernel/rtmutex.c:638
> > in_atomic():1 [00000001], irqs_disabled():1
> 
> Hi Daniel,
>   Can you try this patch and let me know if it fixes your problem?
> 
> -----------------------
> 
> use rcu for root-domain kfree
> 
> Signed-off-by: Gregory Haskins <ghaskins <at> novell.com>
> 
> diff --git a/kernel/sched.c b/kernel/sched.c
> index e6ad493..77e86c1 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
>  <at>  <at>  -339,6 +339,7  <at>  <at>  struct root_domain {
>         atomic_t refcount;
>         cpumask_t span;
>         cpumask_t online;
> +       struct rcu_head rcu;
> 
>         /*
(Continue reading)

Gregory Haskins | 5 Feb 18:13 2008
Picon

Re: CPU hotplug and IRQ affinity with 2.6.24-rt1

>>> On Tue, Feb 5, 2008 at 11:59 AM, in message
<20080205165936.GA18613 <at> dwalker1.mvista.com>, Daniel Walker
<dwalker <at> dwalker1.mvista.com> wrote: 
> On Mon, Feb 04, 2008 at 10:02:12PM -0700, Gregory Haskins wrote:
>> >>> On Mon, Feb 4, 2008 at  9:51 PM, in message
>> <20080205025144.GA31774 <at> dwalker1.mvista.com>, Daniel Walker
>> <dwalker <at> dwalker1.mvista.com> wrote: 
>> > I get the following when I tried it,
>> > 
>> > BUG: sleeping function called from invalid context bash(5126) at
>> > kernel/rtmutex.c:638
>> > in_atomic():1 [00000001], irqs_disabled():1
>> 
>> Hi Daniel,
>>   Can you try this patch and let me know if it fixes your problem?
>> 
>> -----------------------
>> 
>> use rcu for root-domain kfree
>> 
>> Signed-off-by: Gregory Haskins <ghaskins <at> novell.com>
>> 
>> diff --git a/kernel/sched.c b/kernel/sched.c
>> index e6ad493..77e86c1 100644
>> --- a/kernel/sched.c
>> +++ b/kernel/sched.c
>>  <at>  <at>  -339,6 +339,7  <at>  <at>  struct root_domain {
>>         atomic_t refcount;
>>         cpumask_t span;
>>         cpumask_t online;
(Continue reading)


Gmane