Chuck Silvers | 13 Feb 2005 01:32
Favicon

"pmap_unwire: wiring ... didn't change!"

hi,

I looked into the "pmap_unwire: wiring ... didn't change!" problem.
what's happening is that sometimes we're going through uvm_fault() again
for one of the wired pages, and the pmap_enter() which results from that
replaces the wired pmap entry with non-wired one.  so why would we go through
uvm_fault() again?  that's happening because sometimes the TLB entry is
recycled after the page mapped by one side of the entry is wired and before
the other page is wired.  and when MachTLBUpdate() is called for the second
page, the entry it creates only has the second side filled in.  this causes
us to get a "TLB invalid" trap when we access the other half of the TLB entry
instead of the "TLB miss" trap that we would get if there were no TLB entry
at all.  the handler for "TLB invalid" traps from usermode always calls
trap() instead of looking at the PTEs.

so the problem is that the PTEs and the TLB get out of sync, and the trap
handlers aren't expecting that.  I see several possible fixes, in
approximate decreasing order of preference:

 - change MachTLBUpdate() to take both PTEs for the TLB entry for MIPS3.

 - change the MIPS3 MachTLBUpdate() to only update existing TLB entries,
   not create new ones.

 - have the MIPS3 user "TLB invalid" trap handler try to reload from the
   PTEs before calling trap().

 - use MIPS3_TBIS() instead of MachTLBUpdate() for user mappings on MIPS3.

the last one was the easiest, so I implemented that one and it appears to
(Continue reading)

Markus W Kilbinger | 13 Feb 2005 12:02
Picon
Picon

Re: "pmap_unwire: wiring ... didn't change!"

>>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:

    Chuck> I looked into the "pmap_unwire: wiring ... didn't change!"
    Chuck> problem. [...]

Thanks for doing that! :-)

    Chuck> the last one was the easiest, so I implemented that one and
    Chuck> it appears to make the problem go away.

I've applied your patch and I can confirm vanishing of the
"pmap_unwire: ..." messages (so far, 2 hours now).

But I still see (my?) data corruption problem:

  http://mail-index.netbsd.org/port-cobalt/2004/11/04/0000.html

... still 32 bytes long and at 32 bytes boundary, maybe related to

  cpu0 at mainbus0: QED RM5200 CPU (0x28a0) Rev. 10.0 with built-in FPU Rev. 10.0
  cpu0: 32KB/32B 2-way set-associative L1 Instruction cache, 48 TLB entries
  cpu0: 32KB/32B 2-way set-associative write-back L1 Data cache

While digging in that do you see any reason for their appearing?

Regards, Markus.

Chuck Silvers | 13 Feb 2005 14:18
Favicon

Re: "pmap_unwire: wiring ... didn't change!"

On Sun, Feb 13, 2005 at 12:02:39PM +0100, Markus W Kilbinger wrote:
> >>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:
> 
>     Chuck> I looked into the "pmap_unwire: wiring ... didn't change!"
>     Chuck> problem. [...]
> 
> Thanks for doing that! :-)
> 
>     Chuck> the last one was the easiest, so I implemented that one and
>     Chuck> it appears to make the problem go away.
> 
> I've applied your patch and I can confirm vanishing of the
> "pmap_unwire: ..." messages (so far, 2 hours now).

cool.

> But I still see (my?) data corruption problem:
> 
>   http://mail-index.netbsd.org/port-cobalt/2004/11/04/0000.html
> 
> ... still 32 bytes long and at 32 bytes boundary, maybe related to
> 
>   cpu0 at mainbus0: QED RM5200 CPU (0x28a0) Rev. 10.0 with built-in FPU Rev. 10.0
>   cpu0: 32KB/32B 2-way set-associative L1 Instruction cache, 48 TLB entries
>   cpu0: 32KB/32B 2-way set-associative write-back L1 Data cache
> 
> While digging in that do you see any reason for their appearing?

that sounds like a CPU cache problem to me too, probably in bus_dma or the
cache-flushing code itself.  if it's happening during writes to disk rather
(Continue reading)

Izumi Tsutsui | 13 Feb 2005 17:06
Picon
Gravatar

Re: "pmap_unwire: wiring ... didn't change!"

In article <20050213131857.GA4659 <at> spathi.chuq.com>
chuq <at> chuq.com wrote:

> I don't see this problem on my R4400 pmax 5000/260, so it's likely specific
> to either the RM5200/R5000 or systems with no L2 cache.  do RM5200/R5000
> systems with L2 cache (eg. sgimips) see this?  do we support any other
> MIPS variants with no L2 cache?  maybe some of the hpcmips doodads?

My R5000 O2 with enabled L2 cache also has the same problem.
I guess there is something wrong around r5k cache code
but I can't find any particular problem in cache_r5k.c
when I looked at (but I could be wrong).
---
Izumi Tsutsui
tsutsui <at> ceres.dti.ne.jp

Markus W Kilbinger | 13 Feb 2005 21:16
Picon
Picon

Re: "pmap_unwire: wiring ... didn't change!"

>>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:

    >> I've applied your patch and I can confirm vanishing of the
    >> "pmap_unwire: ..." messages (so far, 2 hours now).

    Chuck> cool.

(... 9.5 hours now ;-))

    >> But I still see (my?) data corruption problem:

    Chuck> that sounds like a CPU cache problem to me too, probably in
    Chuck> bus_dma or the cache-flushing code itself. if it's
    Chuck> happening during writes to disk rather than reads from disk
    Chuck> then it's probably in the cache write-back part rather than
    Chuck> the cache invalidate part. I didn't see anything in a brief
    Chuck> look at the code, though.

My mentioned tests, where I can reproduce the data corruption
certainly, involve disk access; _reading_ large data amounts from disk
is enough to get a corruption.

Once I tested my qube2's RAM with pkgsrc/sysutils/memtester where no
errors were reported.

I did not notice any data corruption if using my qube2 in routing
data, but I did no selective stress testing on that.

    Chuck> I don't see this problem on my R4400 pmax 5000/260, so it's
    Chuck> likely specific to either the RM5200/R5000 or systems with
(Continue reading)

Chuck Silvers | 14 Feb 2005 18:41
Favicon

Re: "pmap_unwire: wiring ... didn't change!"

On Sun, Feb 13, 2005 at 09:16:29PM +0100, Markus W Kilbinger wrote:
> >>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:
> 
>     >> I've applied your patch and I can confirm vanishing of the
>     >> "pmap_unwire: ..." messages (so far, 2 hours now).
> 
>     Chuck> cool.
> 
> (... 9.5 hours now ;-))
> 
>     >> But I still see (my?) data corruption problem:
> 
>     Chuck> that sounds like a CPU cache problem to me too, probably in
>     Chuck> bus_dma or the cache-flushing code itself. if it's
>     Chuck> happening during writes to disk rather than reads from disk
>     Chuck> then it's probably in the cache write-back part rather than
>     Chuck> the cache invalidate part. I didn't see anything in a brief
>     Chuck> look at the code, though.
> 
> My mentioned tests, where I can reproduce the data corruption
> certainly, involve disk access; _reading_ large data amounts from disk
> is enough to get a corruption.

so you get different corruption when you read the same file at different times?
that's useful to know.

> Once I tested my qube2's RAM with pkgsrc/sysutils/memtester where no
> errors were reported.
> 
> I did not notice any data corruption if using my qube2 in routing
(Continue reading)

Markus W Kilbinger | 14 Feb 2005 21:37
Picon
Picon

Re: "pmap_unwire: wiring ... didn't change!"

>>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:

    >> My mentioned tests, where I can reproduce the data corruption
    >> certainly, involve disk access; _reading_ large data amounts
    >> from disk is enough to get a corruption.

    Chuck> so you get different corruption when you read the same file
    Chuck> at different times? that's useful to know.

Yes: A (large) file that was intact a first time seems to be corrupt
when reading it later a second time (and vice versa).

For me it seems I can diminish these corrutions by putting some other
load on the qube2. On the other hand pure disk access (e. g. untarring
new base.tgz :-/)) produces quite certainly some corrupt files
(libc.so... :-(). To workaround the latter I run 'nice pax -zvrpe ...'
over a ssh connection, so that pax's '-v' vorbose output produces some
additional load which prevents most file corruptions (not all: some,
especially larger files might still get corrupted).

    >> Hmm, if the problems occurs on quite different hardware, just
    >> having the same mips CPU type, (common) r5k cache handling
    >> seems really to be the most probable cause of the corruption
    >> (correct?). Or ist bus_dma still a candidate?

    Chuck> could be either, we don't know yet. the various versions of
    Chuck> the bus_dma code for all the MIPS3 platforms are pretty
    Chuck> similar.

...despite the fact that the same bus_dma code works for/on your R4k4?
(Continue reading)

Chuck Silvers | 15 Feb 2005 03:46
Favicon

Re: "pmap_unwire: wiring ... didn't change!"

On Mon, Feb 14, 2005 at 09:37:40PM +0100, Markus W Kilbinger wrote:
> >>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:
> 
>     >> My mentioned tests, where I can reproduce the data corruption
>     >> certainly, involve disk access; _reading_ large data amounts
>     >> from disk is enough to get a corruption.
> 
>     Chuck> so you get different corruption when you read the same file
>     Chuck> at different times? that's useful to know.
> 
> Yes: A (large) file that was intact a first time seems to be corrupt
> when reading it later a second time (and vice versa).

ok, so it's clear the corruption can happen when reading from disk.
can you tell if the file is ever corrupted while being written to disk?
you'll need to use a pattern generator to create some files (rather than
reading them from disk).  ideally you'd check the data by reading it on
some other platform that doesn't have this bug, but I'd think that if
all of the data can be read back correctly on the qube at least part of
the time, then it's likely that the corruption is only on the read side.

> For me it seems I can diminish these corrutions by putting some other
> load on the qube2. On the other hand pure disk access (e. g. untarring
> new base.tgz :-/)) produces quite certainly some corrupt files
> (libc.so... :-(). To workaround the latter I run 'nice pax -zvrpe ...'
> over a ssh connection, so that pax's '-v' vorbose output produces some
> additional load which prevents most file corruptions (not all: some,
> especially larger files might still get corrupted).

that makes sense.  the other activity is likely to reuse the cache lines
(Continue reading)

Markus W Kilbinger | 15 Feb 2005 23:33
Picon
Picon

Re: "pmap_unwire: wiring ... didn't change!"

>>>>> "Chuck" == Chuck Silvers <chuq <at> chuq.com> writes:

    >> Maybe I should send-pr now (more precisely)?

    Chuck> please, that would be great.

Done (port-mips/29395).

    Chuck> ok, so it's clear the corruption can happen when reading
    Chuck> from disk. can you tell if the file is ever corrupted while
    Chuck> being written to disk? you'll need to use a pattern
    Chuck> generator to create some files (rather than reading them
    Chuck> from disk). ideally you'd check the data by reading it on
    Chuck> some other platform that doesn't have this bug, but I'd
    Chuck> think that if all of the data can be read back correctly on
    Chuck> the qube at least part of the time, then it's likely that
    Chuck> the corruption is only on the read side.

As described in the pr I wrote a (simple) pattern generator: The
corruptions occur mostly during writing (have a look at the pr if you
are interested in the corruption pattern). Reading seems to corrupt
data only when accessing 2 files simultaneously (as 'cmp' does), but
in general not that much as writing.

BTW: I did _not_ explant my qube2's harddisk for data comparison in
another machine. I've just rebooted the machine multiple times and
repeated the comparison: All the constant diff's are blamed to be
writing related. I hope that's sufficient...

    Chuck> FYI, I'm probably not going to have time to pursue this
(Continue reading)

Izumi Tsutsui | 25 Feb 2005 17:35
Picon
Gravatar

gas warning on mipsX_subr.S

These warnings are harmless or not?
---
# /usr/tools/mipsel/bin/mipsel--netbsd-gcc -mno-abicalls \
-x assembler-with-cpp -traditional-cpp -D_LOCORE -Dcobalt \
-I. -I../../../../arch -I../../../.. -nostdinc -DMIPS3 -DLKM \
-DDIAGNOSTIC -DMAXUSERS=32 -D_KERNEL -D_KERNEL_OPT -c \
../../../../arch/mips/mips/mips3_subr.S
../../../../arch/mips/mips/mipsX_subr.S: Assembler messages:
../../../../arch/mips/mips/mipsX_subr.S:1403: Warning: \
Macro instruction expanded into multiple instructions in a branch delay slot
../../../../arch/mips/mips/mipsX_subr.S:1511: Warning: \
Macro instruction expanded into multiple instructions in a branch delay slot
#
---
Izumi Tsutsui
tsutsui <at> ceres.dti.ne.jp


Gmane