Dave Anderson | 5 Aug 18:21 2015

Re: Wrong RSS field in ps

----- Original Message -----
> On Tue, 4 Aug 2015 16:57:35 -0400 (EDT)
> Dave Anderson <anderson <at> redhat.com> wrote:
> > 
> > 
> > ----- Original Message -----
> > 
> > > > Michael
> > > 
> > > OK, so it would seem to be somewhere in the trail from enumerator_value()
> > > into the gdb_command_funnel() via the GNU_GET_DATATYPE req->command in
> > > datatype_info().
> > > 
> > > I'm having a hard time provisioning an s390x machine internally at this time.
> > > When I get one, I'll see if the problem has been there all along.
> > > 
> > > Dave
> > > 
> > 
> > Michael,
> > 
> > I finally got access to an s390x and I see the same thing.  In fact it also
> > happens on a RHEL6 2.6.32-based kernel, so it appears it's always been an
> > issue.  I'll look into it tomorrow.
> Great, thanks!
> Michael
(Continue reading)

Dave Anderson | 3 Aug 21:06 2015

Re: crash extensions help

----- Original Message -----
> Hi Dave,
> I found you from https://www.redhat.com/mailman/listinfo/crash-utility.
> I was really impressed with your contribution to the crash utility.
> I tried subscribing to the crash utility mailing list. But no luck.

Sorry about that -- I was on vacation for a couple of weeks until today.
You're on the list now.

> I started writing the crash extensions,
> I am unable to read the all the members of task_struct  and mm_struct structures.
> I am trying to read mm->hiwater_vm, the defs.h header file do not have
> struct_mm_hiwater_mm variable as a offset_table member.
> Could you please suggest me the ways that I could read all the members of
> the task_struct and mm_struct.
> Thanks in Advance.
> --
>   Thanks & Regards
>   Rupesh P

For structure members that are not in the built-in offset_table, you can just 
use MEMBER_OFFSET() directly, for example:

  MEMBER_OFFSET("mm_struct", "hiwater_rss")

(Continue reading)

Michael Holzheu | 3 Aug 16:31 2015

[PATCH] crash: Do not use bt -t flag in panic_search()

Hi Dave,

I got a dump where a process "gmain" was incorrectly marked as running:

crash> ps | grep gmain
>   217      1   5      8bec23420     IN   0.0  463276  18240  gmain

The reason was that the "brute force" way parsing the "bt -t -o"
output in panic_search() found the symbol "panic" on the stack:

crash> bt -t -o 8bec23420
PID: 217    TASK: 8bec23420         CPU: 5   COMMAND: "gmain"
              START: __schedule at 83f650
  [       8b662b900] (null) at 0
  [       8b662b950] (null) at 0
  [       8b662b978] __schedule at 83f650
  [       8b662b990] (null) at 0
  [       8b662bb18] (null) at 0
  [       8b662bb40] panic at 83679a  <<<<<--------------
  [       8b662bb58] _ehead at 280da

The real stack trace was as follows:

crash> bt  8bec23420
Detaching after fork from child process 15508.
PID: 217    TASK: 8bec23420         CPU: 5   COMMAND: "gmain"
 #0 [8b662b8f0] __schedule at 83f650
 #1 [8b662b958] schedule at 83fade
 #2 [8b662b970] schedule_hrtimeout_range_clock at 842fc8
(Continue reading)

Qu Wenruo | 27 Jul 05:11 2015

Crash fails to analyse kernel dump from 4.2-rc4

Hi all,

I updated my kernel to 4.2-rc4 but suddenly crash failed to analyse the 
kernel with the following error message:
crash: invalid kernel virtual address: 170000001d  type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 3800000046  type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 240000002d  type: "online"
WARNING: cannot read cpu_online_map
crash: invalid kernel virtual address: 570000006e  type: "active"
WARNING: cannot read cpu_active_map
crash: cannot determine base kernel version
crash: /home/adam/linux-btrfs/vmlinux and vmcore do not match!

Crash 7.1.0 and 7.1.2 fails with the same error.

Previously I'm testing 4.2-rc1 kernel and 7.1.0 works like a charm.

So is there something wrong with the new kernel?


Michael Holzheu | 27 Jul 19:34 2015

Wrong RSS field in ps

Hello Dave,

On s390 (kernel 4.2.0-rc2) the "RSS" field in "ps" is wrong.

The reason is that in rss_page_types_init() enumerator_value("MM_ANONPAGES",
&anonpages) returns zero for "anonpages" and therefore we add MM_FILEPAGES
twice instead of adding MM_ANONPAGES.

Example: Process that allocated 500 MB:

ps 2152
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
   2152   1061   4      7aff0000      IN   0.0  514024   2236  eat_mem

crash> print/x ((struct task_struct *) 0x7aff0000)->mm->rss_stat
$1 = {
  count = {{
      counter = 0x113
    }, {
      counter = 0x1f414
    }, {
      counter = 0x0

Any idea why enumerator_value() is not working?


(Continue reading)

Aaron Tomlin | 27 Jul 16:40 2015

[PATCH] dis: Consolidate cmd_dis

Hi Dave,

No functionality change (hopefully). This patch is essentially a clean up
in preparation to introduce the "-f" option.

Please let me know your thoughts.

Signed-off-by: Aaron Tomlin <atomlin <at> redhat.com>
 kernel.c | 265 ++++++++++++++++++++++-----------------------------------------
 1 file changed, 91 insertions(+), 174 deletions(-)

diff --git a/kernel.c b/kernel.c
index 73d1983..3107014 100644
--- a/kernel.c
+++ b/kernel.c
 <at>  <at>  -1589,212 +1589,129  <at>  <at>  cmd_dis(void)

-		do_load_module_filter = module_symbol(req->addr, NULL, NULL, 
-			NULL, *gdb_output_radix);
+		req->command = GNU_RESOLVE_TEXT_ADDR;
+		gdb_interface(req);
+		req->flags &= ~GNU_COMMAND_FAILED;
+		if (reverse || req->flags & GNU_FUNCTION_ONLY) {
+			if (sp) {
+				savename = sp->name;
+				if ((sp = next_symbol(NULL, sp)))
+					req->addr2 = sp->value;
(Continue reading)

"Zhou, Wenjian/周文剑" | 21 Jul 10:14 2015

[PATCH]remind of using --zero_exlcuded

The option --zero_excluded can be used to analyze the incomplete
So it is needed to remind users of using it when trying to analyze
incomplete dumpfile without --zero_excluded.

Zhou Wenjian
The option --zero_excluded can be used to analyze the incomplete
So it is needed to remind users of using it when trying to analyze
incomplete dumpfile without --zero_excluded.


Zhou Wenjian
Rabin Vincent | 16 Jul 09:50 2015

[PATCH] extensions/trace: max_buffer is optional

max_buffer is optional in the kernel (depends on the
CONFIG_TRACE_MAX_TRACE option).  Don't fail if it isn't available.
 extensions/trace.c |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/extensions/trace.c b/extensions/trace.c
index 9f81568..c269f4c 100644
--- a/extensions/trace.c
+++ b/extensions/trace.c
 <at>  <at>  -34,6 +34,10  <at>  <at>  static int encapsulated_current_trace;
  * trace_buffer is supported
 static int trace_buffer_available;
+ * max_buffer is supported
+ */
+static int max_buffer_available;

 #define koffset(struct, member) struct##_##member##_offset

 <at>  <at>  -163,8 +167,10  <at>  <at>  static int init_offsets(void)

 	if (trace_buffer_available) {
 		init_offset(trace_array, trace_buffer);
-		init_offset(trace_array, max_buffer);
 		init_offset(trace_buffer, buffer);
+		if (max_buffer_available)
+			init_offset(trace_array, max_buffer);
(Continue reading)

Dave Anderson | 13 Jul 17:20 2015

[ANNOUNCE] crash version 7.1.2 is available

Download from: http://people.redhat.com/anderson

The master branch serves as a development branch that will contain all 
patches that are queued for the next release:

  $ git clone git://github.com/crash-utility/crash.git


 - Enhancement of the ARM64 backtrace capability.  Without the patch,
   backtraces of the active tasks start at the function that is saved
   in each per-cpu ELF note.  With the patch, the backtrace will start
   at the "crash_kexec" function on the panicking cpu, and at the 
   "crash_save_cpu" function on the other active cpus.  By doing so,
   the backtrace will display the exception handling functions leading
   to crash_kexec() or crash_save_cpu(), as well as the exception frame
   register set as it was at the time of the fatal exception on the 
   panic cpu, or when the shutdown IPI was received on the other cpus. 
   (anderson <at> redhat.com)

 - Enabled the "bt -R" option on the ARM64 architecture.  Without the
   patch, the option fails with the message "bt: -R option not supported
   or applicable on this architecture or kernel".
   (anderson <at> redhat.com)

 - Enabled the "crash --log vmcore" command line option on the ARM64 
   architecture.  Without the patch, the option fails with the message 
(Continue reading)

Dave Anderson | 7 Jul 17:20 2015

[PATCH] address compiler warning for extensions/trace.c

Hello Qiao,

With more recent versions of gcc, extensions/trace.c generates this warning:

  $ make extensions
  gcc -Wall -g -shared -rdynamic -o trace.so trace.c -fPIC -DX86_64 -DLZO -DSNAPPY -DGDB_7_6
  /tmp/ccSOIphT.o: In function 'ftrace_show':
  /root/crash.git/extensions/trace.c:1560: warning: the use of 'mktemp' is dangerous, better use 'mkstemp'

I've attached an untested patch that replaces the mktemp() call with mkstemp().
I believe the only behavioral difference is that mkstemp() will create the file
automatically with permissions 600 and open flags of O_EXCL, whereas it is 
currently being opened with permissions 644 and flags (O_WRONLY | O_CREAT | O_TRUNC).

Can you either ACK the patch, or address the warning as you would prefer? 

Attachment (use_mkstemp.patch): text/x-patch, 895 bytes

Hello Qiao,

With more recent versions of gcc, extensions/trace.c generates this warning:

  $ make extensions
(Continue reading)

Aaron Tomlin | 6 Jul 10:20 2015

[PATCH 1/2] dis: Make the starting address of a function special for hexadecimal targets too

If no count is not entered, and the target address is the starting address
of a function, disassemble the entire function.
However this is only true, if the given target is symbolic.

This patch simply removes that restriction when the target is a hexadecimal

For example, each of the following commands will return the same result.

 - Where 0xffffffff80137e9d is assumed to be the address of the
   first instruction in do_fork()

  crash> dis 0xffffffff80137e9d
  crash> dis do_fork

Signed-off-by: Aaron Tomlin <atomlin <at> redhat.com>
 kernel.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel.c b/kernel.c
index 4932282..628779c 100644
--- a/kernel.c
+++ b/kernel.c
 <at>  <at>  -1436,6 +1436,8  <at>  <at>  cmd_dis(void)
 				unfiltered = TRUE;
+			if (!offset)
+				req->flags |= GNU_FUNCTION_ONLY;
(Continue reading)