Andre Richter | 1 Oct 22:20 2009
Picon

Re: Nr. lost kernel samples

Hi Robert,

I'm still investigating those lost kernel samples.
Today i installed a new Debian Lenny 5.0 stable System.
I tested 2 kernels, 2.6.18.8 under Xen 3.3.1. Stock kernel from the  
Xen repositories.
And 2.6.26 that comes by default with lenny.

There are two strange behaviors i observed.
2.6.26 doesn't lose a kernel sample. Whereas 2.6.18 under Xen does.
The more interesting fact is that, when i increase the callgraph depth  
from 0 to 16, there is a huge shift in the received samples as well as  
in the lost kernel samples under 2.6.18.
I did some math with my results, but I still can't figure out what's  
goin on.

The buffer overflows seem to be a problem that i can get rid of by  
adapting the sample rate / buffer sizes.
But is the huge gap in the sample counts an expected behavior when i  
increase the callgraph depth?

4 files are attached, filenames explain their setup.
While profiling, the lenny box acted as an iperf server for 1 GBit UDP  
incoming traffic.
Attachment (log-2.6.18xen0-cg0-1min): application/octet-stream, 766 bytes
Attachment (log-2.6.18xen0-cg16-1min): application/octet-stream, 786 bytes
Attachment (log-2.6.26-cg0-1min): application/octet-stream, 942 bytes
Attachment (log-2.6.26-cg16-1min): application/octet-stream, 953 bytes

(Continue reading)

Maynard Johnson | 4 Oct 18:44 2009
Picon

Re: Problems with libc6 symbols

Eran Rom wrote:
> Hi Maynard,
> Any quick & dirty fix for the meanwhile :-)
Sorry, I've been tied up with some critical customer engagement work, and haven't been able to look at this. 
And I likely won't have much time in the coming week either.  But my first pass on solving this problem would
be to look at using the same type of fix-up that's now done for ppc64/64-bit bfd's for all architectures.  If
you want to look at building such a patch, that would be great.  But for a quick and dirty fix for you, you can
just comment out the last check in libutil++/bfd_support.cpp:boring_symbol -- i.e., the SEC_LOAD
check.  I *think* that will be a workaround for you.  That can't be a final fix, because that check is needed to
prevent the "samples_range(): start > end " error that can occur for certain classes of symbols.

-Maynard
> 
> Thanks very much,
> Eran
> 
> Maynard Johnson <maynardj <at> us.ibm.com> wrote on 29/09/2009 23:00:29:
> 
>> Eran Rom wrote:
>>> Maynard Johnson <maynardj <at> us.ibm.com> wrote on 29/09/2009 21:33:41:
>>>> hmmm . . . I get correct results (i.e., symbol resolution into libc)
>>>> when using
>>>> 64-bit libc debuginfo, but I get same results as you when using
>>> 32-bit.  :-(
>>>> Need to investigate more.
>>> Maynard, thanks very much for the reply.
>>> I did forget to mention that I am using OProfile 0.9.5 over 64 bit
> version
>>> of Ubuntu so the libc debug info is 64bit
>> I've discovered the reason 64-bit works OK for me is because I
(Continue reading)

Leo Barnes | 6 Oct 14:38 2009
Picon

Recompiling kernel with OProfile for ARM

Hello!

I am having trouble getting OProfile to run on a arm linux system. I can compile OProfile without any trouble, but when I try to start it I get some error messages telling me that my kernel does not have the OProfile module.

I have tried recompiling my kernel, but since I am quite new to Linux, and extremely new when it comes to recompiling the kernel, this is giving me some trouble. I followed http://www.falkotimme.com/howtos/debian_kernel2.6_compile/ and successfully compiled both a kernel with the OProfile option set to included and one with it set as module. I can't seem to get mkinitrd to work though (or rather, can't find it) but after searching around a bit, I found that mkinitramfs is probably what I want to use. I successfully created a ramdisk and successfully started the system from the new kernel and ramdisk, but I still get the same error complaining about no kernel support. Have I missed something? When I do modprobe -l oprofile in my desktop computer, i find the oprofile module, but when I do the same on the arm system, it finds nothing. Since I have never recompiled the kernel, and don't really know how modules work, I am more or less at a loss about what to do next.

Grateful for any suggestions
//Leo

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list
Richard Purdie | 6 Oct 14:56 2009
Picon

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled

Hi,

On Tue, 2009-10-06 at 13:17 +0100, Mark.Pashley <at> selex-comms.com wrote:
> I have now upgraded the kernel to version 2.6.30.  Unfortunately it
> has not made any difference and I still get the same problem, namely
> all of my samples being lost. Have you got any suggestions where to
> start looking for why these samples are being lost? 

Ok, the next step is to figure out whether anything is getting to
userspace. When oprofile is running there should be a daemon taking
kernel buffers and emptying them into userspace. Is that daemon running
and does it notice data is there for collection?

My memory is fuzzy but there should be some way to make the output from
oprofiled more verbose so you can see what its doing...

Cheers,

Richard

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
Richard Purdie | 6 Oct 15:46 2009
Picon

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled

On Tue, 2009-10-06 at 14:41 +0100, Mark.Pashley <at> selex-comms.com wrote:
> If so this is the file that is filled with the message: 
>         oprofiled: open
> of
/var/lib/oprofile/samples//current/{root}//mnt/spa-terminal-test-stripped/CPU_CYCLES.10000.0.all.all.all
failed: Invalid argument 

Right, this is certainly the problem.

> If I look inside /var/lib/oprofile/samples the following driectory
> structure exists: 
> 
>  
/var/lib/oprofile/samples/current/{root}/mnt/spa-terminal-test-stripped/{dep}/{root}/mnt/spa-terminal-test-stripped 

Ah, what kind of filesystem is this directory running on? jffs2 by any
chance? Is so, try mounting a tmpfs on /var/lib/oprofile as oprofile has
issues with jffs2 filesystems.

Cheers,

Richard

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
Mark.Pashley | 6 Oct 14:17 2009

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled


Hi Richard,

I have now upgraded the kernel to version 2.6.30.  Unfortunately it has not made any difference and I still get the same problem, namely all of my samples being lost. Have you got any suggestions where to start looking for why these samples are being lost?

Regards

Mark.





Richard Purdie <rpurdie <at> rpsys.net>

11/08/2009 17:32

To
maynardj <at> us.ibm.com
cc
Mark.Pashley <at> selex-comms.com, oprofile-list <at> lists.sourceforge.net
Subject
Re: Problems running oprofiled on embedded arm - Invalid Argument        in oprofiled





Hi,

On Tue, 2009-08-11 at 11:07 -0500, Maynard Johnson wrote:
> Richard, as someone with embedded ARM experience, do you have any suggestions
> for Mark.  Something must not be set up right.

Two things jump out at me:

> >> I am trying to run oprofile on an embedded arm platform, the details are
> >
> >> as follows:
> >> Kernel Version:          2.6.17 (it has been customised)

This is an old kernel...

> >> Repeats for most of the file, followed by....
> >> // ---------------
> >> Thu Aug  6 17:04:43 2009
> >>
> >> Nr. sample dumps: 41
> >> Nr. non-backtrace samples: 1245235
> >> Nr. kernel samples: 1025437
> >> Nr. lost samples (no kernel/user): 0
> >> Nr. lost kernel samples: 0
> >> Nr. incomplete code structs: 6
> >> Nr. samples lost due to sample file open failure: 6687
> >> Nr. samples lost due to no permanent mapping: 0
> >> Nr. event lost due to buffer overflow: 2081823
> >> Nr. samples lost due to no mapping: 4306
> >> Nr. backtraces skipped due to no file mapping: 0
> >> Nr. samples lost due to no mm: 0
> >> Nr. samples lost cpu buffer overflow: 67940
> >> Nr. samples received: 2334860
> >> Nr. backtrace aborted: 0

and there are *large* numbers of samples being lost here, probably all
of them. This would point to something in kernel space, not userspace
being the problem.

OProfile support on arm back in 2.6.17 wasn't well used and probably had
bugs so I'd start there. If you can't upgrade the kernel you need to
work out why the samples are all being lost.

Cheers,

Richard





------------------------------------------------------------
This email and any attached files contains company confidential information which may be legally privileged. It is intended only for the person(s) or entity to which it is addressed and solely for the purposes set forth therein. If you are not the intended recipient or have received this email in error please notify the sender by return, delete it from your system and destroy any local copies. It is strictly forbidden to use the information in this email including any attachment or part thereof including copying, disclosing, distributing, amending or using for any other purpose.

In addition the sender excludes all liabilities (whether tortious or common law) for damage or breach arising or related to this email including but not limited to viruses and libel.
SELEX Communications Limited is a Private Limited Company registered in England and Wales under Company Number 964533 and whose Registered Office is Lambda House, Christopher Martin Rd, Basildon, SS14 3EL. England.
------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list
Mark.Pashley | 6 Oct 18:22 2009

Output from arm


Hi,

Now I have finally got oprofile working I was wondering how to get better information from it. Currently I am using the following steps:

Start the Profiler:
   /mnt/bin # ash opcontrol --no-vmlinux --image=/mnt/spa-terminal-test-stripped --event=CPU_CYCLES:10000:0:1:1
   /mnt/bin # ash opcontrol --start


Run the program
   ../spa-terminal-test-stripped

When the program exists, run opreport:
   opreport --accumulated --callgraph --debug-info --symbols

This gives the following output:
CPU: ARM/XScale PMU2, speed 0 MHz (estimated)
Counted CPU_CYCLES events (clock cycles counter) with a unit mask of 0x00 (No unit mask) count 10000
samples  cum. samples  %        cum. %     linenr info                 symbol name
-------------------------------------------------------------------------------
3482     3482          100.000  100.000    (no location information)   /mnt/spa-terminal-test-stripped
3482     3482          100.000  100.000    (no location information)   /mnt/spa-terminal-test-stripped [self]
-------------------------------------------------------------------------------

How to I get more information from opreport? I  assume that location information is missing because the target executable is stripped, unfortunately there is not enough space on the device to hold debug versions of the executable.

I have the unstripped version of this file on my linux build server.

Regards

Mark
------------------------------------------------------------
This email and any attached files contains company confidential information which may be legally privileged. It is intended only for the person(s) or entity to which it is addressed and solely for the purposes set forth therein. If you are not the intended recipient or have received this email in error please notify the sender by return, delete it from your system and destroy any local copies. It is strictly forbidden to use the information in this email including any attachment or part thereof including copying, disclosing, distributing, amending or using for any other purpose.

In addition the sender excludes all liabilities (whether tortious or common law) for damage or breach arising or related to this email including but not limited to viruses and libel.
SELEX Communications Limited is a Private Limited Company registered in England and Wales under Company Number 964533 and whose Registered Office is Lambda House, Christopher Martin Rd, Basildon, SS14 3EL. England.
------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list
Mark.Pashley | 6 Oct 15:41 2009

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled


Hi,

I believe the daemon is running - ps shows the following process:
        329 root       2312 R   /mnt/bin/oprofiled --session-dir=/var/lib/oprofile --

Does this daemon create the file:
        /var/lib/oprofile/samples/oprofiled.log

If so this is the file that is filled with the message:
        oprofiled: open of /var/lib/oprofile/samples//current/{root}//mnt/spa-terminal-test-stripped/CPU_CYCLES.10000.0.all.all.all failed: Invalid argument

If I look inside /var/lib/oprofile/samples the following driectory structure exists:
        /var/lib/oprofile/samples/current/{root}/mnt/spa-terminal-test-stripped/{dep}/{root}/mnt/spa-terminal-test-stripped

This directory contains the file CPU_CYCLES.10000.0.all.all.all, which is filld with NULLs

Regards

Mark.




Richard Purdie <rpurdie <at> rpsys.net>

06/10/2009 13:56

To
Mark.Pashley <at> selex-comms.com
cc
oprofile-list <at> lists.sourceforge.net
Subject
Re: Problems running oprofiled on embedded arm - Invalid        Argument        in oprofiled





Hi,

On Tue, 2009-10-06 at 13:17 +0100, Mark.Pashley <at> selex-comms.com wrote:
> I have now upgraded the kernel to version 2.6.30.  Unfortunately it
> has not made any difference and I still get the same problem, namely
> all of my samples being lost. Have you got any suggestions where to
> start looking for why these samples are being lost?

Ok, the next step is to figure out whether anything is getting to
userspace. When oprofile is running there should be a daemon taking
kernel buffers and emptying them into userspace. Is that daemon running
and does it notice data is there for collection?

My memory is fuzzy but there should be some way to make the output from
oprofiled more verbose so you can see what its doing...

Cheers,

Richard



------------------------------------------------------------
This email and any attached files contains company confidential information which may be legally privileged. It is intended only for the person(s) or entity to which it is addressed and solely for the purposes set forth therein. If you are not the intended recipient or have received this email in error please notify the sender by return, delete it from your system and destroy any local copies. It is strictly forbidden to use the information in this email including any attachment or part thereof including copying, disclosing, distributing, amending or using for any other purpose.

In addition the sender excludes all liabilities (whether tortious or common law) for damage or breach arising or related to this email including but not limited to viruses and libel.
SELEX Communications Limited is a Private Limited Company registered in England and Wales under Company Number 964533 and whose Registered Office is Lambda House, Christopher Martin Rd, Basildon, SS14 3EL. England.
------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list
William Cohen | 6 Oct 23:09 2009
Picon

[Fwd: [PATCH] "opcontrol --start-daemon" doesn't work correctly on ia64]

IA64 machines are not popular. However, I would like to get this patch into
upstream oprofile. Does anyone have comments on the proposed patch? It has been
tested out on IA64 and addresses reported problem.

-Will

-------- Original Message --------
Subject: [PATCH] "opcontrol --start-daemon" doesn't work correctly on ia64
Date: Tue, 01 Sep 2009 12:06:42 -0400
From: William Cohen <wcohen <at> redhat.com>
To: oprofile-list <oprofile-list <at> lists.sourceforge.net>

The ia64 handles things a bit differently than other architectures. It starts up
a number of children processes to control perfmon on each of the processors in
the machine. However, these children processes do not close the io. The results
in children processes still being connected to the console.  There are more
details about this problem at:

https://bugzilla.redhat.com/show_bug.cgi?id=518480

I have made a pass at a patch to fix this particular problem and I am looking
for feedback on the patch:

2009-09-01  William Cohen <wcohen <at> redhat.com>

	* daemon/opd_perfmon.c: Fix ia64 output to allow spawning daemon.

Signed-off-by: William Cohen <wcohen <at> redhat.com>

-Will

diff -up oprofile-0.9.4/daemon/init.c.orig oprofile-0.9.4/daemon/init.c
diff -up oprofile-0.9.4/daemon/opd_perfmon.c.orig oprofile-0.9.4/daemon/opd_perfmon.c
--- oprofile-0.9.4/daemon/opd_perfmon.c.orig	2009-09-01 10:45:52.000000000 -0400
+++ oprofile-0.9.4/daemon/opd_perfmon.c	2009-09-01 11:42:19.000000000 -0400
 <at>  <at>  -141,7 +141,6  <at>  <at>  static void child_sigusr2(int val __attr

 static void child_sigterm(int val __attribute__((unused)))
 {
-	printf("Child received SIGTERM, killing parent.\n");
 	kill(getppid(), SIGTERM);
 }

 <at>  <at>  -156,8 +155,7  <at>  <at>  static void set_affinity(size_t cpu)
 	int err = sched_setaffinity(getpid(), sizeof(set), &set);

 	if (err == -1) {
-		fprintf(stderr, "Failed to set affinity: %s\n",
-			    strerror(errno));
+		perror("Failed to set affinity: ");
 		exit(EXIT_FAILURE);
 	}
 }
 <at>  <at>  -214,8 +212,7  <at>  <at>  static void create_context(struct child 

 	err = perfmonctl(0, PFM_CREATE_CONTEXT, &ctx, 1);
 	if (err == -1) {
-		fprintf(stderr, "CREATE_CONTEXT failed: %s\n",
-		        strerror(errno));
+		perror("CREATE_CONTEXT failed: ");
 		exit(EXIT_FAILURE);
 	}

 <at>  <at>  -304,8 +301,7  <at>  <at>  static void notify_parent(struct child *
 		if (ret == sizeof(size_t))
 			break;
 		if (ret < 0 && errno != EINTR) {
-			fprintf(stderr, "Failed to write child pipe with %s\n",
-			        strerror(errno));
+			perror("Failed to write child pipe with ");
 			exit(EXIT_FAILURE);
 		}
 	}
 <at>  <at>  -333,6 +329,11  <at>  <at>  static void run_child(size_t cpu)

 	notify_parent(self, cpu);

+	/* Close IO to allow this child process to be run as a daemon */
+	close(0);
+	close(1);
+	close(2);
+
 	for (;;) {
 		sigset_t sigmask;
 		sigfillset(&sigmask);
 <at>  <at>  -341,15 +342,11  <at>  <at>  static void run_child(size_t cpu)
 		sigdelset(&sigmask, SIGTERM);

 		if (self->sigusr1) {
-			printf("PFM_START on CPU%d\n", (int)cpu);
-			fflush(stdout);
 			perfmon_start_child(self->ctx_fd);
 			self->sigusr1 = 0;
 		}

 		if (self->sigusr2) {
-			printf("PFM_STOP on CPU%d\n", (int)cpu);
-			fflush(stdout);
 			perfmon_stop_child(self->ctx_fd);
 			self->sigusr2 = 0;
 		}
 <at>  <at>  -368,13 +365,10  <at>  <at>  static void wait_for_child(struct child 
 		if (ret == sizeof(size_t))
 			break;
 		if (ret < 0 && errno != EINTR) {
-			fprintf(stderr, "Failed to read child pipe with %s\n",
-			        strerror(errno));
+			perror("Failed to read child pipe with ");
 			exit(EXIT_FAILURE);
 		}
 	}
-	printf("Perfmon child up on CPU%d\n", (int)tmp);
-	fflush(stdout);

 	close(child->up_pipe[0]);
 	close(child->up_pipe[1]);
 <at>  <at>  -422,21 +416,18  <at>  <at>  void perfmon_init(void)
 		int ret;

 		if (pipe(children[i].up_pipe)) {
-			perror("Couldn't create child pipe.\n");
+			perror("Couldn't create child pipe: ");
 			exit(EXIT_FAILURE);
 		}

 		ret = fork();
 		if (ret == -1) {
-			fprintf(stderr, "Couldn't fork perfmon child.\n");
+			perror("Couldn't fork perfmon child: ");
 			exit(EXIT_FAILURE);
 		} else if (ret == 0) {
-			printf("Running perfmon child on CPU%d.\n", (int)i);
-			fflush(stdout);
 			run_child(i);
 		} else {
 			children[i].pid = ret;
-			printf("Waiting on CPU%d\n", (int)i);
 			wait_for_child(&children[i]);
 		}
 	}

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list
chandraiah gobburu | 7 Oct 12:12 2009
Picon

Problem with Opcontrol

Hello,
 
I am using oprofile of version 0.9.5. I have compiled oprofile for ARM and then i have executed the following command from the terminal on the targetboard.
 
opcontrol --no-vmlinux
 
and the following is the error I am getting.
 
opcontrol: line 1114: syntax error: Bad substitution
 
Could anybody help me to resolve this problem?

--
Thanks and regards,
Chandraiah
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Gmane