Re: Nr. lost kernel samples

Hi Robert,

I'm still investigating those lost kernel samples.
Today i installed a new Debian Lenny 5.0 stable System.
I tested 2 kernels, 2.6.18.8 under Xen 3.3.1. Stock kernel from the  
Xen repositories.
And 2.6.26 that comes by default with lenny.

There are two strange behaviors i observed.
2.6.26 doesn't lose a kernel sample. Whereas 2.6.18 under Xen does.
The more interesting fact is that, when i increase the callgraph depth  
from 0 to 16, there is a huge shift in the received samples as well as  
in the lost kernel samples under 2.6.18.
I did some math with my results, but I still can't figure out what's  
goin on.

The buffer overflows seem to be a problem that i can get rid of by  
adapting the sample rate / buffer sizes.
But is the huge gap in the sample counts an expected behavior when i  
increase the callgraph depth?

4 files are attached, filenames explain their setup.
While profiling, the lenny box acted as an iperf server for 1 GBit UDP  
incoming traffic.

Re: Problems with libc6 symbols

Eran Rom wrote:
> Hi Maynard,
> Any quick & dirty fix for the meanwhile 
Sorry, I've been tied up with some critical customer engagement work, and haven't been able to look at this. 
And I likely won't have much time in the coming week either.  But my first pass on solving this problem would
be to look at using the same type of fix-up that's now done for ppc64/64-bit bfd's for all architectures.  If
you want to look at building such a patch, that would be great.  But for a quick and dirty fix for you, you can
just comment out the last check in libutil++/bfd_support.cpp:boring_symbol -- i.e., the SEC_LOAD
check.  I *think* that will be a workaround for you.  That can't be a final fix, because that check is needed to
prevent the "samples_range(): start > end " error that can occur for certain classes of symbols.

-Maynard
> 
> Thanks very much,
> Eran
> 
> Maynard Johnson <maynardj <at> us.ibm.com> wrote on 29/09/2009 23:00:29:
> 
>> Eran Rom wrote:
>>> Maynard Johnson <maynardj <at> us.ibm.com> wrote on 29/09/2009 21:33:41:
>>>> hmmm . . . I get correct results (i.e., symbol resolution into libc)
>>>> when using
>>>> 64-bit libc debuginfo, but I get same results as you when using
>>> 32-bit.  
>>>> Need to investigate more.
>>> Maynard, thanks very much for the reply.
>>> I did forget to mention that I am using OProfile 0.9.5 over 64 bit
> version
>>> of Ubuntu so the libc debug info is 64bit
>> I've discovered the reason 64-bit works OK for me is because I

Recompiling kernel with OProfile for ARM

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled

Hi,

On Tue, 2009-10-06 at 13:17 +0100, Mark.Pashley <at> selex-comms.com wrote:
> I have now upgraded the kernel to version 2.6.30.  Unfortunately it
> has not made any difference and I still get the same problem, namely
> all of my samples being lost. Have you got any suggestions where to
> start looking for why these samples are being lost? 

Ok, the next step is to figure out whether anything is getting to
userspace. When oprofile is running there should be a daemon taking
kernel buffers and emptying them into userspace. Is that daemon running
and does it notice data is there for collection?

My memory is fuzzy but there should be some way to make the output from
oprofiled more verbose so you can see what its doing...

Cheers,

Richard

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled

On Tue, 2009-10-06 at 14:41 +0100, Mark.Pashley <at> selex-comms.com wrote:
> If so this is the file that is filled with the message: 
>         oprofiled: open
> of
/var/lib/oprofile/samples//current/{root}//mnt/spa-terminal-test-stripped/CPU_CYCLES.10000.0.all.all.all
failed: Invalid argument 

Right, this is certainly the problem.

> If I look inside /var/lib/oprofile/samples the following driectory
> structure exists: 
> 
>  
/var/lib/oprofile/samples/current/{root}/mnt/spa-terminal-test-stripped/{dep}/{root}/mnt/spa-terminal-test-stripped 

Ah, what kind of filesystem is this directory running on? jffs2 by any
chance? Is so, try mounting a tmpfs on /var/lib/oprofile as oprofile has
issues with jffs2 filesystems.

Cheers,

Richard

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Output from arm

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Re: Problems running oprofiled on embedded arm - Invalid Argument in oprofiled

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

[Fwd: [PATCH] "opcontrol --start-daemon" doesn't work correctly on ia64]

IA64 machines are not popular. However, I would like to get this patch into
upstream oprofile. Does anyone have comments on the proposed patch? It has been
tested out on IA64 and addresses reported problem.

-Will

-------- Original Message --------
Subject: [PATCH] "opcontrol --start-daemon" doesn't work correctly on ia64
Date: Tue, 01 Sep 2009 12:06:42 -0400
From: William Cohen <wcohen <at> redhat.com>
To: oprofile-list <oprofile-list <at> lists.sourceforge.net>

The ia64 handles things a bit differently than other architectures. It starts up
a number of children processes to control perfmon on each of the processors in
the machine. However, these children processes do not close the io. The results
in children processes still being connected to the console.  There are more
details about this problem at:

https://bugzilla.redhat.com/show_bug.cgi?id=518480

I have made a pass at a patch to fix this particular problem and I am looking
for feedback on the patch:

2009-09-01  William Cohen <wcohen <at> redhat.com>

	* daemon/opd_perfmon.c: Fix ia64 output to allow spawning daemon.

Signed-off-by: William Cohen <wcohen <at> redhat.com>

-Will

diff -up oprofile-0.9.4/daemon/init.c.orig oprofile-0.9.4/daemon/init.c
diff -up oprofile-0.9.4/daemon/opd_perfmon.c.orig oprofile-0.9.4/daemon/opd_perfmon.c
--- oprofile-0.9.4/daemon/opd_perfmon.c.orig	2009-09-01 10:45:52.000000000 -0400
+++ oprofile-0.9.4/daemon/opd_perfmon.c	2009-09-01 11:42:19.000000000 -0400
 <at>  <at>  -141,7 +141,6  <at>  <at>  static void child_sigusr2(int val __attr

 static void child_sigterm(int val __attribute__((unused)))
 {
-	printf("Child received SIGTERM, killing parent.\n");
 	kill(getppid(), SIGTERM);
 }

 <at>  <at>  -156,8 +155,7  <at>  <at>  static void set_affinity(size_t cpu)
 	int err = sched_setaffinity(getpid(), sizeof(set), &set);

 	if (err == -1) {
-		fprintf(stderr, "Failed to set affinity: %s\n",
-			    strerror(errno));
+		perror("Failed to set affinity: ");
 		exit(EXIT_FAILURE);
 	}
 }
 <at>  <at>  -214,8 +212,7  <at>  <at>  static void create_context(struct child 

 	err = perfmonctl(0, PFM_CREATE_CONTEXT, &ctx, 1);
 	if (err == -1) {
-		fprintf(stderr, "CREATE_CONTEXT failed: %s\n",
-		        strerror(errno));
+		perror("CREATE_CONTEXT failed: ");
 		exit(EXIT_FAILURE);
 	}

 <at>  <at>  -304,8 +301,7  <at>  <at>  static void notify_parent(struct child *
 		if (ret == sizeof(size_t))
 			break;
 		if (ret < 0 && errno != EINTR) {
-			fprintf(stderr, "Failed to write child pipe with %s\n",
-			        strerror(errno));
+			perror("Failed to write child pipe with ");
 			exit(EXIT_FAILURE);
 		}
 	}
 <at>  <at>  -333,6 +329,11  <at>  <at>  static void run_child(size_t cpu)

 	notify_parent(self, cpu);

+	/* Close IO to allow this child process to be run as a daemon */
+	close(0);
+	close(1);
+	close(2);
+
 	for (;;) {
 		sigset_t sigmask;
 		sigfillset(&sigmask);
 <at>  <at>  -341,15 +342,11  <at>  <at>  static void run_child(size_t cpu)
 		sigdelset(&sigmask, SIGTERM);

 		if (self->sigusr1) {
-			printf("PFM_START on CPU%d\n", (int)cpu);
-			fflush(stdout);
 			perfmon_start_child(self->ctx_fd);
 			self->sigusr1 = 0;
 		}

 		if (self->sigusr2) {
-			printf("PFM_STOP on CPU%d\n", (int)cpu);
-			fflush(stdout);
 			perfmon_stop_child(self->ctx_fd);
 			self->sigusr2 = 0;
 		}
 <at>  <at>  -368,13 +365,10  <at>  <at>  static void wait_for_child(struct child 
 		if (ret == sizeof(size_t))
 			break;
 		if (ret < 0 && errno != EINTR) {
-			fprintf(stderr, "Failed to read child pipe with %s\n",
-			        strerror(errno));
+			perror("Failed to read child pipe with ");
 			exit(EXIT_FAILURE);
 		}
 	}
-	printf("Perfmon child up on CPU%d\n", (int)tmp);
-	fflush(stdout);

 	close(child->up_pipe[0]);
 	close(child->up_pipe[1]);
 <at>  <at>  -422,21 +416,18  <at>  <at>  void perfmon_init(void)
 		int ret;

 		if (pipe(children[i].up_pipe)) {
-			perror("Couldn't create child pipe.\n");
+			perror("Couldn't create child pipe: ");
 			exit(EXIT_FAILURE);
 		}

 		ret = fork();
 		if (ret == -1) {
-			fprintf(stderr, "Couldn't fork perfmon child.\n");
+			perror("Couldn't fork perfmon child: ");
 			exit(EXIT_FAILURE);
 		} else if (ret == 0) {
-			printf("Running perfmon child on CPU%d.\n", (int)i);
-			fflush(stdout);
 			run_child(i);
 		} else {
 			children[i].pid = ret;
-			printf("Waiting on CPU%d\n", (int)i);
 			wait_for_child(&children[i]);
 		}
 	}

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Problem with Opcontrol

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

_______________________________________________
oprofile-list mailing list
oprofile-list <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oprofile-list