Derek Glidden | 3 Feb 2003 23:27

no ORDER BY in selects?


Is there a reason that the "get*Nodes" methods in, e.g.
NetworkElementFactory don't do SELECTs with ORDER BY using some
human-identifiable field, like nodeLabel? 

A lot of the screens come up with nodes listed in effectively random
order (order of database select which is more or less undefined but
correlates vaguely to "the order in which nodes were discovered or
changes made to that node's fields") and makes it really hard to find a
specific node if you're browsing through, for example, the
"manage/unmanage" screen on the admin section.

Adding ORDER BY NODELABEL would make many screens a lot more coherent. 
I can't think of a single reason NOT to add this, except if you WANT to
see the nodes in roughly-discovery-or-last-modification order.  You'd
have to hanve thousands and thousands of nodes in the database for this
to have any significant impact on performance, and even then, it would
be minor compared to the number of connection setups/teardowns the
current database model is using to retrieve a lot of data from the
database.

--

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#!/usr/bin/perl -w
$_='while(read+STDIN,$_,2048){$a=29;$b=73;$c=142;$t=255; <at> t=map
{$_%16or$t^=$c^=($m=(11,10,116,100,11,122,20,100)[$_/16%8])&110;
$t^=(72, <at> z=(64,72,$a^=12*($_%16-2?0:$m&17)),$b^=$_%64?12:0, <at> z)
[$_%8]}(16..271);if(( <at> a=unx"C*",$_)[20]&48){$h=5;$_=unxb24,join
"", <at> b=map{xB8,unxb8,chr($_^$a[--$h+84])} <at> ARGV;s/...$/1$&/;$d=
unxV,xb25,$_;$e=256|(ord$b[4])<<9|ord$b[3];$d=$d>>8^($f=$t&($d
(Continue reading)

Ian Wallace | 4 Feb 2003 01:49
Favicon

Re: no ORDER BY in selects?

Derek - I think this is already in Bugzilla (bug id 443).  But if you
want to pull the CVS tree and submit some patches I'll be more than
happy to apply them.  This does sound like a good idea.

cheers
ian

On Mon, 2003-02-03 at 15:27, Derek Glidden wrote:
> 
> Is there a reason that the "get*Nodes" methods in, e.g.
> NetworkElementFactory don't do SELECTs with ORDER BY using some
> human-identifiable field, like nodeLabel? 
> 
> A lot of the screens come up with nodes listed in effectively random
> order (order of database select which is more or less undefined but
> correlates vaguely to "the order in which nodes were discovered or
> changes made to that node's fields") and makes it really hard to find a
> specific node if you're browsing through, for example, the
> "manage/unmanage" screen on the admin section.
> 
> Adding ORDER BY NODELABEL would make many screens a lot more coherent. 
> I can't think of a single reason NOT to add this, except if you WANT to
> see the nodes in roughly-discovery-or-last-modification order.  You'd
> have to hanve thousands and thousands of nodes in the database for this
> to have any significant impact on performance, and even then, it would
> be minor compared to the number of connection setups/teardowns the
> current database model is using to retrieve a lot of data from the
> database.
> 
> -- 
(Continue reading)

Derek Glidden | 4 Feb 2003 17:47

Re: no ORDER BY in selects?

On Mon, 2003-02-03 at 19:49, Ian Wallace wrote:
> Derek - I think this is already in Bugzilla (bug id 443).  But if you
> want to pull the CVS tree and submit some patches I'll be more than
> happy to apply them.  This does sound like a good idea.

be happy to.  I'll work on it today.

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#!/usr/bin/perl -w
$_='while(read+STDIN,$_,2048){$a=29;$b=73;$c=142;$t=255; <at> t=map
{$_%16or$t^=$c^=($m=(11,10,116,100,11,122,20,100)[$_/16%8])&110;
$t^=(72, <at> z=(64,72,$a^=12*($_%16-2?0:$m&17)),$b^=$_%64?12:0, <at> z)
[$_%8]}(16..271);if(( <at> a=unx"C*",$_)[20]&48){$h=5;$_=unxb24,join
"", <at> b=map{xB8,unxb8,chr($_^$a[--$h+84])} <at> ARGV;s/...$/1$&/;$d=
unxV,xb25,$_;$e=256|(ord$b[4])<<9|ord$b[3];$d=$d>>8^($f=$t&($d
>>12^$d>>4^$d^$d/8))<<17,$e=$e>>8^($t&($g=($q=$e>>14&7^$e)^$q*
8^$q<<6))<<9,$_=$t[$_]^(($h>>=8)+=$f+(~$g&$t))for <at> a[128..$#a]}
print+x"C*", <at> a}';s/x/pack+/g;eval 

usage: qrpff 153 2 8 105 225 < /mnt/dvd/VOB_FILENAME \
    | extract_mpeg2 | mpeg2dec - 

         http://www.cs.cmu.edu/~dst/DeCSS/Gallery/
http://www.eff.org/                   http://www.anti-dmca.org/

_______________________________________________
bugs mailing list (bugs <at> lists.opennms.org)
To subscribe, unsubscribe, or change your list options, go to:
http://lists.opennms.org/mailman/listinfo/bugs
(Continue reading)

Derek Glidden | 4 Feb 2003 19:32

Re: no ORDER BY in selects?

On Tue, 2003-02-04 at 11:47, Derek Glidden wrote:
> On Mon, 2003-02-03 at 19:49, Ian Wallace wrote:
> > Derek - I think this is already in Bugzilla (bug id 443).  But if you
> > want to pull the CVS tree and submit some patches I'll be more than
> > happy to apply them.  This does sound like a good idea.
> 
> be happy to.  I'll work on it today.
> 

Umm.. try this patch against the head branch.  (I don't generally make
patches from CVS, so I apologise if this is messed up.)

I tried to only affect selects that pulled the data out for some
human-viewable output and ORDER BY in a way that made sense.  e.g. ORDER
BY inet(IPADDR) for lists of ip addresses, ORDER BY NODELABEL for lists
of nodes, etc.

--

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#!/usr/bin/perl -w
$_='while(read+STDIN,$_,2048){$a=29;$b=73;$c=142;$t=255; <at> t=map
{$_%16or$t^=$c^=($m=(11,10,116,100,11,122,20,100)[$_/16%8])&110;
$t^=(72, <at> z=(64,72,$a^=12*($_%16-2?0:$m&17)),$b^=$_%64?12:0, <at> z)
[$_%8]}(16..271);if(( <at> a=unx"C*",$_)[20]&48){$h=5;$_=unxb24,join
"", <at> b=map{xB8,unxb8,chr($_^$a[--$h+84])} <at> ARGV;s/...$/1$&/;$d=
unxV,xb25,$_;$e=256|(ord$b[4])<<9|ord$b[3];$d=$d>>8^($f=$t&($d
>>12^$d>>4^$d^$d/8))<<17,$e=$e>>8^($t&($g=($q=$e>>14&7^$e)^$q*
8^$q<<6))<<9,$_=$t[$_]^(($h>>=8)+=$f+(~$g&$t))for <at> a[128..$#a]}
print+x"C*", <at> a}';s/x/pack+/g;eval 

(Continue reading)

Ian Wallace | 4 Feb 2003 23:49
Favicon

Re: no ORDER BY in selects?

Derek - Thanks for the patch.  I applied it, but need to compile/build
RPM's before I check it in so that I don't go destroying the build.

Let you know when that succeeds.

cheers
ian

ps : Here's the output of the patch command:

[iwallace <at> iwallace opennms]$ patch -p 0 --verbose -t -b -i
patch.order-by
Hmm...  Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|diff -c -r1.2 GetNodesServlet.java
|***
src/web/src/org/opennms/web/admin/nodeManagement/GetNodesServlet.java      14 Nov 2002 22:19:44
-0000      1.2
|---
src/web/src/org/opennms/web/admin/nodeManagement/GetNodesServlet.java      4 Feb 2003 18:29:53 -0000
--------------------------
Patching file
src/web/src/org/opennms/web/admin/nodeManagement/GetNodesServlet.java
using Plan A...
Hunk #1 succeeded at 46.
Hmm...  The next patch looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|diff -c -r1.2 AssetModel.java
(Continue reading)

Ian Wallace | 5 Feb 2003 20:51
Favicon

Re: no ORDER BY in selects?

Derek - From my limited testing (on Mandrake 9 only) things appear to
work quite well.  I'll check in the changes and lets see what others
think.

Are you able to update bugzilla or do I need to do that?

cheers
ian

On Tue, 2003-02-04 at 15:49, Ian Wallace wrote:
> Derek - Thanks for the patch.  I applied it, but need to compile/build
> RPM's before I check it in so that I don't go destroying the build.
> 
> Let you know when that succeeds.
> 
> cheers
> ian
> 
> ps : Here's the output of the patch command:
> 
> [iwallace <at> iwallace opennms]$ patch -p 0 --verbose -t -b -i
> patch.order-by
> Hmm...  Looks like a new-style context diff to me...
> The text leading up to this was:
> --------------------------
> |diff -c -r1.2 GetNodesServlet.java
> |***
> src/web/src/org/opennms/web/admin/nodeManagement/GetNodesServlet.java      14 Nov 2002 22:19:44
-0000      1.2
> |---
(Continue reading)

Derek Glidden | 5 Feb 2003 20:58

Re: no ORDER BY in selects?

On Wed, 2003-02-05 at 14:51, Ian Wallace wrote:
> Derek - From my limited testing (on Mandrake 9 only) things appear to
> work quite well.  I'll check in the changes and lets see what others
> think.

cool. 

> Are you able to update bugzilla or do I need to do that?

AFAIK I can't do anything to the bugzilla.

--

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#!/usr/bin/perl -w
$_='while(read+STDIN,$_,2048){$a=29;$b=73;$c=142;$t=255; <at> t=map
{$_%16or$t^=$c^=($m=(11,10,116,100,11,122,20,100)[$_/16%8])&110;
$t^=(72, <at> z=(64,72,$a^=12*($_%16-2?0:$m&17)),$b^=$_%64?12:0, <at> z)
[$_%8]}(16..271);if(( <at> a=unx"C*",$_)[20]&48){$h=5;$_=unxb24,join
"", <at> b=map{xB8,unxb8,chr($_^$a[--$h+84])} <at> ARGV;s/...$/1$&/;$d=
unxV,xb25,$_;$e=256|(ord$b[4])<<9|ord$b[3];$d=$d>>8^($f=$t&($d
>>12^$d>>4^$d^$d/8))<<17,$e=$e>>8^($t&($g=($q=$e>>14&7^$e)^$q*
8^$q<<6))<<9,$_=$t[$_]^(($h>>=8)+=$f+(~$g&$t))for <at> a[128..$#a]}
print+x"C*", <at> a}';s/x/pack+/g;eval 

usage: qrpff 153 2 8 105 225 < /mnt/dvd/VOB_FILENAME \
    | extract_mpeg2 | mpeg2dec - 

         http://www.cs.cmu.edu/~dst/DeCSS/Gallery/
http://www.eff.org/                   http://www.anti-dmca.org/

(Continue reading)

Che Cordes | 11 Feb 2003 18:09

Tomcat and/or Java Errors on opennms startup

I have a fresh rpm install of OpenNMS on RedHat 7.2 (I ran up2date after the OpenNMS install and updated the kernel and other necessary packages).  My current kernel version is 2.4.18-24.7.x.  I am running it on a PII 350 MHz with 130MB of ram (swap set to 256MB). 

OpenNMS appears to startup and function correctly for a while.  After some random period of time, however, some of the java processes hang and OpenNMS stops polling and collecting snmp data.  The web interface continues to work but the performance and response time graphs stop showing data.  Also, services are no longer polled OR the web interface is no longer updated when outages occur!  I usually notice this first thing in the morning and the command `service opennms status` just hangs.  The only way to get the `service opennms stop` command to work is to pkill -9 java first.

Any ideas?  Do I need more memory? Here is some info from my messages file during the startup of OpenNMS.  I stopped OpenNMS and tomcat and deleted the contents of /var/tomcat4/work.  The first section was produced after issuing the command `service tomcat4 start`.  The content after the separator resulted from `service opennms start`.

--------------------------------------------------------------------------------------
Feb 11 10:35:11 cslrh1 tomcat4: j2sdk1.4.1_01 startup succeeded
Feb 11 10:36:51 cslrh1 su(pam_unix)[30316]: session opened for user root by ccordes(uid=0)
Feb 11 10:36:51 cslrh1 su: Using CATALINA_BASE:   /var/tomcat4
Feb 11 10:36:51 cslrh1 su: Using CATALINA_HOME:   /var/tomcat4
Feb 11 10:36:51 cslrh1 su: Using CATALINA_TMPDIR: /var/log/tomcat4
Feb 11 10:36:51 cslrh1 su: Using JAVA_HOME:       /usr/java/j2sdk1.4.1_01
Feb 11 10:36:51 cslrh1 su(pam_unix)[30316]: session closed for user root
Feb 11 10:36:51 cslrh1 tomcat4: j2sdk1.4.1_01 startup succeeded
Feb 11 10:36:52 cslrh1 tomcat4: j2sdk1.4.1_01 startup succeeded
-----------------------------------------------------------------------------------------
Feb 11 10:37:20 cslrh1 opennms.sh: Stopping tomcat4:
Feb 11 10:37:20 cslrh1 su(pam_unix)[30603]: session opened for user root by ccordes(uid=0)
Feb 11 10:37:21 cslrh1 su: Using CATALINA_BASE:   /var/tomcat4
Feb 11 10:37:21 cslrh1 su: Using CATALINA_HOME:   /var/tomcat4
Feb 11 10:37:21 cslrh1 su: Using CATALINA_TMPDIR: /var/log/tomcat4
Feb 11 10:37:21 cslrh1 su: Using JAVA_HOME:       /usr/java/j2sdk1.4.1_01
Feb 11 10:37:37 cslrh1 su: Catalina.stop: java.net.ConnectException: Connection refused
Feb 11 10:37:37 cslrh1 su:
Feb 11 10:37:37 cslrh1 su: java.net.ConnectException: Connection refused
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.PlainSocketImpl.socketConnect(Native Method)
Feb 11 10:37:37 cslrh1 su:
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.Socket.connect(Socket.java:426)
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.Socket.connect(Socket.java:376)
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.Socket.<init>(Socket.java:291)
Feb 11 10:37:37 cslrh1 su: ^Iat java.net.Socket.<init>(Socket.java:119)
Feb 11 10:37:37 cslrh1 su: ^Iat org.apache.catalina.startup.Catalina.stop(Catalina.java:831)
Feb 11 10:37:37 cslrh1 su(pam_unix)[30603]: session closed for user root
Feb 11 10:37:37 cslrh1 su: ^Iat org.apache.catalina.startup.Catalina.execute(Catalina.java:683)
Feb 11 10:37:37 cslrh1 su: ^Iat org.apache.catalina.startup.Catalina.process(Catalina.java:179)
Feb 11 10:37:38 cslrh1 su: ^Iat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Feb 11 10:37:38 cslrh1 su: ^Iat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
Feb 11 10:37:38 cslrh1 su: ^Iat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Feb 11 10:37:38 cslrh1 su: ^Iat java.lang.reflect.Method.invoke(Method.java:324)
Feb 11 10:37:38 cslrh1 su: ^Iat org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:243)
Feb 11 10:37:38 cslrh1 opennms.sh: Using CATALINA_BASE:   /var/tomcat4
Feb 11 10:37:38 cslrh1 tomcat4: j2sdk1.4.1_01 startup failed
Feb 11 10:37:38 cslrh1 opennms.sh: Using CATALINA_HOME:   /var/tomcat4
Feb 11 10:37:38 cslrh1 tomcat4: j2sdk1.4.1_01 startup failed
Feb 11 10:37:38 cslrh1 opennms.sh: Using CATALINA_TMPDIR: /var/log/tomcat4
Feb 11 10:37:38 cslrh1 opennms.sh: Using JAVA_HOME:       /usr/java/j2sdk1.4.1_01
Feb 11 10:37:39 cslrh1 opennms.sh: Catalina.stop: java.net.ConnectException: Connection refused
Feb 11 10:37:39 cslrh1 opennms.sh: java.net.ConnectException: Connection refused
Feb 11 10:37:39 cslrh1 opennms.sh: ^Iat java.net.PlainSocketImpl.socketConnect(Native Method)
Feb 11 10:37:39 cslrh1 opennms.sh: ^Iat java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
Feb 11 10:37:39 cslrh1 opennms.sh: ^Iat java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
Feb 11 10:37:39 cslrh1 opennms.sh: ^Iat java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
Feb 11 10:37:39 cslrh1 opennms.sh: ^Iat java.net.Socket.connect(Socket.java:426)
Feb 11 10:37:40 cslrh1 opennms.sh: ^Iat java.net.Socket.connect(Socket.java:376)
Feb 11 10:37:40 cslrh1 opennms.sh: ^Iat java.net.Socket.<init>(Socket.java:291)
Feb 11 10:37:40 cslrh1 opennms.sh: ^Iat java.net.Socket.<init>(Socket.java:119)
Feb 11 10:37:40 cslrh1 opennms.sh: ^Iat org.apache.catalina.startup.Catalina.stop(Catalina.java:831)
Feb 11 10:37:40 cslrh1 opennms.sh: ^Iat org.apache.catalina.startup.Catalina.execute(Catalina.java:683)
Feb 11 10:37:40 cslrh1 opennms.sh: ^Iat org.apache.catalina.startup.Catalina.process(Catalina.java:179)
Feb 11 10:37:41 cslrh1 opennms.sh: ^Iat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Feb 11 10:37:41 cslrh1 opennms.sh: ^Iat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
Feb 11 10:37:41 cslrh1 opennms.sh: ^Iat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Feb 11 10:37:41 cslrh1 opennms.sh: ^Iat java.lang.reflect.Method.invoke(Method.java:324)
Feb 11 10:37:41 cslrh1 opennms.sh: ^Iat org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:243)
Feb 11 10:37:41 cslrh1 opennms.sh:
Feb 11 10:38:01 cslrh1 opennms.sh:
Feb 11 10:38:01 cslrh1 opennms.sh: Tomcat is not shutting down properly.  Please check
Feb 11 10:38:01 cslrh1 opennms.sh: your logs and try again.
Feb 11 10:38:01 cslrh1 opennms.sh:
Feb 11 10:38:01 cslrh1 opennms: opennms.sh startup succeeded
----------------------------------------------------------------------------------

Thanks in advance.

Tarus Balog | 11 Feb 2003 19:02

Re: Tomcat and/or Java Errors on opennms startup


Che Cordes said:
> I am running it on a PII 350 MHz with 130MB of ram (swap set to 256MB).

> OpenNMS appears to startup and function correctly for a while.

> Any ideas?  Do I need more memory?

Yup. The minimum amount you should run OpenNMS with is 256MB, and I would
recommend 512MB RAM and a larger processor.

Check out your log files in /opt/OpenNMS/logs. Look for "fatal" and
"error". My guess is that if you look in the collectd.log file, you may
find some OutOfMemory errors, or "too many open files".

-T

--

-- 
Tarus Balog
Consultant
Sortova Consulting Group, http://www.sortova.com
+1-919-696-7625
tarus <at> sortova.com

_______________________________________________
bugs mailing list (bugs <at> lists.opennms.org)
To subscribe, unsubscribe, or change your list options, go to:
http://lists.opennms.org/mailman/listinfo/bugs

Che Cordes | 11 Feb 2003 19:33

RE: Tomcat and/or Java Errors on opennms startup

Thanks, I will add some memory to continue testing.  This is not a production machine, I am still evaluating. 

I searched the OpenNMS logfile directory for out of memory and too many open files errors and did not find any. 

I did not increase the number of tasks allowed per user (nr_tasks) from 512 because I couldn't figure out how.  I will persist on that if it may help.

When the java processes hang, a hs_err_pid[#].log file gets created in /opt/OpenNMS/etc/.  The log begins:

Unexpected Signal : 11 occurred at PC=0x40606324
Function=size_given_klass__7oopDescP5Klass+0x44
Library=/usr/java/j2sdk1.4.1_01/jre/lib/i386/server/libjvm.so

And Ends:

Local Time = Mon Feb 10 17:28:24 2003
Elapsed Time = 3913
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002E6
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Server VM (1.4.1_01-b01 mixed mode)
#

Che'
__________________________________________________________________

Che Cordes Originally said:
> I am running it on a PII 350 MHz with 130MB of ram (swap set to 256MB).

> OpenNMS appears to startup and function correctly for a while.

> Any ideas?  Do I need more memory?

Yup. The minimum amount you should run OpenNMS with is 256MB, and I would
recommend 512MB RAM and a larger processor.

Check out your log files in /opt/OpenNMS/logs. Look for "fatal" and
"error". My guess is that if you look in the collectd.log file, you may
find some OutOfMemory errors, or "too many open files".

-T

--
Tarus Balog
Consultant
Sortova Consulting Group, http://www.sortova.com
+1-919-696-7625
tarus <at> sortova.com


Gmane