NAN-safe ADD

Hello,

i've implemented a nan-safe add operator (ADDNAN) into rrd. I used it 
to add several incomplete graphs.

NaN + NaN => NaN
  x + NaN => x
  NaN + y => y
  x + y => x + y

The patch is attached.

Best regards
Timo Stripf

_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: NAN-safe ADD

Ups i had a typo ... new patch attached.

At 01:44 06.03.2008, Timo Stripf wrote:
>Hello,
>
>i've implemented a nan-safe add operator (ADDNAN) into rrd. I used 
>it to add several incomplete graphs.
>
>NaN + NaN => NaN
>  x + NaN => x
>  NaN + y => y
>  x + y => x + y
>
>The patch is attached.
>
>Best regards
>Timo Stripf
>
>
>_______________________________________________
>rrd-developers mailing list
>rrd-developers <at> lists.oetiker.ch
>https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Re: NAN-safe ADD

On Thu, Mar 06, 2008 at 01:44:30AM +0100, Timo Stripf wrote:
> Hello,
> 
> i've implemented a nan-safe add operator (ADDNAN) into rrd. I used it 
> to add several incomplete graphs.
> 
> NaN + NaN => NaN
>  x + NaN => x
>  NaN + y => y
>  x + y => x + y

I understand why you would consider this useful.

However, I also think it can be dangerous, especially to newcomers.

Consider two datasources (e.g. two uplinks). One is constant at
(<whatever>, say:) 10Mbps, the other at 20Mbps.

Suddenly something fails somewhere for whatever reason, and getting
stats from one of the two datasources fails.  The user ends up with
a gap in his graph.

I can see the following Q&A happening on the mailinglist, often:
Q: how to fix this?
A: use 'ADDNAN' instead of '+' <period>

Result: only 20 (or 10) Mbps is shown on the graph. The problem is
not fixed. Eventually this user is going to complain that "RRDtool
provides a wrong picture".

Re: NAN-safe ADD

Hi Timo,

> On Thu, Mar 06, 2008 at 01:44:30AM +0100, Timo Stripf wrote:
> > Hello,
> >
> > i've implemented a nan-safe add operator (ADDNAN) into rrd. I used it
> > to add several incomplete graphs.
> >
> > NaN + NaN => NaN
> >  x + NaN => x
> >  NaN + y => y
> >  x + y => x + y
> [...]
> Quite often 'NaN' does not mean 'nothing'.  In many cases substituting
> the average of the two neighbours (not direct neighbours per se) would
> make more sense.
>
> At the very least I think this feature deserves more documentation than
> just a line or two.

How about doing this explicitly:

  x=a,UN,0,a,IF,b,UN,0,b,IF,+

then the effort for doing it has some relationship to the 'danger'
of the operation.

or if you insist to make this simpler, how about implementing a
NANTOZERO operator ?

Re: NAN-safe ADD

Hi Alex,

On Thu, Mar 06, 2008 at 05:13:53AM +0100, Alex van den Bogaerdt wrote:
> I can see the following Q&A happening on the mailinglist, often:
> Q: how to fix this?
> A: use 'ADDNAN' instead of '+' <period>

in my opinion, the user knows better than the application, which has
been the UNIX assumption for decades.

Also I don't think it's helpful to think of RRDTool as a monitoring
solution. If something breaks, some other program should alert the user.
It's not RRDTool's job.

Last but not least, I'm having the problem regularly. The problem is
that first you convert all NaNs to zeros. Then you perform your
calculation. And then you have to check if ALL values were NaN and
change the result to NaN again. Having an operator that's doing that is
an excellent idea.

Tobi, that last step, converting back to NaN if everything is NaN, is
why I'd prefer this special operator over a `NANORZERO' commad - the
if-construction is easy enough.

Actually, I'd like to propose an even simpler syntax for the operator:
I'd simply append a `!' (exclamation mark) to the operators `+', `-',
`*', and `/', i. e. `+!', `-!', `*!', and `/!'. Just an idea though.

Regards,
-octo

Re: NAN-safe ADD

On Thu, Mar 06, 2008 at 06:48:32AM +0100, Tobias Oetiker wrote:

> or if you insist to make this simpler, how about implementing a
> NANTOZERO operator ?

Wouldn't a generic 'IFNAN' operator be more useful? That way I could
substitute any other value, as appropriate for my calculation. You
could still get the NANORZERO with something like:

a,0,IFNAN

Steve

Re: NAN-safe ADD

On Thu, Mar 06, 2008 at 10:47:48AM -0500, Steve Hill wrote:
> On Thu, Mar 06, 2008 at 06:48:32AM +0100, Tobias Oetiker wrote:
> 
> > or if you insist to make this simpler, how about implementing a
> > NANTOZERO operator ?
> 
> Wouldn't a generic 'IFNAN' operator be more useful? That way I could
> substitute any other value, as appropriate for my calculation. You
> could still get the NANORZERO with something like:
> 
> a,0,IFNAN

Take this one step further:

Wouldn't a generic IF be more useful?  You could still get NANORZERO
with something like:

a,UN,0,a,IF

Erm...

--

-- 
Alex van den Bogaerdt
http://www.vandenbogaerdt.nl/rrdtool/

Re: NAN-safe ADD

Hi Steve,

this would be sort of like the COALECE operator in SQL ...

this does not solve the ALL NAN scenario though.

cheers
tobi

Today Steve Hill wrote:

> On Thu, Mar 06, 2008 at 06:48:32AM +0100, Tobias Oetiker wrote:
>
> > or if you insist to make this simpler, how about implementing a
> > NANTOZERO operator ?
>
> Wouldn't a generic 'IFNAN' operator be more useful? That way I could
> substitute any other value, as appropriate for my calculation. You
> could still get the NANORZERO with something like:
>
> a,0,IFNAN
>
> Steve
>
>

--

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten
http://it.oetiker.ch tobi <at> oetiker.ch ++41 62 213 9902

Re: NAN-safe ADD

Hi Tobi,

i'm running a distributed daemon over several servers. On every 
server rrd files are recorded for usage stats and these files are 
exported by NFS. On one control server all rrd files are combined 
using the ADDNAN operator. If one server was not running the value 
would be NaN and it should be treated as 0. If all values are NaN the 
complete system was not running at this time and the result should be 
NaN to be ignored by the TREND line or year average information.

x=a,UN,0,a,IF,b,UN,0,b,IF,+

is not the same since NaN + NaN would be 0.

An alternative for ADDNAN would be

x=a,UN,b,b,UN,a,a,b,+,IF,IF

and this is getting more complex if you like to add three or more 
values this way. A NANTOZERO operator would not help here.

The ADDNAN operator is associative and commutative.

I also thought about adding NAN-safe versions of other arithmetic 
functions. For subtraction NaN could also be treated as 0 but i found 
no real scenario where this would be useful. For multiplication NaN 
could be treated as the neutral element 1. Reasonable? ... i don't 
know. Division and module are far more complex.

One could think about adding NAN-safe MAX/MIN functions. MAXNAN(a, 

Wrong behaviour or not?

Hi,

This seems wrong to me:

CDEF:y=x,b,c,IF

When x is anything but zero, b is returned. Anything includes NaN here.

The problem seems to be in the way C handles NaN comparisons in
combination with this code:

if (x != 0.0) ...

which does indeed branch to TRUE.
(similar code is in rrd_rpncalc.c, case OP_IF: )

I did a quick test:
(crappy C code attached)

Using != 0:

x==nan -> True
x==inf -> True
x==-inf -> True
x==1.000000 -> True
x==0.500000 -> True
x==0.000000 -> False
x==-0.500000 -> True

Comparing to greater than zero also doesn't work out nicely,