Tugrul Erdogan | 12 Sep 08:28 2014
Picon

Rrdtool fetch performance difference between Linux and Freebsd

Hi,

I am porting a project which is developed and used on FreeBSD 10 to Linux. After porting rrdtool funuctionalities, I realized that periodic "rrdtool fetch" calls on CentOS 7 spends two times more time than on FreeBSD. The rrdtool version on FreeBSD is 1.4.7 and on Linux 1.4.8 . 

To examine the situation more deeply, I have transfer same rrd file to FreeBSD and I have write a bash script which reports start and end times of 500 "rrdtool fetch" call with same parameters on same rrd file. The results show that "rrdtool fetch" call on Linux is two times slower than "rrdtool fetch" call on Linux.

What can be the cause of this performance decreasing? How can I improve the perfomance of "rrdtool fetch" call on Linux to make it as fast as FreeBSD "rrdtool fetch" call performance?
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Peter | 7 Jul 23:25 2014
Picon
Picon

RRDCached on Windows

Hi,
is a rrdchached windows service available?
If not, has someone built it from rrd_daemon.c?
Peter

--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/RRDCached-on-Windows-tp7582278.html
Sent from the RRDtool Developers Mailinglist mailing list archive at Nabble.com.
Maciek Kolbusz | 27 Jun 08:09 2014
Picon

Proposing a new instruction for rrdraph: DSN

Hello List,

I migrated my growing collection of RRD files to MySQL several months ago and it really reduced the IO load on the data collecting host as well as improved the speed of drawing graphs. A new problem I faced was that the length of rrdgraph commands increased due to longer DEF statements - with sql/ notation on average the length of a single DEF statement is about 250 characters. I have scripts that generate rrdgraph commands to draw complex graphs with tens or sometimes hundreds of DEFs, CDEFs, GPRINTs for formatted legends etc. and many times it happened that rrdgraph failed to generate a picture, because the command was too long (the length limit on my system is 130k characters). It would help to avoid this issue and shorten the command length if it was possible to define a DSN (Data Source Name) and reuse it in DEFs, ie.:

rrdtool graph ... \
DSN:dsn1=sql//mysql/host=mysql1.example.com/port=3306/dbname=mydb/username=dbuser/password=dbpassword \
... \
DEF:defA=dsn1/rrdminstepsize=60/TABLE_A/timestamp/value/RRDKeyID='"keyA'":avg:AVERAGE \
DEF:defB=dsn1/rrdminstepsize=60/TABLE_B/timestamp/value/RRDKeyID='"keyB'":avg:AVERAGE \
DEF:defC=dsn1/rrdminstepsize=60/TABLE_C/timestamp/value/RRDKeyID='"keyC'":avg:AVERAGE \
...

It's debatable what elements should be included in DSN and what should be left for DEF, however I would suggest that DSN only contained host, port, dbname, dbuser and password, because one database can have multiple tables with different step sizes, so the in DEFs it would be possible to use different tables in the same DB referring to the same DSN, but giving different step sizes and table names.

Another feature that I think would be very useful is reading DSN from files: store DSN definition in a text file and then point to this file in the command, ie.:

rrdtool graph ... \
DSN:dsn1=file:///path/to/file \
...

This way DB credentials would be better protected as there would be no risk that they appear in the commands and later in application logs or elsewhere in the system where other users could see them.

Please share your thoughts and comments on this feature.

--
Regards,
Maciek Kolbusz
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Peter Childs | 13 Jun 11:41 2014
Picon
Picon

about [rrdtool-1.x] Fetch callback (#502)


Reading the comments, and the edit to src/rrd_graph.c

   968 +        // remember that we already got this one
   969 +        g_hash_table_insert(im->rrd_map,gdes_fetch_key(im->gdes[i]),GINT_TO_POINTER(i));

I have noted when I tried to build a custom backend using rrdcached protocol that when drawing a graph it appeared to a little inefficient.

For example
  If I had a RRD with DS(s) of inOct, outOct, inErrs, outErrs and then used 'graph' to generate a single graph with all those DS's from the same RRD I would see 4 FETCH operations via rrdcached.

Since the fetch does not specify the DS required, all the datasources in the RRD need to be returned.   This probably isn't a problem for simple RRD files, but for example if you have 12 metrics, and each metric has a 'cost' to 'get', and you only really need two for your graph, you need to fetch them all anyway (or in my example, you need to fetch them all, then remangle them into the appropriate structures, then return them ..)

My question is -- does this change to the rrd_graph data_fetch() mean that I will not see multiple fetch operations ?

(I guess I should just go and test it ...)

:)

_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Peter | 10 May 11:16 2014
Picon
Picon

Compile rrdtool-1.4.8 in Visual Studio 2013

Hi,

I compiled successfully rrdtool-1.4.8 with Visual Studio 2013 after solving
the following 3 issues. At first pay attention to WIN32-BUILD-TIPS.txt and
load / copy  all libraries.
There are 3 issues left:
1. ==============
use a folder without spaces!
Otherwise later come an post build event error:

Error     20     error MSB3073: The command "copy
c:\Users\Peter\Documents\Visual Studio
2013\rrdtool-1.4.8\win32\\..\contrib\bin\freetype6.dll
c:\Users\Peter\Documents\Visual Studio 2013\rrdtool-1.4.8\win32\Debug\\ ...

2. ====================
Error     2     error C2556: 'int round(double)' : overloaded function
differs only by return type from 'double round(double)'    

Problem:
in math.h is defined
_CRTIMP double __cdecl round(_In_ double _X);

Solution:
comment out:
//__inline int round(double a ){int x = (a + 0.5); return x;}

possible slower as  inline, but in c++ is not allowed overloaded function
differs only by return.. Is there a faster solution?

3. =======================
Error     14     error C1189: #error :  "Don't know how to deal with TIME_T
other than 4 or 8 bytes"     c:\Users\Peter\Documents\Visual Studio
2013\rrdtool-1.4.8\src\rrd_restore.c     235     1     rrdlib

solution from answer
http://stackoverflow.com/questions/16962257/failing-to-compile-rrd
in rrd_restore.c
replace
        #if SIZEOF_TIME_T == 4
                     temp = strtol(( char *)text,NULL, 0);
        #elif SIZEOF_TIME_T == 8
                     temp = strtoll(( char *)text,NULL, 0);       
        #else
        #error "Don't know how to deal with TIME_T other than 4 or 8 bytes"
        #endif  
by 
#ifndef WIN32
        #if SIZEOF_TIME_T == 4
                     temp = strtol(( char *)text,NULL, 0);
        #elif SIZEOF_TIME_T == 8
                     temp = strtoll(( char *)text,NULL, 0);       
        #else
        #error "Don't know how to deal with TIME_T other than 4 or 8 bytes"
        #endif  
#else
                     temp = strtoll(( char *)text, NULL, 0);
#endif

in  rrd_create.c:
replace
#include "../rrd_config.h"
by

#ifndef WIN32
#include "../rrd_config.h"
#endif

After this changes I can compile it without errors and 110 warnigs.

Peter

--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/Compile-rrdtool-1-4-8-in-Visual-Studio-2013-tp7582052.html
Sent from the RRDtool Developers Mailinglist mailing list archive at Nabble.com.
Martin Sperl | 8 May 10:09 2014

rrdcached and --skip-past-updates

Hi!

With regards to template support for rrdcached (on the client side)
which just reacently got merged, I just started to wonder how
"--skip-past-updates" is related to rrdcached.

I wonder if it is really a necessity to have this test in rrd_update.c:
    if (extra_flags != 0) {
        rrd_set_error("The caching daemon cannot be used together with "
                      "templates and skip-past-updates yet.");
        goto out;
    }

It should be fairly easy to "ignore" the error from rrdcached
communication passing extra_opts to rrdc_update and there doing the
sanity check on the response from rrdcached.

Something like that in rrd_client.c in rrdc_update:
/* ignore time update errors in case RRD_SKIP_PAST_UPDATES is set  */
if (status && (extra_flags & RRD_SKIP_PAST_UPDATES))
    if ((strstr(res.message,"illegal attempt to update using time")
        && strstr(res.message," when last update time is "))
            status=0;

Is this a valid observations?

Martin
Martin Sperl | 6 May 15:59 2014

rrd_calc and avoid repeated memory allocations/initial setups

Hi!

It seems as if there is no easy means to store extra data for the whole life-time of the
rpn_calc loop.

The reason why I ask is because I am implementing the predictperc function, which is 
similar to predict and predictsigma but it gives the X percentile.

For this I need to order the data (depending on arguments maybe 50-60 floats),
which means allocating memory (the exactly same amount for every time rpn_calc gets called)
to keep all those values sorted to find the correct percentile.

Unfortunately the way rrd_calc works, there is no data that is kept between calculations,
to avoid the repeated create/teardowns of this structures...

Seems as if an "easy" way would be enhancing rpnp_t by adding this this:
void *extra; /* extra data used during setup */
void (*free_extra)(void *); /* a function used to free the extra structure... 
			     * - if null, then a "normal free" is done */

and some code in rpnstack_free to call the "free" functionpointer with the argument of extra.
(or free directly if the pointer is NULL)

So would something like the above be acceptable?

Thanks,
	Martin
Svante Signell | 29 Apr 14:35 2014
Picon

RFC: [PATCH] Portability by avoiding PATH_MAX

Hello,

Here are updated patches for rrdtool/rrd_client.c and
rrdtool/rrd_daemon.c. We had some discussions in August last year. I
created a branch and the diffs are against latest git. I've run the
code, especially rrd_daemon with valgrind under Linux, but need some
help to check that also rrd_client works OK (maybe rrd_daemon too). Can
you help me with test cases to run the execute the code modified paths.
I know one application using rrdtool, lm-sensors (build-dependency on
librrd2-dev), but am not sure my computers have the sensors to be a good
test case.

Note that the modified functions get_path() and get_abs_path() are
static, so they don't change the API.

Thanks,
Svante Signell
Attachment (rrd_daemon.patch): text/x-patch, 15 KiB
Attachment (rrd_client.patch): text/x-patch, 3960 bytes
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Steve Shipway | 28 Apr 02:37 2014
Picon
Picon

ROL and ROR for RPN

So, splitting this idea out from the separate post on the predict function,
I propose adding the new operations ROL and ROR to the RPN interpreter.

They would rotate the top 3 items left or right, and have this effect on the
stack ...

1,2,3,ROL  -->  2,3,1
1,2,3,ROR -->  3,1,2

These allow more complex arithmetic to be performed and complement the
existing EXC operator.

A context-diff against 1.4 is attached for your consideration; it is pretty
simple and implemented similarly to EXC.  The modified files are
rrd_rpncalc.h and rrd_rpncalc.c

Steve

Steve Shipway
s.shipway <at> auckland.ac.nz

Attachment (rol-ror.patch): application/octet-stream, 2720 bytes
Attachment (smime.p7s): application/pkcs7-signature, 7984 bytes
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Steve Shipway | 23 Apr 23:52 2014
Picon
Picon

Unexpected behaviour of PREDICT and PREDICTSIGMA

So, I’m delving into the PREDICT operations in the RPN, and have found something unexpected.

 

There are two ways of specifying the shift offsets – as a (positive) count of shifts and a list; or as a negative count and a base shift.

 

CDEF:predict1=172800,86400,0,3,3600,x,PREDICT

CDEF:predict2=86400,-3,3600,x,PREDICT

 

My tests (using 1.4.trunk) indicate that the two above have identical behaviour.  My question is, is this intended?  In particular, when giving a negative  shift count, should the resulting list of shifts be based at 0? 

 

Given this:

 

CDEF:predict=s,-3,w,x,PREDICT

 

I had expected to get shifts of s, 2s and 3s.  Instead I get 0, s and 2s.  Is this intentional, or a feature?

 

As a corollary, it means that these two lines are also equal for ANY value of s

 

CDEF:predict1=s,-1,3600,x,PREDICT

CDEF:predict2=x,3600,TRENDNAN

 

Thanks,

 

Steve

 

 

Steve Shipway

University of Auckland

UNIX Systems Design Team Lead

s.shipway <at> auckland.ac.nz

+64 (9) 3737 599 ext 86487

 

Attachment (smime.p7s): application/pkcs7-signature, 7984 bytes
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
AGS IT Helpdesk | 12 Mar 13:16 2014

how to change the x legend from the rrdtool to custom text or default rrdtool


Dear Developers,

Thank you very much for you kind time given to us,

I have a question that in attached email below I have installed the cacti
with rrdtool that came in a preconfigured centos image. But in that graph in
all graphs I see the title named there in right hand Cynunwei.com / Cacti
can you tell me how I can change hit sin cacti or RRDTool.

I search many places and etc in the web but failed to edit and change this
to normal RRDtool text

Thank you in advance 

<http://rrd-mailinglists.937164.n2.nabble.com/file/n7581829/graph_image.php.png> 

--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/how-to-change-the-x-legend-from-the-rrdtool-to-custom-text-or-default-rrdtool-tp7581829.html
Sent from the RRDtool Developers Mailinglist mailing list archive at Nabble.com.

Gmane