Steve Clark | 11 Dec 21:03 2014

threads and rrdcached

Hello List,

I was reading the web page http://oss.oetiker.ch/rrdtool/prog/rrdthreads.en.html
about using threads and it says to use the _r calls like rrd_update_r().

But in examing the code in rrd_update.c for version 1.4.8
it appears calling rrd_update_r() will bypass the use of rrdcached. Am I understanding this
correctly?

Also it looks like rrd_update() eventually calls rrd_update_r() if it doesn't use the
rrdcached daemon.
From rrd_update()
...
    if (! rrdc_is_connected(opt_daemon))
    {
      rc = rrd_update_r(argv[optind], tmplt,
                        argc - optind - 1, (const char **) (argv + optind + 1));
    }
    else /* we are connected */
    {
        rc = rrdc_update (argv[optind], /* file */
                          argc - optind - 1, /* values_num */
                          (const char *const *) (argv + optind + 1)); /* values */
        if (rc > 0)
            rrd_set_error("Failed sending the values to rrdcached: %s",
                          rrd_strerror (rc));
    }
...

Thanks,


--
Stephen Clark
NetWolves Managed Services, LLC.
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.clark <at> netwolves.com
http://www.netwolves.com
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Eduardo Bragatto | 18 Nov 17:23 2014

Pull requests should go to 1.4 or master?

Hi,

I have submitted a couple pull requests for the 1.4 branch and today I noticed Tobias created new pull requests for the same changes to the master branch.

Should I have created pull requests for both branches?

Just asking so in the future I can save you some time and do it properly.

Thanks,
- Eduardo Bragatto
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Jeroen Massar | 24 Oct 12:29 2014
Picon

rrdcached API documentation?

Hola muchachos,

I was checking on adding rrdcached support to the SixXS
statistics daemon[1] to get it a bit out of IO hell...

$ du -hcs rrd
21G	rrd
21G	total

if you get my drift ;) [which might be little for some folks...]

But apart from:
 http://www.netways.de/uploads/media/Sebastian_Harl_How_to_Escape_the_IO_Hell_02.pdf

I can't find any API documentation, using those APIs I googled a bit:

A google(rrdc_connect site:oss.oetiker.ch) is empty.

A google(rrdc_connect site:oetiker.ch) gives only the patches
on the mailinglist.

The Debian librrd-dev does not have anything either and I can't
seem to find it in the source tarballs (see below) unless I am
looking in the wrong location.

Is there a small minimal example application that shows the ropes?

Also, if rrdcached opens the files, how are permissions handled?
Especially when the calling process runs as user X and the daemon
as user Y. Or is it expected that rrdcached runs as X or at least
in the right group?

The "SECURITY CONSIDERATIONS" of
 http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html
has nothing about this...

Greets,
 Jeroen

[1] The thing that collects the data for
    https://www.sixxs.net/misc/traffic/
    but also traffic + latency stats for the individual tunnels

--
rrdtool-1.4.7$ rgrep rrdc_connect .
./src/rrd_update.c:        int status = rrdc_connect(opt_daemon);
./src/rrd_client.c:static int rrdc_connect_unix (const char *path) /* {{{ */
./src/rrd_client.c:} /* }}} int rrdc_connect_unix */
./src/rrd_client.c:static int rrdc_connect_network (const char *addr_orig) /* {{{ */
./src/rrd_client.c:} /* }}} int rrdc_connect_network */
./src/rrd_client.c:int rrdc_connect (const char *addr) /* {{{ */
./src/rrd_client.c:    status = rrdc_connect_unix (addr + strlen ("unix:"));
./src/rrd_client.c:    status = rrdc_connect_unix (addr);
./src/rrd_client.c:    status = rrdc_connect_network(addr);
./src/rrd_client.c:} /* }}} int rrdc_connect */
./src/rrd_client.c:  rrdc_connect(opt_daemon);
./src/rrd_xport.c:        int status = rrdc_connect(im.daemon_addr);
./src/rrd_graph.c:        int status = rrdc_connect(im->daemon_addr);
./src/rrd_flushcached.c:    status = rrdc_connect(opt_daemon);
./src/librrd.sym.in.in:rrdc_connect
./src/rrd_client.h:int rrdc_connect (const char *addr);
./src/rrd_client.h:#	define rrdc_connect(a) 0
./debian/librrd4.symbols: rrdc_connect <at> Base 1.4~rc2
./debian/librrd4.symbols: rrdc_connect <at> Base 1.4~rc2
Tugrul Erdogan | 12 Sep 08:28 2014
Picon

Rrdtool fetch performance difference between Linux and Freebsd

Hi,

I am porting a project which is developed and used on FreeBSD 10 to Linux. After porting rrdtool funuctionalities, I realized that periodic "rrdtool fetch" calls on CentOS 7 spends two times more time than on FreeBSD. The rrdtool version on FreeBSD is 1.4.7 and on Linux 1.4.8 . 

To examine the situation more deeply, I have transfer same rrd file to FreeBSD and I have write a bash script which reports start and end times of 500 "rrdtool fetch" call with same parameters on same rrd file. The results show that "rrdtool fetch" call on Linux is two times slower than "rrdtool fetch" call on Linux.

What can be the cause of this performance decreasing? How can I improve the perfomance of "rrdtool fetch" call on Linux to make it as fast as FreeBSD "rrdtool fetch" call performance?
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Peter | 7 Jul 23:25 2014
Picon
Picon

RRDCached on Windows

Hi,
is a rrdchached windows service available?
If not, has someone built it from rrd_daemon.c?
Peter

--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/RRDCached-on-Windows-tp7582278.html
Sent from the RRDtool Developers Mailinglist mailing list archive at Nabble.com.
Maciek Kolbusz | 27 Jun 08:09 2014
Picon

Proposing a new instruction for rrdraph: DSN

Hello List,

I migrated my growing collection of RRD files to MySQL several months ago and it really reduced the IO load on the data collecting host as well as improved the speed of drawing graphs. A new problem I faced was that the length of rrdgraph commands increased due to longer DEF statements - with sql/ notation on average the length of a single DEF statement is about 250 characters. I have scripts that generate rrdgraph commands to draw complex graphs with tens or sometimes hundreds of DEFs, CDEFs, GPRINTs for formatted legends etc. and many times it happened that rrdgraph failed to generate a picture, because the command was too long (the length limit on my system is 130k characters). It would help to avoid this issue and shorten the command length if it was possible to define a DSN (Data Source Name) and reuse it in DEFs, ie.:

rrdtool graph ... \
DSN:dsn1=sql//mysql/host=mysql1.example.com/port=3306/dbname=mydb/username=dbuser/password=dbpassword \
... \
DEF:defA=dsn1/rrdminstepsize=60/TABLE_A/timestamp/value/RRDKeyID='"keyA'":avg:AVERAGE \
DEF:defB=dsn1/rrdminstepsize=60/TABLE_B/timestamp/value/RRDKeyID='"keyB'":avg:AVERAGE \
DEF:defC=dsn1/rrdminstepsize=60/TABLE_C/timestamp/value/RRDKeyID='"keyC'":avg:AVERAGE \
...

It's debatable what elements should be included in DSN and what should be left for DEF, however I would suggest that DSN only contained host, port, dbname, dbuser and password, because one database can have multiple tables with different step sizes, so the in DEFs it would be possible to use different tables in the same DB referring to the same DSN, but giving different step sizes and table names.

Another feature that I think would be very useful is reading DSN from files: store DSN definition in a text file and then point to this file in the command, ie.:

rrdtool graph ... \
DSN:dsn1=file:///path/to/file \
...

This way DB credentials would be better protected as there would be no risk that they appear in the commands and later in application logs or elsewhere in the system where other users could see them.

Please share your thoughts and comments on this feature.

--
Regards,
Maciek Kolbusz
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Peter Childs | 13 Jun 11:41 2014
Picon
Picon

about [rrdtool-1.x] Fetch callback (#502)


Reading the comments, and the edit to src/rrd_graph.c

   968 +        // remember that we already got this one
   969 +        g_hash_table_insert(im->rrd_map,gdes_fetch_key(im->gdes[i]),GINT_TO_POINTER(i));

I have noted when I tried to build a custom backend using rrdcached protocol that when drawing a graph it appeared to a little inefficient.

For example
  If I had a RRD with DS(s) of inOct, outOct, inErrs, outErrs and then used 'graph' to generate a single graph with all those DS's from the same RRD I would see 4 FETCH operations via rrdcached.

Since the fetch does not specify the DS required, all the datasources in the RRD need to be returned.   This probably isn't a problem for simple RRD files, but for example if you have 12 metrics, and each metric has a 'cost' to 'get', and you only really need two for your graph, you need to fetch them all anyway (or in my example, you need to fetch them all, then remangle them into the appropriate structures, then return them ..)

My question is -- does this change to the rrd_graph data_fetch() mean that I will not see multiple fetch operations ?

(I guess I should just go and test it ...)

:)

_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
Peter | 10 May 11:16 2014
Picon
Picon

Compile rrdtool-1.4.8 in Visual Studio 2013

Hi,

I compiled successfully rrdtool-1.4.8 with Visual Studio 2013 after solving
the following 3 issues. At first pay attention to WIN32-BUILD-TIPS.txt and
load / copy  all libraries.
There are 3 issues left:
1. ==============
use a folder without spaces!
Otherwise later come an post build event error:

Error     20     error MSB3073: The command "copy
c:\Users\Peter\Documents\Visual Studio
2013\rrdtool-1.4.8\win32\\..\contrib\bin\freetype6.dll
c:\Users\Peter\Documents\Visual Studio 2013\rrdtool-1.4.8\win32\Debug\\ ...

2. ====================
Error     2     error C2556: 'int round(double)' : overloaded function
differs only by return type from 'double round(double)'    

Problem:
in math.h is defined
_CRTIMP double __cdecl round(_In_ double _X);

Solution:
comment out:
//__inline int round(double a ){int x = (a + 0.5); return x;}

possible slower as  inline, but in c++ is not allowed overloaded function
differs only by return.. Is there a faster solution?

3. =======================
Error     14     error C1189: #error :  "Don't know how to deal with TIME_T
other than 4 or 8 bytes"     c:\Users\Peter\Documents\Visual Studio
2013\rrdtool-1.4.8\src\rrd_restore.c     235     1     rrdlib

solution from answer
http://stackoverflow.com/questions/16962257/failing-to-compile-rrd
in rrd_restore.c
replace
        #if SIZEOF_TIME_T == 4
                     temp = strtol(( char *)text,NULL, 0);
        #elif SIZEOF_TIME_T == 8
                     temp = strtoll(( char *)text,NULL, 0);       
        #else
        #error "Don't know how to deal with TIME_T other than 4 or 8 bytes"
        #endif  
by 
#ifndef WIN32
        #if SIZEOF_TIME_T == 4
                     temp = strtol(( char *)text,NULL, 0);
        #elif SIZEOF_TIME_T == 8
                     temp = strtoll(( char *)text,NULL, 0);       
        #else
        #error "Don't know how to deal with TIME_T other than 4 or 8 bytes"
        #endif  
#else
                     temp = strtoll(( char *)text, NULL, 0);
#endif

in  rrd_create.c:
replace
#include "../rrd_config.h"
by

#ifndef WIN32
#include "../rrd_config.h"
#endif

After this changes I can compile it without errors and 110 warnigs.

Peter

--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/Compile-rrdtool-1-4-8-in-Visual-Studio-2013-tp7582052.html
Sent from the RRDtool Developers Mailinglist mailing list archive at Nabble.com.
Martin Sperl | 8 May 10:09 2014

rrdcached and --skip-past-updates

Hi!

With regards to template support for rrdcached (on the client side)
which just reacently got merged, I just started to wonder how
"--skip-past-updates" is related to rrdcached.

I wonder if it is really a necessity to have this test in rrd_update.c:
    if (extra_flags != 0) {
        rrd_set_error("The caching daemon cannot be used together with "
                      "templates and skip-past-updates yet.");
        goto out;
    }

It should be fairly easy to "ignore" the error from rrdcached
communication passing extra_opts to rrdc_update and there doing the
sanity check on the response from rrdcached.

Something like that in rrd_client.c in rrdc_update:
/* ignore time update errors in case RRD_SKIP_PAST_UPDATES is set  */
if (status && (extra_flags & RRD_SKIP_PAST_UPDATES))
    if ((strstr(res.message,"illegal attempt to update using time")
        && strstr(res.message," when last update time is "))
            status=0;

Is this a valid observations?

Martin
Martin Sperl | 6 May 15:59 2014

rrd_calc and avoid repeated memory allocations/initial setups

Hi!

It seems as if there is no easy means to store extra data for the whole life-time of the
rpn_calc loop.

The reason why I ask is because I am implementing the predictperc function, which is 
similar to predict and predictsigma but it gives the X percentile.

For this I need to order the data (depending on arguments maybe 50-60 floats),
which means allocating memory (the exactly same amount for every time rpn_calc gets called)
to keep all those values sorted to find the correct percentile.

Unfortunately the way rrd_calc works, there is no data that is kept between calculations,
to avoid the repeated create/teardowns of this structures...

Seems as if an "easy" way would be enhancing rpnp_t by adding this this:
void *extra; /* extra data used during setup */
void (*free_extra)(void *); /* a function used to free the extra structure... 
			     * - if null, then a "normal free" is done */

and some code in rpnstack_free to call the "free" functionpointer with the argument of extra.
(or free directly if the pointer is NULL)

So would something like the above be acceptable?

Thanks,
	Martin
Svante Signell | 29 Apr 14:35 2014
Picon

RFC: [PATCH] Portability by avoiding PATH_MAX

Hello,

Here are updated patches for rrdtool/rrd_client.c and
rrdtool/rrd_daemon.c. We had some discussions in August last year. I
created a branch and the diffs are against latest git. I've run the
code, especially rrd_daemon with valgrind under Linux, but need some
help to check that also rrd_client works OK (maybe rrd_daemon too). Can
you help me with test cases to run the execute the code modified paths.
I know one application using rrdtool, lm-sensors (build-dependency on
librrd2-dev), but am not sure my computers have the sensors to be a good
test case.

Note that the modified functions get_path() and get_abs_path() are
static, so they don't change the API.

Thanks,
Svante Signell
Attachment (rrd_daemon.patch): text/x-patch, 15 KiB
Attachment (rrd_client.patch): text/x-patch, 3960 bytes
_______________________________________________
rrd-developers mailing list
rrd-developers <at> lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers

Gmane