Cacti (home)ForumsDocumentation

"No Data Loss"

Users often want to make sure, that no data will be lost. Even if I do not fully support this way of using rrdtool, let's discuss Howto define a very BIG rra “without data loss”.

You won't really do that! Why? One of the inherent features of rrd's is: they never grow in space. In other words: When creating a new rrd, it is allocated with all space needed. See rrd-beginners tutorial. As usually, you may use the information given here at your own risk.

Basic Knowledge for understanding RRAs

Often, there's a fundamental misunderstanding (which is enforced by cacti's way of defining “rra related parameters”). Basically, there are no daily, weekly, monthly, yearly rra's in any rrd file!

RRDTool defines different levels of consolidation only. It does not define timespans explicitely. It only defines the AMOUNT OF DATAPOINTS for each consolidation level (known as rows in rrdtool lingo). Assuming you are trying to keep only one level of consolidation, this is defined by step in the rra definition. And, if you want to omit consolidation, this equals to step=1. By default, all rrd files will have 4 levels of consolidation, step=1,6,24,288, respectively. Forget about the last three ones (well, they will use some amount of space; but forget about this for the time being). So lets deal with the first rra (step=1) only. If you want to extend this rra to span a longer time, you have to deal with the number of rows. You will have to increase the number of rows until the wanted timespan is reached. You may compute the timespan by multiplying rows * step.

Here we go!

Cacti's logic to generate rrd files works as follows:

  1. create a device (the host that shall be queried)
  2. create a graph for this host (using a graph template or a data query that refers to graph templates)
  3. each graph template refers to a data template
  4. each data template defines one or more data sources
  5. each data template uses one or more round robin archives (rra)
  6. each of the data sources uses the same set of rra's

This tutorial works the way back.

Defining a new round robin archive (rra)

For the following, lets assume you are logged in with admin permissions and use the console tab.

  1. Go to ManagementData SourcesRRAs
  2. Click Add to add a new rra

01-data-sources-rra

Now fill in the data as follows and SAVE:

02-data-sources-rra

Name:you may choose your own
Consolidation function:AVERAGE needed
X-Files Factor:always 0.5
Steps:1 (that is the number of data points to use for consolidation, 1 says: no consolidation at all)
Rows:115200 = 400 days with 24 hours and 12 data points per hour (= 5 min interval)
Timespan:used for displaying 33,053,184 seconds = about 382 days (taken from other cacti rra)

Define a new data template

For ease of use (yes, I'm lazy), please copy an existing template. Goto Data Templates, and check the box on the right of Interface - Traffic:

03-data-template

Then scroll to the bottom of the page, select Duplicate and Go.

04-data-template You will be prompted for a new name of this template:

05-data-template

Of course, you may choose your own name here. Now it is time to modify this template:

06-data-template

  1. You may change the name of the template
  2. Select the just created RRA (Don't worry about the other RRAs in this list; they are needed for the next tutorial …)

Please leave the rest as is; SAVE. Of course, you may define a new data template from scratch. The only thing to keep in mind is to select the appropriate RRA. The data template is now done.

Define a new graph template

Well, you will imagine what comes next. Again, I decided to copy the appropriate graph template. So goto Graph Templates and repeat the steps above for the template Interface Traffic (bits/sec). It will look like this:

07-graph-template

Please pay attention to the next steps! You will have to delete both Graph Item Inputs, as they refer to the wrong data source. Please select the red X to the right of Inbound Data Source as well as Outbound Data Source.

Then you will have to add the newly generated data sources. In order to do that, please select each item of the list of Graph Items, one after the other. This will look like:

08-graph-template

As Data Source you will choose the appropriate data source you generated in the previous step. Don't forget to do this for each and every item of the Graph Item list. When you're done, scroll to the bottom of the Graph Template definition and SAVE.

Modify Data Query to add Graph Template

This example uses Interface Traffic Graph Template. This is referenced by the Data Query SNMP Interface Statistics. Now we're going to add the newly defined Graph Template to this very Data Query.

If you have chosen some other Graph Template, e.g. ucd/net Load Average, you will skip this step.

The Data Query goes like this. Goto Data Queries and select SNMP - Interface Statistics.

09-data-query

Now Add to see this:

10-data-query-add

Define a new name for this Associated Graph and CREATE.

Finally: Create Graphs for this Host

Goto Devices and select your favorite device to see the rra in action. If you have modified the SNMP Interface Statistics Data Query, you may immediately select Create Graphs for this Host to see the following:

11-create-graphs

Select the interface as you would have done for any Traffic Graph. Then Select a graph type from the dropdown list (of course our newly defined Graph Template!) and CREATE. As usually, you will have to wait at least two polling cycles to get the graph generated and filled with the first value. Don't be impatient! Let it run for awhile.

Under the Graphs tab you will notice something like

12-traffic-no-consolidation

Well, this looks like usual, doesn't it? You may wonder about the Outbound traffic displayed negative. Well, this is a little CDEF but is of no matter here. And of course, for the first two days you will not notice anything unusual. This is because the default cacti rra configuration keeps all data points without consolidation for 600 intervals (about 2 days).

Some advice: Please do not click onto the graph too fast. I had to wait some time (don't remember exactly) before clicking gave a result like the next one:

13-traffic-no-consolidation

This is already a zoomed image. You will notice, that my personal laptop isn't online for the whole day ;-).

Now, where's the trick?

At first, you may wonder, whether only this one graph will be displayed. This is, because only one single rra exists. And cacti associated the time interval of the graph with each rra. Only one rra defined gives only one image displayed.

But you may zoom in at any place and will reach down to the 5 min intervals. This is, what had to be proved (q.e.d as the old romans said).

Something to keep in mind

Space allocation with rrdtool

The space needed is calculated from

  • the number of data sources needed (e.g. traffic in and traffic out form two data sources)
  • the number of rra's needed (e.g. one archive for storing original data points, a second one to hold averaged data points for some weeks, a third for holding averaged data points for some months …)
  • the number of data points to be stored in each rra
  • some header space

If you omit consolidation (that is: averaging out some data points), you won't loose data. But you will loose space!

Example: Store data every 300 seconds for a whole year. This leads to 12 (data points each hour) * 24 (hours per day) * 365 (days per year) data point (= 105120). Each data point holds 8 bytes, so the whole rrd will occupy about 840,960 Bytes (plus some header space) for each single data source.

A closer look to rrd file properties

Please have a look at the file sizes on my computer:

-rw-r--r--  1 cactiuser cactiuser   94660 Oct  2 19:40 gandalf_traffic_in_17.rrd
-rw-r--r--  1 cactiuser cactiuser 1844056 Oct  2 19:40 gandalf_traffic_in_71.rrd

They belong to following rrd definitions (see Data Source Debug of that data source)

/usr/bin/rrdtool create \
/var/www/html/cacti-0.8.6f/rra/gandalf_traffic_in_17.rrd \
--step 300  \
DS:traffic_in:COUNTER:600:0:100000000 \
DS:traffic_out:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MIN:0.5:1:600 \
RRA:MIN:0.5:6:700 \
RRA:MIN:0.5:24:775 \
RRA:MIN:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \
RRA:LAST:0.5:1:600 \
RRA:LAST:0.5:6:700 \
RRA:LAST:0.5:24:775 \
RRA:LAST:0.5:288:797 \

and respectively:

/usr/bin/rrdtool create \
/var/www/html/cacti-0.8.6f/rra/gandalf_traffic_in_71.rrd \
--step 300  \
DS:traffic_out:COUNTER:600:0:100000000 \
DS:traffic_in:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:1:115200 \

As you will notice, the newly generated rrd is about 20 times the size of the original one (and this one spreads two years, not only 400 days). So please pay attention, before using this widely. The performance impact for updating and displaying such rrd's in a large installation may not be desired.






Personal Tools