Cacti (home)ForumsDocumentation

Differences

This shows you the differences between two versions of the page.

manual:087:8_rrdtool.03_big_rra [2009/06/21 05:15] (current)
gandalf created
Line 1: Line 1:
 +===== "No Data Loss" =====
  
 +Users often want to make sure, that //no data will be lost//. Even if I do not fully support this way of using rrdtool, let's discuss //Howto define a very BIG rra "without data loss"//.
 +
 +**You won't really do that! Why? One of the inherent features of rrd's is: they never grow in space. In other words: When creating a new rrd, it is allocated with all space needed. See [[http://oss.oetiker.ch/rrdtool|rrd-beginners tutorial]]. As usually, you may use the information given here at your own risk.**
 +
 +==== Basic Knowledge for understanding RRAs ====
 +Often, there's a fundamental misunderstanding (which is enforced by cacti's way of defining "rra related parameters"). Basically, there are no daily, weekly, monthly, yearly rra's in any rrd file!
 +
 +RRDTool defines **different levels of consolidation** only. It does not define timespans explicitely. It only defines the AMOUNT OF DATAPOINTS for each consolidation level (known as **rows** in rrdtool lingo).
 +Assuming you are trying to keep only one level of consolidation, this is defined by **step** in the rra definition. And, if you want to omit consolidation, this equals to **step=1**.
 +By default, all rrd files will have 4 levels of consolidation, step=1,6,24,288, respectively. Forget about the last three ones (well, they will use some amount of space; but forget about this for the time being). So lets deal with the first rra (**step=1**) only.
 +If you want to extend this rra to span a longer time, you have to deal with the number of rows. You will have to increase the number of rows until the wanted timespan is reached. You may compute the timespan by multiplying **rows * step**. 
 +
 +==== Here we go! ====
 +Cacti's logic to generate rrd files works as follows:
 +
 +  - create a device (the host that shall be queried)
 +  - create a graph for this host (using a graph template or a data query that refers to graph templates)
 +  - each graph template refers to a data template
 +  - each data template defines one or more data sources
 +  - each data template uses one or more round robin archives (rra)
 +  - each of the data sources uses the same set of rra's
 +
 +This tutorial works the way back.
 +
 +==== Defining a new round robin archive (rra) ====
 +For the following, lets assume you are logged in with admin permissions and use the **console** tab.
 +
 +  - Go to  **Management** -> **Data Sources** -> **RRAs**
 +  - Click Add to add a new rra
 +
 +{{http://docs.cacti.net/files/images/01-data-sources-rra.png|01-data-sources-rra}}
 +
 +
 +Now fill in the data as follows and SAVE:
 +
 +{{http://docs.cacti.net/files/images/02-rra.png|02-data-sources-rra}}
 +
 +
 +|**Name:**|you may choose your own|
 +|**Consolidation function:**|AVERAGE needed|
 +|**X-Files Factor:**|always 0.5|
 +|**Steps:**|1 (that is the number of data points to use for consolidation, 1 says: no consolidation at all)|
 +|**Rows:**|115200 = 400 days with 24 hours and 12 data points per hour (= 5 min interval)|
 +|**Timespan:**|used for displaying 33,053,184 seconds = about 382 days (taken from other cacti rra)|
 +
 +
 +
 +==== Define a new data template ====
 +For ease of use (yes, I'm lazy), please copy an existing template. Goto **Data Templates**, and check the box on the right of **Interface - Traffic**:
 +
 +{{http://docs.cacti.net/files/images/03-data-template.png|03-data-template}}
 +
 +Then scroll to the bottom of the page, select **Duplicate** and **Go**.
 +
 +{{http://docs.cacti.net/system/files/images/04-data-template_0.png|04-data-template}}
 +You will be prompted for a new name of this template:
 +
 +{{http://docs.cacti.net/files/images/05-data-template.png|05-data-template}}
 +
 +Of course, you may choose your own name here. Now it is time to modify this template:
 +
 +{{http://docs.cacti.net/files/images/06-data-template.png|06-data-template}}
 +
 +  - You may change the name of the template
 +  - Select the just created RRA (Don't worry about the other RRAs in this list; they are needed for the next tutorial ...)
 +
 +Please leave the rest as is; SAVE. Of course, you may define a new data template from scratch. The only thing to keep in mind is to select the appropriate RRA. The data template is now done.
 +
 +==== Define a new graph template ====
 +
 +Well, you will imagine what comes next. Again, I decided to copy the appropriate graph template. So goto **Graph Templates** and repeat the steps above for the template **Interface Traffic (bits/sec)**. It will look like this:
 +
 +{{http://docs.cacti.net/files/images/07-graph-template.png|07-graph-template}}
 +
 +Please pay attention to the next steps! You will have to delete both **Graph Item Inputs**, as they refer to the **wrong** data source. Please select the red X to the right of **Inbound Data Source** as well as **Outbound Data Source**.
 +
 +Then you will have to add the newly generated data sources. In order to do that, please select each item of the list of **Graph Items**, one after the other. This will look like:
 +
 +{{http://docs.cacti.net/files/images/08-graph-template.png|08-graph-template}}
 +
 +As **Data Source** you will choose the appropriate data source you generated in the previous step. Don't forget to do this for each and every item of the **Graph Item** list. When you're done, scroll to the bottom of the **Graph Template** definition and SAVE.
 +
 +==== Modify Data Query to add Graph Template ====
 +
 +This example uses **Interface Traffic** Graph Template. This is referenced by the Data Query **SNMP Interface Statistics**. Now we're going to add the newly defined **Graph Template** to this very **Data Query**.
 +
 +If you have chosen some other **Graph Template**, e.g. **ucd/net Load Average**, you will skip this step.
 +
 +The **Data Query** goes like this. Goto **Data Queries** and select **SNMP - Interface Statistics**.
 +
 +{{http://docs.cacti.net/files/images/09-data-query.png|09-data-query}}
 +
 +Now Add to see this:
 +
 +{{http://docs.cacti.net/files/images/10-data-query-add.png|10-data-query-add}}
 +
 +Define a new name for this **Associated Graph** and CREATE.
 +
 +==== Finally: Create Graphs for this Host ====
 +
 +Goto **Devices** and select your favorite device to see the rra in action. If you have modified the **SNMP Interface Statistics** Data Query, you may immediately select **Create Graphs for this Host** to see the following:
 +
 +{{http://docs.cacti.net/files/images/11-create-graphs.png|11-create-graphs}}
 +
 +Select the interface as you would have done for any Traffic Graph. Then **Select a graph type** from the dropdown list (of course our newly defined **Graph Template**!) and CREATE. As usually, you will have to wait at least two polling cycles to get the graph generated and filled with the first value. Don't be impatient! Let it run for awhile.
 +
 +Under the **Graphs** tab you will notice something like 
 +
 +{{http://docs.cacti.net/files/images/12-traffic-no-consolidation.png|12-traffic-no-consolidation}}
 +
 +Well, this looks like usual, doesn't it? You may wonder about the Outbound traffic displayed negative. Well, this is a little CDEF but is of no matter here. And of course, for the first two days you will not notice anything unusual. This is because the default cacti rra configuration keeps all data points without consolidation for 600 intervals (about 2 days).
 +
 +Some advice:
 +Please do not click onto the graph too fast. I had to wait some time (don't remember exactly) before clicking gave a result like the next one:
 +
 +{{http://docs.cacti.net/files/images/13-traffic-no-consolidation.png|13-traffic-no-consolidation}}
 +
 +This is already a zoomed image. You will notice, that my personal laptop isn't online for the whole day ;-).
 +
 +==== Now, where's the trick? ====
 +
 +At first, you may wonder, whether only this one graph will be displayed. This is, because only one single rra exists. And cacti associated the time interval of the graph with each rra. Only one rra defined gives only one image displayed.
 +
 +But you may zoom in at any place and will reach down to the 5 min intervals. This is, what had to be proved (q.e.d as the old romans said).
 +
 +==== Something to keep in mind ====
 +
 +=== Space allocation with rrdtool ===
 +
 +The space needed is calculated from
 +
 +  * the number of data sources needed (e.g. traffic in and traffic out form two data sources)
 +  * the number of rra's needed (e.g. one archive for storing original data points, a second one to hold averaged data points for some weeks, a third for holding averaged data points for some months ...)
 +  * the number of data points to be stored in each rra
 +  * some header space
 +
 +If you omit consolidation (that is: averaging out some data points), you won't loose data. But you will loose space! 
 +
 +Example: 
 +Store data every 300 seconds for a whole year. This leads to 12 (data points each hour) * 24 (hours per day) * 365 (days per year) data point (= 105120). Each data point holds 8 bytes, so the whole rrd will occupy about 840,960 Bytes (plus some header space) for each single data source.
 +
 +=== A closer look to rrd file properties ===
 +
 +Please have a look at the file sizes on my computer:
 +<code>-rw-r--r--  1 cactiuser cactiuser   94660 Oct  2 19:40 gandalf_traffic_in_17.rrd
 +-rw-r--r--  1 cactiuser cactiuser 1844056 Oct  2 19:40 gandalf_traffic_in_71.rrd</code>
 +
 +They belong to following rrd definitions (see Data Source Debug of that data source)
 +<code>/usr/bin/rrdtool create \
 +/var/www/html/cacti-0.8.6f/rra/gandalf_traffic_in_17.rrd \
 +--step 300&nbsp; \
 +DS:traffic_in:COUNTER:600:0:100000000 \
 +DS:traffic_out:COUNTER:600:0:100000000 \
 +RRA:AVERAGE:0.5:1:600 \
 +RRA:AVERAGE:0.5:6:700 \
 +RRA:AVERAGE:0.5:24:775 \
 +RRA:AVERAGE:0.5:288:797 \
 +RRA:MIN:0.5:1:600 \
 +RRA:MIN:0.5:6:700 \
 +RRA:MIN:0.5:24:775 \
 +RRA:MIN:0.5:288:797 \
 +RRA:MAX:0.5:1:600 \
 +RRA:MAX:0.5:6:700 \
 +RRA:MAX:0.5:24:775 \
 +RRA:MAX:0.5:288:797 \
 +RRA:LAST:0.5:1:600 \
 +RRA:LAST:0.5:6:700 \
 +RRA:LAST:0.5:24:775 \
 +RRA:LAST:0.5:288:797 \</code>
 +
 +and respectively:
 +<code>/usr/bin/rrdtool create \
 +/var/www/html/cacti-0.8.6f/rra/gandalf_traffic_in_71.rrd \
 +--step 300&nbsp; \
 +DS:traffic_out:COUNTER:600:0:100000000 \
 +DS:traffic_in:COUNTER:600:0:100000000 \
 +RRA:AVERAGE:0.5:1:115200 \</code>
 +
 +As you will notice, the newly generated rrd is about 20 times the size of the original one (and this one spreads two years, not only 400 days). So please pay attention, before using this widely. The performance impact for updating and displaying such rrd's in a large installation may not be desired.





Personal Tools