HOME
Library
JRobin API
Utilities
Support
About Us
Forum

 

SourceForge.net Logo 

Get Firefox

 

 

Core API

Create new RRD file

JRobin RRD files are managed through objects of RrdDb class. New RRD file can be created:

  • from the scratch, using RrdDef object
  • from RRDTool's or JRobin's XML file dump
  • from external XML template. See Templates API page for more information.

Create new RRD file from the scratch

Suppose you want to create a JRobin RRD file equivalent to the following RRDTool file (two COUNTER datasources, input and output):

rrdtool create test.rrd --step 300 \
DS:input:COUNTER:600:0:U \
DS:output:COUNTER:600:0:U \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:797 \
RRA:AVERAGE:0.5:288:775 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:797 \
RRA:MAX:0.5:288:775

JRobin API follows exactly the same pattern::

// create new RRD definition
RrdDef def = new RrdDef("test.rrd", 300);
def.addDatasource("input", "COUNTER", 600, 0, Double.NaN);
def.addDatasource("output", "COUNTER", 600, 0, Double.NaN);
def.addArchive("AVERAGE", 0.5, 1, 600);
def.addArchive("AVERAGE", 0.5, 6, 700);
def.addArchive("AVERAGE", 0.5, 24, 797);
def.addArchive("AVERAGE", 0.5, 288, 775);
def.addArchive("MAX", 0.5, 1, 600);
def.addArchive("MAX", 0.5, 6, 700);
def.addArchive("MAX", 0.5, 24, 797);
def.addArchive("MAX", 0.5, 288, 775);

// RRD file definition is completed, create the database!
RrdDb rrd = new RrdDb(def);
// by this point, file "test.rrd" can be found on your disk

Notes:

  • All standard RRDTool data source types are supported (COUNTER, GAUGE, DERIVE, ABSOLUTE), as well as all standard consolidation functions (AVERAGE, MIN, MAX, LAST)
  • All RRDTool creation parameters have the same meaning in JRobin. For the detailed explanation of RRDTool creation parameters, see the original RRDTool documentation.
  • RRDTool uses U to denote unknown values. JRobin uses Double.NaN instead.
  • JRobin RRD files are usually (slightly) smaller than RRDTool files with the same creation parameters (in the last example JRobin takes 92,992 bytes of disk space and RRDTool takes 94,660 bytes)
  • JRobin RRD files are platform-independent, RRDTool files are not.
  • As a consequence, JRobin RRD file format is different from the RRDTool format.
  • JRobin files can be converted to RRDTool files and vice versa through XML import/export process.

Create new RRD file from RRDTool's XML file

Suppose that you have a RRD file named original.rrd created with RRDTool. It is possible to create a JRobin file with the same data in it. First, dump the content of original.rrd to XML file with the following command:

rrdtool dump original.rrd > original.xml

The file original.xml can be used to create a JRobin file from the Java code:

RrdDb rrd = new RrdDb("copy.rrd", "original.xml");

The file copy.rrd is a JRobin equivalent of the original.rrd file.

It's that simple but take care that the import process takes considerable time and memory. As a rule of thumb, increase the ammount of memory available to JVM (use switch like -Xmx128m) whenever importing RRDTool files larger than 1Mb. If you forget this, you'll probably get OutOfMemory error.

Open/close already created RRD file

Not much to say here.. To open RRD file already created on your hard disk just use its filename as RrdDb constructor argument and your file is ready for update/fetch operations:

RrdDb rrd = new RrdDb("rrdfile.rrd");

RrdDb object does not load the whole content of RRD file into memory. Datasource and archive definitions are loaded, but archive values (rows) are read from disk on demand. It's a good practice to keep a reference to an open RRD file if the file is frequently updated, because the opening process takes some time. If your RRD files are updated rarely, open them only when necessary and close them as soon as possible.

It's a good practice to close your RRD file when it is no longer needed:

rrd.close();

If you forget to do this, your RRD file will be eventually closed during the garbage collection process.

RrdDbPool class

Please note that the construction of each RrdDb object is a time consuming process, especially with large RRD files with many datasources and several long archives. It's true that JRobin never loads to complete content of a RRD file into memory. But datasource and archive definitions, its internal states and variables are loaded. With many datasources and archives in a single RRD file this process takes considerable time.

There is one more inportant issue which has to be addressed: in a multithreaded environment you might probably need a reference to the same RRD file from two different threads (for example, RRD file updates are performed in one thread but data fetching and graphing is performed in another one). To make the RrdDb construction process more efficient it might be convenient to open all RRD files in a centralized place. That's the purpose of RrdDbPool class.

How does it work? The typical usage scenario goes like this:

// obtain instance to RrdDbPool object
RrdDbPool pool = RrdDbPool.getInstance();

// request a reference to RrdDb object
String path = "some_relative_or_absolute_path_to_any_RRD_file";
RrdDb rrdDb = pool.requestRrdDb(path);

// reference obtained, do whatever you want with it...
... // (1)

// once you don't need the reference any more, release it!
// DO NOT CALL rrdDb.close()
pool.release(rrdDb);

It's that simple. When the reference is requested for the first time, RrdDbPool will open the RRD file for you and make some internal note that the RRD file is used once. When the reference to the same file (same RRD file path) is requested for the second time, the same RrdDb reference will be returned, and its usage count will be increased by one. When the reference is released its usage count will be decremented by one.

When the reference usage count drops to zero, RrdDbPool will not close the underlying RRD file immediatelly. Instead of it, it will be marked as 'eligible for closing'. If someone requests the same RRD file again (before it gets closed), the same reference will be returned again, otherwise it will be open again.

Let's take a closer look at the code sample above. It is possible that the code marked with (1) triggers an exception. In that case, the release() method never gets called and the usage count for this particular RrdDb reference will never become equal to zero (in other words, the underlying RRD file will never become eligible for closing). It is important that every successful requestRrdDb() method call is matched with a corresponding release() call. A better approach would be to use a try/finally code block:

// obtain instance to RrdDbPool object
RrdDbPool pool = RrdDbPool.getInstance();

// request a reference to RrdDb object
String path = "some_relative_or_absolute_path_to_any_RRD_file";
RrdDb rrdDb = null;
try {
    rrdDb = pool.requestRrdDb(path);

    // reference obtained, do whatever you want with it...
    ... // (1)
} 
finally {
    // once you don't need the reference any more, release it!
    // DO NOT CALL rrdDb.close()
    pool.release(rrdDb); // will succeed, even if rrdDb==null
}

RrdDbPool has a 'garbage collector' which runs in a separate thread and gets activated only when the number of open RRD files is to big (>100, initially). Only RRD files with a reference count equal to zero will be eligible for closing. Unreleased RrdDb references are never invalidated. RrdDbPool keeps track of the time when each RRD file becomes eligible for closing so that the oldest RRD file gets closed first.

Never use RrdDb.close() method on the reference returned from the pool. When the reference is no longer needed, return it to the pool by calling its release() method.

However, you are not forced to use RrdDbPool methods to obtain RrdDb references to RRD files, 'ordinary' RrdDb constructors are still available. But RrdDbPool class offers serious performance improvement especially in complex applications with many threads and many simultaneously open RRD files.

The pool is thread-safe and its initial capacity is set to 100 files. It does not mean that you cannot work with more than 100 RRD files at at the same time. Roughly speaking, it means that the garbage collector (the thread which closes RRD files returned to the pool) will try to close some RRD files no longer in use once this limit is reached. This initial capacity can be changed at runtime (see javadoc for RrdDbPool class).

Update RRD files

To update the RRD file test.rrd created in the first example, RRDTool uses the following pattern:

rrdtool update test.rrd -t input:output timestamp:inputValue:outputValue

or simply:

rrdtool update test.rrd timestamp:inputValue:outputValue

JRobin follows the same pattern:

// open the file
RrdDb rrd = new RrdDb("test.rrd");

// obtain values for timestamp, inputValue and outputValue somehow
long timestamp = ...; // without milliseconds, rounded to the neares second (see notes)
double inputValue = ...;
double outputValue = ...;

// create new, empty data sample using RRD file reference:
Sample sample = rrd.createSample(timestamp);

// put datasource values in your sample
sample.setValue("input", inputValue); // or: sample.setValue(0, inputValue)
sample.setValue("output", outputValue); // or: sample.setValue(1, outputValue);

// update rrd file
sample.update();

Notes:

  • JRobin uses timestamps rounded to the nearest second, without milliseconds (as RRDTool does). If you want to specify 'now' as a sample timestamp, use something like:
long now = (System.currentTimeMillis() + 500L)/ 1000L;
Sample sample = rrd.createSample(now);
// or just:
Sample sampleWithCurrentTimestamp = rrd.createSample();
  • To make things easier, when new Sample object is created all data source values are set to Double.NaN (unknown values). You have to specify only known data source values. However, if you want to be consistent and to specify all values in your java code, note that RRDTool uses U to denote unknown data source values and JRobin uses Double.NaN for that purpose.
  • You can reuse one Sample object as many times you want. Of course, you have to change its timestamp before you call the update() method again (see javadoc for a detailed explanation).
  • The Sample class has several similar methods to set datasource values, check javadoc for full explanation.

Get last update time

To obtain last update time from RRD file, RRDTool uses the following syntax:

rrdtool last file.rrd

It is easy to perform this task with JRobin:

// open the file
RrdDb rrd = new RrdDb("test.rrd");
// get last update time
long timestamp = rrd.getLastUpdateTime();

Timestamp is returned in seconds, without milliseconds.

Fetch data from RRD files

To fetch data from the RRD file RRDTool uses the following syntax (simplified):

rrdtool fetch file.rrd AVERAGE|MIN|MAX|LAST --start startTime --end endTime

For example:

rrdtool fetch test.rrd AVERAGE --start 100100000 --end 100200000

Once again, JRobin follows almost the same pattern:

// open the file
RrdDb rrd = new RrdDb("test.rrd");

// create fetch request using the database reference
FetchRequest request = rrd.createFetchRequest("AVERAGE", 100100000L, 100200000L);

// execute the request
FetchData fetchData = request.fetchData();

// From a logical point of view FetchData object is, simply, 
// an array of FetchPoint objects. Each FetchPoint object 
// represents all datasource values for the specific timestamp.
// Here is how you can produce the same output to stdout 
// as RRDTool's fetch command
for(int i = 0; i < fetchData.getRowCount(); i++) {
    FetchPoint point = fetchData.getRow(i);
    System.out.println(point.dump());
}

Each FetchPoint object has associated timestamp and an array of double values, one for each data source defined in the RRD file. Use getTime() method on a FetchPoint object to obtain the timestamp, and getValue(i) to obtain the value of the i-th data source (see javadoc for the complete explanation).

The FetchData object provides much faster methods to read fetched data column by column. To obtain timestamps for all returned rows use:

long[] timestamps = fetchData.getTimestamps();
To obtain all values of the datasource 'input' which correspond to these timestamps, use:

double[] inputValues = fetchData.getValues("input");

Filtered fetching

If you are already familiar with RRDTool, you know that RRDTool fetch command fetches data for all datasources defined within a RRD file. No problem for a command line utility which prints data to stdout, but this could be a significant performance penalty if you want to fetch data only for a few datasources. JRobin provides simple and efficent mechanisam to get archived values only for selected datasources. For example, if you have a RRD file named 'test.rrd' with three datasources, 'input', 'output' and 'lost', but want to fetch data only for the first two, use the following procedure:

// open the file
RrdDb rrd = new RrdDb("test.rrd");

// create fetch request using the database reference
FetchRequest request = rrd.createFetchRequest("AVERAGE", 100100000L, 100200000L);

// filter the datasources you really need
String[] wantedDatasources = {"input", "output"};
request.setFilter(wantedDatasources);
// if you want only the "input" datasource use:
// request.setFilter("input");

// execute the request
FetchData fetchData = request.fetchData();

...and you save approximately 1/3 of execution time!

Notes:

  • assuming that you have a RRDTool and a JRobin file with the same data in it, JRobin's fetchData() method will return exactly the same number of rows with the same timestamps and values as RRDTool's fetch command would do (for the same consolidation function and the same time span).
  • unknown data source values are represented with Double.NaNs.
  • RRDTool's fetch --resolution parameter is also supported in the JRobin API (see javadoc how to specify fetch resolution)

How to dump RRD file to XML format

RRDTool has a nice utility to view the content of a RRD file:

rrdtool dump file.rrd > file.xml

It is easy to dump the content of your JRobin RRD file in XML format, too:

RrdDb rrd = new RrdDb("file.rrd");
System.out.println(rrd.getXml());

...or to save this XML output to a file:

RrdDb rrd = new RrdDb("file.rrd");
rrd.dumpXml("file.xml");

...or to use XML as a tool for RRD file replication:

RrdDb rrdOriginal = new RrdDb("original.rrd");
rrd.dumpXml("original.xml");
RrdDb rrdCopy = new RrdDb("copy.rrd", "original.xml");

Note:

  • JRobin uses the same XML format as RRDTool. You can use this XML format to convert JRobin files to RRDTool files and vice versa.
  • XML export is much faster and uses less memory than XML import.

RRDToolkit class

Chances are that your RRD files will serve for the purpose initially, but you might understand later that your RRD creation parameters were not good enough. For example, one of your RRA archives could be too small with wrong X-files factor, some datasource could have inadequate heartbeat or maximum value specified at creation time. And if you want to add one datasource and several more archives to your RRD file created many months ago, things will become even worse.

Since version 1.3.0, JRobin has a RrdToolkit class. Use an instance of this class to:

  • add datasource(s)/archive(s) to an already created RRD file;
  • remove datasource(s)/archive(s) from an already created RRD file;
  • change datasource attributes (heartbeat, minimum and maximum value);
  • change archive attributes (X-files factor, number of rows)

All these operations can be performed on a copy of the original RRD file, or on the original file itself (with possible backup file creation). Inspector utility relies on the RrdToolkit functionality heavily, you can use it to play with your RRD files.

XPORT-ing RRD data from JRobin files

JRobin now supports data export functionality, much like the original RRDtool does, with a few extra touches ofcourse.

Since 1.4.0 export has been introduced in the form of RrdExport, RrdExportDef and ExportData classes. It works very much like graphing does, here's a quick example:

// Create a def for export specification
RrdExportDef def = new RrdExportDef( startTime, endTime );
def.datasource( "ds", "test.rrd", "datasource", "AVERAGE" );
def.export( "ds", "A single datasource" );

// Pass the def to the exporter, fetch the export data, but no more than 50 rows
// If you don't specify the maximum number of rows to retrieve, 400 will be used
RrdExport export = new RrdExport( def );
ExportData data = export.fetch( 50 );

// Dump the exported data to XML in a file
data.exportXml( "export-data.xml" );

// Or request a value directly from the exported data
double value = data.getAggregate( "ds", "MAX" );

// We can also request the value as a string, much like with GPRINT
System.out.println( data.print( "ds", "MAX", "Maximum: @5.2" ) );

The ExportData class corresponds to the public RrdDataSet interface, as does the FetchData class (the result of a direct fetch on an RRD). The RrdDataSet interface gives you a number of useful methods that can be used to retrieve and calculate data or for example pass it on to an external graphing or charting library.

UNLIKE with RRDtool, it is not necessary to specify what data to export, by default all datasources requested will be exported. However, as soon you specify a datasource to export, using the export() construct, by default the system will revert to the so called 'strict' export like RRDtool, and only export the datasources you explicitly specified. You can override this behaviour by calling setStrictExport(boolean) method manually. If you disable strict export, you can set legends for only a few datasources, but still have all datasources in the ExportData set.

The RrdExportDef is a subset of the RrdGraphDef and introduces a new XML Template very much alike. An example:

<rrd_export_def>
  <span>
    <start>${start}</start>
    <end>${end}</end>
  </span>
  <datasources>
    <def>
      <name>bytesIn</name>
      <rrd>${rrd}</rrd>
      <source>ifInOctets</source>
      <cf>AVERAGE</cf>
    </def>      
    <def>
      <name>bitsIn</name>
      <rpn>bytesIn,8,*</rpn>
    </def>
  </datasources>
  <exports>
    <export>
      <datasource>bitsIn</datasource>
      <legend>Bits Incoming</legend>
    </export>
    <export>
      <datasource>bytesIn</datasource>
      <legend>Bytes Incoming</legend>
     </export>
  </exports>
</rrd_export_def>

An ExportData object can directly be recreated from an export XML file, like:

ExportData data = new ExportData( new File("export-data.xml"), true );

The second parameter determines if the legends in the export XML should be used to name the datasources. If set to false, the datasources will get named sequentially of the form 'd1', 'd2'... unless you specify a different prefix in the ExportData constructor, like in:

ExportData data = new ExportData( new File("export-data.xml"), "mydata");
// The imported datasources will be named 'mydata1', 'mydata2' ...

Because of the XML import functionality, you can easily import regular RRDtool XPORT xml data into a JRobin ExportData set. Furthermore, ExportData can be set as a datasource in a normal RrdGraphDef, allowing you to generate graphs directly from RRDtool exported data. Simply add a number of ExportData objects to a RrdGraphDef:

RrdGraphDef graphDef = new RrdGraphDef();
...
graphDef.addExportData( exportData1 );
graphDef.addExportData( exportData2 );
...

And there is the RrdGraphDefTemplate XML variation:

<datasources>
  ...
  <export_data>
    <file>/export_data/exportdata1.xml</file>
    <ds_name_prefix>traffic</ds_name_prefix>
  </export_data>

  <export_data>
    <file>/export_data/exportdata2.xml</file>
    <use_legend_names>true</use_legend_names>
  </export_data>

  <export_data>
    <file>/export_data/exportdata3.xml</file>
  </export_data>
  ...
</datasources>

Every RrdGraphDef is also a RrdExportDef. This means you can pass a RrdGraphDef directly to an RrdExport, you can generate the ExportData from the corresponding graph. However, if you are interested in both generating the graph and retrieving the ExportData, it is better to use the export functionality of RrdGraph, in order to generate the data only once. A simple example:

RrdGraphDef graphDef = new RrdGraphDef( startTime, endTime );
...
RrdGraph graph = new RrdGraph( graphDef );

// Save the graph image
graph.saveAsPNG( graphFile );

// Get the export data
ExportData data = graph.getExportData();

// Do something with the exported data
// Store max values in a relational database for example.

Note: RrdGraph also contains two fetchExportData methods. These will calculate the ExportData again based on the passed RrdGraphDef. Use the getExportData() method after creating the image to save a calculation roundtrip, it can save you many CPU cycles, use fetchExportData() if you want to retrieve the export data without generating the graph, or if you want to re-generate it after changing the graphDef or with a lower maxRows limit.

The workings of JRobin export are demonstrated in the combined ExportDemo. The ExportDemo is actually two different apps: ExportExportDemo, that exports data from two rrd files and dumps the result in some XML files:

java -cp jrobin-demo-1.4.0.jar org.jrobin.demo.graph.ExportExportDemo

...and the ExportImportDemo that reads in the output from ExportExportDemo and generates a graph from it:

java -cp jrobin-demo-1.4.0.jar org.jrobin.demo.graph.ExportImportDemo

Lock RRD files for exclusive access

JRobin allowes simultaneous access to a single RRD file from different RrdDb objects by default. This guarantees the fastest fetch/update operations, but could possibly lead to inconsistent RRD files or incorrectly fetched data. In other words, the underlying RRD file is not locked in any way - two different RrdDb objects could modify it at the same time. If you are not aware of this possibility, you can damage your RRD files or fetch strange looking data out of it.

However, if you want to be sure that only one RrdDb object may access the underlying RRD file, you can change the locking mode of JRobin:

RrdDb.setLockMode(RrdDb.WAIT_IF_LOCKED); 

In this mode, underlying RRD files are locked for exclusive access. Any RrdDb object which tries to access already locked RRD file will have to wait until the lock is released (possibly forever). The lock is released when the RrdDb object which holds the lock calls its close() method. If using this locking mode, be sure to close your RrdDb objects as soon as possible (don't keep them open for a long time).

The other possibility is to generate an IOException whenever some RrdDb object tries to access already locked RRD file:

RrdDb.setLockMode(RrdDb.EXCEPTION_IF_LOCKED);

Default locking behaviour (no locks used, fastest possible access) can be restored with the following statement:

RrdDb.setLockMode(RrdDb.NO_LOCKS);

Note that JRobin works much slower when locks are used (up to 50%).

JRobin RRD file structure

Here is a short (but detailed) explanation of the RRD file structure used by JRobin.

Each JRobin RRD file has a single header followed by N datasource segments followed by M archive segments. Once created, RRD file has constant size.

JRobin RRD file:
    Header
    Datasource1
    Datasource2
    ...
    DatasourceN 
    Archive1
    Archive2
    ...
    ArchiveM 
    
Header:
    signature (string*)
    step (long**)
    N = number of datasources (integer***)
    M = number of archives (integer)
    last update time (long)
    
Datasource:
    datasource name (string)
    datasource type (string)
    heartbeat (long)
    min value (double****)
    max value (double)
    last value (double)
    accumulated value (double)
    NaN seconds (long)
    
Archive:
    consolidation function (string)
    X-files factor (double)
    number of archive steps (integer)
    R = number of archive rows (integer)
        archive state1
        round robin1
        archive state2
        round robin2
        ...
        archive stateN 
        round robinN 

Archive state:
    accumulated value (double)
    NaN steps (long)

Round robin:
    pointer (integer)
    value1 (double)
    value2 (double)
    ...
    valueR (double)

*JRobin strings are 20 characters long (if shorter, strings are padded with blanks). Each character is represented with two Unicode bytes. It means that each string in a RRD file takes 40 bytes of diskspace.
**Long primitive takes 8 bytes, high byte comes first.
***Integer primitive takes 4 bytes, high byte comes first.
****Double primitive takes 8 bytes. Before stored, it gets converted to a long using the doubleToLongBits method in class Double.

Total RRD file size

The size of any JRobin RRD file can be calculated from the following expression:

TOTAL SIZE = 64 + 128 * N + 56 * M + 20 * N * M + 8 * N * R
N = number of datasources
M = number of archives
R = total number of rows in all archives

Example: Suppose that we want to create a new RRD file with two datasources:

DS:in:GAUGE:600:U:U
DS:out:GAUGE:600:U:U

and eight archives:

RRA:AVERAGE:0.5:1:600
RRA:AVERAGE:0.5:6:700
RRA:AVERAGE:0.5:24:775
RRA:AVERAGE:0.5:288:797
RRA:MAX:0.5:1:600
RRA:MAX:0.5:1:700
RRA:MAX:0.5:1:775
RRA:MAX:0.5:1:797

Consequently, we have:

N = 2
M = 8
R = (600 + 700 + 775 + 797) * 2 = 5744
TOTAL SIZE = 64 + 128 * 2 + 56 * 8 + 20 * 2 * 8 + 8 * 2 * 5744 
           = 92992 bytes.

Back to the top

Copyright © 2003, 2004 Sasa Markovic & Arne Vandamme. All Rights Reserved.