![]() |
|||||||||||||||||
|
Core APICreate new RRD fileJRobin RRD files are managed through objects of RrdDb class. New RRD file can be created:
Create new RRD file from the scratchSuppose you want to create a JRobin RRD file equivalent to the following RRDTool file (two COUNTER datasources, input and output): rrdtool create test.rrd --step 300 \ DS:input:COUNTER:600:0:U \ DS:output:COUNTER:600:0:U \ RRA:AVERAGE:0.5:1:600 \ RRA:AVERAGE:0.5:6:700 \ RRA:AVERAGE:0.5:24:797 \ RRA:AVERAGE:0.5:288:775 \ RRA:MAX:0.5:1:600 \ RRA:MAX:0.5:6:700 \ RRA:MAX:0.5:24:797 \ RRA:MAX:0.5:288:775 JRobin API follows exactly the same pattern:: // create new RRD definition RrdDef def = new RrdDef("test.rrd", 300); def.addDatasource("input", "COUNTER", 600, 0, Double.NaN); def.addDatasource("output", "COUNTER", 600, 0, Double.NaN); def.addArchive("AVERAGE", 0.5, 1, 600); def.addArchive("AVERAGE", 0.5, 6, 700); def.addArchive("AVERAGE", 0.5, 24, 797); def.addArchive("AVERAGE", 0.5, 288, 775); def.addArchive("MAX", 0.5, 1, 600); def.addArchive("MAX", 0.5, 6, 700); def.addArchive("MAX", 0.5, 24, 797); def.addArchive("MAX", 0.5, 288, 775); // RRD file definition is completed, create the database! RrdDb rrd = new RrdDb(def); // by this point, file "test.rrd" can be found on your disk Notes:
Create new RRD file from RRDTool's XML fileSuppose that you have a RRD file named original.rrd created with RRDTool. It is possible to create a JRobin file with the same data in it. First, dump the content of original.rrd to XML file with the following command: rrdtool dump original.rrd > original.xml The file original.xml can be used to create a JRobin file from the Java code: RrdDb rrd = new RrdDb("copy.rrd", "original.xml"); The file copy.rrd is a JRobin equivalent of the original.rrd file. It's that simple but take care that the import process takes considerable time and memory. As a rule of thumb, increase the ammount of memory available to JVM (use switch like -Xmx128m) whenever importing RRDTool files larger than 1Mb. If you forget this, you'll probably get OutOfMemory error. Open/close already created RRD fileNot much to say here.. To open RRD file already created on your hard disk just use its filename as RrdDb rrd = new RrdDb("rrdfile.rrd");
It's a good practice to close your RRD file when it is no longer needed: rrd.close(); If you forget to do this, your RRD file will be eventually closed during the garbage collection process. RrdDbPool classPlease note that the construction of each RrdDb object is a time consuming process, especially with large RRD files with many datasources and several long archives. It's true that JRobin never loads to complete content of a RRD file into memory. But datasource and archive definitions, its internal states and variables are loaded. With many datasources and archives in a single RRD file this process takes considerable time. There is one more inportant issue which has to be addressed: in a multithreaded environment you might probably need a reference to the same RRD file from two different threads (for example, RRD file updates are performed in one thread but data fetching and graphing is performed in another one). To make the RrdDb construction process more efficient it might be convenient to open all RRD files in a centralized place. That's the purpose of RrdDbPool class. How does it work? The typical usage scenario goes like this: // obtain instance to RrdDbPool object RrdDbPool pool = RrdDbPool.getInstance(); // request a reference to RrdDb object String path = "some_relative_or_absolute_path_to_any_RRD_file"; RrdDb rrdDb = pool.requestRrdDb(path); // reference obtained, do whatever you want with it... ... // (1) // once you don't need the reference any more, release it! // DO NOT CALL rrdDb.close() pool.release(rrdDb); It's that simple. When the reference is requested for the first time, RrdDbPool will open the RRD file for you and make some internal note that the RRD file is used once. When the reference to the same file (same RRD file path) is requested for the second time, the same RrdDb reference will be returned, and its usage count will be increased by one. When the reference is released its usage count will be decremented by one. When the reference usage count drops to zero, RrdDbPool will not close the underlying RRD file immediatelly. Instead of it, it will be marked as 'eligible for closing'. If someone requests the same RRD file again (before it gets closed), the same reference will be returned again, otherwise it will be open again. Let's take a closer look at the code sample above. It is possible that the code marked with (1) triggers an exception. In that case, the // obtain instance to RrdDbPool object RrdDbPool pool = RrdDbPool.getInstance(); // request a reference to RrdDb object String path = "some_relative_or_absolute_path_to_any_RRD_file"; RrdDb rrdDb = null; try { rrdDb = pool.requestRrdDb(path); // reference obtained, do whatever you want with it... ... // (1) } finally { // once you don't need the reference any more, release it! // DO NOT CALL rrdDb.close() pool.release(rrdDb); // will succeed, even if rrdDb==null } RrdDbPool has a 'garbage collector' which runs in a separate thread and gets activated only when the number of open RRD files is to big (>100, initially). Only RRD files with a reference count equal to zero will be eligible for closing. Unreleased RrdDb references are never invalidated. RrdDbPool keeps track of the time when each RRD file becomes eligible for closing so that the oldest RRD file gets closed first. Never use RrdDb.close() method on the reference returned from the pool. When the reference is no longer needed, return it to the pool by calling its release() method. However, you are not forced to use RrdDbPool methods to obtain RrdDb references to RRD files, 'ordinary' RrdDb constructors are still available. But RrdDbPool class offers serious performance improvement especially in complex applications with many threads and many simultaneously open RRD files. The pool is thread-safe and its initial capacity is set to 100 files. It does not mean that you cannot work with more than 100 RRD files at at the same time. Roughly speaking, it means that the garbage collector (the thread which closes RRD files returned to the pool) will try to close some RRD files no longer in use once this limit is reached. This initial capacity can be changed at runtime (see javadoc for RrdDbPool class). Update RRD filesTo update the RRD file test.rrd created in the first example, RRDTool uses the following pattern: rrdtool update test.rrd -t input:output timestamp:inputValue:outputValue or simply: rrdtool update test.rrd timestamp:inputValue:outputValue JRobin follows the same pattern: // open the file RrdDb rrd = new RrdDb("test.rrd"); // obtain values for timestamp, inputValue and outputValue somehow long timestamp = ...; // without milliseconds, rounded to the neares second (see notes) double inputValue = ...; double outputValue = ...; // create new, empty data sample using RRD file reference: Sample sample = rrd.createSample(timestamp); // put datasource values in your sample sample.setValue("input", inputValue); // or: sample.setValue(0, inputValue) sample.setValue("output", outputValue); // or: sample.setValue(1, outputValue); // update rrd file sample.update(); Notes:
long now = (System.currentTimeMillis() + 500L)/ 1000L; Sample sample = rrd.createSample(now); // or just: Sample sampleWithCurrentTimestamp = rrd.createSample();
Get last update timeTo obtain last update time from RRD file, RRDTool uses the following syntax: rrdtool last file.rrd It is easy to perform this task with JRobin: // open the file RrdDb rrd = new RrdDb("test.rrd"); // get last update time long timestamp = rrd.getLastUpdateTime(); Timestamp is returned in seconds, without milliseconds. Fetch data from RRD filesTo fetch data from the RRD file RRDTool uses the following syntax (simplified): rrdtool fetch file.rrd AVERAGE|MIN|MAX|LAST --start startTime --end endTime For example: rrdtool fetch test.rrd AVERAGE --start 100100000 --end 100200000 Once again, JRobin follows almost the same pattern: // open the file RrdDb rrd = new RrdDb("test.rrd"); // create fetch request using the database reference FetchRequest request = rrd.createFetchRequest("AVERAGE", 100100000L, 100200000L); // execute the request FetchData fetchData = request.fetchData(); // From a logical point of view FetchData object is, simply, // an array of FetchPoint objects. Each FetchPoint object // represents all datasource values for the specific timestamp. // Here is how you can produce the same output to stdout // as RRDTool's fetch command for(int i = 0; i < fetchData.getRowCount(); i++) { FetchPoint point = fetchData.getRow(i); System.out.println(point.dump()); } Each FetchPoint object has associated timestamp and an array of double values, one for each data source defined in the RRD file. Use getTime() method on a FetchPoint object to obtain the timestamp, and getValue(i) to obtain the value of the i-th data source (see javadoc for the complete explanation). The FetchData object provides much faster methods to read fetched data column by column. To obtain timestamps for all returned rows use: long[] timestamps = fetchData.getTimestamps();To obtain all values of the datasource 'input' which correspond to these timestamps, use:
double[] inputValues = fetchData.getValues("input"); Filtered fetchingIf you are already familiar with RRDTool, you know that RRDTool fetch command fetches data for all datasources defined within a RRD file. No problem for a command line utility which prints data to stdout, but this could be a significant performance penalty if you want to fetch data only for a few datasources. JRobin provides simple and efficent mechanisam to get archived values only for selected datasources. For example, if you have a RRD file named 'test.rrd' with three datasources, 'input', 'output' and 'lost', but want to fetch data only for the first two, use the following procedure: // open the file RrdDb rrd = new RrdDb("test.rrd"); // create fetch request using the database reference FetchRequest request = rrd.createFetchRequest("AVERAGE", 100100000L, 100200000L); // filter the datasources you really need String[] wantedDatasources = {"input", "output"}; request.setFilter(wantedDatasources); // if you want only the "input" datasource use: // request.setFilter("input"); // execute the request FetchData fetchData = request.fetchData(); ...and you save approximately 1/3 of execution time! Notes:
How to dump RRD file to XML formatRRDTool has a nice utility to view the content of a RRD file: rrdtool dump file.rrd > file.xml It is easy to dump the content of your JRobin RRD file in XML format, too: RrdDb rrd = new RrdDb("file.rrd"); System.out.println(rrd.getXml()); ...or to save this XML output to a file: RrdDb rrd = new RrdDb("file.rrd"); rrd.dumpXml("file.xml"); ...or to use XML as a tool for RRD file replication: RrdDb rrdOriginal = new RrdDb("original.rrd"); rrd.dumpXml("original.xml"); RrdDb rrdCopy = new RrdDb("copy.rrd", "original.xml"); Note:
RRDToolkit classChances are that your RRD files will serve for the purpose initially, but you might understand later that your RRD creation parameters were not good enough. For example, one of your RRA archives could be too small with wrong X-files factor, some datasource could have inadequate heartbeat or maximum value specified at creation time. And if you want to add one datasource and several more archives to your RRD file created many months ago, things will become even worse. Since version 1.3.0, JRobin has a RrdToolkit class. Use an instance of this class to:
All these operations can be performed on a copy of the original RRD file, or on the original file itself (with possible backup file creation). Inspector utility relies on the RrdToolkit functionality heavily, you can use it to play with your RRD files. XPORT-ing RRD data from JRobin filesJRobin now supports data export functionality, much like the original RRDtool does, with a few extra touches ofcourse. Since 1.4.0 export has been introduced in the form of RrdExport, RrdExportDef and ExportData classes. It works very much like graphing does, here's a quick example: // Create a def for export specification RrdExportDef def = new RrdExportDef( startTime, endTime ); def.datasource( "ds", "test.rrd", "datasource", "AVERAGE" ); def.export( "ds", "A single datasource" ); // Pass the def to the exporter, fetch the export data, but no more than 50 rows // If you don't specify the maximum number of rows to retrieve, 400 will be used RrdExport export = new RrdExport( def ); ExportData data = export.fetch( 50 ); // Dump the exported data to XML in a file data.exportXml( "export-data.xml" ); // Or request a value directly from the exported data double value = data.getAggregate( "ds", "MAX" ); // We can also request the value as a string, much like with GPRINT System.out.println( data.print( "ds", "MAX", "Maximum: @5.2" ) ); The ExportData class corresponds to the public RrdDataSet interface, as does the FetchData class (the result of a direct fetch on an RRD). The RrdDataSet interface gives you a number of useful methods that can be used to retrieve and calculate data or for example pass it on to an external graphing or charting library. UNLIKE with RRDtool, it is not necessary to specify what data to export, by default all datasources requested will be exported. However, as soon you specify a datasource to export, using the export() construct, by default the system will revert to the so called 'strict' export like RRDtool, and only export the datasources you explicitly specified. You can override this behaviour by calling setStrictExport(boolean) method manually. If you disable strict export, you can set legends for only a few datasources, but still have all datasources in the ExportData set. The RrdExportDef is a subset of the RrdGraphDef and introduces a new XML Template very much alike. An example: <rrd_export_def> <span> <start>${start}</start> <end>${end}</end> </span> <datasources> <def> <name>bytesIn</name> <rrd>${rrd}</rrd> <source>ifInOctets</source> <cf>AVERAGE</cf> </def> <def> <name>bitsIn</name> <rpn>bytesIn,8,*</rpn> </def> </datasources> <exports> <export> <datasource>bitsIn</datasource> <legend>Bits Incoming</legend> </export> <export> <datasource>bytesIn</datasource> <legend>Bytes Incoming</legend> </export> </exports> </rrd_export_def> An ExportData object can directly be recreated from an export XML file, like: ExportData data = new ExportData( new File("export-data.xml"), true ); The second parameter determines if the legends in the export XML should be used to name the datasources. If set to false, the datasources will get named sequentially of the form 'd1', 'd2'... unless you specify a different prefix in the ExportData constructor, like in: ExportData data = new ExportData( new File("export-data.xml"), "mydata"); // The imported datasources will be named 'mydata1', 'mydata2' ... Because of the XML import functionality, you can easily import regular RRDtool XPORT xml data into a JRobin ExportData set. Furthermore, ExportData can be set as a datasource in a normal RrdGraphDef, allowing you to generate graphs directly from RRDtool exported data. Simply add a number of ExportData objects to a RrdGraphDef: RrdGraphDef graphDef = new RrdGraphDef(); ... graphDef.addExportData( exportData1 ); graphDef.addExportData( exportData2 ); ... And there is the RrdGraphDefTemplate XML variation: <datasources> ... <export_data> <file>/export_data/exportdata1.xml</file> <ds_name_prefix>traffic</ds_name_prefix> </export_data> <export_data> <file>/export_data/exportdata2.xml</file> <use_legend_names>true</use_legend_names> </export_data> <export_data> <file>/export_data/exportdata3.xml</file> </export_data> ... </datasources> Every RrdGraphDef is also a RrdExportDef. This means you can pass a RrdGraphDef directly to an RrdExport, you can generate the ExportData from the corresponding graph. However, if you are interested in both generating the graph and retrieving the ExportData, it is better to use the export functionality of RrdGraph, in order to generate the data only once. A simple example: RrdGraphDef graphDef = new RrdGraphDef( startTime, endTime ); ... RrdGraph graph = new RrdGraph( graphDef ); // Save the graph image graph.saveAsPNG( graphFile ); // Get the export data ExportData data = graph.getExportData(); // Do something with the exported data // Store max values in a relational database for example. Note: RrdGraph also contains two fetchExportData methods. These will calculate the ExportData again based on the passed RrdGraphDef. Use the getExportData() method after creating the image to save a calculation roundtrip, it can save you many CPU cycles, use fetchExportData() if you want to retrieve the export data without generating the graph, or if you want to re-generate it after changing the graphDef or with a lower maxRows limit. The workings of JRobin export are demonstrated in the combined ExportDemo. The ExportDemo is actually two different apps: ExportExportDemo, that exports data from two rrd files and dumps the result in some XML files: java -cp jrobin-demo-1.4.0.jar org.jrobin.demo.graph.ExportExportDemo ...and the ExportImportDemo that reads in the output from ExportExportDemo and generates a graph from it: java -cp jrobin-demo-1.4.0.jar org.jrobin.demo.graph.ExportImportDemo Lock RRD files for exclusive accessJRobin allowes simultaneous access to a single RRD file from different RrdDb objects by default. This guarantees the fastest fetch/update operations, but could possibly lead to inconsistent RRD files or incorrectly fetched data. In other words, the underlying RRD file is not locked in any way - two different RrdDb objects could modify it at the same time. If you are not aware of this possibility, you can damage your RRD files or fetch strange looking data out of it. However, if you want to be sure that only one RrdDb object may access the underlying RRD file, you can change the locking mode of JRobin: RrdDb.setLockMode(RrdDb.WAIT_IF_LOCKED);
In this mode, underlying RRD files are locked for exclusive access. Any RrdDb object which tries to access already locked RRD file will have to wait until the lock is released (possibly forever). The lock is released when the RrdDb object which holds the lock calls its The other possibility is to generate an IOException whenever some RrdDb object tries to access already locked RRD file: RrdDb.setLockMode(RrdDb.EXCEPTION_IF_LOCKED); Default locking behaviour (no locks used, fastest possible access) can be restored with the following statement: RrdDb.setLockMode(RrdDb.NO_LOCKS); Note that JRobin works much slower when locks are used (up to 50%). JRobin RRD file structureHere is a short (but detailed) explanation of the RRD file structure used by JRobin. Each JRobin RRD file has a single header followed by N datasource segments followed by M archive segments. Once created, RRD file has constant size. JRobin RRD file: Header Datasource1 Datasource2 ... DatasourceN Archive1 Archive2 ... ArchiveM Header: signature (string*) step (long**) N = number of datasources (integer***) M = number of archives (integer) last update time (long) Datasource: datasource name (string) datasource type (string) heartbeat (long) min value (double****) max value (double) last value (double) accumulated value (double) NaN seconds (long) Archive: consolidation function (string) X-files factor (double) number of archive steps (integer) R = number of archive rows (integer) archive state1 round robin1 archive state2 round robin2 ... archive stateN round robinN Archive state: accumulated value (double) NaN steps (long) Round robin: pointer (integer) value1 (double) value2 (double) ... valueR (double) *JRobin strings are 20 characters long (if shorter, strings are padded with blanks). Each character is represented with two Unicode bytes. It means that each string in a RRD file takes 40 bytes of diskspace. Total RRD file sizeThe size of any JRobin RRD file can be calculated from the following expression: TOTAL SIZE = 64 + 128 * N + 56 * M + 20 * N * M + 8 * N * R N = number of datasources M = number of archives R = total number of rows in all archives Example: Suppose that we want to create a new RRD file with two datasources: DS:in:GAUGE:600:U:U DS:out:GAUGE:600:U:U and eight archives: RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:600 RRA:MAX:0.5:1:700 RRA:MAX:0.5:1:775 RRA:MAX:0.5:1:797 Consequently, we have: N = 2 M = 8 R = (600 + 700 + 775 + 797) * 2 = 5744 TOTAL SIZE = 64 + 128 * 2 + 56 * 8 + 20 * 2 * 8 + 8 * 2 * 5744 = 92992 bytes. Copyright © 2003, 2004 Sasa Markovic & Arne Vandamme. All Rights Reserved. |