liboml2(1)

liboml2 - OML2 client library

liboml2 is the client library for OML2. It provides an API for application writers to collect measurements from their applications via user-defined Measurement Points (MPs). It also provides a flexible filtering and collection mechanism that allows application users to customize how measurements are processed and stored by an OML-enabled application. Data injected into an MP by the application is serialised into a Measurement Stream (MS), and sent to a Collection Point (e.g., an oml2-server(1) or a file). Measurement samples can also be filtered, pre-processed or aggregated using a set of built-in filters.

This man page documents the command-line user configuration of OML-enabled applications. Application writers who want to learn how to write an application using the liboml2 API should consult liboml2(3). OML also supports an external configuration file, documented in liboml2.conf(5).

When an OML application starts up, it passes the command line arguments to liboml2, which scans them for options that it understands, and uses those to configure itself. liboml2 then removes its own options from the command line so that the application proper does not get confused by them. All OML options start with the prefix --oml-. The --oml-help option lists all the known OML command line options.

The command line options recognized by liboml2 allow for a basic configuration of the measurement library. They are adequate for testing and debugging, but do not offer as much flexibility as the external configuration file format supports. You can pass a configuration file to liboml2 using the --oml-config option. For details, see liboml2.conf(5).

By default, any instrumentation starts with two OML-specific MPs. The first one, _experiment_metadata, contains information about the experiment currently running, as well as application specific details that would not justify a full MP on their own such as version, command line invokation, or unit or precision of some fields. The second automatic MP is _client_instrumentation. When enabled (see option --oml-instr-interval in OML OPTIONS below), the application will periodically inject status report into that stream. Status reports contain the following information

The liboml2 can process measurement tuples in-line before they are sent to a collection point. The filtering mechanism allows to select a subset of the fields of an MP which need to be sent along, apply aggregation or summary functions on them rather than sending them verbatim, and set the periodicity (time- or sample- based) at which aggregate reports are sent (see --oml-interval or --oml-samples in OML OPTIONS below).

When no configuration file is given, liboml2 provides a basic set of filters for each MP, and sends measurements to just one location (using the --oml-collect option). For each measurement point, each element of the measurement point’s injected tuple is given its own filter. The filter created depends on the type of the element and the current sampling policy.

For instance, suppose a measurement point defined with a measurement tuple as follows:

Then liboml2 will create a separate filter for each of source, destination, length, snr, and name. The filters for the first four numeric elements will be an averaging filter (filter type avg), and the last string element will be given a first filter. The first filter keeps the first injected value in the current sampling period and throws away all others, passing the first value on to the measurement output stage.

The exception to this rule is that if --oml-samples was given, and if the argument is 1, then numeric elements are filtered using the first filter, not the averaging filter.

For more information on OML filters and how they are configured, see the documentation for the configuration file format, liboml2.conf(5). For more information on measurement points and how they are defined, see liboml2(3) and omlc_add_mp(3).

Measurement data is serialised using the OML Measurement Stream Protocol (OMSP). OMSP has two variant: binary marshalling, and text mode. The former is presumably more compact, but the latter is easier to read and generate.

Where MSs generated by an OML instrumentation are sent depends on the collection URI (see URI FORMAT below). If using the tcp scheme, it is sent to the named oml2-server(1), for storage in an SQL database, while using the file or flush schemes will create a local file with the given name containing OML text mode protocol which can later on be streamed (e.g., using nc(1)) to an oml2-server. Measurement points are created with a schema, as above, a schema being an ordered list of (name, type) pairs.

OML filters also generate output with a declared schema. For each measurement point, liboml2 generates a single output measurement that is the union of the outputs of all filters attached to the MP. The names of the fields (or columns) of the schema are derived from the names of the original MP fields, and the output schemas of the filters. The schemas can be observed directly with the file collection URI schemes (e.g., file:-) output (identical schemas are sent to the server when the tcp scheme is used). For instance, here is the output schema for a simple example MP that measures a string (label) and an integer (seq_no):

The schema name is generator_lin — a combination of the application name (generator) and the measurement point name (lin). (The number 1 on this line is an index used in the output columns to identify a line of measurement with the schema to which it conforms.) This output was made using --oml-samples 1 on the command line. This creates a first filter for both of the fields of the measurement point. The first filter outputs a single value that has the same type as the filter’s input.

If we change the command line to use --oml-samples 2, then an averaging filter is used for the numeric seq_no field (label is unchanged). The schema therefore changes as well:

(liboml2 always emits schemas on a single line, but this schema is split into several for readability.)

An avg filter picks one field of the MP to filter (in this case seq_no) and then produces a 3-tuple as output (avg, min, max). Therefore liboml2 creates a schema for this filter output that looks like:

This is the general pattern for filters: their output schemas are formed by appending the name of the source MP with the name of the filter output field. (The first filter is an exception in that it just takes the name of the input field and uses that as the output field name.)

When a tcp collection URI is specified, the received oml2-server creates a database table for each measurement point using the combined OML output schema as schema for the table. For instance, the above example would translate to an SQL CREATE TABLE statement like:

Note that even though an MP field may have an integral type, it may be represented as a floating point type in the output because the filter may output floating point values. For instance, the average of a set of integers is real valued because of the division in the averaging operation.

Another situation may arise when a table by the same name, but with a different schema, already exists in the target database. In this case, the oml2-server(1) tries to incrementally number the new table’s name (e.g., mymp_2, mymp_3,… for stream mymp). It makes 9 attempts at renaming the table before failing. Indeed, it would be unwise for experiments that use a large number of instances of the same application with different filter configuration to not explicitly name each of them. This can be done through the use of a configuration file, with these rename attribute of the relevant mp element, see liboml2.conf(5).

The OML client library can be instructed (see URI FORMAT below) to compress its data streams (using the GZip algorithm). In this case, all input data, be it text or binary OMSP.

Currently, OML leaves it up to the zlib(3) deflate() algorithm to decide when to output compressed data, affording a good compression ratio, particularly for the text mode OMSP. However, this means that some delay will generally be introduced between the time when a sample is injected, and that when a compressed block containing it is output towards the collection point. It is therefore not recommended to use compression for applications where (near) real-time collection of data is required.

Note: for compressed streams to be successfully received by the collection point, it needs to have support for this feature. In the case of the oml2-server(1), this means that it should have been compiled and linked against the Zlib.

liboml2 accepts a uri argument for the --oml-collect option that is similar to an IETF URI (see e.g. RFC3986). The OML URI consists of an optional network protocol, a host identifier, and an optional port number, or a mandatory file (or flush )scheme and a local filesystem path. The format of the network server version is:

The formats for the local file version is:

For instance, tcp://collect.example.net:3003 will send measurements to an oml2-server listening on port 3003 on host collect.example.net, using TCP. The // is recommended for URIs with a host part, but not mandatory to preserve backward compatibility. Either a hostname or an IP address can be used as the <host> specifier. Both IPv4 and IPv6 addresses are supported, but it is mandatory to put the latter between square brackets; this is optional for the former. tcp://[2001:db8::1]:3003, tcp://[192.0.2.200]:3003 and tcp://192.0.2.200:3003 are all valid forms. The <port> specifier is optional, defaulting to port 3003. The tcp scheme is the default if this part is omitted.

Alternatively, file:/tmp/myfile.txt writes to the /tmp/myfile.txt file in the local filesystem. Relative paths are also accepted. There should be no double-slash after the colon: file://myfile.txt will try to write the output in the root directory. The special file URL file:- will write output to the standard output. The flush scheme behaves exactly in the same way as file except the file descriptor is flushed after each a measurement line is written out. This allows to prevent any size-based buffering in the C library from delaying the recording of samples. This is useful in case of, e.g., real time graphing of the data based on the contents of the file.

Both URI schemes can be prefixed by gzip+ to enable compression (see STREAM COMPRESSION above), e.g., gzip+tcp://collect.example.net:3003 or gzip+file:/tmp/myfile.txt. Note however that the main scheme is not optional in this case, and both gzip://collect.example.net:3003 and gzip:/tmp/myfile.txt are invalid.

liboml2 recognizes the following environment variables. Note that the equivalent command line options override any value set in an environment variable.

The selection of the first filter when --oml-samples 1 is used can be confusing for numeric MP fields because it results in a different schema in the measurement output compared to other possible configurations available from the command line, which use the avg filter. It is not clear whether this is a feature or a bug.

If a problem you are experiencing is not addressed in the FAQ (http://oml.mytestbed.net/projects/oml/wiki/FAQ_and_Support) nor already present in the list of know bugs (http://oml.mytestbed.net/projects/oml/issues). You could discuss it on the mailing list (details and archives at http://oml.mytestbed.net/tab/show?id=oml).

It is however advisable to open a ticket on our issue tracker at http://oml.mytestbed.net/projects/oml/issues/new. Don’t forget to include details such as client and server logs (at [--oml-log-level|-d] 2). It also helps if you can share the source code of a (minimal, if possible) example reliably triggering the problem.

oml2-server does not use any authentication, and should thus be considered insecure. It is intended to be deployed behind firewalls on a dedicated testbed network. It should not be run as a daemon on an open network. Future versions of OML may be re-designed to be suitable for use in insecure environments.