NAME

liboml2 - OML2 client library

SYNOPSIS

#include <oml2/omlc.h>
int    omlc_init(const char *appname, int *pargc, const char *argv[], o_log_fn log_fn);

OmlMP* omlc_add_mp(const char *name, OmlMPDef *definition);

int    omlc_start(void);

void   omlc_inject(OmlMP *mp, OmlValueU *values);

int    omlc_inject_metadata(OmlMP* mp, const char* key, const OmlValueU* value, OmlValueT type, const char* fname);

oml_guid_t omlc_guid_generate();

int    omlc_close(void);
#define OML_FROM_MAIN // enable oml_register_mps() and necessary storage; only in one file
#include "APPNAME_oml.h"
inline void oml_register_mps();

inline void oml_inject_MPNAME(OmlMP* mp, …);

DESCRIPTION

liboml2 is the client library for OML2. It provides an API for application writers to collect measurements from their applications via user-defined Measurement Points (MPs). It also provides a flexible filtering and collection mechanism that allows application users to customise how measurements are processed and stored by an OML-enabled application.

This man page summarises the API. Each function in the API also has its own detailed man page. To learn how to configure an application that uses the API from the command line, see liboml2(1). To learn about the external configuration file that liboml2 can use, see liboml2.conf(5).

A programmer who wants to use liboml2 to perform measurements in their application must first decide what to measure. OML lets the programmer first define MPs and then inject measurement tuples into them. The core of the library then performs filtering on the injected measurements and, at predefined times, emits the filtered samples to either a local file or a measurement server (see oml2-server(1)).

A program using OML can therefore be divided into two stages:

Initialisation

First the library is initialised and measurement points are defined.

Measurement

Second, the filters are constructed and enabled by calling omlc_start(3). Measurement can proceed, with as many calls to omlc_inject(3) as required.

In some cases, it is necessary to occasionaly inject metadata about one MP, or one of its fields (e.g., unit or quality index of the current samples). The omlc_inject_metadata(3) function can be used for this purpose.

It is also sometimes necessary to link measurement tuples together (e.g., the destination IP of a packet in a network MP and its transport protocol and port in a transport MP). OML supports the concept of a Globally Unique ID (GUID) as a data type. It is a random identifier which can be generated on demand using the omlc_guid_generate() function. It returns an oml_guid_t, which can then be injected into any MP with a field of type :guid (see oml2-scaffold(1)). In case a GUID is not required for the injection of a specific tuple (i.e., no linkage to other measurements), the special value OMLC_GUID_NULL can be used.

For convenience, it is recommended to use oml2-scaffold(1) to generate helper functions for MP registration and injection (oml_register_mps and oml_inject_MPNAME) which take care of preparing the data and passing it to omlc_add_mp(3) and omlc_inject(3), respectively.

It is also best to call omlc_close(3) to shut down the measurement library cleanly once measurement tasks are completed, such as when the application is terminating.

Initialisation

When the application starts up, the programmer should make a call to omlc_init(3) to initialise liboml2. The function accepts a name for the application itself, together with the executable’s command line arguments (the C main() function’s argc and argv parameters), and a fourth argument which is not widely used and can safely be set to NULL:

#include <oml2/omlc.h>

int main(int argc, const char **argv)
{
  int result = omlc_init ("MyOmlClient", &argc, argv, NULL);
  if (result == -1) {
    fprintf (stderr, "Could not initialise OML\n");
    exit (1);
  } else if (result == 1) {
    fprintf (stderr, "OML was disabled by the user, exiting\n");
    exit (0);
  }

  /* Do application stuff... */

  return 0;
}

The omlc_init(3) function gathers configuration information from the command line arguments, the environment, and possibly an external configuration file, and sets internal configuration options of the library. It removes any command line options that it recognises, so that the application itself does not get confused when it tries to interpret the command line. All OML options start with the prefix --oml-. An application should call omlc_init(3) before running its own command line option parser.

After initialising the library, the programmer must define the application’s MPs. Each MP is defined as a typed tuple. OML supports basic integer types, a double-precision floating point type, a string type and a blob type.

Using the registration helper

This is the recommended method as it is easiest.

The oml2-scaffold(1) tool can generate the definition of the desired measurement points, and a single registration function as part of the OML header (APPNAME_oml.h generated with the --oml flag). The registration function, oml_register_mps(), declares theses MPs to the library. It takes no argument, and initialises a structure g_oml_mps_APPNAME which contains named pointer to each measurement point. Taking the example from oml2-scaffold(1),

defApplication('oml:man:foo', 'foo') do |app|
  # ... Some needed code skipped here ...
  app.defMeasurement("memory") do |mp|
    # ... Some more code skipped here ...
  end

  app.defMeasurement("cpu") do |mp|
    # ... Some more code skipped here ...
  end
end

The oml_register_mps() function declares and initialises a structure containing pointers to the OmlMP*. In this example, where the application name is foo, it will be accessible as g_oml_mps_foo, and the OmlMP* for memory and cpu as g_oml_mps_foo→memory and g_oml_mps_foo→cpu, respectively.

This function is made available conditionally from APPNAME_oml.h. Indeed, it requires objects which need only be allocated once, and is done at the same time. To make this function available, one must #define cpp(1) macro OML_FROM_MAIN prior to the #include statement for APPNAME_oml.h. This should be done only in file using oml_register_mps().

Registering MPs manually

In case some or more MPs cannot be known at compile time (e.g., a plugin providing its own MPs), the auto-generated registration code cannot be used. These MPs have to be defined and added manually in the code. An MP definition is represented as an array of OmlMPDef objects, terminated by a NULL object, as shown here:

OmlMPDef mp_def [] =
{
  { "source", OML_UINT32_VALUE },
  { "destination", OML_UINT32_VALUE },
  { "length", OML_UINT32_VALUE },
  { "weight", OML_DOUBLE_VALUE },
  { "protocol", OML_STRING_VALUE },
  { NULL, (OmlValueT)0 }
};

The first member of the OmlMPDef struct is the name of the field of the MP that it represents. The second member is its type. The name appears in the schema for the local file storage, and as part of the database column name for the database created by oml2-server(1).

To register the MP definition with liboml2, the programmer must call omlc_add_mp(3):

OmlMP* mp = omlc_add_mp ("packet_info", mp_def);

if (mp == NULL) {
 fprintf (stderr, "Error: could not register Measurement Point 'packet_info'");
 exit (1);
}

This function returns a handle to the internal MP object, which should be used in subsequent calls to omlc_inject(3), and should be treated as opaque.

Measurement

Once all measurement points have been defined and created using calls to omlc_add_mp(3), the programmer must start the measurement collection process by calling omlc_start(3). This call creates the internal filters according to the current configuration options, and starts the filter output sampling threads for 'interval' style filters (for more information see liboml2.conf(5)). Once omlc_start(3) has been called, no more MPs can be added: further calls to omlc_add_mp(3) will be ignored. Conversely, once omlc_start(3) has been called, calls can be made to perform measurements, and inject there results into the OML reporting path.

Using injection helpers

This is the recommended method as it is easiest and much less error-prone. It is also more future-proof as it shields the application from low-level API changes. These helpers also provide C-compiler type-checking and proper memory management.

The oml2-scaffold(1) tool can generate helper injection functions as part of the OML header (--oml). Their prototype is of the form oml_inject_MPNAME(OmlMP* mp, …), where MPNAME is the name of the MP that is passed as mp. The remaining arguments are C-typed variables constituting the sample to be injected in the MP.

Continuing with the auto-generated example from above, injecting a sample in the memory MP can be done as

oml_inject_memory(g_oml_mps_foo->memory, ram, total, free, used, free_percent, used_percent);

where ram, total, free, used, free_percent and used_percent are assumed to be variables updated by the rest of the application code.

Implementing injection code manually

The omlc_inject(3) function accepts an MP handle and a vector of OmlValueU(3) objects. These values MUST always be initialised first with omlc_zero(3) or omlc_zero_array(3). The programmer should first load up the vector with new values to be measured using the omlc_set_* macros, and then call omlc_inject(3). If string or blobs were affected to some of the OmlValueU(3), these have to be reset with omlc_reset_string(3) or omlc_reset_blob(3) so any allocated memory is properly cleared. For instance, here is how a measurement injection might look for the MP definition above:

...

OmlMP *mp = omlc_add_mp("packet_info", mp_def);

...

uint32_t source_id;
uint32_t destination_id;
uint32_t packet_length;
double weight;
char *protocol;

...

/* Some application-specific code to obtain new values for the variables above */

...

{
   OmlValueU values[5];

   omlc_zero_array(values, 5);

   omlc_set_uint32 (values[0], source_id);
   omlc_set_uint32 (values[1], destination_id);
   omlc_set_uint32 (values[2], packet_length);
   omlc_set_double (values[3], weight);
   omlc_set_string (values[4], protocol);

   omlc_inject (mp, values);

   omlc_reset_string (values[4]); /* Free potentially allocated space */
}

The vector must be loaded with values in the same order as the original definition of the fields of the MP in the call to omlc_add_mp(3). The call to omlc_inject(3) can be repeated as many times as the program wants to make measurements for a given MP.

OML Types

OML uses an opaque data structure, OmlValueU(3) to store arbitrary data types. This structure can contain chunks of allocated memory and should therefore be properly initialised before use (with the omlc_zero(3) and omlc_zero_array(3) functions.

OML defines the following types and setters:

Each of the first four integer types maps to an underlying equivalent type from the C standard header stdint.h(0). These types should be used in calls to the omlc_set_* macros. They are:

  • int32_t

  • uint32_t

  • int64_t

  • uint64_t

The OML_DOUBLE_VALUE type maps to an underlying C double.

The OML_STRING_VALUE maps to a nil-terminated C string. Memory will be dynamically allocated (or reused) when a value of this type is set to an OmlValueU(3).

The OML_BLOB_VALUE maps to an arbitrary block of binary data, of a given size. It is handled in a way similar to OML_STRING_VALUE, save for the nil-termination and automatic size calculation.

The OML_GUID_VALUE type can be used to create logical groups of samples, either within the same MP, or across MPs. For example, parameters from the IP and TCP headers of a TCP packet should be reported in separate MPs (as not all IP packets have TCP payload). However, it is desirable to link these tuples together. Adding an OML_GUID_VALUE field in both MPs, which can be filled with the same GUID returned omlc_guid_generate(3) would allow to join the relevant information about this packet at a later time, during analysis. Similarly, if multiple samples in an MP belong to the same logical group (e.g., an array), they can be linked using the same mechanism, by adding an OML_GUID_VALUE field to that MP, and generating a new oml_guid_t every time a new group needs to be created. To fill OML_GUID_VALUE fields where grouping is not needed, a default NULL value is available as OMLC_GUID_NULL.

The OML_BOOL_VALUE simply encapsulate a boolean value. Anything that is not 0 is assumed to be true. There is however no guarantee that a non-zero boolean will retain the actual (integral) value which was used to set it, beyond its logical truth value.

liboml2 currently also defines an OML_LONG_VALUE that maps to a C long type, but this type is deprecated because it can change size between 32-bit and 64-bit platforms. It will be removed from the API at some point in the future. The library internally clamps OML_LONG_VALUE MP fields to the most positive and negative values that will fit in an int32_t object before sending them to the oml2-server(1) or a local measurement file. OML_LONG_VALUE should not be used in new applications.

Building

Applications using liboml2 must include the -loml2 flag on the linker command line to link against the client library.

EXAMPLE

The following is a fully-functional example application that counts from 1 to 1000 and outputs the value of the counter as an unsigned integer, a string, and a double at each step: 

#include <stdio.h>
#include <stdlib.h>
#include <oml2/omlc.h>

int main (int argc, const char **argv)
{
  int result = omlc_init ("Simple", &argc, argv, NULL);
  if (result == -1) {
    fprintf (stderr, "Could not initialise OML\n");
    exit (1);
  } else if (result == 1) {
    fprintf (stderr, "OML was disabled by the user, exiting\n");
    exit (0);
  }

  OmlMPDef mp_def [] = {
    { "count", OML_UINT32_VALUE },
    { "count_str", OML_STRING_VALUE },
    { "count_real", OML_DOUBLE_VALUE },
    { NULL, (OmlValueT)0 }
  };

  OmlMP *mp = omlc_add_mp ("counter", mp_def);

  if (mp == NULL) {
    fprintf (stderr, "Error: could not register Measurement Point 'counter'");
    exit (1);
  }

  omlc_start();

  int i = 0;
  for (i = 0; i < 1000; i++) {
    uint32_t count = (uint32_t)i;
    char count_str[16];
    double count_real = (double)i;
    OmlValueU values[3];

    omlc_zero_array(values, 3);

    snprintf(count_str, sizeof(count_str), "%d", i);

    omlc_set_uint32 (values[0], count);
    omlc_set_string (values[1], count_str);
    omlc_set_double (values[2], count_real);

    omlc_inject (mp, values);

    omlc_reset_string(values[1]);
  }


  omlc_close();

  return 0;
}

The following command should be sufficient to compile this program: 

$ gcc -o counter counter.c -loml2

Here is an example command line to run this application with a default set of filters: 

$ ./counter --oml-id myid --oml-domain count --oml-collect file:-

The output from this command will appear on the terminal because we told OML to use the standard output (--oml-collect file:-). Here is what it looks like:

INFO   OML Client 2.x.y [OMSPv5] Copyright 2007-2016, NICTA
protocol: 5
domain: count
start-time: 1283160287
sender-id: myid
app-name: Counter
schema: 1 Counter_counter count:uint32 count_str:string count_real:double
content: text


0.797713        1       1       0       0       0.000000
0.797736        1       2       1       1       1.000000
0.797748        1       3       2       2       2.000000
0.797760        1       4       3       3       3.000000
0.797774        1       5       4       4       4.000000
0.797785        1       6       5       5       5.000000
0.797796        1       7       6       6       6.000000
...

An easy way to create more complicated applications is to use oml2-scaffold(1).

PORTING PRE-2.9 APPLICATIONS

In between OML 2.8 and 2.9, a heavy refactoring of the lower layers happened to better track the memory used, and reduce memory leaks. Unfortunately, this change could not be hidden from the higher layers, as the API was extended with the introduction of initialisers destructors for OmlValueU(3) variables, with proper management of dynamic memory allocation.

The shared library’s version has been bumped from 0.8.1 to 9.0.0, so old binaries could still work properly (on distributions supporting multiple library versions, such as Debian). However, this means these old binaries will not benefit from the new features of OML 2.9 onwards.

Porting pre-2.9 instrumentations to OML 2.9 is however relatively easy. Whenever OmlValueU(3), or arrays thereof, are declared, they MUST be properly initialised prior to any use. Not doing so might result in cryptic segmentation faults, or weird ghostly data being collected. Refer to Implementing injection code manually and the example above to see where calls to omlc_zero(3) (or omlc_zero_array(3)), and omlc_reset_string(3) and omlc_reset_blob(3) should be added to your code.

Additionally, omlc_set_const_string() is deprecated, as it did not have much significance. The string is now always duplicated into separate storage to ensure integrity regardless of how the instrumented application manipulates its original data after injection.

All new instrumentations should however avoid using omlc_inject(3) directly, and rely on the code generation capabilities of oml2-scaffold(1) instead. It is also recommended that these changes be taken as an opportunity to port old applications to oml2-scaffold(1) too.

BUGS

If a problem you are experiencing is not addressed in the FAQ (http://oml.mytestbed.net/projects/oml/wiki/FAQ_and_Support) nor already present in the list of know bugs (http://oml.mytestbed.net/projects/oml/issues). You could discuss it on the mailing list (details and archives at http://oml.mytestbed.net/tab/show?id=oml).

It is however advisable to open a ticket on our issue tracker at http://oml.mytestbed.net/projects/oml/issues/new. Don’t forget to include details such as client and server logs (at [--oml-log-level|-d] 2). It also helps if you can share the source code of a (minimal, if possible) example reliably triggering the problem.

SEE ALSO

Manual Pages

oml2-server(1), liboml2(1), liboml2.conf(5), oml2-scaffold(1)

omlc_init(3), omlc_add_mp(3), omlc_start(3), omlc_inject(3), omlc_close(3)

OmlValueU(3)

OML User Manual

http://oml.mytestbed.net/projects/oml/wiki/Documentation

OML Measurement Stream Protocol (OMSP)

http://oml.mytestbed.net/doc/oml/2.12/doxygen/omsp.html