wiki:User/0aInstallation

Version 2 (modified by seskar, 5 years ago) ( diff )

--

Installation

OML Documentation

    The OML instrumentation suite comprises a few distinct components:

    • the OML client library (liboml2, liboml2-dev, libocomm, libocomm-dev);
    • the OML server ( oml2-server );
    • the OML proxy server (for disconnected experiments, oml2-proxy-server );
    • the OML-instrumented applications.

    There are currently two main ways to obtain and install OML, from our package repositories, as explained below, or by getting the source and building it. We recommend using the package repositories for most users, which is described below. This documentation is biased towards Debianish distributions, and primarily uses their package-naming and -subdivision (lib, server, -dev) conventions. However, it also covers installation of the relevant packages on ArchLinux as well as RPM-based systems. Once installed, the execution and testing steps are the same.

    Depending on the desired role in the reporting chain for a gives machine, different sets of packages will need to be installed.

    • All machines which will run OML-enabled applications require the liboml2 package, which provides the OML Client shared library, to be installed.
    • The machine(s) selected to act as collection server(s) also need(s) the oml2-server .
    • Development and build machines will also need the development headers (liboml2-dev[el]). See OML Client Programming for details on how to write own OML measurement applications, or instrument existing ones.

    This installation steps outlined here also applies to the OML enabled Applications, which are similarly packaged, and distributed in the same repositories.

    Distribution Packages

    We maintain packages for Debian/Ubuntu, ArchLinux and RPM-based distros.

    Adding Software Sources

    Debian, Fedora/CentOS

    The Debian and RPM packages are built on ​the OpenSuSE OBS platform. The following links provide source configuration and package installation details for

    The adventurous user will also find staging builds of ​the most recent code there. It would probably be ill-advised to setup this last source on production environments, but it is good for early tests and new features.

    ArchLinux

    OML packages are available in ​the Arch User Repository. The easiest way to install them is by using ​Yaourt, which can built and installed as follows. 

    $ wget https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz | tar xzv
$ cd package-query
$ makepkg
$ sudo pacman -U package-query*.tar.xz
$ cd -
$ wget https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz -O - | tar xzv
$ cd yaourt
$ makepkg
$ sudo pacman -U yaourt*.tar.xz

    Installing OML Components

    On ArchLinux, a single package, oml2 contains all the needed libraries, programs and headers. A PKGBUILD is available in the Arch User Repository, and the same goes for the apps. There also are Git-based packages, which the adventurous user will find by suffixing -git to the name of the previous packages. 

    $ yaourt -S oml2

    Client library

    Debian and RPM distributions offer a finer separation of packages, which can be installed separately depending on the purpose for the machine. The apt-get and yum programs will automatically install all of OML's dependencies as required. (If you find that a required dependency is missing, please let us know, see contact details here.) h4. Injection Point: Client Library (and Proxy Server)

    On Debian, the client library is packaged as liboml2, and also depends on libocomm, which will be pulled at the same time. It is also a good idea to install the dummy oml2-example package, as it allows to test the installation. This is done by running the following command. 

    sudo apt-get install liboml2 oml2-example

    If the measurement nodes are expected to run in disconnected mode, the Proxy Server will also be needed. 

    sudo apt-get install oml2-proxy-server

    On RPM-based distributions, both come in the same package, liboml2, which can be installed with yum 

    sudo yum install liboml2

    Collection Point: Server and Database Tools

    On Debian, the collection server is provided by the oml2-server package. 

    sudo apt-get install oml2-server

    A similar thing works for RPM-based distributions. 

    sudo yum install oml2-server

    If you want to examine the contents of measurement databases, you'll need to manually install the SQLite3 client application. None of the packages directly uses the SQLite3 client, so they don't cause it to be installed automatically. On Debian, this package is named sqlite3, while it is simply named sqlite on ArchLinux and RPM-based distros. In either case, you can install in the same way as the server.

    Development Workstation: Headers

    On Debian, development headers are suffixed -dev. The liboml2-dev will pull the relevant libocomm-dev dependency. This can be done much in the same way as the library itself. 

    sudo apt-get install liboml2-dev

    A very similar approach will work for RPM-based distributions, bearing in mind that development headers are suffixed -devel on these platforms. 

    sudo yum install liboml2-devel

    Testing the Library Installation

    The oml2-example package will have installed a dummy sine generator reporting its data through OML, oml2-generator. It can be used to test that the library has been properly installed, simply by running it with the appropriate parameters to report data to the standard output (so a server is not needed at this stage). 

    oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \
  --oml-id librarytest --oml-domain installtest --oml-collect file:-

    If the library has been properly installed, something similar to the following will be output. 

    Apr 12 10:03:32 INFO    OML Client V2.10.0 [Protocol V4] Copyright 2007-2013, NICTA
INFO    File_stream: opening local storage file '-'
3.141593, 628.318531, 100000
sample-1 1 | 0.000000 0.000000
protocol: 4
domain: installtest
start-time: 1365725012
sender-id: librarytest
app-name: generator
schema: 0 _experiment_metadata subject:string key:string value:string
schema: 1 generator_d_lin label:string seq_no:uint32
schema: 2 generator_d_sin label:string phase:double value:double
content: text

0.104363        1       1       sample-1        1
0.104398        2       1       sample-1        0.000000        0.000000
sample-2 2 | 6.283180 -0.000006
0.205737        1       2       sample-2        2
0.205807        2       2       sample-2        6.283180        -0.000006
sample-3 3 | 6.283136 -0.000049
0.306114        1       3       sample-3        3
0.306154        2       3       sample-3        6.283136        -0.000049
sample-4 4 | 6.283136 -0.000049
0.406637        1       4       sample-4        4
0.406676        2       4       sample-4        6.283136        -0.000049
sample-5 5 | 6.283136 -0.000049
0.507075        1       5       sample-5        5
0.507116        2       5       sample-5        6.283136        -0.000049
sample-6 6 | 6.283136 -0.000049
0.607570        1       6       sample-6        6
0.607609        2       6       sample-6        6.283136        -0.000049
sample-7 7 | 6.283136 -0.000049
0.708090        1       7       sample-7        7
0.708141        2       7       sample-7        6.283136        -0.000049
sample-8 8 | 6.283136 -0.000049
0.808660        1       8       sample-8        8
0.808700        2       8       sample-8        6.283136        -0.000049
sample-9 9 | 6.283136 -0.000049
0.909052        1       9       sample-9        9
0.909092        2       9       sample-9        6.283136        -0.000049
sample-10 10 | 6.283136 -0.000049
1.009559        1       10      sample-10       10
1.009598        2       10      sample-10       6.283136        -0.000049
INFO    Waiting for buffered queue reader thread to drain...
INFO    Buffered queue reader thread finished OK...

    Configuring and Testing the Server

    The server can take command line parameters to change its behaviour, as explained in the manpage and the source:README file. Depending on the distribution, there are different ways to change the default parameters. This might be particularly needed in case the PostgreSQL backend needs to be used.

    Debian

    When installed, the oml2-server program will automatically start whenever the computer starts or reboots. This is done using the standard startup script mechanism of the system. On Debian/Ubuntu, starting or stopping the server manually can be done with 

    sudo service oml2-server stop # Stop the server
sudo service oml2-server start # Start the server again
sudo service oml2-server restart # Cycle the server (stop, then start)

    The server is configured using the options specified in /etc/default/oml2-server. Here are the default contents of this file: 

    OPTS="-l 3003 --logfile=/var/log/oml2-server.log --data-dir=/tmp/"

    The OPTS shell variable is passed to the server command line. All of the switches above are the normal oml2-server command line options, documented here. The default configuration causes the server to listen on port 3003, to write logging information to the file /var/log/oml2-server.log, and to write experiment databases in the /tmp directory. You can add any of the other oml2-server options to /etc/default/oml2-server as well. For instance, this would make the server output lots of debugging log messages to the log file: 

    OPTS="-l 3003 -d4 --logfile=/var/log/oml2-server.log --data-dir=/tmp/"

    ArchLinux and RPM-based distributions

    These distributions now use systemd. Controlling the server is done through systemctl. 

    $ sudo systemctl stop oml2-server.service # Stop the server
$ sudo systemctl start oml2-server.service # Start the server again
$ sudo systemctl restart oml2-server.service # Cycle the server (stop, then start)

    Changing the startup option of the server can be done by copying /usr/lib/systemd/system/oml2-server.service to /etc/systemd/system, before modifying that copy to contain the desired startup parameters. Once done, the new parameters need to be set by re-enabling the service. 

    $ sudo systemctl reenable oml2-server.service

    Using the PostgreSQL Backend

    If you want to use a PostgreSQL backend, you will first need to install the database server somewhere. For simplicity, we only document a database server co-located on the same machine as the OML server, however more complex setups are possible. On Debian, The package is called postgresql-9.1, and can be installed with apt-get similarly to all other packages. You might also want to install the client applications postgresql-client-9.1 if you need to inspect database contents. Additionally, the pgadmin3 package provides a GUI.

    On ArchLinux, the postgresql package contains everything. It can be directly either using yaourt, or sudo pacman -S postgresql. The pgadmin3 tool is also available.

    On RPM-based systems, the server package is called postgresql-server, and the clients are in postgresql. As for the other distros, pgadmin3 is also available under that name.

    The manpage contains details about the use of the PostgreSQL backend. For a quick start, after installing the database server, a user role, say OMLUSER, must first be created, which is allowed to created new databases, but nothing else. The following assumes that DBADMIN is the database admin user, usually postgres. 

    createuser -U DBADMIN --pwprompt --no-superuser --createdb --no-createrole --no-replication OMLUSER

    The --no-replication argument is only valid with PostgreSQL-9.2. In case of error, simply remove it. A password for the newly created database user will be requested. Alternatively, the role can be created through and interactive console. 

    $ psql -U DBADMIN
psql (9.3.2)
Type "help" for help.

DBADMIN=# CREATE USER OMLUSER WITH PASSWORD 'OMLPASSWORD';
CREATE ROLE
DBADMIN=# ALTER ROLE OMLUSER WITH CREATEDB;
ALTER ROLE

    The oml2-server can then be configured, through command line arguments, to use the postgresql backend. Depending on the distribution, and as explained above, the happens either in /etc/default/oml2-server (Debian; in the OPTS variable) or in a copy of /usr/lib/systemd/system/oml2-server.service placed in /etc/systemd/system/oml2-server.service (systemd-based systems: ArchLinux and RPM-based; in the ExecStart statement). In the latter case, running sudo systemctl reenable oml2-server.service is needed for new parameters to be taken into account prior to restarting the service.

    The command-line options need to be extended with parameters to enable the postgresql backend (a line containing them should already be present as a comment in the file). 

    [...] --backend=postgresql --pg-host=localhost --pg-port=5432 --pg-user=OMLUSER --pg-pass=OMLPASS

    Testing

    Local Server Installation

    Testing the server locally can also be done with the dummy sine generator, simply changing its collection URL to send measurement streams to the local server. 

    oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \
  --oml-id localservertest --oml-domain installtest --oml-collect localhost

    As the output no longer contains OMSP data, it is shorter than before. 

    Apr 12 10:07:31 INFO    OML Client V2.10.0 [Protocol V4] Copyright 2007-2013, NICTA
3.141593, 628.318531, 100000
sample-1 1 | 0.000000 0.000000
INFO    OmlNetOutStream: attempting to connect to server at tcp://localhost:3003
sample-2 2 | 6.283180 -0.000006
sample-3 3 | 6.283136 -0.000049
sample-4 4 | 6.283136 -0.000049
sample-5 5 | 6.283136 -0.000049
sample-6 6 | 6.283136 -0.000049
sample-7 7 | 6.283136 -0.000049
sample-8 8 | 6.283136 -0.000049
sample-9 9 | 6.283136 -0.000049
sample-10 10 | 6.283136 -0.000049
INFO    Waiting for buffered queue reader thread to drain...
INFO    Buffered queue reader thread finished OK...

    Inspecting the server log, which defaults to /var/log/oml2-server.log, should show that this client did connect properly. 

    Apr 12 10:02:08 INFO    OML2 Server V2.10.0
Apr 12 10:02:08 INFO    OML Protocol V1--4
Apr 12 10:02:08 INFO    Copyright 2007-2013 NICTA
Apr 12 10:02:08 INFO    sqlite: Creating SQLite3 databases in /var/lib/oml2
Apr 12 10:07:31 INFO    sqlite:installtest: Opening database at '/var/lib/oml2/installtest.sq3'
Apr 12 10:07:31 INFO    server-io:5: New MS schema 0 _experiment_metadata subject:string key:string value:string
Apr 12 10:07:31 INFO    server-io:5: New MS schema 1 generator_d_lin label:string seq_no:uint32
Apr 12 10:07:31 INFO    server-io:5: New MS schema 2 generator_d_sin label:string phase:double value:double
Apr 12 10:07:31 INFO    server-io:5: Client 'installtest:localservertest:generator' ready to send data
Apr 12 10:07:32 INFO    server-io:5: Client 'installtest:localservertest:generator' closed connection
Apr 12 10:07:32 INFO    installtest: Closing database

    The database can also be checked to confirm data has been properly written. When using the default SQLite3 backend, the full path to the database appears in the server log (above). The sqlite3 client can be used to quickly inspect a dump of the data in this file. 

    sqlite3 /var/lib/oml2/installtest.sq3 .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE _senders (name TEXT PRIMARY KEY, id INTEGER UNIQUE);
INSERT INTO "_senders" VALUES('librarytest',1);
CREATE TABLE "_experiment_metadata" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "subject" TEXT, "key" TEXT, "value" TEXT);
INSERT INTO "_experiment_metadata" VALUES(1,NULL,NULL,NULL,NULL,NULL,'table__experiment_metadata','0 _experiment_metadata subject:string key:string value:string');
INSERT INTO "_experiment_metadata" VALUES(2,NULL,NULL,NULL,NULL,NULL,'start_time','1365725251');
INSERT INTO "_experiment_metadata" VALUES(3,NULL,NULL,NULL,NULL,NULL,'table_generator_d_lin','1 generator_d_lin label:string seq_no:uint32');
INSERT INTO "_experiment_metadata" VALUES(4,NULL,NULL,NULL,NULL,NULL,'table_generator_d_sin','2 generator_d_sin label:string phase:double value:double');
CREATE TABLE "generator_d_lin" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "label" TEXT, "seq_no" UNSIGNED INTEGER);
INSERT INTO "generator_d_lin" VALUES(1,1,1,0.0781299999216571,0.088947,'sample-1',1);
INSERT INTO "generator_d_lin" VALUES(2,1,2,0.178464999888092,0.178699,'sample-2',2);
INSERT INTO "generator_d_lin" VALUES(3,1,3,0.27906999969855,0.279303,'sample-3',3);
INSERT INTO "generator_d_lin" VALUES(4,1,4,0.379638999700546,0.37987,'sample-4',4);
INSERT INTO "generator_d_lin" VALUES(5,1,5,0.480201999656856,0.480439,'sample-5',5);
INSERT INTO "generator_d_lin" VALUES(6,1,6,0.580763999372721,0.580999,'sample-6',6);
INSERT INTO "generator_d_lin" VALUES(7,1,7,0.681317999958992,0.681549,'sample-7',7);
INSERT INTO "generator_d_lin" VALUES(8,1,8,0.781868999823928,0.782154,'sample-8',8);
INSERT INTO "generator_d_lin" VALUES(9,1,9,0.882429999299347,0.882665,'sample-9',9);
INSERT INTO "generator_d_lin" VALUES(10,1,10,0.982985999435186,0.983226,'sample-10',10);
CREATE TABLE "generator_d_sin" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "label" TEXT, "phase" REAL, "value" REAL);
INSERT INTO "generator_d_sin" VALUES(1,1,1,0.0781679999781772,0.089038,'sample-1',0.0,0.0);
INSERT INTO "generator_d_sin" VALUES(2,1,2,0.178519999841228,0.178762,'sample-2',6.28317975997925,-5.54720033818512e-06);
INSERT INTO "generator_d_sin" VALUES(3,1,3,0.27909699967131,0.279363,'sample-3',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(4,1,4,0.379664999898523,0.379928,'sample-4',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(5,1,5,0.480227999854833,0.480497,'sample-5',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(6,1,6,0.580789999105036,0.581057,'sample-6',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(7,1,7,0.681343999691308,0.681606,'sample-7',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(8,1,8,0.781895999796689,0.782211,'sample-8',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(9,1,9,0.882456999272108,0.882721,'sample-9',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(10,1,10,0.983012999407947,0.983288,'sample-10',6.28313636779785,-4.89393817133532e-05);
COMMIT;

    When using the PostgreSQL backend, the log should be slightly different (assuming OMLUSER is the database role created while setting the server up). 

    Apr 11 15:17:46 INFO    OML2 Server V2.10.0
Apr 11 15:17:46 INFO    OML Protocol V1--4
Apr 11 15:17:46 INFO    Copyright 2007-2013 NICTA
Apr 11 15:17:46 INFO    psql: Sending experiment data to PostgreSQL server localhost:5432 as user 'OMLUSER'
Apr 12 11:09:11 INFO    psql:installtest: Accessing database
Apr 12 11:09:11 INFO    psql:installtest: Database does not exist, creating it
Apr 12 11:16:47 INFO    server-io:7: New MS schema 0 _experiment_metadata subject:string key:string value:string
Apr 12 11:16:47 INFO    server-io:7: New MS schema 1 generator_d_lin label:string seq_no:uint32
Apr 12 11:16:47 INFO    server-io:7: New MS schema 2 generator_d_sin label:string phase:double value:double
Apr 12 11:16:47 INFO    server-io:7: Client 'installtest:remoteservertest:generator' ready to send data
Apr 12 11:16:48 INFO    server-io:7: Client 'installtest:remoteservertest:generator' closed connection
Apr 12 11:16:48 INFO    installtest: Closing database

    The data can be checked with the pg_dump tool from the PostgreSQL server package 

    pg_dump -U OMLUSER -h localhost installtest

    The output of this tool is quite verbose, it should contain table creation and data sample insertion code, as follows (where [...] indicate truncation of the output). 

    --
-- PostgreSQL database dump
--
[...]
--
-- Name: generator_d_lin; Type: TABLE; Schema: public; Owner: OMLUSER; Tablespace: 
--

CREATE TABLE generator_d_lin (
    id integer NOT NULL,
    oml_sender_id integer,
    oml_seq integer,
    oml_ts_client double precision,
    oml_ts_server double precision,
    label text,
    seq_no bigint
);
[...]
--
-- Data for Name: generator_d_lin; Type: TABLE DATA; Schema: public; Owner: OMLUSER
--

COPY generator_d_lin (id, oml_sender_id, oml_seq, oml_ts_client, oml_ts_server, label, seq_no) FROM stdin;
1       1       1       0.451680999999999999    3.23078500000000002     sample-11
2       1       2       0.551826999999999956    3.23337500000000011     sample-22
3       1       3       0.652055000000000051    3.23441000000000001     sample-33
4       1       4       0.752271000000000023    3.23523199999999989     sample-44
5       1       5       0.852474999999999983    3.23595899999999981     sample-55
6       1       6       0.952718999999999983    3.2368570000000001      sample-66
7       1       7       1.05291800000000002     3.23768000000000011     sample-77
8       1       8       1.15313599999999994     3.23822900000000002     sample-88
9       1       9       1.25338799999999995     3.23879699999999993     sample-99
10      1       10      1.35357499999999997     3.23950799999999983     sample-10       10
\.
[...]
--
-- PostgreSQL database dump complete
--

    Full Reporting Chain

    Testing the full reporting chain is almost exactly the same as testing the server locally. The only difference is that the oml2-generator needs to be run on a different measurement node, and the collection URL should be altered to point to the now-remote server's address or hostname. 

    oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \
  --oml-id remoteservertest --oml-domain installtest --oml-collect SERVERNAME

    Checking that the data was properly recorded can be done exactly as in the previous section.

    Troubleshooting

    In case of problem in the full reporting chain, the first thing to investigate is the existence of firewalling rules somewhere on the path between the injection point (measurement node) and the collection point (server). This is highly likely if the oml2-generator always fails at connecting, and no line saying INFO server-io:XX: Client 'DOMAIN:ID:APP' ready to send data appears in the server's log file. This absence might indicate other negotiation issues; you can diagnose further by running the server at log level DEBUG (-d 1), and check that a line saying DEBUG server-io:XX: New client connected is not present either, which is a definitive confirmation that the client did not connect, rather than connect and did not abide by the protocol. The oml2-server listen for connections over TCP/IPv4 (only, for now), to port 3003 by default. Traffic to this port should be allowed in all the reporting network, and in the local firewall of the reporting point. Under Linux, a rule allowing this can be inserted before any other firewalling rules in place using iptables. 

    sudo iptables -I INPUT -p tcp --dport 3003 -j ACCEPT
sudo ip6tables -I INPUT -p tcp --dport 3003 -j ACCEPT
# optional steps to save firewall configuration across reboots:
sudo apt-get install iptables-persistent
sudo service iptables-persistent save

    Persistence of this rule across reboots is distribution-dependent. Debianishes provide an iptables-persistent package allowing to save (manually) and restore (automatically at boot) firewall rules; the iptables package under ArchLinux does this by default, provided the rules were saved with iptables-save to /etc/iptables/iptables.rules; by default, Fedora 18 saves any active rules on stop.

    Installing from Source

    If you want to make changes to OML itself, you'll need to check out the source from our Git repository. Instructions for this can be found about Getting and building the source from Git (you can also contribute patches if you made changes).

    Note: See TracWiki for help on using the wiki.