| Version 1 (modified by , 6 years ago) ( diff ) |
|---|
Installation
The OML instrumentation suite comprises a few distinct components:
- the OML client library (liboml2, liboml2-dev, libocomm, libocomm-dev);
- the OML server ( oml2-server );
- the OML proxy server (for disconnected experiments, oml2-proxy-server );
- the OML-instrumented applications.
There are currently two main ways to obtain and install OML, from our package repositories, as explained below, or by getting the source and building it. We recommend using the package repositories for most users, which is described below. This documentation is biased towards Debianish distributions, and primarily uses their package-naming and -subdivision (lib, server, -dev) conventions. However, it also covers installation of the relevant packages on ArchLinux as well as RPM-based systems. Once installed, the execution and testing steps are the same.
Depending on the desired role in the reporting chain for a gives machine, different sets of packages will need to be installed.
- All machines which will run OML-enabled applications require the liboml2 package, which provides the OML Client shared library, to be installed.
- The machine(s) selected to act as collection server(s) also need(s) the oml2-server .
- Development and build machines will also need the development headers (liboml2-dev[el]). See OML Client Programming for details on how to write own OML measurement applications, or instrument existing ones.
This installation steps outlined here also applies to the OML enabled Applications, which are similarly packaged, and distributed in the same repositories.
Distribution Packages
We maintain packages for Debian/Ubuntu, ArchLinux and RPM-based distros.
Adding Software Sources
Debian, Fedora/CentOS
The Debian and RPM packages are built on the OpenSuSE OBS platform. The following links provide source configuration and package installation details for
The adventurous user will also find staging builds of the most recent code there. It would probably be ill-advised to setup this last source on production environments, but it is good for early tests and new features.
ArchLinux
OML packages are available in the Arch User Repository. The easiest way to install them is by using Yaourt, which can built and installed as follows.
$ wget https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz | tar xzv $ cd package-query $ makepkg $ sudo pacman -U package-query*.tar.xz $ cd - $ wget https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz -O - | tar xzv $ cd yaourt $ makepkg $ sudo pacman -U yaourt*.tar.xz
Installing OML Components
On ArchLinux, a single package, oml2 contains all the needed libraries, programs and headers. A PKGBUILD is available in the Arch User Repository, and the same goes for the apps. There also are Git-based packages, which the adventurous user will find by suffixing -git to the name of the previous packages.
$ yaourt -S oml2
Client library
Debian and RPM distributions offer a finer separation of packages, which can be installed separately depending on the purpose for the machine. The apt-get and yum programs will automatically install all of OML's dependencies as required. (If you find that a required dependency is missing, please let us know, see contact details here.) h4. Injection Point: Client Library (and Proxy Server)
On Debian, the client library is packaged as liboml2, and also depends on libocomm, which will be pulled at the same time. It is also a good idea to install the dummy oml2-example package, as it allows to test the installation. This is done by running the following command.
sudo apt-get install liboml2 oml2-example
If the measurement nodes are expected to run in disconnected mode, the Proxy Server will also be needed.
sudo apt-get install oml2-proxy-server
On RPM-based distributions, both come in the same package, liboml2, which can be installed with yum
sudo yum install liboml2
Collection Point: Server and Database Tools
On Debian, the collection server is provided by the oml2-server package.
sudo apt-get install oml2-server
A similar thing works for RPM-based distributions.
sudo yum install oml2-server
If you want to examine the contents of measurement databases, you'll need to manually install the SQLite3 client application. None of the packages directly uses the SQLite3 client, so they don't cause it to be installed automatically. On Debian, this package is named sqlite3, while it is simply named sqlite on ArchLinux and RPM-based distros. In either case, you can install in the same way as the server.
Development Workstation: Headers
On Debian, development headers are suffixed -dev. The liboml2-dev will pull the relevant libocomm-dev dependency. This can be done much in the same way as the library itself.
sudo apt-get install liboml2-dev
A very similar approach will work for RPM-based distributions, bearing in mind that development headers are suffixed -devel on these platforms.
sudo yum install liboml2-devel
Testing the Library Installation
The oml2-example package will have installed a dummy sine generator reporting its data through OML, oml2-generator. It can be used to test that the library has been properly installed, simply by running it with the appropriate parameters to report data to the standard output (so a server is not needed at this stage).
oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \ --oml-id librarytest --oml-domain installtest --oml-collect file:-
If the library has been properly installed, something similar to the following will be output.
Apr 12 10:03:32 INFO OML Client V2.10.0 [Protocol V4] Copyright 2007-2013, NICTA INFO File_stream: opening local storage file '-' 3.141593, 628.318531, 100000 sample-1 1 | 0.000000 0.000000 protocol: 4 domain: installtest start-time: 1365725012 sender-id: librarytest app-name: generator schema: 0 _experiment_metadata subject:string key:string value:string schema: 1 generator_d_lin label:string seq_no:uint32 schema: 2 generator_d_sin label:string phase:double value:double content: text 0.104363 1 1 sample-1 1 0.104398 2 1 sample-1 0.000000 0.000000 sample-2 2 | 6.283180 -0.000006 0.205737 1 2 sample-2 2 0.205807 2 2 sample-2 6.283180 -0.000006 sample-3 3 | 6.283136 -0.000049 0.306114 1 3 sample-3 3 0.306154 2 3 sample-3 6.283136 -0.000049 sample-4 4 | 6.283136 -0.000049 0.406637 1 4 sample-4 4 0.406676 2 4 sample-4 6.283136 -0.000049 sample-5 5 | 6.283136 -0.000049 0.507075 1 5 sample-5 5 0.507116 2 5 sample-5 6.283136 -0.000049 sample-6 6 | 6.283136 -0.000049 0.607570 1 6 sample-6 6 0.607609 2 6 sample-6 6.283136 -0.000049 sample-7 7 | 6.283136 -0.000049 0.708090 1 7 sample-7 7 0.708141 2 7 sample-7 6.283136 -0.000049 sample-8 8 | 6.283136 -0.000049 0.808660 1 8 sample-8 8 0.808700 2 8 sample-8 6.283136 -0.000049 sample-9 9 | 6.283136 -0.000049 0.909052 1 9 sample-9 9 0.909092 2 9 sample-9 6.283136 -0.000049 sample-10 10 | 6.283136 -0.000049 1.009559 1 10 sample-10 10 1.009598 2 10 sample-10 6.283136 -0.000049 INFO Waiting for buffered queue reader thread to drain... INFO Buffered queue reader thread finished OK...
Configuring and Testing the Server
The server can take command line parameters to change its behaviour, as explained in the manpage and the source:README file. Depending on the distribution, there are different ways to change the default parameters. This might be particularly needed in case the PostgreSQL backend needs to be used.
Debian
When installed, the oml2-server program will automatically start whenever the computer starts or reboots. This is done using the standard startup script mechanism of the system. On Debian/Ubuntu, starting or stopping the server manually can be done with
sudo service oml2-server stop # Stop the server sudo service oml2-server start # Start the server again sudo service oml2-server restart # Cycle the server (stop, then start)
The server is configured using the options specified in /etc/default/oml2-server. Here are the default contents of this file:
OPTS="-l 3003 --logfile=/var/log/oml2-server.log --data-dir=/tmp/"
The OPTS shell variable is passed to the server command line. All of the switches above are the normal oml2-server command line options, documented here. The default configuration causes the server to listen on port 3003, to write logging information to the file /var/log/oml2-server.log, and to write experiment databases in the /tmp directory. You can add any of the other oml2-server options to /etc/default/oml2-server as well. For instance, this would make the server output lots of debugging log messages to the log file:
OPTS="-l 3003 -d4 --logfile=/var/log/oml2-server.log --data-dir=/tmp/"
ArchLinux and RPM-based distributions
These distributions now use systemd. Controlling the server is done through systemctl.
$ sudo systemctl stop oml2-server.service # Stop the server $ sudo systemctl start oml2-server.service # Start the server again $ sudo systemctl restart oml2-server.service # Cycle the server (stop, then start)
Changing the startup option of the server can be done by copying /usr/lib/systemd/system/oml2-server.service to /etc/systemd/system, before modifying that copy to contain the desired startup parameters. Once done, the new parameters need to be set by re-enabling the service.
$ sudo systemctl reenable oml2-server.service
Using the PostgreSQL Backend
If you want to use a PostgreSQL backend, you will first need to install the database server somewhere. For simplicity, we only document a database server co-located on the same machine as the OML server, however more complex setups are possible. On Debian, The package is called postgresql-9.1, and can be installed with apt-get similarly to all other packages. You might also want to install the client applications postgresql-client-9.1 if you need to inspect database contents. Additionally, the pgadmin3 package provides a GUI.
On ArchLinux, the postgresql package contains everything. It can be directly either using yaourt, or sudo pacman -S postgresql. The pgadmin3 tool is also available.
On RPM-based systems, the server package is called postgresql-server, and the clients are in postgresql. As for the other distros, pgadmin3 is also available under that name.
The manpage contains details about the use of the PostgreSQL backend. For a quick start, after installing the database server, a user role, say OMLUSER, must first be created, which is allowed to created new databases, but nothing else. The following assumes that DBADMIN is the database admin user, usually postgres.
createuser -U DBADMIN --pwprompt --no-superuser --createdb --no-createrole --no-replication OMLUSER
The --no-replication argument is only valid with PostgreSQL-9.2. In case of error, simply remove it. A password for the newly created database user will be requested. Alternatively, the role can be created through and interactive console.
$ psql -U DBADMIN psql (9.3.2) Type "help" for help. DBADMIN=# CREATE USER OMLUSER WITH PASSWORD 'OMLPASSWORD'; CREATE ROLE DBADMIN=# ALTER ROLE OMLUSER WITH CREATEDB; ALTER ROLE
The oml2-server can then be configured, through command line arguments, to use the postgresql backend. Depending on the distribution, and as explained above, the happens either in /etc/default/oml2-server (Debian; in the OPTS variable) or in a copy of /usr/lib/systemd/system/oml2-server.service placed in /etc/systemd/system/oml2-server.service (systemd-based systems: ArchLinux and RPM-based; in the ExecStart statement). In the latter case, running sudo systemctl reenable oml2-server.service is needed for new parameters to be taken into account prior to restarting the service.
The command-line options need to be extended with parameters to enable the postgresql backend (a line containing them should already be present as a comment in the file).
[...] --backend=postgresql --pg-host=localhost --pg-port=5432 --pg-user=OMLUSER --pg-pass=OMLPASS
Testing
Local Server Installation
Testing the server locally can also be done with the dummy sine generator, simply changing its collection URL to send measurement streams to the local server.
oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \ --oml-id localservertest --oml-domain installtest --oml-collect localhost
As the output no longer contains OMSP data, it is shorter than before.
Apr 12 10:07:31 INFO OML Client V2.10.0 [Protocol V4] Copyright 2007-2013, NICTA 3.141593, 628.318531, 100000 sample-1 1 | 0.000000 0.000000 INFO OmlNetOutStream: attempting to connect to server at tcp://localhost:3003 sample-2 2 | 6.283180 -0.000006 sample-3 3 | 6.283136 -0.000049 sample-4 4 | 6.283136 -0.000049 sample-5 5 | 6.283136 -0.000049 sample-6 6 | 6.283136 -0.000049 sample-7 7 | 6.283136 -0.000049 sample-8 8 | 6.283136 -0.000049 sample-9 9 | 6.283136 -0.000049 sample-10 10 | 6.283136 -0.000049 INFO Waiting for buffered queue reader thread to drain... INFO Buffered queue reader thread finished OK...
Inspecting the server log, which defaults to /var/log/oml2-server.log, should show that this client did connect properly.
Apr 12 10:02:08 INFO OML2 Server V2.10.0 Apr 12 10:02:08 INFO OML Protocol V1--4 Apr 12 10:02:08 INFO Copyright 2007-2013 NICTA Apr 12 10:02:08 INFO sqlite: Creating SQLite3 databases in /var/lib/oml2 Apr 12 10:07:31 INFO sqlite:installtest: Opening database at '/var/lib/oml2/installtest.sq3' Apr 12 10:07:31 INFO server-io:5: New MS schema 0 _experiment_metadata subject:string key:string value:string Apr 12 10:07:31 INFO server-io:5: New MS schema 1 generator_d_lin label:string seq_no:uint32 Apr 12 10:07:31 INFO server-io:5: New MS schema 2 generator_d_sin label:string phase:double value:double Apr 12 10:07:31 INFO server-io:5: Client 'installtest:localservertest:generator' ready to send data Apr 12 10:07:32 INFO server-io:5: Client 'installtest:localservertest:generator' closed connection Apr 12 10:07:32 INFO installtest: Closing database
The database can also be checked to confirm data has been properly written. When using the default SQLite3 backend, the full path to the database appears in the server log (above). The sqlite3 client can be used to quickly inspect a dump of the data in this file.
sqlite3 /var/lib/oml2/installtest.sq3 .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE _senders (name TEXT PRIMARY KEY, id INTEGER UNIQUE);
INSERT INTO "_senders" VALUES('librarytest',1);
CREATE TABLE "_experiment_metadata" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "subject" TEXT, "key" TEXT, "value" TEXT);
INSERT INTO "_experiment_metadata" VALUES(1,NULL,NULL,NULL,NULL,NULL,'table__experiment_metadata','0 _experiment_metadata subject:string key:string value:string');
INSERT INTO "_experiment_metadata" VALUES(2,NULL,NULL,NULL,NULL,NULL,'start_time','1365725251');
INSERT INTO "_experiment_metadata" VALUES(3,NULL,NULL,NULL,NULL,NULL,'table_generator_d_lin','1 generator_d_lin label:string seq_no:uint32');
INSERT INTO "_experiment_metadata" VALUES(4,NULL,NULL,NULL,NULL,NULL,'table_generator_d_sin','2 generator_d_sin label:string phase:double value:double');
CREATE TABLE "generator_d_lin" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "label" TEXT, "seq_no" UNSIGNED INTEGER);
INSERT INTO "generator_d_lin" VALUES(1,1,1,0.0781299999216571,0.088947,'sample-1',1);
INSERT INTO "generator_d_lin" VALUES(2,1,2,0.178464999888092,0.178699,'sample-2',2);
INSERT INTO "generator_d_lin" VALUES(3,1,3,0.27906999969855,0.279303,'sample-3',3);
INSERT INTO "generator_d_lin" VALUES(4,1,4,0.379638999700546,0.37987,'sample-4',4);
INSERT INTO "generator_d_lin" VALUES(5,1,5,0.480201999656856,0.480439,'sample-5',5);
INSERT INTO "generator_d_lin" VALUES(6,1,6,0.580763999372721,0.580999,'sample-6',6);
INSERT INTO "generator_d_lin" VALUES(7,1,7,0.681317999958992,0.681549,'sample-7',7);
INSERT INTO "generator_d_lin" VALUES(8,1,8,0.781868999823928,0.782154,'sample-8',8);
INSERT INTO "generator_d_lin" VALUES(9,1,9,0.882429999299347,0.882665,'sample-9',9);
INSERT INTO "generator_d_lin" VALUES(10,1,10,0.982985999435186,0.983226,'sample-10',10);
CREATE TABLE "generator_d_sin" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "label" TEXT, "phase" REAL, "value" REAL);
INSERT INTO "generator_d_sin" VALUES(1,1,1,0.0781679999781772,0.089038,'sample-1',0.0,0.0);
INSERT INTO "generator_d_sin" VALUES(2,1,2,0.178519999841228,0.178762,'sample-2',6.28317975997925,-5.54720033818512e-06);
INSERT INTO "generator_d_sin" VALUES(3,1,3,0.27909699967131,0.279363,'sample-3',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(4,1,4,0.379664999898523,0.379928,'sample-4',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(5,1,5,0.480227999854833,0.480497,'sample-5',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(6,1,6,0.580789999105036,0.581057,'sample-6',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(7,1,7,0.681343999691308,0.681606,'sample-7',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(8,1,8,0.781895999796689,0.782211,'sample-8',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(9,1,9,0.882456999272108,0.882721,'sample-9',6.28313636779785,-4.89393817133532e-05);
INSERT INTO "generator_d_sin" VALUES(10,1,10,0.983012999407947,0.983288,'sample-10',6.28313636779785,-4.89393817133532e-05);
COMMIT;
When using the PostgreSQL backend, the log should be slightly different (assuming OMLUSER is the database role created while setting the server up).
Apr 11 15:17:46 INFO OML2 Server V2.10.0 Apr 11 15:17:46 INFO OML Protocol V1--4 Apr 11 15:17:46 INFO Copyright 2007-2013 NICTA Apr 11 15:17:46 INFO psql: Sending experiment data to PostgreSQL server localhost:5432 as user 'OMLUSER' Apr 12 11:09:11 INFO psql:installtest: Accessing database Apr 12 11:09:11 INFO psql:installtest: Database does not exist, creating it Apr 12 11:16:47 INFO server-io:7: New MS schema 0 _experiment_metadata subject:string key:string value:string Apr 12 11:16:47 INFO server-io:7: New MS schema 1 generator_d_lin label:string seq_no:uint32 Apr 12 11:16:47 INFO server-io:7: New MS schema 2 generator_d_sin label:string phase:double value:double Apr 12 11:16:47 INFO server-io:7: Client 'installtest:remoteservertest:generator' ready to send data Apr 12 11:16:48 INFO server-io:7: Client 'installtest:remoteservertest:generator' closed connection Apr 12 11:16:48 INFO installtest: Closing database
The data can be checked with the pg_dump tool from the PostgreSQL server package
pg_dump -U OMLUSER -h localhost installtest
The output of this tool is quite verbose, it should contain table creation and data sample insertion code, as follows (where [...] indicate truncation of the output).
--
-- PostgreSQL database dump
--
[...]
--
-- Name: generator_d_lin; Type: TABLE; Schema: public; Owner: OMLUSER; Tablespace:
--
CREATE TABLE generator_d_lin (
id integer NOT NULL,
oml_sender_id integer,
oml_seq integer,
oml_ts_client double precision,
oml_ts_server double precision,
label text,
seq_no bigint
);
[...]
--
-- Data for Name: generator_d_lin; Type: TABLE DATA; Schema: public; Owner: OMLUSER
--
COPY generator_d_lin (id, oml_sender_id, oml_seq, oml_ts_client, oml_ts_server, label, seq_no) FROM stdin;
1 1 1 0.451680999999999999 3.23078500000000002 sample-11
2 1 2 0.551826999999999956 3.23337500000000011 sample-22
3 1 3 0.652055000000000051 3.23441000000000001 sample-33
4 1 4 0.752271000000000023 3.23523199999999989 sample-44
5 1 5 0.852474999999999983 3.23595899999999981 sample-55
6 1 6 0.952718999999999983 3.2368570000000001 sample-66
7 1 7 1.05291800000000002 3.23768000000000011 sample-77
8 1 8 1.15313599999999994 3.23822900000000002 sample-88
9 1 9 1.25338799999999995 3.23879699999999993 sample-99
10 1 10 1.35357499999999997 3.23950799999999983 sample-10 10
\.
[...]
--
-- PostgreSQL database dump complete
--
Full Reporting Chain
Testing the full reporting chain is almost exactly the same as testing the server locally. The only difference is that the oml2-generator needs to be run on a different measurement node, and the collection URL should be altered to point to the now-remote server's address or hostname.
oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \ --oml-id remoteservertest --oml-domain installtest --oml-collect SERVERNAME
Checking that the data was properly recorded can be done exactly as in the previous section.
Troubleshooting
In case of problem in the full reporting chain, the first thing to investigate is the existence of firewalling rules somewhere on the path between the injection point (measurement node) and the collection point (server). This is highly likely if the oml2-generator always fails at connecting, and no line saying INFO server-io:XX: Client 'DOMAIN:ID:APP' ready to send data appears in the server's log file. This absence might indicate other negotiation issues; you can diagnose further by running the server at log level DEBUG (-d 1), and check that a line saying DEBUG server-io:XX: New client connected is not present either, which is a definitive confirmation that the client did not connect, rather than connect and did not abide by the protocol. The oml2-server listen for connections over TCP/IPv4 (only, for now), to port 3003 by default. Traffic to this port should be allowed in all the reporting network, and in the local firewall of the reporting point. Under Linux, a rule allowing this can be inserted before any other firewalling rules in place using iptables.
sudo iptables -I INPUT -p tcp --dport 3003 -j ACCEPT sudo ip6tables -I INPUT -p tcp --dport 3003 -j ACCEPT # optional steps to save firewall configuration across reboots: sudo apt-get install iptables-persistent sudo service iptables-persistent save
Persistence of this rule across reboots is distribution-dependent. Debianishes provide an iptables-persistent package allowing to save (manually) and restore (automatically at boot) firewall rules; the iptables package under ArchLinux does this by default, provided the rules were saved with iptables-save to /etc/iptables/iptables.rules; by default, Fedora 18 saves any active rules on stop.
Installing from Source
If you want to make changes to OML itself, you'll need to check out the source from our Git repository. Instructions for this can be found about Getting and building the source from Git (you can also contribute patches if you made changes).
