Changes between Initial Version and Version 1 of User/0aInstallation

Feb 17, 2019, 3:41:35 PM (4 years ago)



  • User/0aInstallation

    v1 v1  
     1= Installation =
     3The OML instrumentation suite comprises a few distinct components:
     4* the OML client library (liboml2, liboml2-dev, libocomm, libocomm-dev);
     5* the OML server ( oml2-server );
     6* the OML proxy server (for disconnected experiments, oml2-proxy-server );
     7* the OML-instrumented applications.
     9There are currently two main ways to obtain and install OML, from our package repositories, as explained below, or by getting the source and building it. We recommend using the package repositories for most users, which is described below. This documentation is biased towards Debianish distributions, and primarily uses their package-naming and -subdivision (lib, server, -dev) conventions. However, it also covers installation of the relevant packages on ArchLinux as well as RPM-based systems. Once installed, the execution and testing steps are the same.
     11Depending on the desired role in the reporting chain for a gives machine, different sets of packages will need to be installed.
     13 * All machines which will run OML-enabled applications require the liboml2 package, which provides the OML Client shared library, to be installed.
     14 * The machine(s) selected to act as collection server(s) also need(s) the oml2-server .
     15 * Development and build machines will also need the development headers (liboml2-dev[el]). See OML Client Programming for details on how to write own OML measurement applications, or instrument existing ones.
     17This installation steps outlined here also applies to the OML enabled Applications, which are similarly packaged, and distributed in the same repositories.
     19== Distribution Packages ==
     20We maintain packages for Debian/Ubuntu, ArchLinux and RPM-based distros.
     22=== Adding Software Sources ===
     24==== Debian, Fedora/CentOS ====
     26The Debian and RPM packages are built on [ the OpenSuSE OBS platform]. The following links provide source configuration and package installation details for
     28* [ oml2 (server and client libraries) ]
     29* [ oml2-apps]
     31The adventurous user will also find staging builds of [ the most recent code] there. It would probably be ill-advised to setup this last source on production environments, but it is good for early tests and new features.
     33==== !ArchLinux ====
     35OML packages are available in [ the Arch User Repository]. The easiest way to install them is by using [ Yaourt], which can built and installed as follows.
     38$ wget | tar xzv
     39$ cd package-query
     40$ makepkg
     41$ sudo pacman -U package-query*.tar.xz
     42$ cd -
     43$ wget -O - | tar xzv
     44$ cd yaourt
     45$ makepkg
     46$ sudo pacman -U yaourt*.tar.xz
     49== Installing OML Components ==
     51On ArchLinux, a single package, oml2 contains all the needed libraries, programs and headers. A PKGBUILD is available in the Arch User Repository, and the same goes for the apps. There also are Git-based packages, which the adventurous user will find by suffixing -git to the name of the previous packages.
     54$ yaourt -S oml2
     57== Client library ==
     59Debian and RPM distributions offer a finer separation of packages, which can be installed separately depending on the purpose for the machine.
     60The apt-get and yum programs will automatically install all of OML's dependencies as required. (If you find that a required dependency is missing, please let us know, see contact details here.)
     61h4. Injection Point: Client Library (and Proxy Server)
     63On Debian, the client library is packaged as liboml2, and also depends on libocomm, which will be pulled at the same time. It is also a good idea to install the dummy oml2-example package, as it allows to test the installation. This is done by running the following command.
     66sudo apt-get install liboml2 oml2-example
     69If the measurement nodes are expected to run in disconnected mode, the Proxy Server will also be needed.
     72sudo apt-get install oml2-proxy-server
     75On RPM-based distributions, both come in the same package, liboml2, which can be installed with yum
     78sudo yum install liboml2
     81=== Collection Point: Server and Database Tools ===
     82On Debian, the collection server is provided by the oml2-server package.
     85sudo apt-get install oml2-server
     88A similar thing works for RPM-based distributions.
     91sudo yum install oml2-server
     94If you want to examine the contents of measurement databases, you'll need to manually install the SQLite3 client application. None of the packages directly uses the SQLite3 client, so they don't cause it to be installed automatically. On Debian, this package is named sqlite3, while it is simply named sqlite on ArchLinux and RPM-based distros. In either case, you can install in the same way as the server.
     96=== Development Workstation: Headers ===
     98On Debian, development headers are suffixed -dev. The liboml2-dev will pull the relevant libocomm-dev dependency. This can be done much in the same way as the library itself.
     101sudo apt-get install liboml2-dev
     104A very similar approach will work for RPM-based distributions, bearing in mind that development headers are suffixed -devel on these platforms.
     107sudo yum install liboml2-devel
     110== Testing the Library Installation ==
     112The oml2-example package will have installed a dummy sine generator reporting its data through OML, oml2-generator. It can be used to test that the library has been properly installed, simply by running it with the appropriate parameters to report data to the standard output (so a server is not needed at this stage).
     115oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \
     116  --oml-id librarytest --oml-domain installtest --oml-collect file:-
     119If the library has been properly installed, something similar to the following will be output.
     122Apr 12 10:03:32 INFO    OML Client V2.10.0 [Protocol V4] Copyright 2007-2013, NICTA
     123INFO    File_stream: opening local storage file '-'
     1243.141593, 628.318531, 100000
     125sample-1 1 | 0.000000 0.000000
     126protocol: 4
     127domain: installtest
     128start-time: 1365725012
     129sender-id: librarytest
     130app-name: generator
     131schema: 0 _experiment_metadata subject:string key:string value:string
     132schema: 1 generator_d_lin label:string seq_no:uint32
     133schema: 2 generator_d_sin label:string phase:double value:double
     134content: text
     1360.104363        1       1       sample-1        1
     1370.104398        2       1       sample-1        0.000000        0.000000
     138sample-2 2 | 6.283180 -0.000006
     1390.205737        1       2       sample-2        2
     1400.205807        2       2       sample-2        6.283180        -0.000006
     141sample-3 3 | 6.283136 -0.000049
     1420.306114        1       3       sample-3        3
     1430.306154        2       3       sample-3        6.283136        -0.000049
     144sample-4 4 | 6.283136 -0.000049
     1450.406637        1       4       sample-4        4
     1460.406676        2       4       sample-4        6.283136        -0.000049
     147sample-5 5 | 6.283136 -0.000049
     1480.507075        1       5       sample-5        5
     1490.507116        2       5       sample-5        6.283136        -0.000049
     150sample-6 6 | 6.283136 -0.000049
     1510.607570        1       6       sample-6        6
     1520.607609        2       6       sample-6        6.283136        -0.000049
     153sample-7 7 | 6.283136 -0.000049
     1540.708090        1       7       sample-7        7
     1550.708141        2       7       sample-7        6.283136        -0.000049
     156sample-8 8 | 6.283136 -0.000049
     1570.808660        1       8       sample-8        8
     1580.808700        2       8       sample-8        6.283136        -0.000049
     159sample-9 9 | 6.283136 -0.000049
     1600.909052        1       9       sample-9        9
     1610.909092        2       9       sample-9        6.283136        -0.000049
     162sample-10 10 | 6.283136 -0.000049
     1631.009559        1       10      sample-10       10
     1641.009598        2       10      sample-10       6.283136        -0.000049
     165INFO    Waiting for buffered queue reader thread to drain...
     166INFO    Buffered queue reader thread finished OK...
     168== Configuring and Testing the Server ==
     170The server can take command line parameters to change its behaviour, as explained in the manpage and the source:README file. Depending on the distribution, there are different ways to change the default parameters. This might be particularly needed in case the PostgreSQL backend needs to be used.
     172=== Debian ===
     174When installed, the oml2-server program will automatically start whenever the computer starts or reboots. This is done using the standard startup script mechanism of the system. On Debian/Ubuntu, starting or stopping the server manually can be done with
     177sudo service oml2-server stop # Stop the server
     178sudo service oml2-server start # Start the server again
     179sudo service oml2-server restart # Cycle the server (stop, then start)
     182The server is configured using the options specified in /etc/default/oml2-server. Here are the default contents of this file:
     185OPTS="-l 3003 --logfile=/var/log/oml2-server.log --data-dir=/tmp/"
     188The OPTS shell variable is passed to the server command line. All of the switches above are the normal oml2-server command line options, documented here. The default configuration causes the server to listen on port 3003, to write logging information to the file /var/log/oml2-server.log, and to write experiment databases in the /tmp directory. You can add any of the other oml2-server options to /etc/default/oml2-server as well. For instance, this would make the server output lots of debugging log messages to the log file:
     191OPTS="-l 3003 -d4 --logfile=/var/log/oml2-server.log --data-dir=/tmp/"
     194=== ArchLinux and RPM-based distributions ===
     196These distributions now use systemd. Controlling the server is done through systemctl.
     199$ sudo systemctl stop oml2-server.service # Stop the server
     200$ sudo systemctl start oml2-server.service # Start the server again
     201$ sudo systemctl restart oml2-server.service # Cycle the server (stop, then start)
     204Changing the startup option of the server can be done by copying /usr/lib/systemd/system/oml2-server.service to /etc/systemd/system, before modifying that copy to contain the desired startup parameters. Once done, the new parameters need to be set by re-enabling the service.
     207$ sudo systemctl reenable oml2-server.service
     210== Using the PostgreSQL Backend ==
     212If you want to use a PostgreSQL backend, you will first need to install the database server somewhere. For simplicity, we only document a database server co-located on the same machine as the OML server, however more complex setups are possible.
     213On Debian, The package is called postgresql-9.1, and can be installed with apt-get similarly to all other packages. You might also want to install the client applications postgresql-client-9.1 if you need to inspect database contents. Additionally, the pgadmin3 package provides a GUI.
     215On ArchLinux, the postgresql package contains everything. It can be directly either using yaourt, or sudo pacman -S postgresql. The pgadmin3 tool is also available.
     217On RPM-based systems, the server package is called postgresql-server, and the clients are in postgresql. As for the other distros, pgadmin3 is also available under that name.
     219The manpage contains details about the use of the PostgreSQL backend. For a quick start, after installing the database server, a user role, say OMLUSER, must first be created, which is allowed to created new databases, but nothing else. The following assumes that DBADMIN is the database admin user, usually postgres.
     222createuser -U DBADMIN --pwprompt --no-superuser --createdb --no-createrole --no-replication OMLUSER
     225The --no-replication argument is only valid with PostgreSQL-9.2. In case of error, simply remove it. A password for the newly created database user will be requested.
     226Alternatively, the role can be created through and interactive console.
     229$ psql -U DBADMIN
     230psql (9.3.2)
     231Type "help" for help.
     236ALTER ROLE
     239The oml2-server can then be configured, through command line arguments, to use the postgresql backend. Depending on the distribution, and as explained above, the happens either in /etc/default/oml2-server (Debian; in the OPTS variable) or in a copy of /usr/lib/systemd/system/oml2-server.service placed in /etc/systemd/system/oml2-server.service (systemd-based systems: ArchLinux and RPM-based; in the ExecStart statement). In the latter case, running sudo systemctl reenable oml2-server.service is needed for new parameters to be taken into account prior to restarting the service.
     241The command-line options need to be extended with parameters to enable the postgresql backend (a line containing them should already be present as a comment in the file).
     244[...] --backend=postgresql --pg-host=localhost --pg-port=5432 --pg-user=OMLUSER --pg-pass=OMLPASS
     247== Testing ==
     249=== Local Server Installation ===
     250Testing the server locally can also be done with the dummy sine generator, simply changing its collection URL to send measurement streams to the local server.
     253oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \
     254  --oml-id localservertest --oml-domain installtest --oml-collect localhost
     257As the output no longer contains OMSP data, it is shorter than before.
     260Apr 12 10:07:31 INFO    OML Client V2.10.0 [Protocol V4] Copyright 2007-2013, NICTA
     2613.141593, 628.318531, 100000
     262sample-1 1 | 0.000000 0.000000
     263INFO    OmlNetOutStream: attempting to connect to server at tcp://localhost:3003
     264sample-2 2 | 6.283180 -0.000006
     265sample-3 3 | 6.283136 -0.000049
     266sample-4 4 | 6.283136 -0.000049
     267sample-5 5 | 6.283136 -0.000049
     268sample-6 6 | 6.283136 -0.000049
     269sample-7 7 | 6.283136 -0.000049
     270sample-8 8 | 6.283136 -0.000049
     271sample-9 9 | 6.283136 -0.000049
     272sample-10 10 | 6.283136 -0.000049
     273INFO    Waiting for buffered queue reader thread to drain...
     274INFO    Buffered queue reader thread finished OK...
     277Inspecting the server log, which defaults to /var/log/oml2-server.log, should show that this client did connect properly.
     280Apr 12 10:02:08 INFO    OML2 Server V2.10.0
     281Apr 12 10:02:08 INFO    OML Protocol V1--4
     282Apr 12 10:02:08 INFO    Copyright 2007-2013 NICTA
     283Apr 12 10:02:08 INFO    sqlite: Creating SQLite3 databases in /var/lib/oml2
     284Apr 12 10:07:31 INFO    sqlite:installtest: Opening database at '/var/lib/oml2/installtest.sq3'
     285Apr 12 10:07:31 INFO    server-io:5: New MS schema 0 _experiment_metadata subject:string key:string value:string
     286Apr 12 10:07:31 INFO    server-io:5: New MS schema 1 generator_d_lin label:string seq_no:uint32
     287Apr 12 10:07:31 INFO    server-io:5: New MS schema 2 generator_d_sin label:string phase:double value:double
     288Apr 12 10:07:31 INFO    server-io:5: Client 'installtest:localservertest:generator' ready to send data
     289Apr 12 10:07:32 INFO    server-io:5: Client 'installtest:localservertest:generator' closed connection
     290Apr 12 10:07:32 INFO    installtest: Closing database
     293The database can also be checked to confirm data has been properly written. When using the default SQLite3 backend, the full path to the database appears in the server log (above). The sqlite3 client can be used to quickly inspect a dump of the data in this file.
     296sqlite3 /var/lib/oml2/installtest.sq3 .dump
     297PRAGMA foreign_keys=OFF;
     300INSERT INTO "_senders" VALUES('librarytest',1);
     301CREATE TABLE "_experiment_metadata" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "subject" TEXT, "key" TEXT, "value" TEXT);
     302INSERT INTO "_experiment_metadata" VALUES(1,NULL,NULL,NULL,NULL,NULL,'table__experiment_metadata','0 _experiment_metadata subject:string key:string value:string');
     303INSERT INTO "_experiment_metadata" VALUES(2,NULL,NULL,NULL,NULL,NULL,'start_time','1365725251');
     304INSERT INTO "_experiment_metadata" VALUES(3,NULL,NULL,NULL,NULL,NULL,'table_generator_d_lin','1 generator_d_lin label:string seq_no:uint32');
     305INSERT INTO "_experiment_metadata" VALUES(4,NULL,NULL,NULL,NULL,NULL,'table_generator_d_sin','2 generator_d_sin label:string phase:double value:double');
     306CREATE TABLE "generator_d_lin" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "label" TEXT, "seq_no" UNSIGNED INTEGER);
     307INSERT INTO "generator_d_lin" VALUES(1,1,1,0.0781299999216571,0.088947,'sample-1',1);
     308INSERT INTO "generator_d_lin" VALUES(2,1,2,0.178464999888092,0.178699,'sample-2',2);
     309INSERT INTO "generator_d_lin" VALUES(3,1,3,0.27906999969855,0.279303,'sample-3',3);
     310INSERT INTO "generator_d_lin" VALUES(4,1,4,0.379638999700546,0.37987,'sample-4',4);
     311INSERT INTO "generator_d_lin" VALUES(5,1,5,0.480201999656856,0.480439,'sample-5',5);
     312INSERT INTO "generator_d_lin" VALUES(6,1,6,0.580763999372721,0.580999,'sample-6',6);
     313INSERT INTO "generator_d_lin" VALUES(7,1,7,0.681317999958992,0.681549,'sample-7',7);
     314INSERT INTO "generator_d_lin" VALUES(8,1,8,0.781868999823928,0.782154,'sample-8',8);
     315INSERT INTO "generator_d_lin" VALUES(9,1,9,0.882429999299347,0.882665,'sample-9',9);
     316INSERT INTO "generator_d_lin" VALUES(10,1,10,0.982985999435186,0.983226,'sample-10',10);
     317CREATE TABLE "generator_d_sin" (id INTEGER PRIMARY KEY, oml_sender_id INTEGER, oml_seq INTEGER, oml_ts_client REAL, oml_ts_server REAL, "label" TEXT, "phase" REAL, "value" REAL);
     318INSERT INTO "generator_d_sin" VALUES(1,1,1,0.0781679999781772,0.089038,'sample-1',0.0,0.0);
     319INSERT INTO "generator_d_sin" VALUES(2,1,2,0.178519999841228,0.178762,'sample-2',6.28317975997925,-5.54720033818512e-06);
     320INSERT INTO "generator_d_sin" VALUES(3,1,3,0.27909699967131,0.279363,'sample-3',6.28313636779785,-4.89393817133532e-05);
     321INSERT INTO "generator_d_sin" VALUES(4,1,4,0.379664999898523,0.379928,'sample-4',6.28313636779785,-4.89393817133532e-05);
     322INSERT INTO "generator_d_sin" VALUES(5,1,5,0.480227999854833,0.480497,'sample-5',6.28313636779785,-4.89393817133532e-05);
     323INSERT INTO "generator_d_sin" VALUES(6,1,6,0.580789999105036,0.581057,'sample-6',6.28313636779785,-4.89393817133532e-05);
     324INSERT INTO "generator_d_sin" VALUES(7,1,7,0.681343999691308,0.681606,'sample-7',6.28313636779785,-4.89393817133532e-05);
     325INSERT INTO "generator_d_sin" VALUES(8,1,8,0.781895999796689,0.782211,'sample-8',6.28313636779785,-4.89393817133532e-05);
     326INSERT INTO "generator_d_sin" VALUES(9,1,9,0.882456999272108,0.882721,'sample-9',6.28313636779785,-4.89393817133532e-05);
     327INSERT INTO "generator_d_sin" VALUES(10,1,10,0.983012999407947,0.983288,'sample-10',6.28313636779785,-4.89393817133532e-05);
     331When using the PostgreSQL backend, the log should be slightly different (assuming OMLUSER is the database role created while setting the server up).
     333Apr 11 15:17:46 INFO    OML2 Server V2.10.0
     334Apr 11 15:17:46 INFO    OML Protocol V1--4
     335Apr 11 15:17:46 INFO    Copyright 2007-2013 NICTA
     336Apr 11 15:17:46 INFO    psql: Sending experiment data to PostgreSQL server localhost:5432 as user 'OMLUSER'
     337Apr 12 11:09:11 INFO    psql:installtest: Accessing database
     338Apr 12 11:09:11 INFO    psql:installtest: Database does not exist, creating it
     339Apr 12 11:16:47 INFO    server-io:7: New MS schema 0 _experiment_metadata subject:string key:string value:string
     340Apr 12 11:16:47 INFO    server-io:7: New MS schema 1 generator_d_lin label:string seq_no:uint32
     341Apr 12 11:16:47 INFO    server-io:7: New MS schema 2 generator_d_sin label:string phase:double value:double
     342Apr 12 11:16:47 INFO    server-io:7: Client 'installtest:remoteservertest:generator' ready to send data
     343Apr 12 11:16:48 INFO    server-io:7: Client 'installtest:remoteservertest:generator' closed connection
     344Apr 12 11:16:48 INFO    installtest: Closing database
     347The data can be checked with the pg_dump tool from the PostgreSQL server package
     349pg_dump -U OMLUSER -h localhost installtest
     352The output of this tool is quite verbose, it should contain table creation and data sample insertion code, as follows (where [...] indicate truncation of the output).
     356-- PostgreSQL database dump
     360-- Name: generator_d_lin; Type: TABLE; Schema: public; Owner: OMLUSER; Tablespace:
     363CREATE TABLE generator_d_lin (
     364    id integer NOT NULL,
     365    oml_sender_id integer,
     366    oml_seq integer,
     367    oml_ts_client double precision,
     368    oml_ts_server double precision,
     369    label text,
     370    seq_no bigint
     374-- Data for Name: generator_d_lin; Type: TABLE DATA; Schema: public; Owner: OMLUSER
     377COPY generator_d_lin (id, oml_sender_id, oml_seq, oml_ts_client, oml_ts_server, label, seq_no) FROM stdin;
     3781       1       1       0.451680999999999999    3.23078500000000002     sample-11
     3792       1       2       0.551826999999999956    3.23337500000000011     sample-22
     3803       1       3       0.652055000000000051    3.23441000000000001     sample-33
     3814       1       4       0.752271000000000023    3.23523199999999989     sample-44
     3825       1       5       0.852474999999999983    3.23595899999999981     sample-55
     3836       1       6       0.952718999999999983    3.2368570000000001      sample-66
     3847       1       7       1.05291800000000002     3.23768000000000011     sample-77
     3858       1       8       1.15313599999999994     3.23822900000000002     sample-88
     3869       1       9       1.25338799999999995     3.23879699999999993     sample-99
     38710      1       10      1.35357499999999997     3.23950799999999983     sample-10       10
     391-- PostgreSQL database dump complete
     395== Full Reporting Chain ==
     396Testing the full reporting chain is almost exactly the same as testing the server locally. The only difference is that the oml2-generator needs to be run on a different measurement node, and the collection URL should be altered to point to the now-remote server's address or hostname.
     399oml2-generator --amplitude 1 --frequency 1000 --samples 10 --sample-interval .1 \
     400  --oml-id remoteservertest --oml-domain installtest --oml-collect SERVERNAME
     403Checking that the data was properly recorded can be done exactly as in the previous section.
     405== Troubleshooting ==
     407In case of problem in the full reporting chain, the first thing to investigate is the existence of firewalling rules somewhere on the path between the injection point (measurement node) and the collection point (server). This is highly likely if the oml2-generator always fails at connecting, and no line saying INFO server-io:XX: Client 'DOMAIN:ID:APP' ready to send data appears in the server's log file. This absence might indicate other negotiation issues; you can diagnose further by running the server at log level DEBUG (-d 1), and check that a line saying DEBUG server-io:XX: New client connected is not present either, which is a definitive confirmation that the client did not connect, rather than connect and did not abide by the protocol. The oml2-server listen for connections over TCP/IPv4 (only, for now), to port 3003 by default. Traffic to this port should be allowed in all the reporting network, and in the local firewall of the reporting point. Under Linux, a rule allowing this can be inserted before any other firewalling rules in place using iptables.
     410sudo iptables -I INPUT -p tcp --dport 3003 -j ACCEPT
     411sudo ip6tables -I INPUT -p tcp --dport 3003 -j ACCEPT
     412# optional steps to save firewall configuration across reboots:
     413sudo apt-get install iptables-persistent
     414sudo service iptables-persistent save
     417Persistence of this rule across reboots is distribution-dependent. Debianishes provide an iptables-persistent package allowing to save (manually) and restore (automatically at boot) firewall rules; the iptables package under ArchLinux does this by default, provided the rules were saved with iptables-save to /etc/iptables/iptables.rules; by default, Fedora 18 saves any active rules on stop.
     419== Installing from Source ==
     421If you want to make changes to OML itself, you'll need to check out the source from our Git repository. Instructions for this can be found about Getting and building the source from Git (you can also contribute patches if you made changes).