Changes between Initial Version and Version 1 of General/2bTutorials


Ignore:
Timestamp:
Feb 18, 2019, 10:37:48 PM (6 years ago)
Author:
seskar
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • General/2bTutorials

    v1 v1  
     1= = GUID Tutorial for OML 2.10 =
     2
     3In this tutorial we discuss the use of the new GUID data type (oml_guid_t) which is introduced in OML 2.10. The name GUID is an acronym for "globally unique ID". Within OML we use GUIDs to group together a collection of related measurements and/or to uniquely identify a particular measurement (e.g. to provide a unique identity for a BLOB). To illustrate these uses we shall consider a simple RF monitor application which does some simple signal processing and records results using OML.
     4
     5== Using GUIDs to group data within a measurement stream ==
     6In this example we assume that a software radio or other source is providing us with a time series of quadrature (I/Q) samples. The purpose of the RFMon example application is to collect a number of samples, perform an FFT on them and identify the peaks within the frequency domain. For every peak we discover we record information about it using OML. Lets take a look at the application template for RFMon:
     7
     8{{{
     9 1# simple rfmon application configuration template
     10 2#
     11 3defApplication('omf:app:rfmon', 'rfmon') do |app|
     12 4
     13 5  app.version(1,0)
     14 6  app.shortDescription = "RF monitor"
     15 7  app.description = %{'rfmon' is an application that identifies peaks in the RF spectrum.}
     16 8
     17 9  app.defMeasurement("peaks") do |peak|
     1810    peak.defMetric('freq', :int32)
     1911    peak.defMetric('amplitude', :int32)
     2012    peak.defMetric('sample_id', :guid)
     2113  end
     2214
     2315end
     24}}}
     25
     26Here we define a single "peak" measurement point (at lines 9-13) to represent a peak in the frequency domain for the captured samples. Each measurement records the relative frequency and amplitude of the peak. There maybe many different peaks in the frequency domain output produced by the FFT and so we assign a sample ID so that we can group all of the different measurements that occurred in the same FFT together. The following fragment of the C code for this application to shows how we generate a unique GUID and assign it to each of the related measurements.
     27
     28{{{
     29 1void
     30 2run(opts_t* opts, oml_mps_t* oml_mps)
     31 3{
     32 4  for(;;) {
     33 5    /* perform peak detection */
     34 6    _Complex samples[N_SAMPLES];
     35 7    read_samples(N_SAMPLES, samples);
     36 8    int32_t bins[N_BINS];
     37 9    fft(N_SAMPLES, samples, bins);
     3810    struct peak peaks[N_BINS];
     3911    size_t n_peaks = identify_peaks(N_BINS, bins, peaks);
     4012    /* record results */
     4113    if(n_peaks > 0) {
     4214      size_t i;
     4315      oml_guid_t id = omlc_guid_generate();
     4416      for(i = 0; i < n_peaks; i++) {
     4517        oml_inject_peaks(oml_mps->peaks, peaks[i].freq, peaks[i].ampl, id);
     4618      }
     4719    }
     4820  }
     4921}
     50}}}
     51
     52This code is performing the peak extraction in lines 6-11 that populates a struct peak array. Lines 12-18 are responsible for injecting the resulting collection of peak values to OML. At line 13 the id is defined and initialized to the return value of omlc_guid_generate().
     53
     54{{{
     55   oml_guid_t id = omlc_guid_generate();
     56}}}
     57
     58This has the effect of generating a new unique GUID value which is then used across a number of subsequent calls to oml_inject_peaks() in lines 15-17. Every measurement injected will share the same value so identifying them as all belonging to the same group. We can see this by executing a query against the database:
     59
     60{{{
     61 1rfmon=# SELECT * FROM RFMON_PEAKS;
     62 2 id  | oml_sender_id | oml_seq | oml_ts_client | oml_ts_server |  freq  | amplitude |      sample_id       
     63 3-----+---------------+---------+---------------+---------------+--------+-----------+----------------------
     64 4   1 |             1 |       1 |    4714.25265 |   4714.430439 | -87467 |       -29 |  1354744092159542987
     65 5   2 |             1 |       2 |   4714.252666 |    4714.43091 | -74973 |       -33 |  1354744092159542987
     66 6   3 |             1 |       3 |   4714.252668 |   4714.431136 | -49957 |       -42 |  1354744092159542987
     67 7   4 |             1 |       4 |   4714.252669 |   4714.431338 | -36089 |       -33 |  1354744092159542987
     68 8   5 |             1 |       5 |   4714.252671 |   4714.431536 | -30001 |       -26 |  1354744092159542987
     69 9   6 |             1 |       6 |   4714.252672 |   4714.431739 | -12488 |       -43 |  1354744092159542987
     7010   7 |             1 |       7 |   4714.252677 |   4714.431935 |  -6085 |       -35 |  1354744092159542987
     7111   8 |             1 |       8 |   4714.252678 |   4714.432135 |     13 |        39 |  1354744092159542987
     7212   9 |             1 |       9 |   4714.252679 |   4714.432331 |  12490 |       -43 |  1354744092159542987
     7313  10 |             1 |      10 |   4714.252681 |   4714.432527 |  18647 |       -33 |  1354744092159542987
     7414  11 |             1 |      11 |   4714.252684 |   4714.432742 |  49961 |       -52 |  1354744092159542987
     7515  12 |             1 |      12 |   4715.253405 |   4715.253506 | -87483 |       -26 |  1179636728160628604
     7616  13 |             1 |      13 |   4715.253415 |   4715.264028 | -74968 |       -38 |  1179636728160628604
     7717  14 |             1 |      14 |   4715.253417 |   4715.264234 | -49988 |       -44 |  1179636728160628604
     7818  15 |             1 |      15 |   4715.253418 |   4715.264433 | -36108 |       -30 |  1179636728160628604
     7919  16 |             1 |      16 |   4715.253419 |   4715.264631 | -30021 |       -25 |  1179636728160628604
     8020  17 |             1 |      17 |   4715.253421 |    4715.26485 | -12528 |       -42 |  1179636728160628604
     8121  ...
     82}}}
     83
     84== Using GUIDs to associate different measurement streams ==
     85
     86We can demonstrate how GUIDs can link different measurements by extending the RFMon application. As it stands we record the fact that a peak is discovered but we do not keep the samples from which we extracted the peak. Lets modify the application to have a second measurement point that will record the samples and related information.
     87
     88{{{
     89 1# simple rfmon application configuration
     90 2#
     91 3defApplication('omf:app:rfmon', 'rfmon') do |app|
     92 4
     93 5  app.version(1,1)
     94 6  app.shortDescription = "RF monitor"
     95 7  app.description = %{'rfmon' is an application that identifies and records peaks in the RF spectrum.}
     96 8
     97 9  app.defMeasurement("peaks") do |peak|
     9810    peak.defMetric('freq', 'int32')
     9911    peak.defMetric('amplitude', 'int32')
     10012    peak.defMetric('sample_id', 'guid')
     10113  end
     10214
     10315  app.defMeasurement("samples") do |sample|
     10416    sample.defMetric('sample_id', 'guid')
     10517    sample.defMetric('device', 'string')
     10618    sample.defMetric('center_freq', 'int64')
     10719    sample.defMetric('sample', 'blob')
     10820  end
     10921
     11022end
     111}}}
     112
     113In the listing above we have added a new sample measurement point to record our samples. This consists of an sample_id field of type GUID that identifies this measurement. The measurement also contains a number of additional fields that describe the device was captured from, the center frequency of the capture device and finally the samples themselves stored as a BLOB. The peak and sample measurements are linked by the GUID - for any sample_id in the peak measurements there will be a sample entry whose sample_id has that same value. The relationship between the peak and sample is, as we'd expect, a many-to-one relationship. So what difference does it make to the code? The listing below shows a revised version of RFMon which injects both the sample and the peaks.
     114
     115{{{
     116 1void
     117 2run(opts_t* opts, oml_mps_t* oml_mps)
     118 3{
     119 4  for(;;) {
     120 5    /* perform peak detection */
     121 6    _Complex samples[N_SAMPLES];
     122 7    read_samples(N_SAMPLES, samples);
     123 8    int32_t bins[N_BINS];
     124 9    fft(N_SAMPLES, samples, bins);
     12510    struct peak peaks[N_BINS];
     12611    size_t n_peaks = identify_peaks(N_BINS, bins, peaks);
     12712    /* record results */
     12813    if(n_peaks > 0) {
     12914      size_t i;
     13015      oml_guid_t id = omlc_guid_generate();
     13116      oml_inject_samples(oml_mps->samples, id, "n210", 434075000, samples, N_SAMPLES * sizeof(_Complex));
     13217      for(i = 0; i < n_peaks; i++) {
     13318        oml_inject_peaks(oml_mps->peaks, peaks[i].freq, peaks[i].ampl, id);
     13419      }
     13520    }
     13621  }
     13722}
     13823
     139}}}
     140
     141The only difference in the two programs is that we've inserted a calll to oml_inject_samples() at line 16. This will record the actual sample. Note that we record the sample before we record the peaks that refer to it. This ensures that the sample_id for any entry in the rfmon_peaks table will refer to a valid entry in the rfmon_samples table. Now that we have two sets of measurements linked by GUID we can use it to perform database operations that link the tables. The following query, for example, lists the peaks in terms of their absolute frequency by adding the relative frequency of the rfmon_peaks table to the center_freq of the rfmon_samples.
     142
     143{{{
     144 1rfmon=# select rfmon_peaks.freq + rfmon_samples.center_freq from rfmon_peaks, rfmon_samples where rfmon_peaks.sample_id = rfmon_samples.sample_id;
     145 2 ?column? 
     146 3-----------
     147 4 433987533
     148 5 434000027
     149 6 434025043
     150 7 434038911
     151 8 434044999
     152 9 434062512
     15310 434068915
     15411 434075013
     15512 434087490
     15613 434093647
     15714 434124961
     15815 433987517
     15916 434000032
     16017 434025012
     16118 434038892
     16219 434044979
     16320 434062472
     16421 434068894
     16522 434075043
     16623 434087461
     16724 ...
     168}}}
     169
     170== Advanced issues ==
     171With the basic GUID mechanism we can link groups of measurements and link different kinds of measurements together. In this section we'll discuss some of the more advanced concepts: NULL valued GUIDs and the possibility of collisions.
     172
     173=== Null or "non-existent" GUID values ===
     174It maybe the case that we wish to record a "NULL" GUID to indicate that there is nothing else associated with a given measurement. This is straightforward and achieved by assigning the GUID to OMLC_GUID_NULL as shown here:
     175
     176{{{
     177   oml_guid_t id = OMLC_GUID_NULL;
     178}}}
     179
     180Where a GUID is injected with this null value the database back-end will write a NULL-valued database entry.
     181
     182=== Collisions ===
     183Internally the GUID is represented as a number in the range [1, 2^64^). The mechanism to create GUIDs is probabilistically unique and this means that after collecting more than 2^32^ GUIDs have been created the probability of the same GUID being assigned twice is P(.5). The likelihood of collision is, therefore, very small even for large data sets but it is possible that GUIDs will be created that have previously been issued and it is worth bearing this in mind when very large data sets are in use.