WAVEMERGER

version 1.0

Thomas Feser
university of Mainz
Feb, 2002, south pole

introduction

The goal of the software is to merge TWR DAQ events and muon DAQ events into a single stream of data. This is mainly done by selecting those events of both data streams which have the closest GPS times to each other.

Besides the main wavemerger executable there is a second application called cleanup.wm.pl, which basicaly does some clean up work and keeps you (the poor winterovers...) informed if something goes seriously wrong. This will hopefully never happen. ;-)

The package constists basically of two executable files:

  1. wavemerger - the actual application.
  2. cleanup.wm.pl - a little perl script that does the dirty work for us.
Since everything is quite primitive at the moment, you find all the source code and the Makefile together with the object files and the two executables above in one directory; currently ~monitor/wavemerger.

how to handle the WAVEMERGER package:

  1. start/stop

    2. To start WAVEMERGER,
    1. log in as user monitor on bos2
    2. change into ~/wavemerger
    3. confirm that there is no other instance of wavemerger running (ps -x |grep wave)
    4. remove the lock file
      (usually /home/monitor/wavemerger/log/lock.wavemerger)
      if it is present
    5. say crontab -e. Below the monitoring entries, enter the following lines:
      0,15,30,45 * * * * /home/monitor/wavemerger/wavemerger &> /home/monitor/wavemerger/log/msg.txt &
      40 * * * * /home/monitor/wavemerger/cleanup.wm.pl &> /home/monitor/wavemerger/log/cuscr.txt &
      I also added these lines to the monitoring cron_job_all file; so if you restart the monitoring, the cron daemon will know about wavemerger, too.

    The stopping procedure is a bit primitive...
    1. just delete the crontab entries for wavemerger by
      crontab -e. (not crontab -r!!!)
    2. If no new input files arrive, WAVEMERGER will stop running after a while in a controlled manner. If you can't or don't want to wait, kill it. (Look for the pid with the command
      top
      or by saying
      ps ax | grep wav and ps aux | grep cleanup.
      Then say
      kill -9 pid
      for both processes. This, however, will result in the lock file remaining at its location. Wavemerger can not be restarted unless this file is removed by hand.

  2. Configuration

    3. The confoguration of WAVEMERGER is all done in its header file. There you will find everything you desire - input directories etc. Be careful when changing directories - the last "/" is important, since acces to the files is done by concatenating dir names and file names. If you find that the incoming directories of wavemerger grow too big, you can decrease the time for which the incoming files are kept there. Since that task is handled by the cleanup script, you have to configure this number there.

  3. The Log File

    4. location can be changed - as all other configurable parameters - in the file wavemerger.h. The default place for it is
    /home/monitor/wavemerger/log/log.wavemerger.
    Wavemerger keeps track about what it's doing in this file. The file is always opened in append mode. If it grows too big, just delete it while the program is NOT running. The cleanup perl script cleanup.wm.pl will also keep an eye on this. It tries to delete the log file as soon as it grows larger than 100 Megs, however, it will only be able to do so if wavemerger is not running at the moment of that check. A secure way is to stop wavemerger and delete the file by hand, should that be neccessary. If you realise that the code of wavemerger has a bug (which it surely has), want to fix it and need debug output, you can set the debug-flags in wavemerger.h or in the code itself. A hellish load of debug output will then go into that logfile. Depending on how closely you followed the crontab instructions above, there will also be the file
    /home/monitor/wavemerger/log/msg.txt.
    This holds the short information wavemerger screams when it is ivoked; you can check by the date and contents of this file how the last attempt to start wavemerger ended. This does not provide much information, though. Rather go for the central log file.

  4. The File Handling

    5. of WAVEMERGER is done by the two classes
    1. f2k_handler
      2. and
    2. wv_handler
    that are very similiar to each other. The most important methods in both classes are These methods make up the nucleus of the file handling. If the incoming events of TWR and muon daq are very far from each other, the methods are called; which, however, only call the routines above in an appropriate way.

    Both file handling methods keep track of the input files they have processed already in specific files in the log directory (default: /home/monitor/wavemerger/log)

    1. wm_wv_handler_processed_files.txt
    2. wm_f2k_handler_processed_files.txt.

    When jumping to a new input file, the new incoming file is compared to the contents of these file lists. You can edit or delete these lists if you wish wavemerger to restart processing the files in its incoming directories.

    The file handling routines basically assume that the incoming data files arrive ordered by time. If this is not the case, nothing serious should happen. However, whenever the central loop described below reaches the end of a file, this file will be considered as processed and not being opened again. This leads to a possible loss of data, but I can't think of a managable way to avoid this.

    Incoming f2k and twr files are transferred into the directories
    /net/disk3/twr_online/incoming_f2k_files and
    /net/disk3/twr_online/incoming_twr_files
    by polechomper. WAVEMERGER writes its outgoing data files into
    /net/disk3/twr_online/outgoing_f2k_files.
    Further monitoring histograms are written into
    /net/disk3/twr_online/merger_monitoring.

  5. The Event Merging

    6. is handled by wavemerger.C and relies heavily on the two classes
    1. f2kev_buffer
      2. and
    2. wv_buffer.
    These buffers spit out the right events once the times of the muon DAQ events reach the time of the twr event - that is the time at which we consider to merge two events.

    The inner merging structure located in wavemerger.C is summarized by the following algorithm. The parameter epsilon turns up at two positions in the code and is a very important value to configure... the sake of our merging effort depends on it!

    while (both new f2k and new twr files are present)
    while (f2k event time < (twr event time - epsilon))
    get next f2k event and write it into buffer
    search the f2k event buffer for the event 
    (1) closest to our last twr event
    (2) that has a multiplicity 80 trigger flag
    search the twr event buffer for the event
    closest to the previously selected f2k event
    if (the time gps time difference between these two events is smaller than epsilon)
    MERGE!
    while (twr event time < f2k event time)
    get next twr event and write it into buffer

  6. The Output Files

    7. always follow the file names of the current f2k input files. WAVEMERGER extracts year, run, day and file number of the incoming data files and forms output file names like (for instance)
    ab_2002_032_5153_015.data.mu.wf.f2k.do_not_transfer.
    Except for the very last events in a f2k file, all merged events will end up in a file with the same time stamps carried by the file they originated from. This exception happens because the output files are prepared at the same time as the input files are opened, whereas the events are written into these files only when they run in danger to be overwritten in the ring buffer. Besides files like the one above, you will also find names like
    ab_2002_032_5153_015.data.mu.wf000.f2k.do_not_transfer.
    Such a name is given when the last present twr file is finished while the current f2k file - in our example ab_2002_032_5153_015.data.mu.wf.f2k.do_not_transfer - is not yet fully processed. When WAVEMERGER is opened the next time, this (the last) f2k file is still considered as not processed. It will be opened again, as will the corresponding output file. At this point, however, WAVEMERGER will prevent the already existing output file from being overwritten. Instead, a different filename will be given.

    The handling of output files is quite difficult to understand. Don't mess with it unless you're sure you have to!

Finally I apologize to the poor folks who undertake the effort to understand this software. Especially the buffer classes are not very well constructed, and also the file handling is far from being perfect. However, WAVEMERGER merges, and it merges fast, and after all, we're at south pole.

From May, you can mail to Thomas.Feser@spole.gov with some hope to get an answer. ;-)