Gennaro Gelao,Giuseppe Raso,Giuseppe Zito
Sezione INFN Bari
We will report here on the experience of using HTML hypertext as the support of a group of around 50 people responsible for monitoring a piece of the Aleph detector called Hadron Calorimeter(Hcal). The hypertext has been called "Hcal Control Room" since it is a kind of virtual control room used to monitor the detector. This hypertext contains all information relevant for Hcal monitoring. But it is more than an information system since it allows the collaborators to communicate among them through hypertext (thus replacing the paper logbook of the experiment) and also to monitor the apparatus directly from the hypertext.
In the past pen and paper were the main tools used to document monitoring procedures:usually a paper logbook was kept for this purpose. There is nothing more easy to use than a paper logbook, so why replace it with a Web hypertext? The main reason is the possibility of the Web hypertext to be accessed by the expert, from everywhere in the world,instantly. The Web technology makes this possible from any platform,without the necessity to install special software or to connect by remote login to the online computer: a dial-up connection to Internet is enough to access the Hcal Control Room hypertext and start monitor the apparatus and communicate with other collaborators.
Another reason for the use of the Web is that you can, not only document monitoring procedures, but actually start them directly from the hypertext. This is accomplished with links that activate scripts on the online computers and this is,of course not possible from paper! But we can, not only start monitoring tasks, but also "collect" their results,previously printed on paper, attaching them to the hypertext through hyperlinks.
The last important feature of the Web hypertext as used in this application,is the possibility to make easy the access of informations to novices. Since experiments like Aleph, run for many years the problem of the training of new collaborators is crucial.
2. Monitoring procedures documentation and execution before the Web hypertext.
The Aleph Hadron Calorimeter ,Hcal in short, is an iron-plastic streamer tube sampling calorimeter with a readout of 4608 analog signals coming from "towers" and 137 976 digital (i.e. yes-no) signals coming from readout strips. The active part is a sandwich of streamer tube planes inside the Aleph magnet iron return yoke. This is a massive iron cylindrical box with 2 meter thick walls, enclosing the inner part of the detector. Integral part of Hcal is the Muon detector wich consists of 88 muon chambers in two layers which completely surround the Aleph detector. Here the active elements are planes of streamer tubes read by a set of orthogonal strips, to measure the point where the particle has crossed the chamber. The total number of strips is 89 144. Each fired strip will give a digital (yes/no) signal. Up to now monitoring procedures for these two subdetectors were documented in many different ways.
The first problem for the collaborator is to find the most updated description of some procedure:then he will follow the steps described and perform the procedure. Up to now, the only way to cope with the complexity of such environment, was to go with the expert and look him carefully to learn how the procedure was carried out.
Another problem connected with monitoring,was that of collecting the paper output of monitoring tasks and archiving it in paper binders. The access of this information was impossible to collaborators outside Cern, and difficult even for collaborators in Cern since you had to go in the experimental area to browse through it.
For example, a monitoring procedure involved the start of a special task that would collect data and compute a "plateau" . This was done every day from people in charge;after the data was collected it was printed , inspected and then filed in a binder. If the distribution showed any problem the collaborator would note it in the logbook and then take the appropriate action. Note that almost all these activities were connected to computers, so they could be in principle automated.
3. The Hcal Control Room hypertext
Now with Web hypertext, it is possible to have all information added in a uniform way and connected to a single address:the URL of the Hcal Control Room hypertext. Computer procedures are included as scripts in the hypertext and started directly from the documentation. The output of monitoring tasks is also automatically connected to the hypertext, replacing the old paper binders. Last but not least, we have the electronic counterpart of the paper logbook with one page for each day.
The main page of the Hcal Control Room is shown in figure 1 There are four main sections:
3.1 Hcal procedure
Since most monitoring procedures are carried automatically from batch jobs,the main task of the collaborator is to check that all these computer jobs are running. This is done by clicking on the Job Status line in the main page. The correct running of these tasks is the responsibility of a number of experts that will intervene if something goes wrong.
Of course, the collaborator in shift may decide to start a monitoring procedure immediately by using the other three lines in this section. As an example,figure 2 shows the page produced by clicking on Check list. Here each line will start some monitoring task, producing an immediate result that will inform the people in shift on what is happening in that moment.
3.2 Hcal logbook
The automatic monitoring tasks will produce typically a daily output that will be attached to the daily logbook page. So, before the collaborator starts writing his comments to the daily page, he has already all monitoring tasks outputs for this day collected and attached to it :this can be seen in figure 3a . He will go through these listings by clicking on them one by one and then, if there is something wrong, write an appropriate comment attacching,if necessary, a figure (for example an event display or a plot) to clarify the problem(figure 3b). A calendar (see figure 4 ) allows access to the whole logbook. This can be seen also as a unique document figure 5 with all comments put toghether.
3.2 Hcal Performance
In this section (figure 1) you have access to all the collections of periodic reports produced either by daily monitoring tasks, by programs analyzing the data of long periods, or by the collaborators themseves ( coordinator reports). Figure 6 shows for example, what one gets if he wants to see all plateaus collected during the year.
4. Use of the Hcal Control room hypertext during the first year and problems encountered
In starting this new technology we had to face the following problems:
4.1 Training of collaborators
The first idea was that each collaborator having information to add, should write it in the HTML format and add it to the hypertext. For this reason a short course was done at the beginning of the year, before data taking. But it came out, that only very few people (5 out of 50) bothered to get enough expertise to write hypertext: so a group of Web experts has been formed. This group has tried to make more easy the addition of information to hypertext for the rest of the collaboration, by writing forms like the one shown in figure 7. The same group is also in charge for the maintenance of the hypertext.
4.2 Setting up the server,protection.
We have used the Cern httpd server modified for the Openvms platform.This runs on a Openvms Alpha workstation. A protection scheme was setup involving a password from remote users. This seems to work very well and we don't have to report any problem concerning security:but it should be said that the harm that could be done to our detector by misuse of the monitoring procedures is very little; so this issue is not critical for us.
4.3 Starting developing the logbook system.
We decided to use the tools available on our platform to ease our work,
from batch queues to spawning, from DCL command files to CERN-made programs,
and last but not least some public domain programs.
After a first attempt made us more confident with the HTML language, then we had to decide how to write command files to make this logbook really interactive. We decided not to use compiled languages because of the lenght in debugging , furthermore we did not need too much speed, because many application were simple.
Our collaboration uses mostly VMS computers, so many people were able to program in DCL and this suggested us to use this language in this work. DCL seems well suited for this application because of simplicity in programming, because it does not require compilation and because it provides powerful string manipulation functions and procedures to access and to modify computers resources.
Next problems was to understand completly the connection between DCL command files and the httpd server. The main problems were about debugging command files when the output was corrupted because of bugs, so that MOSAIC refused to display the output page. It was not always very simple, especially when command files grow more and more sophisticated, but we reduced debugging work using standard development tools and some simple debugging tricks.
All old on-line control programs have been used, but some have been converted to create html output, other have been interfaced with DCL command files which format their output, but many simple new utilities have been written in DCL and just few in fortran.
As the restricted group gained experience about programming and problem solving, the recruitment of experts for writing pages and command files about their specific expertice in monitoring begun.
We split information in many files grouped in subdirectories to have a clear structure;for information acquired every day we decided to use a file for each kind of information and for every day. These files had the date in the filename and as much as possible easy-to-understand filenames were appointed to them.
4.4 Tuning the electronic logbook.
At beginning daily files were stored in html format, in such a way that these informations were available as complete html files. In this way we made a classic software development mistake by not separating the application logic(logbook content) from the interface logic (how the logbook content is presented to users in a html document). For example every time we modified the content arrangment in these files we had to modify the command files that created them and to worry about compatibility with the old format. We realized that it would be better to save in files just the real information, and to access these through command files( figure 8), which create on flight the HTML structure, with hyperlinks and some few formatting instructions. With this strategy there are little backward compatibility problems , and every change in format can be obtained just modifing few lines in command files. The overhead in creating HTML structure on-flight gives no slow-down sensation to user, because the operation is really fast on ALPHA-VAX.
The final tuning of the electronic logbook was accomplished with the help of those of us with bigger experience with apparatus maintenance: they pointed out which were their primary needs during monitoring shifts. For example, they sorted the information in HCAL PROCEDURE [figura], as they felt more confortable for 8 o'clock daily first check of apparatus. They suggested to create the CHECKLIST page [figura] to make 8:00 o'clock check more fast, while the standard changing of colours of already used hyperlinks shows which checks are still to be done.
4.5 Images: paragrafo da eliminare ?
For immages, we used free-software and Dec-Windows utility to grab and save immages from windows of the screen. In this case we used pages as a guide for non expert users explaining who to start and use these programs. At the end of the process, a new hyperlink is put in the TODAY page poiting to the new grabbed immage. Having All files ordered on disk, it is easy and very fast to make research through the files, this would be not so on paper databes !
How reliable is this new technology? After one year of use, although we have to report a few problems, the rewards are so big, that we heartily recommend it to any collaboration.