Validating CMS event display tools with real data

This is a report on the work done for a talk to CHEP09 Conference , Prag 21-27 March 2009.

Abstract

An ideal general purpose event display for HEP experiments should:
- shield completely the user from data complexity providing an user friendly GUI
- be fast enough to allow scanning of hundreds, if not thousands, of events in a few minutes
- be light enough to allow access everywhere, also from an Internet cafe'
- produce quality images and animations for publishing and press
- highlight the most relevant event features

This is indeed a complex task and what is presented here is how the
tools now available in CMS cope with this goal. For this reason they were used to access, 
scan and analyze visually cosmic runs containing millions of events.
The results of these tests provide a picture of what to expect in the LHC environment 
and give feedback for the improvement of these tools.

Introduction

Looking at the data taken in the last cosmic runs , I found some strange events containing a peculiar pattern in tracker endcaps: strings of clusters aligned with beam line and apparently with no connection with the cosmic track. I found only one or two every 10,000 events. Since most of the time these aligned clusters weren't reconstructed as tracks, you couldn't study them with usual analysis metods (i.e. without using the event display).

At this point I would like as I said in the abstract to "access ,scan and analyze visually cosmic runs containing millions of events" in order to have some answers to questions like these:

• Were these events present in all runs?
• Were these events present in equal percentage during the whole run?
• Were these strings distributed uniformly in space (i.e. were all tracker modules interested or only a part of them)?
• Were these events present without magnetic field?
• Were these events present in Montecarlo samples?

So, you wan't find here the score card you expected: the available event display tools are all fine programs but in the brave new world of LHC experiments , if we want to use these fine tools we have to develop a system to assist them and the main purpose of this presentation is to say what and why we need this. (The answer of the riddle of the aligned clusters is at the end).

The Challenge : some numbers about CRAFT

• Number of events with cosmics tracks : around 300M divided in around 140 good runs ; but the number of events with tracks is only 7M. So expect around 1 event with tracks every 50 events.
• Events taken in around 20 days at the rate of around 300/sec.
• Each event replicated at least 3 times in around 6 different PDS
• Many thousands of dqm plots available for each run and directly accessible from disc(afs area).These plots are filled for all events in a run.
• Average number of events for each run 2.5 M stored in around 100 files of 20,000 events each one but with some very big runs (up to 15M) and very small (<1M). Each file is available on Cern cluster lxplus from castor mass storage.

• You can access a single dqm plot for all runs and process them in around 1 minute (300M processed)
• You can do plots of quantities already contained in a file in 1 minute(20,000 events processed but sometime you need to load the file from castor to disk)
• Time necessary to do a simple plot on all events of a 6 M events run: 10 hours inside CMSSW with maximum storage 1500MB; 5 hour with bare Root with maximum storage 350MB.

Three programs and a Framework

Root+ FWLite will allow the user to visualize quantities (like track pt) that can be computed with a simple method from stored quantities. FWLite is an integral part of CMSSW and it is distributed with CMSSW . It is a kind of programming user interface created to shield the user from CMSSW complexity allowing him to analyze data with simple scripts in CINT or python. Although it can in principle do everything that CMSSW can do,in practice (at the moment I write this) it is used only for simple interactive analyses of data.

Then you have the full framework (called CMSSW) that allows the user to access any class and any CMS environment data stored on CMS databases. An event display completely embedded inside CMSSW has the advantage it can use any CMSSW class and access any data contained in CMSSW databases. Unfortunately this also means an heavy use of computer and network resources that slows down considerably the visualization degrading the user experience. All the four event display tools I used for this test have dealt in some way with this problem. First of all we have the two CMS general purpose event display tools:

1. Iguana will run both completely embedded in CMSSW or as a so called "iView browser" that will display events in a special format created by normal CMSSW tasks with no graphics. The event in the special format can be read from a normal file or provided by a server with VisNet protocol.
2. Fireworks uses FWLite and in this way shields the user from CMSSW complexity. The use of FWLite is also done in such a way to optimize the use of computer resources and the result is an easier to use and faster program.

Frog works only outside CMSSW. It is based on a custom event format completely independent from CMS (it can be used to display any hep experimental data). In this way it can be used on many platforms (Windows,Linux,Mac). The CMS specific part is a normal CMSSW task that will access CMS events and store them in the frog format. The data stored contains not only geometric data but also any other kind of data that can be used later in the visualizer either to be displayed as it is (when picking the object: i.e. track pt) or to apply cuts.

The last tool used is the trackermap. A visualization tool specialized for tracker. This is a synoptic view of the tracker and is implemented both in "Iguana embedded" and as single class that uses no other CMSSW services and allow its use like a Root histogram.

Because of the use case I was validating what mattered for me more in a tool was speed and the possibility to access 3D view (I had to check tracks in space). A test of speed done scanning visually 551 events (same machine, local access to events) gave me the following result:

fireworks	2'
iguana embedded	8':30 sec
iguana iView browser	30 sec
frog	30 sec

These results and the fact that fireworks at the time I started the study didn't have 3D view and iguana iView was not yet available , forced me to use frog . I used also extensively the trackermap outside iguana. Note: at this point people that were expecting a score card will become extremely disappointed about how I am proceeding. I have discarded from the validation the two CMS ufficial programs because they are not ready. Unfortunately this is the truth. Try to look with Iguana or Fireworks at a 20,000 events sample as I did with frog: it is impossible unless you want to spend days on it. And the reason for this isn't because iguana or foreworks are slow as display program but because they are embedded in CMSSW. In fact iguana outside CMSSW is as fast as frog! So ,at this time, I had two possibilities: use frog or wait until iguana outside CMSSW was ready.I choose of course the first approach since my experience would benefit the iguana and fireworks developers. Also CMSSW developers could try , from my experience, if CMSSW could be optimized for this use case.

What means looking at events in an experiment like CMS

So, if we still want to use an event display to look at ALL events in a run, the only solution is to select the possibly interesting events and look only at them. Taking in account that we would like to look at all events and that a person , even with a a fast program, cannot scan more than a few hundreds events in an hour , than an optimum cut should take only around 1 event every 100 events in a run.

For example , for tracker monitoring, we could request only events with number of tracks > 0. If the filter is fast, this will allow us to scan one million events , in a few hours.

The user should also be able to change easily his/her selection strategy , based on the features of selected events. For example scanning the sample with tracks he could find some events with a very big number of clusters and he wants to know if there are events with large number of clusters but NO TRACKS.

Another important thing is that the user should be able easily to change from run to run without bothering with list of filenames.

Since you have so many events that you cannot even in principle look, it would be nice to have access to synoptic views that would show in a single image the main features of a run. This image should be created processing all events and used to give the user a rough idea of what's happening in the run. By putting toghether all 150 miniatures of these images(one for each run), you have a kind of visual summary of your data.

All these new requirements aren't met by the event display programs and require the use and sometime the development of new tools.

Trying to use event display to monitor CMS tracker : my experience

The result of this scan was that the events with tracks in tracker are very few (around 500 out of 20000) and that there were around 10 events with some very obvious noise.

A quick scan of other runs was made possible in frog by an automatic procedure set up by Nikita Beliy and showed that this happened in all files examined.

At this point a more thorough study was necessary. First of all I tried to classify the noisy events. There were two obvious noise patterns.

1. Events with a very big number of clusters positioned randomly in one or more of the 4 tracker parts named TIB/TOB/TID/TEC. Those events had no tracks only because the track finding algorithm will protect itself in this cases by doing nothing.
2. Events with a peculiar pattern of clusters mostly in TEC. Clusters seem to be organized in a "string" parallel to beam line. Although the modules in different layers are aligned, the clusters aren't. But the track finding algorithms manages sometime to fit one or more tracks completely included in TEC.

The first noise pattern has already been studied in detail by other people and a cause has been found (it depends on trigger) so no further study was necessary. The second type of noise was new and required now some more exhaustive visual study of events to try to characterize it. First of all it was necessary to study it during the whole run and from run to run.

We found the following possible strategy in three steps to do this and make some progress:

1. Using a synoptic view of the tracker (trackermap) do a kind of tracker x-ray showing for each module the total number of rechits during the run. This would be done for ALL EVENTS IN A RUN , for ALL RUNS and also for MC data. Comparison of these images would clarify if some anomaly was apparent in TEC and how this anomaly changed in time.
2. When this kind of visual summary of all CRAFT data is created, classify also ALL EVENTS creating small ad-hoc Root trees for each run. For each event a "classification ntuple" is stored containing around 10 quantities necessary to characterize the event (for example: number of clusters, number of tracks). This information will be used for further fast study of the problem and as basis for the third strategy step.(What I did here is to create so called "tags". Neverthless I wouldn't use this name since it refers to a global strategy to access data.)
3. Select a relatively small subsamble of events presenting anomalies. These subsamples should contain MOSTLY noise events that we could inspect visually one by one.

The implementation of this strategy required around a month mostly to learn new "tricks" necessary to process so large amount of data and then another month to do the analysis. It is incredible how difficult it is to scale from the single file analysis to the single run analysis (which contains up to 1,000 files).Considering that the single file analysis requires less than a minute with Root using an interpreted program, one would naively think that scaling to 1 hundred files using a compiled program will require less than an hour. Unfortunately also here the complexity of CMSSW will transform this simple task in a gargantuan task requiring a ridiculous amount of resources. It is clear that here some optimization is needed if we would like to debug quickly the detector when we start taking real data.If you need at least a day to obtain some simple histogram on a complete run, you react very slowly to new problems.

Implementing the strategy

The creation of the image for each run is done by a CMSSW task that requires around 1 day . It processes all events filling the trakermap. In the same time it stores a small amount of information for each event in a root tree. Although it is in principle possible to run in parallel all tasks one for each run: in practice the scarce disc resources available to a normal user on AFS wan't allow me to run more than a few runs in parallel. The complete processing of all runs that could in principle be carried out in a week, required for this reason around one month. Interestingly enough, the building of this summary would allow the discovery of new problems in tracker :for example the two blue wedges that you see in two of the discs in the bottom layer were discovered tracing their origin to a wrong cabling. The same visualization showed also clearly when problems had appeared and when they stopped: for example you can see clearly that the hole present in all rectangular layers disappears starting from this run. It also gave a quick feedback in cases when the problem could be solved by reprocessing the data.

It would help a lot if this procedure could be completely integrated in normal tracker data DQM processing. Both the trackermap and the "event classification tree" could be added as additional material to the thousands of plots already automatically available.

Once the "classification ntuple" is available, selecting the interesting events from the castor dataset can be performed automatically by running a root macro that will process the "classification tree" , create a list of events and then run the CMSSW task that creates the file in frog format reading only the selected events. This operation is relatively fast (compared to full run processing in CMSSW) and can be performed on a typical run in only an hour.

The results

1. Full CMSSW :build a skim of the events presenting this pattern and then look at them. Because of the amount of resources needed, this can be run only in batch using LSF service.
2. Root +FWLite without the classification ntuples. This can be run interactively on lxplus.
3. We create the selection using the classification ntuples. Then we use this list of events to look directly at events (or produce a skim). This second step requires a CMSSW task but uses very little cpu time and so can be run interactively on lxplus.

1. The Selector: this job is specialized for selecting events. This can be done using directly the events (method 2) or using the event classification (method 3). In this case doing interactively a list of all events in a run with an arbitrary cut (i.e. tracks in TID and in pixel) takes less than 1 minute. Otherwise using FWLite it needs 4 hours.
2. The Analyzer. A CMSSW task creating the file in the format requested by the visualizer for the selected events extracts quickly from events the information needed by the Visualizer. As soon as a new event is ready it is made available to the Visualizer. This job in method 2 works in parallel with 1 and so it takes no time.
3. The Visualizer. After a few minutes let me look at the first selected events. In an hour I can look at all selected events in a 6M events run with method 3(4 hours with method 2) . This job can run on any computer with an Internet connection.

Note that both The Selector and The Analyzer could be implemented as centralized services available to the CMS community through the Web if the CMS management decides to do so. This would clearly profit the CMS community making the task of displaying any selection of events incredibly easy in comparison to what we have now. The Selector would be a central service to select events with arbitrary queries returning the list of events satisfying the query. The Analyzer would instead digest these selections returning the events in a format ready to be seen by the Visualizer. Note that the attempt done now for an "Event Server" returning the complete events aren't good for event display for obvious reason : the complete event needs all CMSSW software to be visualized (instead of the Visualizer that can be downloaded and used even in a Internet cafe') and also needs a fast link because of the size of data to be sent.

Conclusion

The proposed strategy requires an huge amount of resources . Its use by a normal user is impractical and for this reason I propose here to include it in normal DQM monitoring adding to the thousands of plots already built for each run the following new items:

1. synoptic views of subdetectors
2. a "database" of all events with a few informations necessary to classify them.

The answer of the riddle

1. Processing a run without magnetic field showed that this pattern is absent
2. Processing a run with MC generated data showed that the pattern was present although in a slightly smaller percentage of events.
3. Processing 3 complete runs I could look at a sample of 1000 events containing this pattern. It was clear that, in almost all cases ,when the primary cosmic was present in tracker, you could find that the line along the string of clusters was connected in 3D to the cosmic track. Sometimes ,like in this event , you could see the complete track originating from the primary track.
4. So these strings of aligned clusters are secondary tracks generated by interaction of the primary track with detector materials. These low momentum tracks spiral along the magnetic field generating the pattern
5. A plot of the cluster charge confirmed this explanation. See this post to commissioning hypernews

References

• CMS event visualization (unofficial) FAQ
• Comments on draft on Framework and Edm Development hypernews