Distributing and storing required data efficiently by means of specifically tailored data formats in the ATLAS collaboration

With the start of the LHC physics program, the ATLAS experiment started to record vast amounts of data. This data has to be distributed and stored on the world-wide computing grid in a smart way in order to enable an effective and efficient analysis by physicists. There are in principle two classes of analysis that are required. In the commissioning phase of the ATLAS experiment, low-level Event Summary Data (ESD), the result of the event reconstruction, has to be analyzed to evaluate the performance of the individual subdetectors, the performance of the reconstruction and particle identification algorithms, and to obtain calibration coefficients. For later physics analysis, it is usually sufficient to use the less detailed Analysis Object Data (AOD), which is a less-detailed version of the ESD. In the grid model of distributed analysis, these data must be transferred to Tier-2 sites before they can be analyzed. However, the large size of ESD (~1 MByte/event) constrains the amount of detailed data that can be distributed on the grid and is available on disks. In order to overcome this constraint and make the data fully available, new data sets - collectively known as Derived ESD (DESD) - have been designed. Each DESD set contains a subset of the ESD data, tailored to specific needs of the subdetector and object reconstruction and identification performance groups. Filtering algorithms perform a selection based on physics conten ts and trigger response, further reducing the data volume. Thanks to these techniques, the total volume of DESD to be distributed on the grid amounts to 20% of the initial ESD data. Contrary to the ESD, the full AOD is distributed to the Tier-2s. However, the vast number of events and the still sizable volume (~150 kByte/event) render physics analyzes directly on these datasets very inefficient. Several classes of event signatures have been identified and based on these, several Derived AODs (DAODs) can be produced. An event selection can be performed based on physics contents and trigger response with the goal to reduce the data volume of each of these new DAODs to about 1% of the initial AOD and thus reducing the required processing time for a given analysis by a factor 100. In all cases, the selection criteria and other relevant information are stored inside the DESDs and DAODs as meta-data and a connection to external databases is also established