CTA Data Analysis Use Case

From ESCAPE_WIKI
Revision as of 09:25, 16 December 2020 by Ghughes (talk | contribs) (Summary)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Summary

This is the use case for the analysis of CTA data via the ESAP.

Actors

The actors could be one of the following:

  • CTA Project PI
  • Science User
  • Member of the public

Requirements

Science Platform with ability to:

  • search the data lake for CTA data
  • search for relavent computing resources
  • search for relavent notebooks
  • the ability to submit jobs/workflows to distributed computing

Flow

  1. Search for Data
    cone search RA, DEC, solid angle, time period
    cone search Alt, AZ, solid angle, time period
    target name (Simbad), solid angle, time period
    by Source class, time period
    select data level (binned or unbinned)
    by project ID (maybe)
  2. Select data from search results or select all (Add to Basket)
  3. Find IRF that corresponds to the data selected - if data is not binned. (Add to Basket)
  4. Find corresponding metadata, log files etc .... (Download)
  5. Make quality selection cuts based on log files
    Compile the selected data with the correct IRFs (if data is not binned) into a runlist.
  6. The data can now be analysed either in interactive or batch sessions:
    1. Interactive (best for binned data):
      • Search for Jupyter hub with appropriate modules/software
      • Runlist or links to runlist are also available
      • Search for and upload appropriate notebook
    2. Batch (best for unbinned data):
      • Search for software/container appropriate for the data
      • Calculate resources needed to analyse the data
      • Find resources (eg via DIRAC)
      • Submit to runlist to compute resources (eg via DIRAC)
      • Monitor job, via GUI or other tool
      • When job is finished a report should be sent to the user.
      • Output data, metadata, logs etc ... transferred to chosen location
  7. Data verification (maybe)