WP2 - DIOS

From ESCAPE_WIKI
Jump to: navigation, search

Data Infrastructure for Open Science

1 Introduction

The goal of the ESCAPE WP2 is to build a cloud of data services, referred to as datalake. Such an infrastructure should be seen as a modular ecosystem of services. The following sections describe the datalake architecture and define the functional elements.

2 Datalake Architecture

3 Datalake Components and Reference Implementation

3.1 Storage Technologies

Datalake Status

Institute STORAGE

Technology


Version


Quota (inode limit)

Min Free Space XROOT HTTPS GSIFTP ESCAPE VO (X509) Token Based Auth/Z perfSONAR Hosts

(bandwidth & latency)

perfSONAR PoC Storage PoC
CERN EOS v4.8.20 300 TB 30 TB YES YES YES YES NO psb01-gva.cern.ch

psl01-gva.cern.ch

Rizart Dona Rizart Dona

Riccardo Di Maria

Xavier Espinal

LAPP Federated DPM

(ALPAMED)

1.14.2 100 TB 10 TB YES YES YES YES YES lapp-ps01.in2p3.fr

lapp-ps02.in2p3.fr

P.Seraphin, F.Chollet via support-grid Stephane Jezequel

Frederique Chollet

LAPP dCache 5.2 20 TB 1 TB NO YES NO YES WiP lapp-ps01.in2p3.fr

lapp-ps02.in2p3.fr

P.Seraphin, F.Chollet via support-grid Frederic Gillardo
LAPP webdav 200 GB 150 GB NO YES NO YES NO lapp-ps01.in2p3.fr

lapp-ps02.in2p3.fr

P.Seraphin, F.Chollet via support-grid Berkay Turk
LAPP aws 200 GB 100 GB NO YES NO YES NO Not Applicable Not Applicable Frederic Gillardo
SURFsara dCache 6.0.9 98 TB 140 GB YES YES YES YES YES perfsonar-bandwidth.grid.surfsara.nl

perfsonar-latency.grid.surfsara.nl

Alexander Verkooijen Alexander Verkooijen
GSI xRootD 4.12 20 TB (1M inodes) 1 TB YES YES NO YES YES dclxdlperfsonar1.gsi.de

dclxdlperfsonar2.gsi.de

Paul-Niklas Kramp Paul-Niklas Kramp

Marek Szuba

GSI xRootD 4.11 1 TB 10 GB YES YES NO YES YES dclxdlperfsonar1.gsi.de

dclxdlperfsonar2.gsi.de

Paul-Niklas Kramp Paul-Niklas Kramp

Marek Szuba

INFN-CNAF StoRM 1.11.19 10 TB (100k inodes) 1 TB NO YES NO YES YES perfsonar-ps.cnaf.infn.it

perfsonar-ow.cnaf.infn.it

Daniele Cesini

Lucia Morganti

Andrea Ceccanti

Marcelo Soares

INFN-ROMA1 DPM ? 2 TB 200 GB YES YES YES NO ? perfsonar1.roma1.infn.it

perfsonar2.roma1.infn.it

Alessandro De Salvo Alessandro De Salvo
INFN-Napoli DPM local storage 1.14.2 68 TB 5 TB YES YES YES YES ? perfsonar.na.infn.it

perfsonar2.na.infn.it

Alessandra Doria Alessandra Doria

Bernardino Spisso

INFN-Napoli DPM Federated storage 1.14.2 46 TB 5 TB YES YES YES YES ? perfsonar.na.infn.it

perfsonar2.na.infn.it

Alessandra Doria Alessandra Doria

Bernardino Spisso

DESY dCache 6.2.2 40 TB 4 TB YES YES YES YES YES perfsonar-ps-04.desy.de

perfsonar-ps-03.desy.de

Michael Steder Aleem Sarwar

Paul Millar

CC-IN2P3 (IN2P3-CC-DCACHE) dCache 6.2.29 60 TB 1 TB YES YES YES YES YES ccperfsonar1.in2p3.fr

ccperfsonar2.in2p3.fr

Laurent Caillat Vallet Paul Musset

Adrien Georget

CC-IN2P3 (IN2P3-CC-LSST-DEST) dCache 6.2.29 20 TB 500 GB NO YES NO YES YES ccperfsonar1.in2p3.fr

ccperfsonar2.in2p3.fr

Laurent Caillat Vallet Lionel Schwarz

Paul Musset Adrien Georget

CC-IN2P3 (IN2P3-CC-LSST-SOURCE) dCache 6.2.29 20 TB 500 GB NO YES NO YES YES ccperfsonar1.in2p3.fr

ccperfsonar2.in2p3.fr

Laurent Caillat Vallet Lionel Schwarz

Paul Musset Adrien Georget

PIC dCache 5.2.20 28 TB 27.99 TB YES YES YES YES YES psb01.pic.es

psl01.pic.es

Vanessa Acín

Gonzalo Merino

Elena Planas

Vanessa Acín

Gonzalo Merino

AARNET EOS ? ? WiP WiP WiP WiP WiP ?

?

? ?
  • Here's the JIRA ticketing system. Please email/ rocket chat Rosie Bolton to request an account.

3.2 Asynchronous Data Transfer Services and Networking

3.2.1 The WLCG/OSG perfSONAR Infrastructure

WLCG [1] and OSG [2] jointly operate a network of perfSONAR [3] agents deployed world-wide, which provides an open platform that can be used to baseline network performance and debug any potential issues. In the context of the ESCAPE project, Datalake sites are encouraged to setup two perfSONAR hosts (one for bandwidth and one for latency) and then join the network. The monitoring tools that are already in place as well as the tests configuration platform are connected to the OSG perfSONAR system. Sites that want to join the network must follow the guidelines that are described here [4] after having installed the perfsonar-toolkit [5] bundle.

[1] https://wlcg.web.cern.ch/
[2] https://opensciencegrid.org/
[3] https://www.perfsonar.net/
[4] https://opensciencegrid.org/networking/
[5] https://docs.perfsonar.net/install_options.html

Configuration Platform (for admins)
perfSONAR Web Admin

The network testing concerns:

  • Latency tests that use the owping tool in order to perform the one-way delay test between the hosts
    • These tests run continuously in the background and only IPv4 is tried
  • Traceroute tests that use the traceroute tool in order to measure the number of hops that are required for hosts to reach each other
    • These tests run every 10 minutes and no IP version is forced (both IPv4 and IPv6 are tried as supported)
  • Throughput tests that use the iperf3 tool in order to perform the throughput test between the hosts
    • These tests run every 23 hours, each test has a duration of 25 seconds and the TCP protocol is used. Both IPv4 and IPv6 are tried given that they are supported by the hosts

More details can be found in the Configuration Platform.

Monitoring
ESCAPE MadDash (public)
ESCAPE CHECK_MK (needs grid certificate)
CERN Monit Grafana (public)

3.3 Storage Orchestration Service

3.3.1 Deterministic vs non-deterministic RSE

Thanks to Martin Barisits for the following explanation:

For Rucio, a deterministic RSE is an RSE where the path of a replica can purely be calculated based on the DID scope and name. (+static protocol/prefix information of the storage)

e.g. scope=test,name=file1

pathes like

  • test/file1
  • hash(scope)[0:2]/hash(name)[0:2]/test/file1
  • etc.

are all Rucio-deterministic. You can see some of the deterministic lfn2pfn methods here

https://github.com/rucio/rucio/blob/a106bb113b00803dd5b2f37a008fe211c75a313f/lib/rucio/rse/protocols/protocol.py#L118

  • __hash
  • __identity
  • __belleii
  • __xenon
  • etc.

The main motivation of this is, if you have the scope+name, you actually do not need to contact Rucio to find the location of a file on storage.


A non-deterministic RSE is an RSE where the "registration process" (outside of rucio) just defines the path based on external knowledge. So it could be anything, the process just defines this path and the path is stored in the Rucio catalog. (For deterministic RSEs, no path is stored for the replica, since it can always be quickly calculated)

Now you might ask though, what happens if you want to transfer data (already in Rucio on a deterministic RSE) to a non-deterministic RSE. That's where it gets tricky. We also use algorithms there (historically we call them surl algorithms). They are very similar to the deterministic ones, but in contrast to these you are allowed to use additional information in these algorithms, such as the dataset a file is member of, metadata, ...

Eg. this is the one used in ATLAS https://github.com/rucio/rucio/blob/a106bb113b00803dd5b2f37a008fe211c75a313f/lib/rucio/common/utils.py#L529

It will use dataset information and build the path. This has the advantage that eg. tape systems can colocate the files of the same dataset on the same tape.

And that is the main difference, just with the scope+name you cannot find out the location of a file on a non-deterministic RSE. You would have to additionally query the rucio catalog, for metadata, or for dataset membership (etc.) to be able to calculate the path. (Or just ask the Rucio catalog what the file path is).

So in short

  • deterministic RSE: path can be calculated just with scope+name
  • non-deterministc RSE: Additional metadata can be used to calculate path, or even define the path just outside of Rucio.

3.4 Content Delivery and Caching

INFN/CNAF - Maintainer: CIANGOTTINI Diego


IN2P3/LAPP - Maintainer: MUSSET Paul


CERN - Maintainer: DI MARIA Riccardo

3.5 Authentication, Authorization, and Identity Services

The IAM Service Documentation: https://indigo-iam.github.io/escape-docs/

The ESCAPE IAM Service: https://iam-escape.cloud.cnaf.infn.it/

CLI for managing OpenID Connect tokens: https://github.com/indigo-dc/oidc-agent

3.6 Event-Driven Data Management and Data Processing

3.7 Information and Configuration System

General:

  • In order to configure Rucio, the CRIC (Computing Resource Information Catalogue) information system is utilized.
  • The current instance is hosted inside CERN and is maintained by the CRIC team.
  • Storage providers must hold an account. Via the CRIC WebUI they are able to configure the following for their RSEs:
    • FTS instance to be used when triggering replica creation (file transfers via TPC).
    • Custom parameters for their RSEs. These get mapped directly as Rucio attributes.
    • QoS class (as a custom parameter).
    • Various Rucio relevant flags (verify_checksum, write_availability, read_availability, etc.).
    • Protocols to be used when interacting with the RSE (flavour, endpoint, basepath, Rucio priority).
    • Distances to other RSEs. Every RSE (source, destination) pair has a ranking value that describes how far or close those RSEs are (the interpretation is arbitrary, could be bandwidth/latency or something equivalent). This is a relative value (lower is better, 0 is best) and is used from Rucio whenever it has to satisfy rules when no explicit RSE is defined (for example triggering replica creation only based on QoS tags).
  • The mapping logic between what is configured in CRIC and what gets mapped to Rucio can be found here. This code must be executed periodically in order to ensure proper config synchronization.
  • The CRIC configuration is also available as a JSON via REST API.

Notes:

  • The permission schema, as implemented to this day (9/4/2021), restricts the following
    • Only assigned users can modify certain SITES
    • Only assigned users can modify certain RSEs
  • The permission schema, as implemented to this day (9/4/2021), allows the following:
    • Everybody can modify custom parameters to all RSEs
    • Everybody can modify service protocols for all RSEs
  • Modifying the distances from the WebUI is currently not encouraged as the full functionality is not there yet.
    • Currently there are 17 RSEs that are active. All RSE pairs have a distance ranking value of "15". This was added manually in order to make it easier in the future to bring an RSE pair closer (just by subtracting 1).
    • Full functionality pending, until then manual intervention has to be requested (contact here).
  • If you need an account, please contact here.


4 Datalake Integration and Deployment

4.1 Monitoring

Instructions on how to gain access to the ESCAPE Grafana instance can be found here.

Once you have completed the instructions you can access the ESCAPE Grafana homepage here.

Architecture

4.2 Functional and Stress Testing

Gfal tests Gfal tests performing direct transfers between sites are up and running, reporting to the dashboard here: https://monit-grafana.cern.ch/d/TMScKNjWk/gfal-testing?orgId=51&var-bin=$__auto_interval_bin&var-status=All

A quick check to see which sites are failing is to look with "status=FAILED" - https://monit-grafana.cern.ch/d/TMScKNjWk/gfal-testing?orgId=51&var-bin=$__auto_interval_bin&var-status=FAILED

Rucio-level tests SKAO has spun up an instance on the STFC cloud machine that will be used to run continuous automatic datalake tests. WP2 colleagues are welcome to collaborate and can be given access to this machine by getting in touch with James Collinson and Rohini Joshi.


The automated rucio testing framework can be found here:

https://github.com/ESCAPE-WP2/rucio-analysis An example test has been defined in a yaml file (https://github.com/ESCAPE-WP2/rucio-analysis/blob/master/etc/tests.yml), this is run by a cron job. Tests are run under the rucio_testing scope. Test parameters cover: file size, number of files to upload, and sites to target in the source / destination matrix.

Now that we have this framework in place, we welcome suggestions for versions of this test that represent the range of use cases we have in ESCAPE - collaborators can add new yaml files, and (if needed) new python tests (to define different data moving logic) to the github or discuss test requirements with the SKA / DepOps team (use the rocket chat channel).

Please read the "read me" for more info and get in touch on the "Datalake_status_and_support" rocket chat channel.

4.3 Pilot Phase

4.4 Prototype Phase

Presentations and Talks

Indico@IN2P3

Conferences/Workshops

WP2 Fortnightly Meetings

Date Topics Speakers Notes
March 7th, 2019 News

Rucio workshop: outcome in the scope of ESCAPE

Simone Campana

Mario Lassnig, Martin Barisits

April 4th, 2019 News

QoS and ideas for Task 2.2

Simone Campana

Paul Millar

April 18th, 2019 Caching Daniele Spiga
May 2nd, 2019 News

HammerCloud

Simone Campana

Aristeidis Fkiaras, Jaroslava Schovancova

May 16th, 2019 Datalake Ideas from FAIR Kilian Schwarz Minutes
June 6th, 2019 F2F Preparations, Technical Input

AAI

Simone Campana, Rosie Bolton

Andrea Ceccanti

Minutes
June 20th, 2019 LinkedIn

Preparation of the WP2/WP5 F2F

Simone Campana Minutes
September 4th, 2019 Status of Deliverable D2.1

Status of the ESCAPE Rucio Testbed

IAM Integration with Experiment Specific Auth

Simone Campana

Aristeidis Fkiaras

M. Delfino

Minutes
September 25th, 2019 Status of Deliverable D2.1

PV2020

Simone Campana Minutes
October 9th, 2019 ESCAPE RockChat as Tool for a Discussion Forum

Task 2.1: Status of Pilot and Next Steps

DataLake Status

CERN XCache Status

Simone Campana

Xavier Espinal

Aristeidis Fkiaras

Riccardo Di Maria

Minutes
October 23rd, 2019 Hiring Update: SKA Job Advert, Deadline November 29th

Task 2.3: Integration with Compute

CERN XCache Plan

Rosie Bolton

Yan Grange

Riccardo Di Maria

Minutes
November 20th, 2019 Task 2.4: Networking and perfSONAR Rizart Dona Minutes
December 4th, 2019 News

Task 2.5: Authentication and Authorization

Simone Campana

Andrea Ceccanti

Minutes
December 18th, 2019 News

Task 2.2: Data Lake Orchestration

Simone Campana

Paul Millar

Minutes
January 1st, 2020 News

Task 2.1: status of pilot and next steps

Simone Campana

Aristeidis Fkiaras, Xavier Espinal

January 22nd, 2020 News and Preparation of ESCAPE Progress Meeting

Work Packages Round Table: Task 2.1

Work Packages Round Table: Task 2.2

Simone Campana

Xavier Espinal

Paul Millar

Minutes
January 29th, 2020 News and Preparation of ESCAPE Progress Meeting

Discussion on ESCAPE Compute

Round Table

Simone Campana

Riccardo Di Maria

Paul Millar

Minutes
February 12th, 2020 News on Tape-ISO certification meeting TBC

Action Plan

Simone Campana

Xavier Espinal

Minutes
February 26th, 2020 News

Task 2.1: status of pilot and next steps

Simone Campana

Aristeidis Fkiaras, Xavier Espinal

March 11th, 2020 Task Updates: Current Activities and Short-Term Plans

LOFAR Data Injection, Distribution and Access: Status Update

WP2-Tasks Leaders

Riccardo Di Maria

Minutes
March 25th, 2020 News and Rolling Plan

Datalake Data Access: CMS Experiment Use Case for Open and Embargoed Data

Xavier Espinal

Diego Ciangottini

Minutes
April 8th, 2020 News and Short Tasks Update

From the Detector to the DataLake: Gamma-Ray Telescopes Data Injection Tests at PIC

WP2-Tasks Leaders

Agustin Bruzzese

Minutes
April 22nd, 2020 Action List Review

Early Implementation and First Tests of Storage QoS

Xavier Espinal

Aleem Sarwar

Minutes
May 6th, 2020 Proposal: ESCAPE Data Lake Deployment and Operations

LOFAR Data Injection, Distribution and Processing

Xavier Espinal

Yan Grange

Minutes
May 20th, 2020 News and DepOps Team Discussion

LOFAR Data Injection, Distribution and Processing (Part 2)

ATLAS Data Analysis in the ESCAPE Datalake

Xavier Espinal

V.N.Pandey

Stéphane Jezequel

Minutes
June 3rd, 2020 News

Involvement of the ESCAPE Datalake in the MECHANICS Project (ICT-40-2020)

Task Updates: Activities and Short-Term Plans

XCache Caching Layer Experience in CC-IN2P3

Xavier Espinal

Xavier Espinal

WP2-Tasks Leaders

Paul Musset

Minutes
June 17th, 2020 News

Deployment and Operations Team - Kick-Off

Monitoring and Dashboards Update

Xavier Espinal

Rosie Bolton

Rizart Dona

Minutes
July 1st, 2020 News

PIC Gamma-Ray Data Injector Prototype Update

Content Delivery and Data Caching Activities

Xavier Espinal

Agustin Bruzzese

Riccardo Di Maria

Minutes
July 15th, 2020 News

QoS: Experiments Input and Storage Endpoints Deployment

Content Delivery and Data Caching Activities

Xavier Espinal

Paul Millar

Riccardo Di Maria

Minutes
July 29th, 2020 Integration with Compute

Content Delivery and Data Caching Activities

Yan Grange

Riccardo Di Maria

Minutes
August 26th, 2020 News

Rucio/JupyterLab Integration Project

Rucio/JupyterLab Integration Project - Next Steps

Xavier Espinal

Muhammad Aditya Hilmy

Riccardo Di Maria

Minutes
September 9th, 2020 News

DataLake Status and Plan

Pilot DataLake Assessment: Towards a Dress Rehearsal Exercise

Xavier Espinal

Riccardo Di Maria

WP2-Tasks Leaders

Minutes
September 23rd, 2020 News

Update on XCache Activities in IN2P3

Pilot DataLake Assessment: Tasks Progress Updates

Xavier Espinal

Ghita Rahal, Paul Musset

WP2-Tasks Leaders

Minutes
October 7th, 2020 News

Pilot DataLake Assessment: EOS EULAKE Update

Pilot DataLake Assessment: Experiment Data Injection Update

ESCAPE Namespace Authorization Proposal

Pilot DataLake Assessment: QoS/Data Lifecycles Update

Pilot DataLake Assessment: DataLake Automated Tests and Monitoring

Xavier Espinal

Xavier Espinal

Riccardo Di Maria

Andrea Ceccanti

Muhammad Aleem Sarwar

Rizart Dona

Minutes
October 21st, 2020 News

WP2 Workshop Preparation

Data Injection Demonstrators (Discussion)

Automated Rucio Transfer Tests

Xavier Espinal

Xavier Espinal

Riccardo Di Maria

Rob Barnsley

Minutes
November 4th, 2020 WP2 Workshop Preparation

Dress Rehearsal: Execution Plans

Data Geolocation Visualiser

Rosie Bolton

WP2-Tasks Leaders

Berkay Turk, Frédéric Gillardo

Minutes
November 18th, 2020 News

DataLake FDR: Experiment Tests and Logbook

Rosie Bolton

Experiments, Sites, and WP2-Tasks Leaders

Minutes
December 2nd, 2020 News

LOFAR Long-Term Archive Data Transfers

Xavier Espinal

Yan Grange

Minutes
January 13th, 2021 News

Deliverable 2.2 Preparation: Assessment and Analysis of the Performance of the Pilot Data Lake

Xavier Espinal

Rosie Bolton

Minutes
January 27th, 2021 News

Updates from Participating Institutes

AOB

Xavier Espinal

IFAE, NIKHEF, ASTRON, RUG, INFN

AOB

Minutes
February 10th, 2021 News

Updates from Participating Institutes

AOB

Rosie Bolton, Xavier Espinal

Surf/SARA, GSI, NIKHEF, ASTRON, DESY

AOB

Minutes
February 24th, 2021 News

Updates from Participating Institutes

AOB

Rosie Bolton, Xavier Espinal

SKAO, CERN, LAPP

AOB

Minutes
March 10th, 2021 News

EOSC - Future update

ESCAPE Data Lake and HPC integration: CMS and CINECA

Rosie Bolton, Xavier Espinal

Ian Bird

Diego Ciangottini

Minutes
March 24th, 2021 News

DIOS Second Phase, towards DAC21

Rosie Bolton, Xavier Espinal

Andrea Ceccanti, Paul Millar, Riccardo Di Maria, Rosie Bolton, Xavier Espinal, Yan Grange

Minutes
April 21th, 2021 News

Report and discussion from the DIOS-ESAP joint meeting

Discussion on DIOS Second Phase and DAC21

Depops hot topics

Rosie Bolton, Xavier Espinal

Yan Grange

All

Depops weekly chair

Minutes
May 5th, 2021 News

DIOS Second Phase/DAC21: roadmap and task forces

Progress report on commercial clouds-datalake integration

AOB

Rosie Bolton, Xavier Espinal

Rosie Bolton, Xavier Espinal

Frederic Gillardo

Riccardo Di Maria, Rizart Dona

Minutes
May 26th, 2021 News

A1 - Data Injection, Replication and Processing

A2 - Multi-RUCIO, continuous testing and Monitoring

A3 - AAI, token integration, ACLs and networks

Rosie Bolton, Xavier Espinal

Riccardo, Yan and Frederic

Rosie, Rizart, Agustin

Andrea, Paul, Marek

Minutes
June 2nd, 2021 News

Short updates from DAC21 Activities

Depops hot topics

Rosie Bolton, Xavier Espinal

Agustin Bruzzese, Andrea Ceccanti, Marek Szuba, Paul Millar, Riccardo Di Maria, Rizart Dona, Rosie Bolton, Yan Grange, Frederic Gillardo

Depops weekly chair

Minutes
June 16th, 2021 News

Activity-2: Multi-RUCIO, continuous testing and Monitoring

Updates from DAC21 Activity Groups

Rosie Bolton, Xavier Espinal

Agustin Bruzzese, Rizart Dona, Rosie Bolton

Agustin Bruzzese, Andrea Ceccanti, Marek Szuba, Paul Millar, Riccardo Di Maria, Rizart Dona, Rosie Bolton, Yan Grange, Frederic Gillardo

Minutes
June 30th, 2021 News

Advancements towards the token-based authentication

Activity-One progress update: Data Injection, Replication and Processing

Short updates from DAC21 Activity Groups

Rosie Bolton, Xavier Espinal

Rizart Dona

Riccardo Di Maria, Yan Grange, Frederic Gillardo

Agustin Bruzzese, Andrea Ceccanti, Marek Szuba, Paul Millar, Riccardo Di Maria, Rizart Dona, Rosie Bolton, Yan Grange, Frederic Gillardo

Minutes


July 14th, 2021 News

Using Redmine to describe RI goals, metrics and KPIs fro DAC21

Partners and RIs round table: summer/fall status and plans

Short updates from DAC21 Activity Groups

Rosie Bolton, Xavier Espinal

Jutta Schnabel, Rosie Bolton

All

Agustin Bruzzese, Andrea Ceccanti, Marek Szuba, Paul Millar, Riccardo Di Maria, Rizart Dona, Rosie Bolton, Yan Grange, Frederic Gillardo

Minutes


July 28th, 2021 News

Metadata-related progress

Short updates from DAC21 Activity Groups

Rosie Bolton, Xavier Espinal

Dr James Collinson, Dr Rob Barnsley, Rohini Joshi

Agustin Bruzzese, Andrea Ceccanti, Marek Szuba, Paul Millar, Riccardo Di Maria, Rizart Dona, Rosie Bolton, Yan Grange, Frederic Gillardo

Minutes
August 25th, 2021 News

Updates on ESAP and Data Lake integration

Short updates from DAC21 Activity Groups

Rosie Bolton, Xavier Espinal

Yan Grange

Agustin Bruzzese, Andrea Ceccanti, Marek Szuba, Paul Millar, Riccardo Di Maria, Rizart Dona, Rosie Bolton, Yan Grange, Frederic Gillardo

Minutes
September 8th, 2021 News

DAC21 - Activity 2 updates: Multi-Rucios, Testing and Monitoring

DAC21 - Activity 3 updates: AAI, token integration, data-access ACLs

(tbc) Data Lake as a Service Demo

Rosie Bolton, Xavier Espinal

Agustin Bruzzese, Rizart Dona, Rosie Bolton

Andrea Ceccanti, Marek Szuba, Paul Millar

Muhammad Aditya, Riccardo Di Maria

Minutes
September 16th, 2021 News

Data Lake as a Service demo

Monitoring update

Discussion on potential use cases for EGI-ACE

Rosie Bolton, Xavier Espinal

Muhammad Aditya, Riccardo Di Maria

Alba Vendrell Moya

Marek Szuba

Minutes
October 7th, 2021 ATLAS activities during the DAC21 week

KM3Net activities during the DAC21 week

FAIR activities during the DAC21 week

CMS activities during the DAC21 week

Arturo Sanchez Pineda, Stephane Jezequel

Mieke Bouwhuis

Marek Szuba

Daniele Spiga, Diego Ciangottini

Minutes

WP2 Tasks Meetings

Date Topics Task Leader Notes
November 15th, 2019 Task 2.3: First Round Table Yan Grange (ASTRON, the Netherlands Institute for Radio Astronomy) Minutes
December 11th, 2019 Task 2.1: Christmas Checkpoint Xavier Espinal (CERN) Minutes
December 18th, 2019 Task 2.3: Updates on Use Cases and System Yan Grange (ASTRON, the Netherlands Institute for Radio Astronomy) Minutes
February 5th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
February 19th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
February 19th, 2020 Task 2.5: ESCAPE Progress Meeting - AAI Session Preparation Andrea Ceccanti (INFN) Minutes
February 24th-25th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Martin, Mario, Aris, Paul and Aleem at CERN Report
March 9th, 2020 Task 2.4: Networking Task Catch-Up Rosie Bolton (SKAO) Minutes
March 11th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
March 18th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
April 1st, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Aleem Sarwar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
April 15th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
April 29th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
May 7th, 2020 Task 2.1/2.3: ESCAPE Data Caching Initiative Riccardo Di Maria (CERN) Minutes
May 13th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
May 27th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
June 10th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
June 22nd, 2020 Task 2.1/2.3: ESCAPE Data Caching Initiative Riccardo Di Maria (CERN) Minutes
June 24th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting

ESCAPE CTA QoS workflows

Paul Millar (DESY, The Deutsches Elektronen-Synchrotron)

Frederic Gillardo & Nadine Neyroud (CNRS (CC-IN2P3))

Minutes
July 8th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting

ESCAPE SKA QoS workflows

Paul Millar (DESY, The Deutsches Elektronen-Synchrotron)

Rosie Bolton & Rohini Joshi (SKA)

Minutes
July 22nd, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting

ESCAPE FAIR QoS workflows

Paul Millar (DESY, The Deutsches Elektronen-Synchrotron)

Marek Szuba (GSI)

Minutes
July 24th, 2020 Task 2.1/2.3: ESCAPE Data Caching Initiative

Disk-Caching-On-The-Fly vs. XCache

Riccardo Di Maria (CERN)

Serhat Atay (GSI)

Minutes
August 19th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
September 2nd, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
September 16th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
September 30th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
September 30th, 2020 Data Injection Demonstrators Riccardo Di Maria (CERN) Minutes
October 14th, 2020 Data Injection Demonstrators Riccardo Di Maria (CERN) Minutes
October 14th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
October 28th, 2020 Data Injection Demonstrators Riccardo Di Maria (CERN) Minutes
October 28th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
November 4th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
November 11th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
November 11th, 2020 Data Injection Demonstrators: FDR Preparation Riccardo Di Maria (CERN) Minutes
November 25th, 2020 Data Injection Demonstrators: Post-FDR Riccardo Di Maria (CERN) Minutes
November 25th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
December 16th, 2020 Task 2.2: ESCAPE QoS Fortnightly Meeting

PANDA QoS Architecture

Paul Millar (DESY, The Deutsches Elektronen-Synchrotron)

Marek Szuba (GSI)

Minutes
January 20th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
February 3rd, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
February 17th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
March 3rd, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting

QoS for LOFAR

Paul Millar (DESY, The Deutsches Elektronen-Synchrotron)

Pandey & Yan (ASTRON)

Minutes
March 17th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
March 31st, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
April 14th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
April 28th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
May 12th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
June 2nd, 2021 ActivityOne - Data Injection, Replication, and Processing Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
June 2th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
June 9th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
June 9th, 2021 Task 2.2: ESCAPE QoS Fortnightly Meeting Paul Millar (DESY, The Deutsches Elektronen-Synchrotron) Minutes
June 23th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
July 7th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
July 21th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
August 4th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
August 11th, 2021 ActivityOne - QoS Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
August 18th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
September 1st, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes
September 23th, 2021 ActivityOne - Data Injection, Replication, and Processing Riccardo Di Maria, Yan Grange, Frederic Gillardo Minutes

Webinar

Date Topics Speakers Sites participated Notes
January 13th, 2020 Task 2.5: OpenID-Connect authentication for dCache sites Paul Millar

Christian Voss

Aleem Sarwar

IN2P3, LAPP, PIC, SURFsara Minutes
April 2nd, 2020 Task 2.5: ESCAPE AAI Webinar Andrea Ceccanti - Q&A Session

Others

Data and Analysis Challenge (DAC21)

This section provides information about different use cases from ESFRI facilities for DAC21.

Services

Use cases

Documentation

Link to other Work Packages