WP2 - DIOS
Contents
- 1 Data Infrastructure for Open Science
- 1.1 1 Introduction
- 1.2 2 Datalake Architecture
- 1.3 3 Datalake Components and Reference Implementation
- 1.3.1 3.1 Storage Technologies
- 1.3.2 3.2 Asynchronous Data Transfer Services and Networking
- 1.3.3 3.3 Storage Orchestration Service
- 1.3.4 3.4 Content Delivery and Caching
- 1.3.5 3.5 Authentication, Authorization, and Identity Services
- 1.3.6 3.6 Event-Driven Data Management and Data Processing
- 1.3.7 3.7 Information and Configuration System
- 1.4 4 Datalake Integration and Deployment
- 2 Presentations and Talks
- 3 Data and Analysis Challenge (DAC21)
- 4 Services
- 5 Use cases
- 6 Documentation
- 7 Link to other Work Packages
Data Infrastructure for Open Science
1 Introduction
The goal of the ESCAPE WP2 is to build a cloud of data services, referred to as datalake. Such an infrastructure should be seen as a modular ecosystem of services. The following sections describe the datalake architecture and define the functional elements.
2 Datalake Architecture
3 Datalake Components and Reference Implementation
3.1 Storage Technologies
Datalake Status
|
- Here's the JIRA ticketing system. Please email/ rocket chat Rosie Bolton to request an account.
3.2 Asynchronous Data Transfer Services and Networking
3.2.1 The WLCG/OSG perfSONAR Infrastructure
WLCG [1] and OSG [2] jointly operate a network of perfSONAR [3] agents deployed world-wide, which provides an open platform that can be used to baseline network performance and debug any potential issues. In the context of the ESCAPE project, Datalake sites are encouraged to setup two perfSONAR hosts (one for bandwidth and one for latency) and then join the network. The monitoring tools that are already in place as well as the tests configuration platform are connected to the OSG perfSONAR system. Sites that want to join the network must follow the guidelines that are described here [4] after having installed the perfsonar-toolkit [5] bundle.
[1] https://wlcg.web.cern.ch/
[2] https://opensciencegrid.org/
[3] https://www.perfsonar.net/
[4] https://opensciencegrid.org/networking/
[5] https://docs.perfsonar.net/install_options.html
Configuration Platform (for admins)
perfSONAR Web Admin
The network testing concerns:
- Latency tests that use the owping tool in order to perform the one-way delay test between the hosts
- These tests run continuously in the background and only IPv4 is tried
- Traceroute tests that use the traceroute tool in order to measure the number of hops that are required for hosts to reach each other
- These tests run every 10 minutes and no IP version is forced (both IPv4 and IPv6 are tried as supported)
- Throughput tests that use the iperf3 tool in order to perform the throughput test between the hosts
- These tests run every 23 hours, each test has a duration of 25 seconds and the TCP protocol is used. Both IPv4 and IPv6 are tried given that they are supported by the hosts
More details can be found in the Configuration Platform.
Monitoring
ESCAPE MadDash (public)
ESCAPE CHECK_MK (needs grid certificate)
CERN Monit Grafana (public)
3.3 Storage Orchestration Service
3.3.1 Deterministic vs non-deterministic RSE
Thanks to Martin Barisits for the following explanation:
For Rucio, a deterministic RSE is an RSE where the path of a replica can purely be calculated based on the DID scope and name. (+static protocol/prefix information of the storage)
e.g. scope=test,name=file1
pathes like
- test/file1
- hash(scope)[0:2]/hash(name)[0:2]/test/file1
- etc.
are all Rucio-deterministic. You can see some of the deterministic lfn2pfn methods here
- __hash
- __identity
- __belleii
- __xenon
- etc.
The main motivation of this is, if you have the scope+name, you actually do not need to contact Rucio to find the location of a file on storage.
A non-deterministic RSE is an RSE where the "registration process"
(outside of rucio) just defines the path based on external knowledge. So
it could be anything, the process just defines this path and the path is
stored in the Rucio catalog. (For deterministic RSEs, no path is stored
for the replica, since it can always be quickly calculated)
Now you might ask though, what happens if you want to transfer data (already in Rucio on a deterministic RSE) to a non-deterministic RSE. That's where it gets tricky. We also use algorithms there (historically we call them surl algorithms). They are very similar to the deterministic ones, but in contrast to these you are allowed to use additional information in these algorithms, such as the dataset a file is member of, metadata, ...
Eg. this is the one used in ATLAS https://github.com/rucio/rucio/blob/a106bb113b00803dd5b2f37a008fe211c75a313f/lib/rucio/common/utils.py#L529
It will use dataset information and build the path. This has the advantage that eg. tape systems can colocate the files of the same dataset on the same tape.
And that is the main difference, just with the scope+name you cannot find out the location of a file on a non-deterministic RSE. You would have to additionally query the rucio catalog, for metadata, or for dataset membership (etc.) to be able to calculate the path. (Or just ask the Rucio catalog what the file path is).
So in short
- deterministic RSE: path can be calculated just with scope+name
- non-deterministc RSE: Additional metadata can be used to calculate path, or even define the path just outside of Rucio.
3.4 Content Delivery and Caching
INFN/CNAF - Maintainer: CIANGOTTINI Diego
IN2P3/LAPP - Maintainer: MUSSET Paul
CERN - Maintainer: DI MARIA Riccardo
3.5 Authentication, Authorization, and Identity Services
The IAM Service Documentation: https://indigo-iam.github.io/escape-docs/
The ESCAPE IAM Service: https://iam-escape.cloud.cnaf.infn.it/
CLI for managing OpenID Connect tokens: https://github.com/indigo-dc/oidc-agent
3.6 Event-Driven Data Management and Data Processing
3.7 Information and Configuration System
General:
- In order to configure Rucio, the CRIC (Computing Resource Information Catalogue) information system is utilized.
- The current instance is hosted inside CERN and is maintained by the CRIC team.
- Storage providers must hold an account. Via the CRIC WebUI they are able to configure the following for their RSEs:
- FTS instance to be used when triggering replica creation (file transfers via TPC).
- Custom parameters for their RSEs. These get mapped directly as Rucio attributes.
- QoS class (as a custom parameter).
- Various Rucio relevant flags (verify_checksum, write_availability, read_availability, etc.).
- Protocols to be used when interacting with the RSE (flavour, endpoint, basepath, Rucio priority).
- Distances to other RSEs. Every RSE (source, destination) pair has a ranking value that describes how far or close those RSEs are (the interpretation is arbitrary, could be bandwidth/latency or something equivalent). This is a relative value (lower is better, 0 is best) and is used from Rucio whenever it has to satisfy rules when no explicit RSE is defined (for example triggering replica creation only based on QoS tags).
- The mapping logic between what is configured in CRIC and what gets mapped to Rucio can be found here. This code must be executed periodically in order to ensure proper config synchronization.
- The CRIC configuration is also available as a JSON via REST API.
Notes:
- The permission schema, as implemented to this day (9/4/2021), restricts the following
- Only assigned users can modify certain SITES
- Only assigned users can modify certain RSEs
- The permission schema, as implemented to this day (9/4/2021), allows the following:
- Everybody can modify custom parameters to all RSEs
- Everybody can modify service protocols for all RSEs
- Modifying the distances from the WebUI is currently not encouraged as the full functionality is not there yet.
- Currently there are 17 RSEs that are active. All RSE pairs have a distance ranking value of "15". This was added manually in order to make it easier in the future to bring an RSE pair closer (just by subtracting 1).
- Full functionality pending, until then manual intervention has to be requested (contact here).
- If you need an account, please contact here.
4 Datalake Integration and Deployment
4.1 Monitoring
Instructions on how to gain access to the ESCAPE Grafana instance can be found here.
Once you have completed the instructions you can access the ESCAPE Grafana homepage here.
4.2 Functional and Stress Testing
Gfal tests Gfal tests performing direct transfers between sites are up and running, reporting to the dashboard here: https://monit-grafana.cern.ch/d/TMScKNjWk/gfal-testing?orgId=51&var-bin=$__auto_interval_bin&var-status=All
A quick check to see which sites are failing is to look with "status=FAILED" - https://monit-grafana.cern.ch/d/TMScKNjWk/gfal-testing?orgId=51&var-bin=$__auto_interval_bin&var-status=FAILED
Rucio-level tests SKAO has spun up an instance on the STFC cloud machine that will be used to run continuous automatic datalake tests. WP2 colleagues are welcome to collaborate and can be given access to this machine by getting in touch with James Collinson and Rohini Joshi.
The automated rucio testing framework can be found here:
https://github.com/ESCAPE-WP2/rucio-analysis An example test has been defined in a yaml file (https://github.com/ESCAPE-WP2/rucio-analysis/blob/master/etc/tests.yml), this is run by a cron job. Tests are run under the rucio_testing scope. Test parameters cover: file size, number of files to upload, and sites to target in the source / destination matrix.
Now that we have this framework in place, we welcome suggestions for versions of this test that represent the range of use cases we have in ESCAPE - collaborators can add new yaml files, and (if needed) new python tests (to define different data moving logic) to the github or discuss test requirements with the SKA / DepOps team (use the rocket chat channel).
Please read the "read me" for more info and get in touch on the "Datalake_status_and_support" rocket chat channel.
4.3 Pilot Phase
4.4 Prototype Phase
Presentations and Talks
Indico@IN2P3
Conferences/Workshops
- ESCAPE Press Release
- H2020 ESCAPE Kick-Off Meeting
- Joint ESCAPE WP2/WP5 Workshop + WP5 1st Deliverable Discussion, July 1st-4th, 2019, Amsterdam Science Park
- H2020 ESCAPE Progress Meeting, February 26-27th, 2020, Royal Library of Belgium, Brussels
- 2nd ESCAPE WP2/DIOS Workshop, December 9th-10th, 2020, Virtual-Zoom
- WP5 AAI Workshop, January 19th-20th, 2021, Virtual-Zoom
- IAM Users Workshop, January 27th-28th, 2021, Virtual-Zoom
- WP2 - WP5 integration Workshop, April 6th-7th, 2021, Virtual-Zoom
- RUCIO/experiments mini-workshop, May 4th, 2021, Virtual-Zoom
- Data Lake as a Service for Open Science, September 17th, 2021, Virtual-Zoom
- 3rd ESCAPE DIOS Workshop, March 21st-23rd, 2022, Virtual-Zoom
WP2 Fortnightly Meetings
|
WP2 Tasks Meetings
|
Webinar
|
Others
Data and Analysis Challenge (DAC21)
This section provides information about different use cases from ESFRI facilities for DAC21.
