Test QoS endpoints in ESCAPE datalake

From ESCAPE_WIKI
Jump to: navigation, search

Early implementation and first tests of storage QoS

This page provides the purpose and information about the demo shown on WP2 meeting on 22.04.2020 regarding Test QoS endpoints in ESCAPE datalake.

The purpose of the demo to show early implementation and first tests of storage QoS. This would be helpful to get the feedback of the community. This is not the final go and work is in progress to make a reliable system.

It was a limited demo based on available system. Right now active work is going on rucio side. And it's expected in production Q2 2020.

The demo consist of three parts.

  • Create a RSE
  • Add attribute on RSE
  • Upload file and move from one RSE to another applying QoS transition

Create a RSE

This step is optional, if you are both fine with changing attributes of an existing RSE and allowed to do so you may skip it. Moreover, keep in mind that the newly created storage element has neither access protocols nor distances to other RSEs configured, therefore it will not be possible to actually replicate files to it without further configuration.

List the available RSE's

This step can be performed multiple times to list the available RSE's in ESCAPE Datalake. At the moment there are 12 RSE's.

$ rucio list-rses
CNAF-STORM
PIC-DCACHE
DESY-DCACHE
...

Adding a new RSE QOS-A-PIC

This step is performed once. And the reason to choose this name is because PIC-DCACHE RSE is already available.

$ rucio-admin rse add QOS-A-PIC 
Added new deterministic RSE: QOS-A-PIC

Add attribute on RSE

This step is performed once. For the demo purpose we used QOS A and its suggested that LABEL should describe the characteristics and not the technology (Use FAST instead of SSD, HIGH-Latency instead of TAPE).

$ rucio-admin rse set-attribute --rse QOS-A-PIC --key QOS --value A
Added new RSE attribute for QOS-A-PIC: QOS-A

Upload file and move from one RSE to another applying QoS transition

Upload a file

This step is performed once. File name test_qos is uploaded using scope testing in DESY-DCACHE RSE.

$ rucio upload test_qos --scope testing --rse DESY-DCACHE
2020-04-20 16:01:52,856	INFO	Preparing upload for file test_qos
2020-04-20 16:01:53,037	INFO	Successfully added replica in Rucio catalogue at DESY-DCACHE
2020-04-20 16:01:53,178	INFO	Successfully added replication rule at DESY-DCACHE
2020-04-20 16:01:53,799	INFO	Trying upload with davs to DESY-DCACHE
2020-04-20 16:01:54,821	INFO	Successfully uploaded file test_qos

Make the file avaiable on a QOS=A Site

This step is performed once.

$ rucio add-rule testing:test_qos 1 'QOS=A'
1f87409a72934e0bab2e9168ae3f5d58

Check file transfer status

This step can be performed multiple times.

$ rucio list-rules testing:test_qos 
ID                                ACCOUNT    SCOPE:NAME        STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
--------------------------------  ---------  ----------------  ----------------------  ----------------  --------  ---------------  -------------------
1a97227b69034c5d9c3528d542831bde  root       testing:test_qos  OK[1/0/0]               DESY-DCACHE              1                   2020-04-20 14:01:53
1f87409a72934e0bab2e9168ae3f5d58  root       testing:test_qos  REPLICATING[0/1/0]      QOS=A                    1                   2020-04-20 14:06:47
  • Special thanks to Aris and Paul