WP2 - DIOS - MultiVO XCache Recipes

From ESCAPE_WIKI
Jump to: navigation, search

HTTP/WebDAV protocol: token based

The considered scenario involves a minumum of two servers where one acts as remote custodial site (origin) and the other as an XCache instance fetching and storing data from the custodial site (that exposes an http endpoint) while serving the client HTTP request.

The AuthN/Z model for this scenario can be managed purely with Escape IAM tokens.

AuthN/Z model

Escape http.png
  • To request a file in the namespace, for instance, /cms request a client should provide the xcache server with an ESCAPE access token with the following scopes:
    • storage.read:/cms
    • storage.write:/cms
  • This scopes are granted by IAM automatically when the user is added to the CMS IAM group
    • the policy for the group<->namespace mapping can only be managed by an ESCAPE IAM admin
  • The cache server is able to forward the access token provided to the custodial site

Requirements

  • At least two hosts with a docker daemon or a plain centos7 os for the manual recipe
  • Two host certificates for the servers

Docker images

Origin server

You can find the docker and configuration files for the origin server here. With the following commands you will bring up a server running an xrootd server exposing on 8443 an HTTP/WebDAV service.

 mkdir -p /mydata
 # allow docker xrootd user to write and read from your data folder
 chown -R 999:997 /mydata
 # start the server
 docker run -d --name origin \
            --net host \
            -v <hostcerts dir>/hostcert.pem:/etc/grid-security/hostcert.pem \
            -v <hostcerts dir>/hostcert.key:/etc/grid-security/hostcert.key \
            -v <ca dir>/<your CA>.pem:/etc/grid-security/certificates/DODAS.pem \
            -v /tmp/mydata:/data \
            dodasts/xrootd-escape-http

If you do not have an host certificate you can set the environment variable XRD_HOST when running the container to generate one trusted by the dev CA here

 docker run -d --name origin \
            --net host \
            -e XRD_HOST=<your machine hostname to be trusted> \
            -v /tmp/mydata:/data \
            dodasts/xrootd-escape-http


XCache server

The xcache docker and config files are defined here

Just like the origin server above you will need the following instructions.

 mkdir -p /mydata
 # allow docker xrootd user to write and read from your data folder
 chown -R 999:997 /mydata
 # start the server
 docker run -d --name xcache \ 
            --net host \
            -v <hostcerts dir>/hostcert.pem:/etc/grid-security/hostcert.pem \
            -v <hostcerts dir>/hostcert.key:/etc/grid-security/hostcert.key \
            -v <ca dir>/<your CA>.pem:/etc/grid-security/certificates/DODAS.pem \
            -e ORIGIN_HOST=https://<address of the origin server> \
            -e ORIGIN_XRD_PORT=8443 \
            -e CACHE_RAM_GB=4 \
            -v /tmp/mydata:/data \
            dodasts/xcache-escape-http


And if you do not have an host certificate, try:


 docker run -d --name xcache \
            --net host \
            -e XRD_HOST=<your machine hostname to be trusted> \
            -e ORIGIN_HOST=https://<address of the origin server> \
            -e ORIGIN_XRD_PORT=8443 \
            -e CACHE_RAM_GB=4 \
            -v /tmp/mydata:/data \
            dodasts/xcache-escape-http


Local demo with docker compose

Live demo of the docker compose environment

If you are looking for a local demo environment you can benefit from the docker compose file provided here

To build the images and run the compose file you can just do:

 git clone https://github.com/DODAS-TS/dodas-docker-images.git
 cd dodas-docker-images/docker/cachingondemand/xcache-escape-http/
 docker-compose build
 docker-compose up -d

Bare metal instructions

Host preparation

To install all the needed packages starting from a Centos7 image you can use the this script

Origin configuration

  1. cat /etc/xrootd/scitokens.cfg
 [Issuer ESCAPE] 
 issuer = https://iam-escape.cloud.cnaf.infn.it/  
 base_path = / 
 map_subject = False 
 default_user = xrootd 
  1. cat /etc/xrootd/xrootd-http.cfg
 all.export / 
 oss.localroot /data/ 
 xrootd.trace info 
 xrd.trace info 
 acc.trace debug 
 ofs.trace debug 
 sec.trace debug 
 xrd.port 8443 
 xrd.protocol XrdHttp:8443 libXrdHttp.so   
 #http.trace 
 # Uncomment the following line to get extensive debugging information 
 http.trace all debug 
 # Enable the SciTokens authorization library. 
 ofs.authorize 
 ofs.authlib libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg 
 # This particular authfile allows anonymous writes. 
 acc.authdb /etc/xrootd/Auth-file-http  
 http.header2cgi Authorization authz 
 # Boiler-plate HTTPS configuration 
 http.cadir /etc/grid-security/certificates 
 http.cert  /etc/grid-security/hostcert.pem 
 http.key   /etc/grid-security/hostcert.key 
 http.listingdeny yes 
 http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt 
  1. cat /etc/xrootd/Auth-file-http
 u * / lr 

Cache configuration

  1. cat /etc/xrootd/scitokens.cfg
 [Issuer ESCAPE] 
 issuer = https://iam-escape.cloud.cnaf.infn.it/  
 base_path = / 
 map_subject = False 
 default_user = xrootd
  1. cat /etc/xrootd/Auth-file-http
 u * / lr
  1. cat /etc/xrootd/xrootd-http.cfg
 all.export / 
 oss.localroot /data/ 
 xrootd.trace info 
 xrd.trace info 
 acc.trace debug 
 ofs.trace debug 
 sec.trace debug 
 xrd.port 8443 
 xrd.protocol XrdHttp:8443 libXrdHttp.so   
 #http.trace 
 # Uncomment the following line to get extensive debugging information 
 http.trace all debug 
 # Enable the SciTokens authorization library. 
 ofs.authorize 
 ofs.authlib libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg 
 # This particular authfile allows anonymous writes. 
 acc.authdb /etc/xrootd/Auth-file-http  
 http.header2cgi Authorization authz 
 # Boiler-plate HTTPS configuration 
 http.cadir /etc/grid-security/certificates 
 http.cert  /etc/grid-security/hostcert.pem 
 http.key   /etc/grid-security/hostcert.key 
 http.listingdeny yes 
 http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
 pss.origin https//https://<address of the origin server>:8443
 pfc.diskusage 0.85 0.95
 pfc.ram       4g
 pfc.blocksize   512k
 pfc.prefetch    0
 pss.setopt ParallelEvtLoop 10
 pss.setopt RequestTimeout 25
 pss.setopt ConnectTimeout 25
 pss.setopt ConnectionRetry 2

XRootD protocol: X509 voms from IAM

The considered scenario involves a minumum of two servers where one acts as remote custodial site (origin) and the other as an XCache instance fetching and storing data from the custodial site that exposes an xrootd endpoint while serving the client xrd request.

The AuthN/Z model for this scenario can be managed through X509 VOMS groups provided by Escape IAM.

AuthN/Z model

Escape xrd.png
  • To request a file in the namespace, for instance, /cms request a client should provide a valid X509 proxy with the ESCAPE voms extension and with the following group:
    • /escape/cms
  • The groups are associated to the user by IAM VO (CMS in this example) admin that is responsible for including only VO member in the IAM group
    • the policy for the group<->namespace mapping is managed at server configuration level
  • The cache server must have a super-user proxy to be allowed to read from multiple VO namespaces
  • ACLs between cache and origin server must be kept in sync

Requirements

  • At least two hosts with a docker daemon or a plain centos7 os for the manual recipe
  • Two host certificates for the servers
  • One valid service proxy on the cache server

Docker images

Origin server

You can find the docker and configuration files for the origin server here. With the following commands you will bring up a server running an xrootd server exposing on 1094 an XRootD service.

 mkdir -p /mydata
 # allow docker xrootd user to write and read from your data folder
 chown -R 999:997 /mydata
 # start the server
 docker run -d --name origin \
            --net host \
            -v <hostcerts dir>/hostcert.pem:/etc/grid-security/hostcert.pem \
            -v <hostcerts dir>/hostcert.key:/etc/grid-security/hostcert.key \
            -v <ca dir>/<your CA>.pem:/etc/grid-security/certificates/DODAS.pem \
            -v /tmp/mydata:/data \
            dodasts/xrootd-escape-xrd

If you do not have an host certificate you can set the environment variable XRD_HOST when running the container to generate one trusted by the dev CA here

 docker run -d --name origin \
            --net host \
            -e XRD_HOST=<your machine hostname to be trusted> \
            -v /tmp/mydata:/data \
            dodasts/xrootd-escape-xrd


XCache server

The xcache docker and config files are defined here

Just like the origin server above you will need the following instructions.

 mkdir -p /mydata
 # allow docker xrootd user to write and read from your data folder
 chown -R 999:997 /mydata <path to your service proxy>
 # start the server
 docker run -d --name xcache \ 
            --net host \
            -v <hostcerts dir>/hostcert.pem:/etc/grid-security/hostcert.pem \
            -v <hostcerts dir>/hostcert.key:/etc/grid-security/hostcert.key \
            -v <ca dir>/<your CA>.pem:/etc/grid-security/certificates/DODAS.pem \
            -v <path to your service proxy>:/tmp/x509_u999 \
            -e ORIGIN_HOST=<address of the origin server> \
            -e ORIGIN_XRD_PORT=1094 \
            -e CACHE_RAM_GB=4 \
            -v /tmp/mydata:/data \
            dodasts/xcache-escape-xrd

And if you do not have an host certificate, try:

 docker run -d --name xcache \
            --net host \
            -e XRD_HOST=<your machine hostname to be trusted> \
            -v <path to your service proxy>:/tmp/x509_u999
            -e ORIGIN_HOST=<address of the origin server> \
            -e ORIGIN_XRD_PORT=1094 \
            -e CACHE_RAM_GB=4 \
            -v /tmp/mydata:/data \
            dodasts/xcache-escape-xrd

Local demo with docker compose

If you are looking for a local demo environment you can benefit from the docker compose file provided here

To build the images and run the compose file you can just do:

 git clone https://github.com/DODAS-TS/dodas-docker-images.git
 cd dodas-docker-images/docker/cachingondemand/xcache-escape-http/
 docker-compose build
 docker-compose up -d

Then log into the xcache server and init a valid voms proxy. After that simply restart the cache with ./restart.sh command.

Bare metal instructions

To install all the needed packages starting from a Centos7 image you can use the this script

Origin configuration

Same as used in docker file /etc/xrootd/xrootd-escape.cfg

 all.export /
 oss.localroot /data/
 xrootd.seclib /usr/lib64/libXrdSec.so
 xrootd.trace info
 xrd.trace info
 sec.trace debug
 sec.protocol /usr/lib64 gsi \
   -certdir:/etc/grid-security/certificates \
   -cert:/etc/grid-security/hostcert.pem \
   -key:/etc/grid-security/hostcert.key \
   -d:3 \
   -ca:1 -crl:0 \
   -gridmap:/dev/null  \
   -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=raw|dbg
 ofs.authorize 1
 acc.audit deny
 acc.authdb /etc/xrootd/Authfile-auth-X509-vo
 sec.protbind * gsi

With /etc/xrootd/Authfile-auth-X509-vo

 g /escape/cms /cms a
 g /escape /scratch a
 g /cms /test a

Cache configuration

Same as used in docker file /etc/xrootd/xrootd-escape.cfg

 all.export /
 all.role  server
 oss.localroot /data/
 xrd.port 1094
 ofs.osslib   libXrdPss.so
 pss.cachelib libXrdFileCache.so
 pss.origin <originaddress>:1094
 pss.config streams 256
 xrootd.seclib /usr/lib64/libXrdSec.so
 xrootd.trace info
 xrd.trace info
 sec.trace debug
 pfc.trace debug
 sec.protocol /usr/lib64 gsi \
   -certdir:/etc/grid-security/certificates \
   -cert:/etc/grid-security/hostcert.pem \
   -key:/etc/grid-security/hostcert.key \
   -d:3 \
   -ca:1 -crl:0 \
   -gridmap:/dev/null  \
   -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=raw|dbg
 ofs.authorize 1
 acc.audit deny
 acc.authdb /etc/xrootd/Authfile-auth-X509-vo
 sec.protbind * gsi
 pfc.diskusage 0.85 0.95
 pfc.ram       4g
 pfc.blocksize   512k
 pfc.prefetch    0
 pss.setopt ParallelEvtLoop 10
 pss.setopt RequestTimeout 25
 pss.setopt ConnectTimeout 25
 pss.setopt ConnectionRetry 2

With /etc/xrootd/Authfile-auth-X509-vo

 g /escape/cms /cms a
 g /escape /scratch a
 g /cms /test a