WP3 - OSSR
Contents
Open-source scientific Software and Service Repository (OSSR)
The aim of WP3 is to expose the tools of the ESCAPE ESFRI projects in a repository under the EOSC catalogue of services, see the repository description
The main objectives are
- To support a community-based approach for continuous development, deployment, exposure and preservation of domain-specific open-source scientific software/tools/services in the global context of the EOSC, catalogue of services
- To enable open science interoperability, software re-use and development for the data analysis of the ESCAPE ESFRI projects.
Beta implementations are based on:
Common resources
| Resource | Description | Link | Access |
|---|---|---|---|
| Software repository primer | Software repository test ground (GitLab) | Gitlab group @in2p3.fr | open |
| Meetings | Calendar, video room access and meeting-specific material | WP 3 Indico | Event access: standard ESCAPE password |
| Shared documents | including deliverables, minutes, focus-group specific material | ESCAPE cloud - WP3 | ESCAPE personal login |
| Project management | Deliverable and milestone overview, internal wiki | OSSR project platform + wiki | ESCAPE personal login |
| Chat | Rocket chat | ESCAPE chat | ESCAPE personal login |
| Email list | WP-wide mailing list | e-ossr@PROJECTESCAPE.EU | sign up |
| Virtual meeting rooms | two zoom rooms, free to use for all members anytime | WP3 Zoom details | - |
ESCAPE on OpenAIRE: https://explore.openaire.eu/search/project?projectId=corda__h2020::12147f5add27acd4a1f5bf3f7a358d52
Division of work
Tasks
The main objectives of the work package are formulated as tasks in the work package description. The tasks are
- Task 3.1: Management Activities, Policy and Support Action (MAPS) - Providing assistance and support for work package related activities
- Task 3.2: ESFRI Software and Services Collection (ESSC) - Systematically list available software and requirements to facilitate integration into the repository
- Task 3.3: Common Approaches: Software and Services (CASS) - Fostering common devepmentsand facilitate sharing of software
- Task 3.4: Foundation of Competence for Software and Service Innovation (COSSI) - Establish competence group for mainstreaming new approaches to data analysis (e.g. deep learning)
- Task 3.5: Repository Implementation and Deployment (RIAD) - Setting up a demonstrator for a common software repository
Focus groups
In order to easily facilitate the work within the tasks, focus groups have been established. They are dedicated to specific working items within the work package tasks or intersecting the tasks.
- Focus group 1: Collecting Software requirements (related to Task 3.2)
- Focus group 2: Technical implementation of the repository (related to Task 3.5)
- Focus group 3: Innovative workflows (related to Task 3.4)
- Focus group 4: Distributed computing
- Focus group 5: Common approaches to Corsika (related to Task 3.3)
Current activities
Work package common activities
Objectives
Work package wide meetings and activities aim to coordinate the individual work efforts, serve as an entrance point for future contributions, and integrate the work on upcoming milestones and deliverables.
Organization
- Phone calls: Approximately every 2 month, for the next call see the indico
FG1 Software requirements
Objectives
The aim is to produce a software collection of relevant ESFRI software. As a first step, a template to collect all relevant information on the specific software packages is developed.
Organization
- Meetings: Bi-weekly phone meetings for discussion on the current objectives, see the indico for next dates.
- Shared material: Focus group folder on the cloud
- Software collection template: Form (under discussion) on the survey tool
- Example software project: at the ESCAPE gitlab.
OSSR onboarding
The time for integration of software into the OSSR has started. For this, each partner/ESFRI is requested to present information on the software they plan to share and onboard their software. We have noted down the requirements on your software and the procedure at the OSSR pages - start the onboarding procedure checklist now!
FG2 Technical implementation
Objectives
Setting up and testing of a software repository, drawing on existing technologies. GitLab and Zenodo are currently considered as reference technologies and platforms.
Organization
- Implementation: A group at the Gitlab instance of IN2P3 to test the Gitlab environment - Gitlab group @in2p3.fr, including an example project.
- Shared material: Focus group folder on the cloud
OSSR portal
OSSR entry point for developers and users: http://purl.org/escape/ossr
OSSR Open Curation
In order to open the curation process and have multiple curators participating in the process and checking records comply to the OSSR guidelines, I have created the following repository and opened two merge requests for the two pending requests in Zenodo: https://gitlab.in2p3.fr/escape2020/wp3/ossr-curation/-/merge_requests
You may use these merge requests for the curation process and check that everything is in order for these records. I have copied our current requirements as checklist to help you in the process. Once we have a couple of approval, I will accept the records in the escape2020 community.
This process is a first round of test towards a more complete workflow (under development) using gitlab CI and features to have multiple curators automatically notified when a request is made for a record to be added to the escape2020 community on Zenodo. This should also allow OSSR contributors to take part to the curation process.
OSSR Policy document
A first version of the OSSR policy and guidelines were delivered as deliverable D3.7. However, we knew already at the time that these would probably evolve with time. In particular, the EOSC will also give recommendations that we should follow in our policy. In order to develop the OSSR policy, version and publish it, I have setup the ossr-policy repository. This draft is based on our current policy (from D3.7). You are all welcome to contribute, either by raising issues to be discussed (open issues), or directly contributing through merge requests. We are aiming for a first published version next spring. Once published, these guidelines will of course be used for the curation process.
FG3 Innovative workflows
Objectives
Goal is the establishment of an innovation competence group. To this end, calls for the exchange of current practices are established and a workshop will be organized in 2020.
Organization
- Meetings: Bi-weekly phone meetings, see the indico and below for topics.
- Shared material: Focus group folder on the cloud
- Mailing List: Mailing list ossr-innovation@projectescape.eu subscription link
- Workshop 2020: Overview on the projects platform.
Focus group call topics
Focus group calls are intended as regular meetings with brief presentations introducing current developments of workflows and applications.
A template for these presentations can be found in the focus group folder.
Overview
| Date | Contribution | Links |
|---|---|---|
| 06/12/2019 | Deep Learning workflow for KM3NeT/ORCA (Stefan Reck, Simona Maria Stellacci) | indico |
| 10/01/2020 | Hangar - Open Source versioning tool (Filippo Quarenghi) | indico |
| 24/01/2020 | GammaLearn - Deep Learning for CTA data analysis (Thomas Vuillaume) | indico |
| 21/02/2020 | Workflow of solar high-resolution observations (Meetu Verma) | indico |
| 20/03/2020 | Machine Learning pipeline for GW signal CCSN classification (Alberto Iess) | indico |
| 03/04/2020 | Workflows from the WP5 perspective (Zheng Meyer-Zhao) | indico |
| 17/04/2020 | Deep learning in WP4 (Martino Romaniello, Nima Sedaghat) | indico |
| 22/05/2020 | Galaxy/quasar/star classification with machine learning (Alex Clarke) | indico |
| 12/06/2020 | Workflows in dark matter searches at the LHC and beyond (Caterina Doglioni) | indico |
| 03/07/2020 | Integration of HANGAR and the GW machine learning pipeline (Alberto Iess) | indico |
| 30/07/2020 | Update Hangar @ VIRGO (Filippo Quarenghi) | indico |
| 09/10/2020 | Real time multi-messenger analysis: ideas and ongoing projects (Barbara Patricelli) | indico |
Miscellaneous Notes
FG4 Distributed computing
Objectives
Understanding and integrating the role of distributed computing mechanisms in the implementation of the software repository, e.g. the use of DIRAC and FairROOT.
Organization
- Shared material: Focus group folder on the cloud
- next meeting in coordination with WP5 tech calls early 2020.
FG5 CORSIKA
Objectives
This groups coordinates the common development and integration efforts, mostly between CTA and KM3NeT. Two target groups work on 1) the setup of containers with a common CORSIKA installation and 2) on new developments within CORSIKA.
Subprojects and activities
- ConCORDIA i.e. Containers for CORSIKA on DIRAC. This is a set of ready-made containers that come with preset CORSIKA running parameters and performance estimators as plots and graphs concerning the expected energy and particle spectrum. Reference persons: Daniel Nieto (CTA), Bernardino Spisso and Simona Maria Stellacci (KM3NeT).
- ConCORDIA DIRAC App: an application hosted on a DIRAC instance that can be used both in Web-GUI interactive mode and in scripting mode to fine-tune CORSIKA running parameters or to launch and monitor jobs. Reference persons: Luigi Fusco (KM3NeT) and Andrei Tsaregorodtsev.
- CORSIKA benchmarks: Study on performance optimization for CORSIKA. Contact person: Simona Maria Stellacci
- Fluorescence radiation in CORSIKA: An add-on module (distributed as a patch) for CORSIKA versions 7 to add the simulation of fluorescence radiation to the standard CORSIKA package. Contact person: José Luìs Contreras.
- pyCORSIKAIO: reader package for CORSIKA output files in Python. Contact person: Daniel Nieto.
- DIRAC/ESAP integration: A major step of integration between DIRAC and the ESAP platform. Contact person: Gareth Hughes.
Organization
- Meetings: Phone meetings are organized within the two subgroups, see the indico.
- Shared material: Focus group folder on the cloud
- Mailing List: Mailing list ossr-corsika@projectescape.eu subscription link
- Project overview: Sub-project on the ESCAPE project platform.
Find an overview of the work and related projects in the Concordia Overview
How to join
If you are new to WP3 or you want to join a specific task, please see WP3 - OSSR/How to join.
Open polls for new dates can also be found in the internal wiki on the project platform.
