README.md 7 KB
Newer Older
emarti2's avatar
emarti2 committed
1
# *FAIRsoft* ETL and indicators evaluation pipeline 
emarti2's avatar
emarti2 committed
2

emarti2's avatar
emarti2 committed
3
*FAIRsoft* are a set of research software *FAIRness* indicators, specifically devised to be assesed automatically. In order to test them, we have developed a pipeline (see flowchart below) to gather metadata about research software specific to Computational Biology, harmonize and integrate it and then perform an evaluation of their compliance with *FAIRsoft* indicators. 
emarti2's avatar
emarti2 committed
4

emarti2's avatar
emarti2 committed
5
6
7
8
9
10
This repository contains the code relative to the:
    - Extraction of software metadata from Bioconda, Bioconductor, GalaxyToolshed, bio.tools, OpenEBench, SourceForge, Bitbucket and GitHub.
    - Harmonization of the prevourly obtained metadata.
    - Integration of pieces of metadata belonging to the same software.
    - Calculation of *FAIRsoft* indicators compliance and FAIRsoft scores.

Eva Martin del Pico's avatar
readme    
Eva Martin del Pico committed
11
#### :construction: Overview 
emarti2's avatar
emarti2 committed
12

emarti2's avatar
emarti2 committed
13
```mermaid
emarti2's avatar
emarti2 committed
14
%%{init: {"theme": "base", "securityLevel": "loose", "themeCSS": ".label foreignObject { font-size: 16px}"} }%%
emarti2's avatar
emarti2 committed
15

emarti2's avatar
emarti2 committed
16
flowchart TB
emarti2's avatar
emarti2 committed
17
%%{config {'fontSize': 16, 'rankSpacing':60, 'curve':'basis', 'nodeSpacing':45}}%%
emarti2's avatar
emarti2 committed
18
19
    alambique[(Alambique)]
    style alambique fill:#ffff,stroke:#000000,stroke-width:1px
emarti2's avatar
emarti2 committed
20
    subgraph "Data Importation";
emarti2's avatar
emarti2 committed
21

Eva Martin del Pico's avatar
Eva Martin del Pico committed
22
23
24
25
26
27
28
29
30
31
32
33
34
35
        subgraph A["Toolshed importer"];

            reposFetcher["fa:fa-cog GalaxyToolshed repositories <br> metadata importer"] -->  galaxy_metadata.json["Toolshed repositories metadata (JSON)"]
            click reposFetcher "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/tools_importers/toolshed_imp/repos_metadata_importer.py" _blank


            galaxy_metadata.json--> toolshedConfigImporter["fa:fa-cog Repositories Config<br> metadata extractor"]

            galaxy_metadata.json--> toolshedRepoMetaImporter["fa:fa-cog Repositories Metadata<br> metadata extractor"]

            style galaxy_metadata.json fill:#ffff, stroke:#111111, stroke-width:1px
            click A "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/tools_importers/toolshed_imp/importer.py" _blank

        end
emarti2's avatar
emarti2 committed
36

emarti2's avatar
emarti2 committed
37
38
39
        bioconductor/importer.py["fa:fa-cog Bioconductor importer"]
        click bioconductor/importer.py "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/tree/FAIRsoft/tools_importers/bioconductor/importer.py" _blank
        
Eva Martin del Pico's avatar
Eva Martin del Pico committed
40
        bioconda_biotools["fa:fa-cog Bioconda, GalaxyEU and bio.tools importer"]
emarti2's avatar
emarti2 committed
41
        click bioconda_biotools "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/tools_importers/bioconda_biotools_galaxy/importer.py" _blank
emarti2's avatar
emarti2 committed
42
43

        sourceforge["fa:fa-cog Sourceforge importer"]
emarti2's avatar
emarti2 committed
44
        click sourceforge "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/tools_importers/repositories/sourceforge/importer.py" _blank
emarti2's avatar
emarti2 committed
45

Eva Martin del Pico's avatar
Eva Martin del Pico committed
46
47
        bioconda_recipes["fa:fa-cog Bioconda importer"]

emarti2's avatar
emarti2 committed
48
49
50
        config_opeb["config"] -.->opeb_metrics/importer.py["fa:fa-cog OEB Importer"]
        click config_opeb "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/tools_importers/opeb_metrics/config.py" _blank
        click opeb_metrics/importer.py "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/tools_importers/opeb_metrics/importer.py" _blank
emarti2's avatar
emarti2 committed
51
52
        style config_opeb fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
        
emarti2's avatar
emarti2 committed
53
        repo_config[config]-.->repo_importer["fa:fa-cog GitHub and <br> Bitbucket enricher"] 
emarti2's avatar
emarti2 committed
54
        style repo_config fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
emarti2's avatar
emarti2 committed
55

emarti2's avatar
emarti2 committed
56
57
58
    end
    
    repo_importer -->alambique;
Eva Martin del Pico's avatar
Eva Martin del Pico committed
59
60
    toolshedRepoMetaImporter --> alambique;
    toolshedConfigImporter --> alambique;
emarti2's avatar
emarti2 committed
61
62
    opeb_metrics/importer.py-->alambique;
    bioconductor/importer.py-->alambique;
emarti2's avatar
emarti2 committed
63
64
    bioconda_biotools-->alambique;
    sourceforge --> alambique;
Eva Martin del Pico's avatar
Eva Martin del Pico committed
65
    bioconda_recipes["fa:fa-cog Bioconda importer"] --> alambique;
emarti2's avatar
emarti2 committed
66
    
emarti2's avatar
emarti2 committed
67
68
69
70
    
    pretools[(Pretools)];
    style pretools fill:#ffff,stroke:#000000,stroke-width:1px

emarti2's avatar
emarti2 committed
71
    subgraph "Data harmonization";
emarti2's avatar
emarti2 committed
72
        direction LR
emarti2's avatar
emarti2 committed
73
74
75
76
77
        config_pretools["config"] -.->raw2pretools[fa:fa-cog Transformation];
        click config_pretools "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/e536ae8725bdf0f3bc71b3a70ce407d3d40210d4/transformation/config_pretools.yaml" _blank
        style config_pretools fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
        click raw2pretools "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/e536ae8725bdf0f3bc71b3a70ce407d3d40210d4/transformation/transform_raw.py" _blank;
        
emarti2's avatar
emarti2 committed
78
79
80
        end;
    alambique --> raw2pretools;
    raw2pretools--> pretools;
emarti2's avatar
emarti2 committed
81
82
83
84
    
    tools[(Tools)];
    style tools fill:#ffff,stroke:#000000,stroke-width:1px
    
emarti2's avatar
emarti2 committed
85
    subgraph "Tools Integration";
emarti2's avatar
emarti2 committed
86
87
88
        config_integration["config"] -.-> integration[fa:fa-cog Integration]
        click integration "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/e536ae8725bdf0f3bc71b3a70ce407d3d40210d4/integration/integration.py" _blank;
        style config_integration fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5           
emarti2's avatar
emarti2 committed
89
    end;
emarti2's avatar
emarti2 committed
90
91
92
    
    pretools-->integration;
    integration --> tools;
emarti2's avatar
emarti2 committed
93

emarti2's avatar
emarti2 committed
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
    subgraph "Analysis"
        FAIR_calc["fa:fa-cog FAIRsoft indicators calculation"];
        click FAIR_calc "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/FAIRsoft/FAIRsoft/indicators%20_evaluation/FAIR_indicators_eval.py" _blank;
        analysis["fa:fa-cog Tools data exploration and analysis"];
    end;

    tools-->FAIR_calc;
    tools-->analysis;
    FAIR_calc --> FAIRscores(FAIRsoft scores);
    style FAIRscores fill:#ffff,stroke:#000000,stroke-width:1px
    analysis --> vis(Tools Statistics);
    style vis fill:#ffff,stroke:#000000,stroke-width:1px

    subgraph FAIRs["FAIRsoft library"]
        FAIRsoft_metatrans["`FAIRsoft.meta_transformers`"] -.dep.- raw2pretools;
        FAIRsoft_utils["`FAIRsoft.utils`"] -.dep.- FAIR_calc
        FAIRsoft_utils -.dep.- FAIRsoft_metatrans
        FAIRsoftintegration["`FAIRsoft.integration`"] -.dep.- FAIR_calc
        FAIRsoftintegration -.dep.- integration
        
        style FAIRs fill:#ffff
        style FAIRsoft_metatrans fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
        style FAIRsoft_utils fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
        style FAIRsoftintegration fill:#ffff,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
        click FAIRsoft_utils "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/FAIRsoft/FAIRsoft/utils.py" _blank
        click FAIRsoft_metatrans "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/FAIRsoft/FAIRsoft/FAIRsoft/meta_transformers.py" _blank
        click FAIRsoftintegration "https://gitlab.bsc.es/inb/elixir/software-observatory/FAIRsoft_ETL/-/blob/e536ae8725bdf0f3bc71b3a70ce407d3d40210d4/FAIRsoft/FAIRsoft/integration.py" _blank
emarti2's avatar
emarti2 committed
121
122
    end;

emarti2's avatar
emarti2 committed
123
124
```
<sub>***FAIRsoft* pipeline**. "dep" relationships indicate a dependencies between code in this repository.</sub>