|
Virtual Research Environments integrate tools and pipelines to enforce a research community. We offer the possibility to integrate your application in pone of these analytical platforms. Please, read though this documentation and **contact us for any doubt** or suggestion you may have.
|
|
Virtual Research Environments integrate tools and pipelines to enforce a research community. We offer the possibility to integrate your application in one of these analytical platforms. Please, read though this documentation and **contact us for any doubt** or suggestion you may have.
|
|
|
|
|
|
#### Table of contents
|
|
#### Table of contents
|
|
* [Why?](#why)
|
|
* [Why?](#why)
|
... | @@ -19,6 +19,7 @@ open Virtual Research Environment offers a number of **benefits** for developers |
... | @@ -19,6 +19,7 @@ open Virtual Research Environment offers a number of **benefits** for developers |
|
#### Requirements
|
|
#### Requirements
|
|
The application or pipeline to be integrated should:
|
|
The application or pipeline to be integrated should:
|
|
- Be free and **open source code**
|
|
- Be free and **open source code**
|
|
|
|
- Dockerized
|
|
- Run in **non-interactive mode** in linux-based operating system:
|
|
- Run in **non-interactive mode** in linux-based operating system:
|
|
- Ubuntu 18.04
|
|
- Ubuntu 18.04
|
|
- Ubuntu 20.04
|
|
- Ubuntu 20.04
|
... | @@ -42,16 +43,17 @@ There are some steps to follow for achieving the integration of a your applicati |
... | @@ -42,16 +43,17 @@ There are some steps to follow for achieving the integration of a your applicati |
|
***
|
|
***
|
|
|
|
|
|
## How to bring in a new tool?
|
|
## How to bring in a new tool?
|
|
Essentially, VRE will need two elements, (1) your application or workflow wrapped within a **VRE RUNNER**, and (2) **metadata** annotating it (*i.e.* input files requirements, descriptions). The following steps describe how to achieve it.
|
|
Essentially, VRE will need thre elements, (1) a docker image for your tool (2) your application or workflow wrapped within a **VRE RUNNER**, and (2) **metadata** annotating it (*i.e.* input files requirements, descriptions). The following steps describe how to achieve it.
|
|
|
|
|
|
- [Step 1](#step-1-pre-define-your-tool-to-build-a-test-set-of-VRE-job-execution-files) Define your tool's input and output files. Build job execution files for testing the RUNNER.
|
|
- [Step 1](#step-1-pre-define-your-tool-to-build-a-test-set-of-VRE-job-execution-files) Define your tool's input and output files. Build job execution files for testing the RUNNER.
|
|
- [Step 2](#step-2-prepare-a-VRE-RUNNER-wrapping-your-application) Prepare a new VRE RUNNER wrapping your application.
|
|
- [Step 2](#step-2-prepare-a-VRE-RUNNER-wrapping-your-application) Prepare a VRE RUNNER script for your application
|
|
- [Step 3](#step-3-annotate-and-submit-the-new-VRE-tool) Annotate and submit the new VRE tool
|
|
- [Step 3](#step-3-annotate-and-submit-the-new-VRE-tool) Annotate and submit the new VRE tool
|
|
- [Step 4](#step-4-test-and-debug-the-new-tool-from-the-VRE) Test and debug the new tool from the VRE user interface
|
|
- [Step 4](#step-4-test-and-debug-the-new-tool-from-the-VRE) Test and debug the new tool from the VRE user interface
|
|
- [Step 5](#step-5-prepare-a-web-page-to-display-a-summary-report-on-each-execution) (optional) Prepare a web page to display a summary report on each execution
|
|
- [Step 5](#step-5-prepare-a-web-page-to-display-a-summary-report-on-each-execution) (optional) Prepare a web page to display a summary report on each execution
|
|
- [Step 6](#step-6-provide-documentation-for-the-new-tool) Provide documentation for the new tool
|
|
- [Step 6](#step-6-provide-documentation-for-the-new-tool) Provide documentation for the new tool
|
|
<br/>
|
|
<br/>
|
|
|
|
|
|
|
|
|
|
#### STEP 1: Pre-define your tool to build a test set of VRE job execution files
|
|
#### STEP 1: Pre-define your tool to build a test set of VRE job execution files
|
|
|
|
|
|
VRE **job execution files** are two JSON files. In production, these will be generated by the VRE server on each execution initiated by the user at the web interface. This data should be consumed by the RUNNER wrapping your application.
|
|
VRE **job execution files** are two JSON files. In production, these will be generated by the VRE server on each execution initiated by the user at the web interface. This data should be consumed by the RUNNER wrapping your application.
|
... | @@ -87,11 +89,11 @@ Defining which are the input files and arguments that your tool will consume is |
... | @@ -87,11 +89,11 @@ Defining which are the input files and arguments that your tool will consume is |
|
|
|
|
|
<br/>
|
|
<br/>
|
|
|
|
|
|
#### STEP 2: Prepare a VRE RUNNER wrapping your application
|
|
#### STEP 2: Prepare a VRE RUNNER script for your application
|
|
|
|
|
|
VRE RUNNERs are pieces of code that work as adapters between the VRE server and each of the integrated applications or pipelines. Eventually, the RUNNER should:
|
|
VRE RUNNERs are pieces of code that work as adapters between the VRE server and each of the integrated applications or pipelines. Eventually, the RUNNER should:
|
|
1. Consume the VRE job execution files that will be generated when a user submits a new job from the web interface,
|
|
1. Consume the VRE job execution files that will be generated when a user submits a new job from the web interface,
|
|
2. Run locally the wrapped application or pipeline, and
|
|
2. Run locally the wrapped application or pipeline (the command that would be the CMD for the docker image of the tool), and
|
|
3. Generate a list of output files, information that the VRE server will use to register and display the files at users' workspace.
|
|
3. Generate a list of output files, information that the VRE server will use to register and display the files at users' workspace.
|
|
|
|
|
|
For preparing the RUNNER, the easiest option is to **take as reference a `RUNNER template `** and use it as skeleton to wrap your own application. The template includes a couple of python classes that parse and load VRE job execution files into python dictionaries. The template includes a method that you can customize at your convenience to call the application, module or pipeline to be integrated.
|
|
For preparing the RUNNER, the easiest option is to **take as reference a `RUNNER template `** and use it as skeleton to wrap your own application. The template includes a couple of python classes that parse and load VRE job execution files into python dictionaries. The template includes a method that you can customize at your convenience to call the application, module or pipeline to be integrated.
|
... | @@ -101,11 +103,11 @@ For preparing the RUNNER, the easiest option is to **take as reference a `RUNNER |
... | @@ -101,11 +103,11 @@ For preparing the RUNNER, the easiest option is to **take as reference a `RUNNER |
|
1. **Fork or clone** the repository of the RUNNER template in your local development environment.
|
|
1. **Fork or clone** the repository of the RUNNER template in your local development environment.
|
|
| RUNNER template repository | |
|
|
| RUNNER template repository | |
|
|
| ------------- | ------------- |
|
|
| ------------- | ------------- |
|
|
| https://github.com/inab/vre_template_tool | [documentation](https://vre-template-tool.readthedocs.io/en/latest/reference/classes.html) |
|
|
| https://github.com/inab/vre_template_tool_dockerized | [documentation](https://vre-template-tool.readthedocs.io/en/latest/reference/classes.html) |
|
|
|
|
|
|
2. (optional) Run the `hello_word` example. The RUNNER template is initially configured to "wrap" an application called `hello.py`. It demonstrates the overall flow of a VRE RUNNER.
|
|
2. (optional) Run the `demo` example. The RUNNER template is initially configured to "wrap" an application called `demo`. It demonstrates the overall flow of a VRE RUNNER.
|
|
- How to: https://github.com/inab/vre_template_tool#run-the-wrapper
|
|
- How to: https://github.com/inab/vre_template_tool#run-the-wrapper
|
|
3. **Include your own job execution files** in the repository. You can copy the JSON files generated in *STEP 1* into the `test/` folder of the repository to replace the basic `hello_word` example. They should contain the input files and arguments for a test execution of your tool. You can try again to run the RUNNER as above, but now it's going to fail, as the RUNNER is still expecting the arguments and the input files of the `hello_word` example.
|
|
3. **Include your own job execution files** in the repository. You can copy the JSON files generated in *STEP 1* into the `test/` folder of the repository to replace the basic `demo` example. They should contain the input files and arguments for a test execution of your tool. You can try again to run the RUNNER as above, but now it's going to fail, as the RUNNER is still expecting the arguments and the input files of the `demo` example.
|
|
|
|
|
|
Make sure that the absolute path of the working directory and the input files defined in these JSON files are accessible.
|
|
Make sure that the absolute path of the working directory and the input files defined in these JSON files are accessible.
|
|
|
|
|
... | | ... | |