|
|
|
Virtual Research Environments integrate tools and pipelines to enforce a research community. We offer the possibility to integrate your application in the analytical platform. Please, read though this documentation and contact us for any doubt or suggestion you may have.
|
|
|
|
|
|
|
|
#### Table of contents
|
|
|
|
* [Why](#why)
|
|
|
|
* [How it works](#How-it-works?)
|
|
|
|
* [How it works](#How-it-works)
|
|
|
|
* [How to?](#how-to)
|
|
|
|
|
|
|
|
### Why
|
|
|
|
### Why?
|
|
|
|
open Virtual Research Environment offers a number of benefits for developers willing to integrate their tools in the platform:
|
|
|
|
|
|
|
|
- Open access service publicly available
|
|
|
|
- A user-friendly web interface with user support utilities, authentication service and data handlers for local and remote datasets and repositories.
|
|
|
|
- Visibility for your tool and organization, with ownership recognition, tailored web forms, help pages and customized viewers
|
|
|
|
- The possibility to add extra value to your tool by complementing it with other related tools already in the platform.
|
|
|
|
- A complete administration panel for monitoring, logging and managing your tool.
|
|
|
|
|
|
|
|
#### Requirements
|
|
|
|
The application or pipeline to be integrated should be
|
|
|
|
- free and open source code
|
|
|
|
- Run in non-interactively mode in linux-based virtual machines.
|
|
|
|
|
|
|
|
### How it works?
|
|
|
|
|
|
|
|
### How to?
|
|
|
|
Step 1 - Define your tool's input and output files. The information is used to build job execution files for testing the RUNNER.
|
|
|
|
|
|
|
|
Step 2 - Prepare a new VRE RUNNER wrapping your application.
|
|
|
|
|
|
|
|
Step 3 - Define the metadata of your tool into a tool-specification file
|
|
|
|
|
|
|
|
Step 4- Submit the RUNNER code and the tool-specification file to VRE administrators, who will install and register the new tool.
|
|
|
|
|
|
|
|
Step 5- Test and debug the new tool from the VRE user interface
|
|
|
|
|
|
|
|
Step 6- (optional) Prepare a web page to display a summary report on each execution
|
|
|
|
|
|
|
|
Step 7- Provide documentation for the new tool
|
|
|
|
|
|
|
|
|
|
|
|
>> Step 1 -- Define input files, arguments and output files of the new tool. Use them to build job execution files for testing the RUNNER
|
|
|
|
|
|
|
|
VRE job execution files are 2 JSON files. In production, these will be generated by the VRE server on each execution initiated by the user at the web interface. This data will be consumed by your tool RUNNER.
|
|
|
|
|
|
|
|
- Run configuration file (i.e. config.json): contains the list of input file selected by the user for a particular run, the values of the arguments, and the list of expected output files.
|
|
|
|
|
|
|
|
- Input files metadata: (in_metadata.json): contains the metadata of the input files listed in config.json, including the absolute file path.
|
|
|
|
|
|
|
|
Additionally, is handy to have a shell script with the command line of the RUNNER. The 2 previous files are passed in as arguments (test.sh).
|
|
|
|
|
|
|
|
Defining which are the input files and arguments that your tool will consume is essential to build these test data. There are 2 ways of creating it:
|
|
|
|
|
|
|
|
1.a - Manual approach:
|
|
|
|
|
|
|
|
Manually generate the 2 files following the corresponding JSON schema and taking as reference some examples
|
|
|
|
|
|
|
|
- Examples:
|
|
|
|
|
|
|
|
Test data of the RUNNER template : https://github.com/inab/vre_template_tool/tree/master/tests/basic
|
|
|
|
|
|
|
|
Test data of the dpfrep RUNNER (example of a R-based tool): https://github.com/inab/vre_dpfrep_executor/tree/master/tests/basic
|
|
|
|
|
|
|
|
- Schemas:
|
|
|
|
|
|
|
|
euCanSHare tool schemas: https://github.com/euCanSHare/vre/tree/master/tool_schemas/tool_specification
|
|
|
|
|
|
|
|
1.b - VRE web interface approach:
|
|
|
|
|
|
|
|
Through web forms that (1) allows the edition and validation of a JSON document gathering data about the input files and arguments, and (2) asks data related to your local development environment. The result is a downloadable config.json and in_metadata.json with the corresponding shell script ready to be locally executed (in step 2).
|
|
|
|
|
|
|
|
- Where? https://vre.eucanshare.bsc.es/ -> Admin -> My Tools -> Development -> (+) Add new tool
|
|
|
|
|
|
|
|
- Requirements: VRE user account with "tool developer" rights
|
|
|
|
|
|
|
|
|
|
|
|
>> Step 2 -- Prepare a new VRE RUNNER wrapping your application
|
|
|
|
|
|
|
|
VRE RUNNERs are pieces of code that work as adaptors between the VRE server and each of the integrated applications or pipelines. The RUNNER (1) consumes the VRE job execution files generated when a user submits a new job at the web interface, (2) runs the wrapped application, and (3) generates the list of output files, data that the VRE server will eventually register and display at users' workspace.
|
|
|
|
|
|
|
|
For preparing the RUNNER, the easiest option is to take as reference the RUNNER template repository and adapt some of its methods. The template includes a couple of python classes that parse VRE job execution files into python dictionaries. These are passed to the `run` method of the `VRE_Tool` class, function that you can customize at your convenience to call the application, module or pipeline to be integrated.
|
|
|
|
|
|
|
|
- RUNNER template repository : https://github.com/inab/vre_template_tool
|
|
|
|
|
|
|
|
- RUNNER template documentation : https://vre-template-tool.readthedocs.io/en/latest/reference/classes.html
|
|
|
|
|
|
|
|
2.a - Fork or clone the repository of the RUNNER template in your local development environment.
|
|
|
|
|
|
|
|
How to: https://github.com/inab/vre_template_tool
|
|
|
|
|
|
|
|
2.b - (optional) Run the `hello_word` example. The RUNNER template is initially configured to "wrap" an application called `hello.py`. It demonstrates the overall flow of a VRE RUNNER.
|
|
|
|
|
|
|
|
How to: https://github.com/inab/vre_template_tool#run-the-wrapper
|
|
|
|
|
|
|
|
2.c - Pass the job execution files generated in step 1 as parameters of the VRE_RUNNER. These should contain the input files and arguments for a test execution of your tool. You can copy them into the `test/` folder of the repository to replace the basic `hello_word` example.
|
|
|
|
|
|
|
|
Make sure that the absolute path of the working directory and the input files defined in these JSON files are accessible.
|
|
|
|
|
|
|
|
2.d - Implement the `run` method of the `VRE_Tool` so that the function executes the application, module or pipeline to be integrated. The input file locations and argument values as defined in the job execution files are going to be content of parameters received in the `run` method.
|
|
|
|
|
|
|
|
2.e - The RUNNER will be ready when the wrapped application is properly executed and the output files are generated in the working directory, under the location specified in `output_files[].file.path`, defined either in `config.json` or in `out_metadata.json`.
|
|
|
|
|
|
|
|
>> Step 3 -- Define the metadata of your tool into a tool-specification file
|
|
|
|
|
|
|
|
|
|
|
|
Notes:
|
|
|
|
|
|
|
|
`VRE_RUNNER` should have executable permissions
|
|
|
|
|
|
|
|
Make sure that all output files are generated at the root of the working directory ( defined at the argument with key `execution`. Read from `test/config.jso |
|
|
\ No newline at end of file |
