... | @@ -12,23 +12,48 @@ |
... | @@ -12,23 +12,48 @@ |
|
* [Repository Interfaces](#Repository-Interfaces)
|
|
* [Repository Interfaces](#Repository-Interfaces)
|
|
|
|
|
|
### What is openVRE
|
|
### What is openVRE
|
|
|
|
|
|
|
|
openVRE is a cloud-based working environment that allow to rapidly build your own computational platform. It offers:
|
|
|
|
|
|
|
|
- a user-friendly web-based interface that integrates a number of pluggable resources:
|
|
|
|
- analysis tools or pipelines,
|
|
|
|
- interfaces to external data repositories,
|
|
|
|
- visualizaters
|
|
|
|
- an scalable backend for cloud computing compatible with OCCI middlewares like OpenNebula or OpenStack.
|
|
|
|
|
|
|
|
|
|
### Architecture
|
|
### Architecture
|
|
|
|
|
|
The [main repository](https://github.com/inab/openVRE) contains the source code of openVRE, including the web front-end, which can be easily customizable to your project, and the core code, responsible of managing data, metadata, processing units (analyses and visualizers) and computational resources. This figure shows an overview of one of the possible layouts for a computational infrastructure based on openVRE:
|
|
The [main repository](https://github.com/inab/openVRE) contains the source code of openVRE, including the web front-end, which can be easily customizable to your project, and the core code, responsible of managing data, metadata, processing units (analyses and visualizers) and computational resources. This figure shows an overview of one of the possible layouts for a computational infrastructure based on openVRE:
|
|
![arch_1](https://github.com/inab/openVRE/raw/master/public/assets/layouts/layout/img/architecture%20openvre%20(1).png)
|
|
|
|
![arch_1](https://github.com/inab/openVRE/raw/master/public/assets/layouts/layout/img/wiki/architecture_overview.png)
|
|
![arch_1](https://github.com/inab/openVRE/raw/master/public/assets/layouts/layout/img/wiki/architecture_overview.png)
|
|
|
|
|
|
The different components makes possible to provide the computational environment for an analysis or visualization tool initiated by a user at the VRE web interface. The petition is processed by one of the two schedulers available, PMES or SGE-oneflow. They both end up triggering the tool in the particular virtual machine where the selected tool is encapsulated (see bellow the differences between both enactors). In any case, resources are dynamically assigned based on user demands as the number of dedicated VMs is proportional to the number of job petitions, as long as cloud computational resources allow it. Those VMs have access to a number of data volume where input files are accessible, either private (coming from user's workspace) or public datasets are available, and where output files will be written.
|
|
The different components makes possible to provide the computational environment for an analysis or visualization tool initiated by a user at the VRE web interface. The petition is processed by one of the two schedulers available, PMES or SGE-oneflow. They both end up triggering the tool in the particular virtual machine where the selected tool is encapsulated (see bellow the differences between both enactors). In any case, resources are dynamically assigned based on user demands as the number of dedicated VMs is proportional to the number of job petitions, as long as cloud computational resources allow it. Those VMs have access to a number of data volume where input files are accessible, either private (coming from user's workspace) or public datasets are available, and where output files will be written.
|
|
|
|
|
|
Once job's petition reaches the VM, tool execution starts. In any case. tools are generally wrapped by the mg-tool API, that acts as intermediary agent between the MuGVRE and the tool code itself to uniformelly communicate with it. Thought the API, the application or pipelines is eventually executed.
|
|
Once job's petition reaches the allocated VM, tool execution starts. Plugged-in VRE tools are independent software components developed by third parties (i.e. bioinformatics, researchers, statisticians) and eventually deployed in the on-premises VMs of the compute platform. They all show the common structure proposed by the [openvre-tool-api](https://github.com/inab/openvre-tool-api) so that openVRE core can uniformly trigger all tools.
|
|
|
|
|
|
|
|
##### Docker-based architecture
|
|
|
|
|
|
|
|
The [branch Dockerized](https://github.com/inab/openVRE/tree/dockerized) of the main repository contains not only a containerized version of openVRE, but also the required components to deploy a minimal and functional computational cloud infrastructure. This figure shows the different parts:
|
|
|
|
|
|
![arch_2](https://github.com/inab/openVRE/raw/master/public/assets/layouts/layout/img/wiki/architecture_docker.png)
|
|
![arch_2](https://github.com/inab/openVRE/raw/master/public/assets/layouts/layout/img/wiki/architecture_docker.png)
|
|
|
|
|
|
### Components
|
|
### Components
|
|
### Puggable resources
|
|
|
|
|
|
- **Queue system + oneflow:** Grid Engine is a general process scheduler typically used in cluster-based infrastructures. Here openVRE core acts as a submitter node that dispatch jobs to VM/s hosting the tool code, the workers. In fact, a queue is set up for each tool, and several VM replicas are associated to each queue. The availability of these replicas is controlled by oneFlow, an OpenNebula service that dynamically deploys VMs according to a set of configurable system metrics like the VM load. In this way, the number of queue workers can automatically grow or shrink on demand, with the only restriction of having at least one VM already deployed and ready to accept the first job petitions.
|
|
|
|
|
|
|
|
- **PMES (Programming Model Enactment Service):** Service that remotely controls the execution of jobs in an underlying cloud platform through an Open Cloud Computing Interface (OCCI), regardless the cloud middleware in use (i.e, OpenNebula, OpenStack). PMES controls VM creation and contextualization, application execution, and VM destruction.
|
|
|
|
|
|
|
|
- **openvre-tool-api:** VRE tool skeleton written in Python and including a set of parsing and modeling classes that triggers any batch and asynchronous execution while offering the common openVRE command-line client. In this way, openvre-tool-api acts as an adaptor, a uniform entrypoint between openVRE core and the application code itself.
|
|
|
|
|
|
|
|
### Pluggable resources
|
|
|
|
|
|
|
|
openVRE considers three different types of plug-gable pieces of software that can be integrated in a functional VRE in a modular by a VRE administrator. Depending on the
|
|
|
|
|
|
##### Tools
|
|
##### Tools
|
|
|
|
|
|
|
|
|
|
##### Visualizers
|
|
##### Visualizers
|
|
|
|
|
|
##### Repository Interfaces
|
|
##### Repository Interfaces
|
|
|
|
|