Skip to content
GitLab

User guide · Changes

Page history
Updated wiki to EAR version 3.3 authored by Lluis Alonso's avatar Lluis Alonso
Show whitespace changes
Inline Side-by-side
User-guide.md 0 → 100644
View page @ c03aa01d
Running applications with EAR
------------------------------
With EAR's SLURM plugin, running an application with EAR is as easy as submitting a job with either `srun`, `sbatch` or `mpirun` with SLURM. There are multiple configuration settings that can be set to customize EAR's behaviour, which are explained below as well as examples on how to run applications with each method.
## Job submission with EAR
The following EAR options can be specified when running `srun` and/or `sbatch`, and are supported with `srun`/`sbatch`/`salloc`:
| Options | Description |
| -------------------------- | -------------------------------------------------------------------- |
| --ear=on/off(**) | Enables/disables EAR library. |
| --ear-policy=policy | Selects an energy policy for EAR. See the [Policies page](EAR-policies) for more info |
| --ear-cpufreq=frequency(*) | Specifies the starting frequency to be used by the chosen EAR policy (in KHz). |
| --ear-policy-th=value(*) | Specifies the ear_threshold to be used by the chosen EAR policy {`value=[0...1]`}. |
| --ear-user-db=file | Specifies the files where the user applications' metrics summary will be stored {'file.nodename.csv'}. If not defined, these files will not be created. |
| --ear-verbose=value | Specifies the level of verbosity {value=[0...2]}; the default is 0. |
| --ear-tag=tag | Selects an energy tag. |
| --ear-learning=p_state(*) | Enables the learning phase for a given P_STATE {`p_state=[1...n]`}. |
For more information consult `srun --help` output or see configuration options sections for more detailed description.
(*) Option requires _ear privileges_ to be used.
(**) Does not require _ear privileges_ but values might be limited by EAR configuration.
## MPI versions supported
When using sbacth/srun or salloc, Intel MPI and OpenMPI are 100% supported. When using mpi commands to start applications (mpirun, mpiexec.hydra etc), There are minor differences explained in the following examples.
## Examples
### `srun` examples
EAR plugin reads `srun` options and contacts with EARD. Invalid options are filtered to default values, so behaviour depends on system configuration.
- Executes application with EAR on/off (depending on the configuration) with default values:
```
srun -J test -N 1 -n 24 --tasks-per-node=24 application
```
- Executes application with EAR on with default values (policy, default frequency,etc) and verbose set to 1:
```
srun --ear-verbose=1 -J test -N 1 -n 24 --tasks-per-node=24 application
```
- Executes application with EAR on and verbose set to 1. It user is authorised, job will be executed at 2.0GHz as default frequency and with power policy set to monitoring. Otherwise, default values will be applied:
```
srun --ear-cpufreq=2000000 --ear-policy=monitoring --ear-verbose=1 -J test -N 1 -n 24 --tasks-per-node=24 application
```
- Executes application with EAR. If users is authorised to select the “memory-intensive” tag, its application will be executed according to the definition of the tag in the EAR configuration:
```
srun --ear-tag=memory-intensive --ear-verbose=1 -J test -N 1 -n 24 --tasks-per-node=24 application
```
### `sbatch` examples
When using `sbatch` EAR options can be specified in the same way. If more than one srun is included in the job submission, EAR options can be inherited from `sbatch` to the different `srun`s or specifically modified in each individual `srun`.
The following example will set the ear verbose mode for all the job steps to 1. First job step will be executed with default settings and second one with monitoring as policy.
```
#!/bin/bash
#SBATCH -N 1
#SBATCH -e test.%j.err
#SBATCH -o test.%j.out
#SBTACH --ntasks=24
#SBATCH --tasks-per-node=24
#SBATCH --cpus-per-task=1
#SBATCH --ear-verbose=1
srun application
srun --ear-policy=monitoring application
```
### Running EAR with `mpirun` (in SLURM systems)
#### Intel MPI
When running EAR with `mpirun` rather than `srun`, we have to specify the utilisation of `srun` as bootstrap. Otherwise jobs will not go through the SLURM plugin and any EAR options will not be recognised. The API depends on Intel version. Versions older or equal than 2018 use two mpirun arguments to specify the bootstrap and extra SLURM flags (to be passed to the SLURM).
The following example will run application with min_time_to_solution policy:
```mpirun -n 10 -bootstrap slurm -bootstrap-exec-args="--ear-policy=min_time” application```
Version 2019 and newer offers two environment variables rather than mpirun arguments.
```
export I_MPI_HYDRA_BOOTSTRAP=slurm
export I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS="--ear-policy=monitoring --ear-verbose=1"
mpiexec.hydra -n 10 application
```
#### OpenMPI
Bootstrap is an Intel® MPI option but not an OpenMPI option. For OpenMPI `srun` must be used for an automatic EAR support, or use the `erun` program explained below.
ERUN
----
ERUN is a program that simulates all the SLURM and EAR SLURM Plugin pipeline. It comes with the EAR package and is compiled automatically. You can find it in in `bin` folder in your installation path. It must be used when a set of nodes does not have SLURM installed or when using OpenMPI `mpirun` which does not contact with SLURM. You can launch ERUN instead directly your application. In example:
```
mpirun -n 4 /path/to/erun --program="hostname --alias"
```
In this example, MPIRUN would run 4 ERUN processes. Then, ERUN would launch the application hostname with its alias parameter. You can use as many parameters as you want but the semicolons have to cover all the parameters in case there are more than just the program name. ERUN would simulate in the remote node both the local and remote pipelines for all created processes. It has an internal system to avoid repeating functions that are executed just one time per job or node, like SLURM does with its plugins.
```
> erun --help
This is the list of ERUN parameters:
Usage: ./erun [OPTIONS]
Options:
--job-id=<arg> Set the JOB_ID.
--nodes=<arg> Sets the number of nodes.
--program=<arg> Sets the program to run.
--clean Removes the internal files.
SLURM options:
...
```
The `--job-id` and `--nodes` parameters, creates the environment variables that SLURM would have created automatically, because it is possible that your application make use of them. The `--clean` option removes the temporal files created to synchronize all ERUN processes.
Also you have to load the EAR environment module or define its environment variables in your environment or script:
| Variable | Parameter |
| ------------------------- | ---------------------- |
| EAR_INSTALL_PATH=\<path\> | prefix=\<path\> |
| EAR_TMP=\<path\> | localstatedir=\<path\> |
| EAR_ETC=\<path\> | sysconfdir=\<path\> |
| EAR_DEFAULT=\<on/off\> | default=<on/off\> |
Lastly, the tipical SLURM parameters can be passed to ERUN in the same way they were written to SRUN or SBATCH. In example:
```
mpirun -n 4 /path/to/erun --program="myapp" --ear-policy=monitoring --ear-verbose=2
```
User commands
-------------
The only command available to users is `eacct`. With `eacct` a user can see their previously executed jobs with the information that EAR monitors (time, average power, number of nodes and average frequency among others) and a number of options to manipulate said output. Some data will not be available if a job is not executed with EARL.
Note that a user can only see their own applications/jobs unless they are a privileged user and specified as such in the `ear.conf` configuration file.
For more information, check its [Commands section](Commands#energy-account-eacct).