|
|
|
[[_TOC_]]
|
|
|
|
|
|
|
|
# EAR general questions
|
|
|
|
|
|
|
|
**Q: What is EAR?**
|
|
|
|
|
|
|
|
**A:** EAR is a system software for energy and power optimization, accounting and management. EAR components interact with each other to perform system monitoring, job accounting and energy optimisation on HPC systems.
|
|
|
|
|
|
|
|
The main goals of EAR are to provide energy optimization and monitoring about power consumption about the system or jobs.
|
|
|
|
|
|
|
|
**Q: Must all EAR components be available to get EAR working?**
|
|
|
|
|
|
|
|
**A:** No. Each set of components was designed to provide different services.
|
|
|
|
|
|
|
|
The minimal functionality is the system power monitoring, provided by the EAR Daemon (EARD). In order to get the information collected by the EARD, another service is needed to store such, i.e., the EAR Database Daemon (EARDBD).
|
|
|
|
|
|
|
|
In addition, EARD can offer job accounting if EAR is integrated with the system batch scheduler (e.g., SLURM). Current release has support for SLURM, PBSPro, OpenPBS and OAR. These integrations can load the EAR Library (EARL) when an application is starting in order to get performance metrics and apply energy optimisation policies.
|
|
|
|
|
|
|
|
Finally, the EAR Global Manager Daemon (EARGMD) is a cluster wide component offering cluster energy monitoring and capping. Its main goal is to provide features to apply power capping policies for systems with power consumption restrictions.
|
|
|
|
|
|
|
|
**Q: Which programming models does EARL support?**
|
|
|
|
|
|
|
|
**A:** EARL can be loaded automatically on pure MPI, MPI+OpenMP, OpenMP, MKL, CUDA, and python applications. For other programming models it must be loaded on demand.
|
|
|
|
|
|
|
|
Depending on the programming model, EARL offers different features. Also, the approach to load it may differ depending on which kind of application is launched. See the [user guide](User%20guide#use-cases) to read more detailed information on how to run jobs with the EAR.
|
|
|
|
|
|
|
|
**Q: Is EAR providing per-process metrics?**
|
|
|
|
|
|
|
|
**A:** The current version reports per jobid/stepid/nodeid metrics, that is, metrics reported are or the accumulated or the averages of the metrics for all the processes of the application in the node. However, the EAR library collects per core and per-process metrics so it will be available in next versions.
|
|
|
|
|
|
|
|
**Q: OK, so will I get per-thread metrics on Hybrid MPI+OpenMP apps?**
|
|
|
|
|
|
|
|
**A:** Unfortunately no, you won't.
|
|
|
|
|
|
|
|
EAR does not provide metrics at thread level at this moment. For hybrid applications, EAR will provide metrics for each MPI process. You can run OpenMP apps with EARL, but you'll get process metrics too.
|
|
|
|
|
|
|
|
# Using EAR flags with SLURM plug-in
|
|
|
|
|
|
|
|
**Q: How to see EAR configuration and metrics at runtime?**
|
|
|
|
|
|
|
|
**A:** Use `--ear-verbose=1` flag in your submission command (e.g., srun, mpirun) to enable verbosity.
|
|
|
|
|
|
|
|
Check the [user guide](User%20guide#ear-job-submission-flags) to see how to tune EAR configuration at submission time.
|
|
|
|
|
|
|
|
**Q: Why EAR flags are not working?**
|
|
|
|
|
|
|
|
**A:** The following list of EAR flags are only allowed to Authorized users (contact with your system administrator):
|
|
|
|
|
|
|
|
- `--ear-cpufreq`
|
|
|
|
- `--ear-tag`
|
|
|
|
- `--ear-learning`
|
|
|
|
- `--ear-policy-th`
|
|
|
|
- `--ear-policy` \*
|
|
|
|
|
|
|
|
\* It depends on the system administrator whether this flag is restricted to authorized users. Check next question.
|
|
|
|
|
|
|
|
Below there are `ear.conf` fields that specify the list of authorized users/accounts/groups:
|
|
|
|
|
|
|
|
```INI
|
|
|
|
AuthorizedUsers=user1,user2
|
|
|
|
AuthorizedAccounts=acc1,acc2,acc3
|
|
|
|
AuthorizedGroups=xx,yy
|
|
|
|
```
|
|
|
|
|
|
|
|
If a user is not authorized, non-working flags is the expected behaviour.
|
|
|
|
|
|
|
|
**Q: Why is a different energy policy other than the selected one being applied? I validated it with `--ear-verbose=1`.**
|
|
|
|
|
|
|
|
**A:** The selected policy may not be enabled for all users.
|
|
|
|
|
|
|
|
Energy policies can be configured to be enabled to all users or not. Contact with your system administrator.
|
|
|
|
|
|
|
|
Check policy configuration (ear.conf) and user authorization (ear.conf).
|
|
|
|
|
|
|
|
```INI
|
|
|
|
#Enabled to all users
|
|
|
|
Policy=monitoring Settings=0 DefaultFreq=2.4 Privileged=0
|
|
|
|
#Enabled to authorized users
|
|
|
|
Policy=monitoring Settings=0 DefaultFreq=2.4 Privileged=1
|
|
|
|
```
|
|
|
|
|
|
|
|
If not enabled or not authorized therefore this is is the expected behaviour.
|
|
|
|
|
|
|
|
**Q: How to disable the EAR library explicitly?**
|
|
|
|
|
|
|
|
**A:** Use `–ear=off` flag at submission time.
|
|
|
|
|
|
|
|
**Q: Can I apply EAR settings to all runs (e.g., srun, mpirun) inside a batch script?**
|
|
|
|
|
|
|
|
**A:** Yes, you can provide EAR flags to the batch scheduler integration by setting EAR options on the header of your batch script. For example, in SLURM systems:
|
|
|
|
|
|
|
|
```INI
|
|
|
|
#!/bin/bash
|
|
|
|
#SBATCH -N 1
|
|
|
|
#SBATCH –ear-policy=min_time # application 1 and 2 will run with min_time
|
|
|
|
|
|
|
|
srun application1
|
|
|
|
|
|
|
|
srun --ear-verbose=1 application2 # This step will show EAR messages.
|
|
|
|
```
|
|
|
|
|
|
|
|
**Q: How can I know which energy policies are installed?**
|
|
|
|
|
|
|
|
**A:** Type `srun --help`. The output will show which policies you can provide with the flag `--ear-policy`.
|
|
|
|
|
|
|
|
It is possible that non-authorized users are not allowed to select EAR policy. Contact with your system administrator.
|
|
|
|
|
|
|
|
**Q: How to set EAR flags with `mpirun` if I run my application with Intel MPI?**
|
|
|
|
|
|
|
|
**A:** Depending on the Intel MPI version. Check the user-guide.
|
|
|
|
|
|
|
|
Before version 2019, `mpirun` had 2 parameters to specify SLURM options.
|
|
|
|
|
|
|
|
```INI
|
|
|
|
mpirun –bootstrap=slurm -bootstrap-exec-args=”—ear-verbose=1”
|
|
|
|
```
|
|
|
|
|
|
|
|
Since version 2019, SLURM options must be specified using environment variables:
|
|
|
|
|
|
|
|
```INI
|
|
|
|
export I_MPI_HYDRA_BOOTSTRAP=slurm
|
|
|
|
export I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS="--ear-verbose=1"
|
|
|
|
```
|
|
|
|
|
|
|
|
**Q: How to set EAR flags with `mpirun` if I run my application with OpenMPI?**
|
|
|
|
|
|
|
|
**A:** OpenMPI needs an extra support when `srun` is not used. EAR's **erun** command must be used.
|
|
|
|
|
|
|
|
```INI
|
|
|
|
mpirun erun –ear-policy=min_energy --program=application
|
|
|
|
```
|
|
|
|
|
|
|
|
**Q: How to collect paraver traces?**
|
|
|
|
|
|
|
|
**A:** Use the environment variables to enable the trace collection and to specify the path.
|
|
|
|
|
|
|
|
```INI
|
|
|
|
SLURM_EAR_TRACE_PLUGIN$EAR_INSTALL_PATH/lib/plugins/tracer/tracer_paraver.so
|
|
|
|
SLURM_EAR_TRACE_PATH=TRACES_PARAVER/
|
|
|
|
```
|
|
|
|
|
|
|
|
# Using additional MPI profiling libraries/tools
|
|
|
|
|
|
|
|
EAR uses the `LD_PRELOAD` mechanism to be loaded and the PMPI API for
|
|
|
|
a transparent loading. In order to be compatible with other profiling libraries
|
|
|
|
EAR is not replacing the MPI symbols, it just calls the next symbol in the list.
|
|
|
|
So it is compatible with other tools or profiling libraries. In case of conflict, the
|
|
|
|
EARL can be disabled by setting `--ear=off` flag at submission time.
|
|
|
|
|
|
|
|
# Jobs executed without the EAR Library: Basic Job accounting
|
|
|
|
|
|
|
|
For applications not executed with the EARL loaded (e.g., `srun` is not used or
|
|
|
|
programming models or applications not loaded by default with EARL), EAR provides a default monitoring.
|
|
|
|
In this case a subset of metrics will be reported to the Databse:
|
|
|
|
|
|
|
|
| Metric | Unit |
|
|
|
|
| ------ | ---- |
|
|
|
|
| Accumulated DC energy | Joules |
|
|
|
|
| Accumulated DRAM energy | Joules |
|
|
|
|
| Accumulated CPU package energy | Joules |
|
|
|
|
| EDP | Energy–Delay Product |
|
|
|
|
| Maximum DC node power detected | Watts |
|
|
|
|
| Minimum DC node power detected | Watts |
|
|
|
|
| Execution time | Seconds |
|
|
|
|
| CPU average frequency | kHz\* |
|
|
|
|
| CPU default frequency | kHz\* |
|
|
|
|
|
|
|
|
\* The unit showed by `eacct` output is GHz.
|
|
|
|
|
|
|
|
DC node energy includes the CPU and GPU energy if there are.
|
|
|
|
These metrics are reported per node, Job and Step IDs, so they can be seen per
|
|
|
|
job and job and step when using `eacct` command.
|
|
|
|
|
|
|
|
# Troubleshooting
|
|
|
|
|
|
|
|
|
|
|
|
**User asks for application metrics with `eacct` command and no information appears in some of the columns in the output.**
|
|
|
|
|
|
|
|
This means EARL was not loaded with the application or the application fails before the MPI_Finalize call, nor reporting application data.
|
|
|
|
|
|
|
|
**After some time, user asks for an application metrics with eacct and application is not reported.**
|
|
|
|
|
|
|
|
Try again after some minutes (applications are not reported immediately). |
