EAR-environment-variables.md

 [[_TOC_]]
 
-# Introduction
+## Introduction
 
 EAR offers some environment variables in order to provide users the opportunity to
 tune or request some of EAR features.
@@ -18,9 +18,9 @@ This design may only have a real effect on SLURM systems, but it makes it easier
 
 All examples showing the usage of below environment variables assume a system using SLURM.
 
-# Loading EAR Library
+## Loading EAR Library
 
-## EAR_LOADER_APPLICATION
+### EAR_LOADER_APPLICATION
 
 Rules the EAR Loader to load the EAR Library for a specific application that does not follow any of the current programming models (or maybe a sequential app) supported by EAR.
 Your system must have installed the non-MPI version of the Library (ask your system administrator).
@@ -38,7 +38,7 @@ srun --ntasks 1 --job-name=my_job_name ./my_exec_file
 
 See the [Use cases](User-guide#use-cases) section to read more information about how to run jobs with EAR.
 
-## EAR_LOAD_MPI_VERSION
+### EAR_LOAD_MPI_VERSION
 
 Forces to load a specific MPI version of the EAR Library.
 This is needed, for example, when you want to load the EAR Library for Python + MPI applications, where the Loader is not able to detect the MPI implementation the application is going to use.
@@ -66,9 +66,9 @@ srun --ear-policy=min_time \
 
 See the [Use cases](User-guide#use-cases) section to read more information about how to run jobs with EAR.
 
-# Report plug-ins
+## Report plug-ins
 
-## EAR_REPORT_ADD
+### EAR_REPORT_ADD
 
 Specify a report plug-in to be loaded. The value must be a shared object file, and it must be located at `$EAR_INSTALL_PATH/lib/plugins/report` or at the path from where the job was launched.
 Alternatively, you can provide the full path (absolute or relative) of the report plug-in.
@@ -81,9 +81,9 @@ export SLURM_EAR_REPORT_ADD=my_report_plugin.so
 srun -n 10 my_mpi_app
 ```
 
-# Verbosity
+## Verbosity
 
-## EARL_VERBOSE_PATH
+### EARL_VERBOSE_PATH
 
 Specify a path to create a file (one per node involved in a job) where to print messages from the EAR Library.
 This is useful when you run a job in multiple nodes, as EAR verbose information for each of them can result in lots of messages mixed at stderr (EAR messages default channel).
@@ -111,9 +111,9 @@ mpirun -np 96 -ppn 48 my_app
 
 After the above job example completion, in the same directory where the application was submitted, there will be a directory called *ear_logs_dir_name* with two files, i.e., one for each node, called *earl_logs.0.0.<job_step>.<job_id>* and *earl_logs.1.0.<job_step>.<job_id>*, respectively.
 
-# Frequency management
+## Frequency management
 
-## EAR_GPU_DEF_FREQ
+### EAR_GPU_DEF_FREQ
 
 Set a GPU frequency (in kHz) to be fixed while your job is running.
 The same frequency is set for all GPUs used by the job.
@@ -137,7 +137,7 @@ srun --cpu-bind=core --ear-policy=min_energy gmx_mpi mdrun \
     -s $GROMACS_INPUT -noconfout -ntomp 1
 ```
 
-## EAR_JOB_EXCLUSIVE_MODE
+### EAR_JOB_EXCLUSIVE_MODE
 
 Indicate whether the job will run in a node exclusively (non-zero value).
 EAR will reduce the CPU frequency of those cores not used by the job.
@@ -155,7 +155,7 @@ export SLURM_EAR_JOB_EXCLUSIVE_MODE=1
 srun -n 10 --ear=on ./mpi_mpi_app
 ```
 
-## Controlling Uncore/Infinity Fabric frequency
+### Controlling Uncore/Infinity Fabric frequency
 
 EARL offers the possibility to control the Integrated Memory Controller (IMC) for Intel(R)
 architectures and Infinity Fabric (IF) for AMD architectures.
@@ -163,7 +163,7 @@ On this page we will use the term *uncore* to refer both of them.
 Environment variables related to uncore control covers [policy specific settings](#ear_set_imcfreq) or
 the chance for a user to [fix it](#ear_max_imcfreq-and-ear_min_imcfreq) during an entire job.
 
-### EAR_SET_IMCFREQ
+#### EAR_SET_IMCFREQ
 
 Enables/disables EAR's [eUFS](Home#publications) feature.
 Type `ear-info` to see whehter eUFS is enabled by default.
@@ -185,7 +185,7 @@ export SLURM_EAR_POLICY_IMC_TH=0.035
 srun [...] my_app
 ```
 
-### EAR_MAX_IMCFREQ and EAR_MIN_IMCFREQ
+#### EAR_MAX_IMCFREQ and EAR_MIN_IMCFREQ
 
 Set the maximum and minimum values (in kHz) at which *uncore* frequency should be.
 Two variables were designed because Intel(R) architectures let to set a range of
@@ -205,7 +205,7 @@ export SLURM_EAR_MIN_IMCFREQ=2000000
 srun [...] my_app
 ```
 
-## Load Balancing
+### Load Balancing
 
 By default, EAR policies try to set the best CPU (and uncore, if [enabled](#controlling-uncore-infinity-fabric-frequency)) frequency according to node grain metrics.
 This behaviour can be changed telling EAR to detect and deal with unbalanced workloads, i.e., there is no equity between processes regarding their MPI/computational activity.
@@ -216,7 +216,7 @@ Please, contact with [ear-support@bsc.es](mailto:ear-support@bsc.es) if you want
 
 > A correct CPU binding it's required to get the most benefit of this feature. Check the documentation of your application programming model/vendor/flavour or yur system batch scheduler.
 
-### EAR_LOAD_BALANCE
+#### EAR_LOAD_BALANCE
 
 Enables/Disables EAR's Load Balance strategy in energy policies.
 Type `ear-info` to see whether this feature is enabled by default.
@@ -238,7 +238,7 @@ export SLURM_EAR_LOAD_BALANCE_TH=0.89
 srun [...] my_app
 ```
 
-## Support for Intel(R) Speed Select Technology
+### Support for Intel(R) Speed Select Technology
 
 Since version 4.2, EAR supports the interaction with [Intel(R) Speed Select Technology (Intel(R) SST)](https://www.intel.com/content/www/us/en/architecture-and-technology/speed-select-technology-article.html)
 which lets the user to have more fine grained control over per-CPU Turbo frequency.
@@ -263,7 +263,7 @@ If you enable [EARL verbosity](User-guide#ear-job-submission-flags) you will see
 the mapping of the CLOS set for each CPU in the node.
 Note that a `-1` value means that no change was done on the specific CPU.
 
-### EAR_PRIO_TASKS
+#### EAR_PRIO_TASKS
 
 A list that specifies the CLOS that CPUs assigned to tasks must be set.
 This variable is useful because you can configure your application transparently
@@ -341,7 +341,7 @@ PRIO3: MAX GHZ - 0.0 GHz (low)
 [120,-1] [121,-1] [122,-1] [123,-1] [124,-1] [125,-1] [126,-1] [127,-1] 
 ```
 
-### EAR_PRIO_CPUS
+#### EAR_PRIO_CPUS
 
 A list of priorities that should have the same length as the number of CPUs your
 job is using.
@@ -355,16 +355,16 @@ But it becomes more flexible when the user has more control over the affinity se
 to its application, because you can discriminate between different CPUs assigned to the same task.
 Moreover, this is the only way to set different priorities over different threads in no-MPI applications.
 
-## Disabling EAR's affinity masks usage
+### Disabling EAR's affinity masks usage
 
 For both [Load Balancing](load-balancing) and [Intel(R) SST](#support-for-intel-r-speed-select-technology)
 support, EAR uses processes' affinity mask read at the beginning of the job.
 If you are working on an application that changes (or may change) the affinty mask of tasks, this can lead some miss configuration not detected by EAR.
-To avoid any unexpected problem, **we highly recommend you** to export `EAR_NO_AFFINITY_MASK` environment variable (**even your are not planning to work with some of the mentioned features**).
+To avoid any unexpected problem, **we highly recommend you** to export `EAR_NO_AFFINITY_MASK` environment variable **even you are not planning to work with some of the mentioned features**.
 
-# Data gathering
+## Data gathering
 
-## EAR_GET_MPI_STATS
+### EAR_GET_MPI_STATS
 
 Use this variable to generate two files at the end of the job execution that will contain global, per process MPI information.
 You must specify the prefix (optionally with a path) of the filename. One file (*[path/]prefix.ear_mpi_stats.full_nodename.csv*) will contain a resume about MPI throughput (per-process), while the other one (*[path/]prefix.ear_mpi_calls_stats.full_nodename.csv*) will contain a more fine grained information about different MPI call types.
@@ -441,7 +441,7 @@ Below table shows fields available by **ear_mpi_calls_stats** file:
 | t_SendRecv | Time (in microseconds) spent in **blocking** SendRecv calls.
 | t_Wait | Time (in microseconds) spent in **blocking** Wait calls.
 
-## EAR_TRACE_PLUGIN
+### EAR_TRACE_PLUGIN
 
 EAR offers the chance to generate Paraver traces to visualize runtime metrics with the [Paraver tool](https://tools.bsc.es/paraver).
 Paraver is a visualization tool developed by CEPBA-Tools team and currently maintained by the Barcelona Supercomputing Center’s tools team.
@@ -451,7 +451,7 @@ You must set the value of this variable to `tracer_paraver.so` to load the trace
 This shared object comes with the official EAR distribution and it is located at `$EAR_INSTALL_PATH/lib/plugins/tracer`.
 Then you need to set the `EAR_TRACE_PATH` variable (see below) to specify the destination path of the generated Paraver traces.
 
-## EAR_TRACE_PATH
+### EAR_TRACE_PATH
 
 Specify the path where you want to store the trace files generated by the EAR Library. The path must be fully created. Otherwise, the Paraver tracer plug-in won’t be loaded.
 
@@ -469,7 +469,7 @@ srun -n 10 --ear=on ./mpi_app
 
 ```
 
-## REPORT_EARL_EVENTS
+### REPORT_EARL_EVENTS
 
 Use this variable (i.e., `export SLURM_REPORT_EARL_EVENTS=1`) to make EARL send internal events to the [Database](EAR-Database).
 These events are useful to have more information about Library's behaviour, like
@@ -487,7 +487,7 @@ a table of all reported events:
 | Value | The value stored with the event. Categorical events explained below. |
 | node_id | The node from where the event was reported. |
 
-### Event types
+#### Event types
 
 Below are listed all kind of event types you can get when requesting job events.
 For categorical event values, the (value, category) mapping is explained.