What the EvaSuite addon does internally?

The workflow

The eva-suite is executed automatically within the job chain (after simulation is ready) or it can be started manually by typing './subchain eva-suite'. Then, the script eva-suite.job.sh is called which you find in your experiment directory, e.g. ${SPDIR}/chain/gcm2icon/sp001_eva-suite/scripts_eva-suite . This job-script uses the settings defined in the file https://hereon-coast.atlassian.net/wiki/spaces/SPICE/pages/983205 located in the same directory. The job-script is dedicated for the evaluation with EObs or ERA5 and manages to:

• write configuration files for the addon Evasuite

• start the addon Evasuite for each variable separately and for each type of evaluation separately (referred as evaluation steps afterwards)

• copy the results of the Evasuite to the final directory ${EVALRESULTSDIR}/graphics  (definition in https://hereon-coast.atlassian.net/wiki/spaces/SPICE/pages/983205 )

The variables are temperature (mean, min, max), radiation, precipitation and optionally cloud cover and precipitable water mean sea level pressure. The types of evaluation are timeseries of monthly BIAS (each subregion), timeseries of yearly cycle BIAS (monthly and seasonal, 1D and 2D), the BIAS over the whole period, a probability density distribution and a Quantile-Quantile scatter plot. If you define a ${REFID} in eva-suite_settings , the Evasuite also computes a Advanced mean squared error skill score (as 2D plot) and a score table for each parameter and region using a R script.

To manage all this types of evaluation, the Evasuite is executed automatically by eva-suite.job.sh specifying 4 input files and redirect the output of the console to specific logging directories ( ${SPDIR}/chain/scratch/${EXPID}/eva-suite_configs_and_loggings/ ). This logging directories are created for each evaluation step separately and should be examined for evaluation errors.

The first two input files config_general.yaml and config_visualization.yaml  are configurations written in YAML format and define the options of the eva-suite, for instance which parameter to evaluate and which metric. Both files are created based on templates, which can be found in ${SPDIR}/src/eva-suite/ . The last two input files ${OBSDEF} and modeldata_settings.yaml  are database definitions written in YAML format and define the metadata of the datasets, e.g. the location of files and the variable names. The observational datasets are fixed in spice and thus the  ${OBSDEF}  is static (${SPDIR/src/eva-suite/reference_datasets.yaml ). The second database definition is generated by the spice environment because here the name of the model simulation and the filenames are needed. See the template in ${SPDIR}/src/eva-suite/model_datasets_template.yaml .  It is important to note, that the definition of the model datasets is very sensitive to filenames. If you have some special application or file naming of your model output, you need to adapt this template.

Outputs

The intermediate steps of the evasuite (interpolation, ...) are done in the working directory ${SPDIR}/chain/scratch/${EXPID}/eva-suite which is deleted after each separate evaluation step. Only the final results are then copied by the Evasuite from the working directory to the backup directory ${EVALRESULTSDIR}/data

As the filenames of the figures in the backup directory do not follow the naming conventions of former evaluation tools, the figures from the backup directory are copied to the final 'evaluation results directory' ${EVALRESULTSDIR}/graphics. If you have specified a reference run in eva-suite_settings, additional figures were created for the advance mean squared error skill score.