nicetoolbox.evaluation.config_handler.ConfigHandler¶

class nicetoolbox.evaluation.config_handler.ConfigHandler(eval_config_file: str, machine_specifics_file: str)[source]¶

Bases: object

Handles loading and parsing of configuration files for evaluation.

Parameters:

eval_config_file (str) – Path to the evaluation configuration TOML file.
machine_specifics_file (str) – Path to the machine-specifics TOML file.

Methods

`get_evaluation_and_dataset_configs`	Generator that yields tuples of (RunConfig, DatasetProperties, EvaluationConfig) for each dataset defined in the run configurations.
`save_experiment_config`	Save the effective configuration for the overall evaluation experiment run.

Attributes

`cfg_loader`
`auto_placeholders`
`runtime_placeholders`
`machine_specific_config`
`global_settings`
`io_config`
`metric_type_configs`
`summaries_configs`
`experiment_config`
`experiment_io`
`component_algorithm_mapping`
`all_run_configs`
`all_dataset_properties`

get_evaluation_and_dataset_configs() → Iterator[Tuple][source]¶

Generator that yields tuples of (RunConfig, DatasetProperties, EvaluationConfig) for each dataset defined in the run configurations.

Each tuple defines a task for the EvaluationEngine to process. This method is thus the main loop over datasets and their evaluation settings and tasks.

Yields:: Iterator[Tuple[RunConfig, DatasetProperties, EvaluationConfig]] – Tuples containing the run configuration, dataset properties, and evaluation configuration for each dataset.

save_experiment_config(output_folder: Path) → None[source]¶

Save the effective configuration for the overall evaluation experiment run. This includes the evaluation config, run config, dataset properties, detector config, and machine-specific config.

Parameters:: output_folder (Path) – The folder where the configuration will be saved.