nicetoolbox.evaluation.runner

Dataset runner for executing evaluations on a single dataset.

Classes

DatasetRunner

Runner for evaluating a single dataset with given configurations.
class nicetoolbox.evaluation.runner.DatasetRunner(io_manager: IO, run_config: DetectorsRunConfig, dataset_properties: DatasetConfig, evaluation_config: FinalEvaluationConfig)[source]

Runner for evaluating a single dataset with given configurations.

Initialize the DatasetRunner.

Parameters:

  • io_manager (IO) – IO manager for file operations.

  • run_config (RunConfig) – Configuration for the current run.

  • dataset_properties (DatasetProperties) – Properties of the dataset.

  • evaluation_config (EvaluationConfig) – Current evaluation task config.

run() → None[source]

Execute the evaluation process for the dataset.

This includes: 1. Initializing the dataset in the IO manager. 2. Discovering work items using the DiscoveryEngine. 3. Setting up the EvaluationDataset and DataLoader. 4. Running the MetricRunner to compute metrics. 5. Saving the evaluation results using the IO manager.