Conducting experiments

One of the main features of DiscoPygal package it to easily and quickly define and conduct experiments that run and compare different solvers. It can be different solver algorithms, same algorithm but with different parameters or any combination of the two.

This feature consist of two parts:

scenarios_runner - To use within a python script
run_experiment - To use from CLI

1. Scenarios runner

discopygal.experiments.scenarios_runner

A utility to run scenarios, repeat them and output the results to dir that contains a csv files.
It is intended to be used within your own python script, which means you need to import it, create your own list of scenarios
and call it’s function.

1.1. Scenario

A scenario is a definition for a case experiment to run - it defines which solver to run on which scene.
It has several configurations, part of them are mandatory while others are optional and have default value if not defined explicitly.
These configurations are described in Scenario

1.2. Custom handlers

It is possible to add also custom handlers that can handle the result of the solver and return more info or calculations relevant for the experiment.
These handlers will be called at each iteration after running the solver.
If added, these results will appear in the result csv files under their given name (including automatically taking there average and standard deviation)
These handlers are passed as a dict to “extra_result_handlers” in run_scenarios()
in the format of “<field_name>”: handler.
handler should be a func that receives the path collection as the first parameter and the solver object as the second
and should return a numeric result.
It is optional to add your own custom handlers.

The default results that are calculated for each scenario (and you don’t need to add yourself) are:

total_path_length - The sum of lengths of all the robots’ paths in the solution
makespan - The makespan of the solution (time it takes)
calc_time - The time it took to calculate the solution (the time the planning process took)

1.3. Using scenarios_runner

To use define a list of scenarios (objects of class: Scenario) and invoke the function run_scenarios()
At the end, each scenario contains it’s csv file with the results of all the repetitions that were done
and also there is a main csv file (called ‘results.csv’) that contains the averages and standard deviation for each scenario and is a summary for the whole experiment.
Results dir is created under the given result_root_dir arg in a dir with the timestamp of the time the experiment began

To summarize, the steps that should be done in your python script are:

Import discopygal.experiments.scenarios_runner
Create a list of scenarios you would like to run
Create a dict of custom handlers (optional)
Call: run_scenarios() with the desired parameters

1.4. Example

from discopygal.experiments.scenarios_runner import run_scenarios, Scenario
from discopygal.solvers.rrt.drrt_star import dRRT_star

SCENE_PATH = "examples/scenes/2_pocket_maze_tight.json"

scenarios = [Scenario(dRRT_star, SCENE_PATH, {"prm_num_landmarks": 200, "num_expands": 40}),
             Scenario(dRRT_star, SCENE_PATH, {"prm_num_landmarks": 2000}),
             Scenario(dRRT_star, SCENE_PATH, repetitions=5),
             Scenario(dRRT_star, SCENE_PATH, {"prm_num_landmarks": 100}, 30, 20)]

extra_handlers = {"size_of_roadmap": lambda _, solver: len(solver.roadmap.edges), # This handler returns the number of edges in roadmap
                  "num_of_nodes": lambda _, solver: len(solver.roadmap.points)}   # This handler returns the number of nodes in roadmap

run_scenarios(scenarios, "results_drrt_star", extra_handlers)

2. Run experiment

A CLI tool that uses scenarios_runner. To use, run in CLI:

scenarios_runner <args...>

scenarios_runner.exe <args...>

From the repo run:

python .\src\discopygal\experiments\run_experiment.py <args...>

2.1. Modes

It has 4 modes:

2.1.1. Full experiment

Run all scenarios specified in scenarios_file in the global list named SCENARIOS (it must be defined in scenarios_file).

Also possible to define in scenarios_file a global dir named RESULT_HANDLERS to add your custom result handlers.

Command:

scenarios_runner <result_root_dir> <scenarios_file>

Arg:

result_root_dir - Root dir where the results are created (in dir with the timestamp of the time the experiment began)
scenarios_file - Path to scenarios_file as described above

2.1.2. Continue last experiment

Continue running all scenarios specified in scenarios_file from where the last experiment that ran has stopped (according to the timestamp in dir).
scenarios_file must be the same as the last experiment to continue.
Useful in case that it is required to stop the experiment and continue afterwards or if the execution has failed.

Command:

scenarios_runner <result_root_dir> <scenarios_file> resume

Arg:

result_root_dir - Root dir where the results are created (in dir with the timestamp of the time the last experiment began)
scenarios_file - Path to scenarios_file as described in Full experiment

2.1.3. Single Chunk

Divides the list of scenarios into equal chunks and runs the specified chunk.

Good for dividing the experiment into small chunks are running simultaneously.

Command:

scenarios_runner <result_root_dir> <scenarios_file> <number_of_chunks> <chunk_number>

Arg:

result_root_dir - Root dir where the results are created (in dir “chunk_<chunk_number>/results_<timestamp>” where timestamp is the time the experiment began)
scenarios_file - Path to scenarios_file as described in Full experiment
number_of_chunks - Number of chunks to divide the scenarios list to.
chunk_number - Chunk index to run (zero based). Executes all scenarios in the chunk

2.1.4. Summarize chunks

Summarize all last chunks into one dir that merges all results (results per scenario and overall results).
Execute after running all chunks.
All chunks must be under same root dir and part of the same experiment
Merged results are under <results_root_dir>/all

Command:

scenarios_runner <result_root_dir> <scenarios_file> <number_of_chunks> end

Arg:

result_root_dir - Root dir where the results are created (in dir “all/results_<timestamp>” where timestamp is the time the experiment began)
scenarios_file - Path to scenarios_file as described in Full experiment
number_of_chunks - Number of chunks to divide the scenarios list to.

2.2. Example of scenarios_file

import itertools

from discopygal.experiments.scenarios_runner import Scenario
from discopygal.solvers.rrt.drrt_star import dRRT_star

random_sample_counter_list = [0, 1, 10, 20, 30, 50, 75, 100, 200, 500]

scenes = ["examples/scenes/tunnels_disc.json",
          "examples/scenes/coffee_shop/coffee_shop.json",
          "examples/scenes/2_discs_corridor.json",
          "examples/scenes/2_pocket_maze_tight.json"]

SCENARIOS = [Scenario(dRRT_star, scene, {"prm_num_landmarks": 1000, "num_landmarks": 100, "random_sample_counter": random_sample_counter})
             for random_sample_counter, scene in itertools.product(random_sample_counter_list, scenes)]

RESULT_HANDLERS = {"size_of_roadmap": lambda _, solver: len(solver.roadmap.edges),
                   "num_of_nodes": lambda _, solver: len(solver.roadmap.points)