.. _experiments: Conducting experiments ====================== .. sectnum:: One of the main features of DiscoPygal package it to easily and quickly define and conduct experiments that run and compare different solvers. It can be different solver algorithms, same algorithm but with different parameters or any combination of the two. This feature consist of two parts: #. scenarios_runner - To use within a python script #. run_experiment - To use from CLI Scenarios runner ---------------- :class:`discopygal.experiments.scenarios_runner` | A utility to run scenarios, repeat them and output the results to dir that contains a csv files. | It is intended to be used within your own python script, which means you need to import it, create your own list of scenarios | and call it's function. Scenario ~~~~~~~~ | A scenario is a definition for a case experiment to run - it defines which solver to run on which scene. | It has several configurations, part of them are mandatory while others are optional and have default value if not defined explicitly. | These configurations are described in :class:`~discopygal.experiments.scenarios_runner.Scenario` Custom handlers ~~~~~~~~~~~~~~~ | It is possible to add also custom handlers that can handle the result of the solver and return more info or calculations relevant for the experiment. | These handlers will be called at each iteration after running the solver. | If added, these results will appear in the result csv files under their given name (including automatically taking there average and standard deviation) | These handlers are passed as a dict to "extra_result_handlers" in :func:`~discopygal.experiments.scenarios_runner.run_scenarios` | in the format of "": :token:`handler`. | :token:`handler` should be a func that receives the path collection as the first parameter and the solver object as the second | and should return a numeric result. | It is optional to add your own custom handlers. The default results that are calculated for each scenario (**and you don't need to add yourself**) are: #. total_path_length - The sum of lengths of all the robots' paths in the solution #. makespan - The makespan of the solution (time it takes) #. calc_time - The time it took to calculate the solution (the time the planning process took) Using scenarios_runner ~~~~~~~~~~~~~~~~~~~~~~ | To use define a list of scenarios (objects of class: :class:`~discopygal.experiments.scenarios_runner.Scenario`) and invoke the function :func:`~discopygal.experiments.scenarios_runner.run_scenarios` | At the end, each scenario contains it's csv file with the results of all the repetitions that were done | and also there is a main csv file (called 'results.csv') that contains the averages and standard deviation for each scenario and is a summary for the whole experiment. | Results dir is created under the given :token:`result_root_dir` arg in a dir with the timestamp of the time the experiment began To summarize, the steps that should be done in your python script are: #. Import :class:`discopygal.experiments.scenarios_runner` #. Create a list of scenarios you would like to run #. Create a dict of custom handlers (optional) #. Call: :func:`~discopygal.experiments.scenarios_runner.run_scenarios` with the desired parameters Example ~~~~~~~ .. literalinclude:: /../experiments/example_experiment.py Run experiment -------------- A CLI tool that uses scenarios_runner. To use, run in CLI: .. tabs:: .. tab:: Unix :: scenarios_runner .. tab:: Windows :: scenarios_runner.exe From the repo run: :: python .\src\discopygal\experiments\run_experiment.py Modes ~~~~~ It has 4 modes: .. _full_experiment: Full experiment *************** | Run all scenarios specified in :token:`scenarios_file` in the global list named :token:`SCENARIOS` (it must be defined in :token:`scenarios_file`). | Also possible to define in :token:`scenarios_file` a global dir named :token:`RESULT_HANDLERS` to add your custom result handlers. Command: :: scenarios_runner Arg: #. result_root_dir - Root dir where the results are created (in dir with the timestamp of the time the experiment began) #. scenarios_file - Path to :token:`scenarios_file` as described above Continue last experiment ************************ | Continue running all scenarios specified in :token:`scenarios_file` from where the last experiment that ran has stopped (according to the timestamp in dir). | :token:`scenarios_file` must be the same as the last experiment to continue. | Useful in case that it is required to stop the experiment and continue afterwards or if the execution has failed. Command: :: scenarios_runner resume Arg: #. result_root_dir - Root dir where the results are created (in dir with the timestamp of the time the last experiment began) #. scenarios_file - Path to :token:`scenarios_file` as described in :ref:`Full experiment ` Single Chunk ************ | Divides the list of scenarios into equal chunks and runs the specified chunk. | Good for dividing the experiment into small chunks are running simultaneously. Command: :: scenarios_runner Arg: #. result_root_dir - Root dir where the results are created (in dir "chunk_/results_" where timestamp is the time the experiment began) #. scenarios_file - Path to :token:`scenarios_file` as described in :ref:`Full experiment ` #. number_of_chunks - Number of chunks to divide the scenarios list to. #. chunk_number - Chunk index to run (zero based). Executes all scenarios in the chunk Summarize chunks **************** | Summarize all last chunks into one dir that merges all results (results per scenario and overall results). | Execute after running all chunks. | All chunks must be under same root dir and part of the same experiment | Merged results are under /all Command: :: scenarios_runner end Arg: #. result_root_dir - Root dir where the results are created (in dir "all/results_" where timestamp is the time the experiment began) #. scenarios_file - Path to :token:`scenarios_file` as described in :ref:`Full experiment ` #. number_of_chunks - Number of chunks to divide the scenarios list to. Example of scenarios_file ~~~~~~~~~~~~~~~~~~~~~~~~~ :: import itertools from discopygal.experiments.scenarios_runner import Scenario from discopygal.solvers.rrt.drrt_star import dRRT_star random_sample_counter_list = [0, 1, 10, 20, 30, 50, 75, 100, 200, 500] scenes = ["examples/scenes/tunnels_disc.json", "examples/scenes/coffee_shop/coffee_shop.json", "examples/scenes/2_discs_corridor.json", "examples/scenes/2_pocket_maze_tight.json"] SCENARIOS = [Scenario(dRRT_star, scene, {"prm_num_landmarks": 1000, "num_landmarks": 100, "random_sample_counter": random_sample_counter}) for random_sample_counter, scene in itertools.product(random_sample_counter_list, scenes)] RESULT_HANDLERS = {"size_of_roadmap": lambda _, solver: len(solver.roadmap.edges), "num_of_nodes": lambda _, solver: len(solver.roadmap.points)