From 354d73d1e868d1e47b4bece46ea5213e628d2195 Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Tue, 4 Jul 2023 14:25:07 +0200
Subject: [PATCH 1/9] First draft of CLI documentation (WIP)

---
 docs/getting-started/cli.rst | 130 +++++++++++++++++++++++++++++++++++
 docs/index.rst               |   1 +
 2 files changed, 131 insertions(+)
 create mode 100644 docs/getting-started/cli.rst

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
new file mode 100644
index 000000000..b5e4a5696
--- /dev/null
+++ b/docs/getting-started/cli.rst
@@ -0,0 +1,130 @@
+======================
+Command Line Interface
+======================
+
+Many features of the core library are accessible via the command line interface built
+using the `Sacred <https://github.com/idsia/sacred>`_ package.
+
+Sacred is used to configure and run the algorithms of the core library.
+It is centered around the concept of `experiments <https://sacred.readthedocs.io/en/stable/experiment.html>`_
+which are composed of reusable `ingredients <https://sacred.readthedocs.io/en/stable/ingredients.html>`_.
+Each experiment and each ingredient has its own configuration namespace.
+Named configurations are used to specify a coherent set of configuration values.
+It is recommended to at least read the
+`Sacred documentation about the command line interface <https://sacred.readthedocs.io/en/stable/command_line.html>`_.
+
+The :py:mod:`scripts <imitation.scripts>` package contains a number of sacred experiments to either execute algorithms or perform utility tasks.
+The most important :py:mod:`ingredients <imitation.scripts.ingredients>` for imitation learning are:
+
+- :py:mod:`Environments <imitation.scripts.ingredients.environment>`
+- :py:mod:`Expert Policies <imitation.scripts.ingredients.expert>`
+- :py:mod:`Expert Demonstrations <imitation.scripts.ingredients.demonstrations>`
+- :py:mod:`Reward Functions <imitation.scripts.ingredients.reward>`
+
+
+Usage Examples
+==============
+
+Here we demonstrate some usage examples for the command line interface.
+You can always find out all the configurable values by running:
+
+.. code-block:: bash
+
+    python -m imitation.scripts.<script> print_config
+
+Run BC on the `CartPole-v1` environment with a pre-trained PPO policy as expert:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    python -m imitation.scripts.train_imitation bc with \
+        cartpole \
+        demonstrations.n_expert_demos=50 \
+        bc.train_kwargs.n_batches=2000 \
+        expert.policy_type=ppo \
+        expert.loader_kwargs.path=tests/testdata/expert_models/cartpole_0/policies/final/model.zip
+
+50 expert demonstrations are sampled from the PPO policy that is included in the testdata folder.
+2000 batches are enough to train a good policy.
+
+Run DAgger on the `CartPole-v0` environment with a random policy as expert:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    python -m imitation.scripts.train_imitation dagger with \
+        cartpole \
+        dagger.total_timesteps=2000 \
+        demonstrations.n_expert_demos=10 \
+        expert.policy_type=random
+
+This will not produce any meaningful results, since a random policy is not a good expert.
+
+
+Run AIRL on the `MountainCar-v0` environment with a expert from the HuggingFace model hub:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    python -m imitation.scripts.train_adversarial airl with \
+        seals_mountain_car \
+        total_timesteps=50000 \
+        expert.policy_type=ppo-huggingface \
+        demonstrations.n_expert_demos=500
+
+TODO: tweak above parameters to get good results
+
+
+Run GAIL on the `seals/Swimmer-v0` (named config) environment with an ensemble of reward networks:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+    xxx
+
+Algorithm Scripts
+=================
+
+What script to call for which algorithm?
+
++---------------------------------+----------+--------------------------+
+|  Algorithm                      | Script   |  command line arguments  |
++=================================+==========+==========================+
+|  BC                             |  xxx     |  xxx                     |
++---------------------------------+----------+--------------------------+
+| DAgger                          |  xxx     |  xxx                     |
++---------------------------------+----------+--------------------------+
+| AIRL                            |  xxx     |  xxx                     |
++---------------------------------+----------+--------------------------+
+| GAIL                            |  xxx     |  xxx                     |
++---------------------------------+----------+--------------------------+
+| Preference Comparison           |  xxx     |  xxx                     |
++---------------------------------+----------+--------------------------+
+| MCE IRL                         |  none    |  xxx                     |
++---------------------------------+----------+--------------------------+
+| Density Based Reward Estimation |  none    |  xxx                     |
++---------------------------------+----------+--------------------------+
+
+
+Utility Scripts
+===============
+
++--------------------------------+----------+--------------------------+
+| Functionality                  | Script   |  command line arguments  |
++================================+==========+==========================+
+|Reinforcement Learning          |  xxx     |  xxx                     |
++--------------------------------+----------+--------------------------+
+| Evaluating a Policy            |  xxx     |  xxx                     |
++--------------------------------+----------+--------------------------+
+| Parallel Execution             |  xxx     |  xxx                     |
++--------------------------------+----------+--------------------------+
+| Converting Trajectory Formats  |  xxx     |  xxx                     |
++--------------------------------+----------+--------------------------+
+| Analyzing Experimental Results |  xxx     |  xxx                     |
++--------------------------------+----------+--------------------------+
+
+
+Output Directories
+==================
+
+Where do all the files go and what is their purpose?
\ No newline at end of file
diff --git a/docs/index.rst b/docs/index.rst
index f3ec53b03..891ab063f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -47,6 +47,7 @@ If you use ``imitation`` in your research project, please cite our paper to help
    getting-started/what-is-imitation
    getting-started/variable-horizon
    getting-started/first-steps
+   getting-started/cli
 
 .. toctree::
    :maxdepth: 2

From 722b25235888b1b3bf9dde8b7627e67ee0260387 Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Wed, 5 Jul 2023 13:51:13 +0200
Subject: [PATCH 2/9] Add more examples for the CLI, improve some wording, add
 a overview table for the algorithm scripts.

---
 docs/getting-started/cli.rst | 64 +++++++++++++++++++++---------------
 1 file changed, 38 insertions(+), 26 deletions(-)

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
index b5e4a5696..3613b9e67 100644
--- a/docs/getting-started/cli.rst
+++ b/docs/getting-started/cli.rst
@@ -5,7 +5,7 @@ Command Line Interface
 Many features of the core library are accessible via the command line interface built
 using the `Sacred <https://github.com/idsia/sacred>`_ package.
 
-Sacred is used to configure and run the algorithms of the core library.
+Sacred is used to configure and run the algorithms.
 It is centered around the concept of `experiments <https://sacred.readthedocs.io/en/stable/experiment.html>`_
 which are composed of reusable `ingredients <https://sacred.readthedocs.io/en/stable/ingredients.html>`_.
 Each experiment and each ingredient has its own configuration namespace.
@@ -38,7 +38,7 @@ Run BC on the `CartPole-v1` environment with a pre-trained PPO policy as expert:
 .. code-block:: bash
 
     python -m imitation.scripts.train_imitation bc with \
-        cartpole \
+        cartpole \  # This is a named configuration
         demonstrations.n_expert_demos=50 \
         bc.train_kwargs.n_batches=2000 \
         expert.policy_type=ppo \
@@ -68,43 +68,55 @@ Run AIRL on the `MountainCar-v0` environment with a expert from the HuggingFace
 
     python -m imitation.scripts.train_adversarial airl with \
         seals_mountain_car \
-        total_timesteps=50000 \
+        total_timesteps=5000 \
         expert.policy_type=ppo-huggingface \
         demonstrations.n_expert_demos=500
 
-TODO: tweak above parameters to get good results
+Note: the small number of total timesteps is only for demonstration purposes and will not produce a good policy.
+
+
+Run GAIL on the `seals/Swimmer-v0` environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Here we do not use the named configuration for the seals environment, but instead specify the gym_id directly.
+The :code:`seals:` prefix ensures that the seals package is imported and the environment is registered.
 
-Run GAIL on the `seals/Swimmer-v0` (named config) environment with an ensemble of reward networks:
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Note that the Swimmer environment needs `mujoco_py` to be installed.
 
 .. code-block:: bash
 
-    xxx
+    python -m imitation.scripts.train_adversarial gail with \
+            environment.gym_id="seals:seals/Swimmer-v0" \
+            total_timesteps=5000 \
+            demonstrations.n_expert_demos=50
+
 
 Algorithm Scripts
 =================
 
-What script to call for which algorithm?
-
-+---------------------------------+----------+--------------------------+
-|  Algorithm                      | Script   |  command line arguments  |
-+=================================+==========+==========================+
-|  BC                             |  xxx     |  xxx                     |
-+---------------------------------+----------+--------------------------+
-| DAgger                          |  xxx     |  xxx                     |
-+---------------------------------+----------+--------------------------+
-| AIRL                            |  xxx     |  xxx                     |
-+---------------------------------+----------+--------------------------+
-| GAIL                            |  xxx     |  xxx                     |
-+---------------------------------+----------+--------------------------+
-| Preference Comparison           |  xxx     |  xxx                     |
-+---------------------------------+----------+--------------------------+
-| MCE IRL                         |  none    |  xxx                     |
-+---------------------------------+----------+--------------------------+
-| Density Based Reward Estimation |  none    |  xxx                     |
-+---------------------------------+----------+--------------------------+
+Call the algorithm scripts like this:
+
+.. code-block:: bash
 
+    python -m imitation.scripts.<script> [command] with <named_config> <config_values>
+
++---------------------------------+------------------------------+----------+
+|  algorithm                      | script                       |  command |
++=================================+==============================+==========+
+| BC                              | train_imitation              |  bc      |
++---------------------------------+------------------------------+----------+
+| DAgger                          | train_imitation              |  dagger  |
++---------------------------------+------------------------------+----------+
+| AIRL                            | train_adversarial            |  airl    |
++---------------------------------+------------------------------+----------+
+| GAIL                            | train_adversarial            |  gail    |
++---------------------------------+------------------------------+----------+
+| Preference Comparison           | train_preference_comparisons |  -       |
++---------------------------------+------------------------------+----------+
+| MCE IRL                         | none                         |  -       |
++---------------------------------+------------------------------+----------+
+| Density Based Reward Estimation | none                         |  -       |
++---------------------------------+------------------------------+----------+
 
 Utility Scripts
 ===============

From c2d2d4419f0f0b38e1cc867cac776bb0dd99edae Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Wed, 5 Jul 2023 14:05:06 +0200
Subject: [PATCH 3/9] Add documentation of the script output format.

---
 docs/getting-started/cli.rst | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
index 3613b9e67..d6f67a37b 100644
--- a/docs/getting-started/cli.rst
+++ b/docs/getting-started/cli.rst
@@ -139,4 +139,20 @@ Utility Scripts
 Output Directories
 ==================
 
-Where do all the files go and what is their purpose?
\ No newline at end of file
+The results of the script runs are stored in the following directory structure:
+
+.. code-block::
+
+    output
+    ├── <algo>
+    │   └── <environment>
+    │       └── <timestamp>
+    │           ├── log
+    │           ├── monitor
+    │           └── sacred -> ../../../sacred/<script_name>/1
+    └── sacred
+        └── <script_name>
+            ├── 1
+            └── _sources
+
+It contains the final model, tensorboard logs, sacred logs and the sacred source files.

From 6596b3f4b9c7b42f45d7c7b711e8557e8fa5f45f Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Wed, 5 Jul 2023 14:05:23 +0200
Subject: [PATCH 4/9] Remove comment from command line example.

---
 docs/getting-started/cli.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
index d6f67a37b..dc63332d6 100644
--- a/docs/getting-started/cli.rst
+++ b/docs/getting-started/cli.rst
@@ -35,10 +35,12 @@ You can always find out all the configurable values by running:
 Run BC on the `CartPole-v1` environment with a pre-trained PPO policy as expert:
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Note: the cartpole environment is specified via a named configuration
+
 .. code-block:: bash
 
     python -m imitation.scripts.train_imitation bc with \
-        cartpole \  # This is a named configuration
+        cartpole \
         demonstrations.n_expert_demos=50 \
         bc.train_kwargs.n_batches=2000 \
         expert.policy_type=ppo \

From 80a4ca5dd1409228434028acf59737e265a75423 Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Thu, 6 Jul 2023 14:33:55 +0200
Subject: [PATCH 5/9] Add table for the utility scripts.

---
 docs/getting-started/cli.rst | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
index dc63332d6..23285451c 100644
--- a/docs/getting-started/cli.rst
+++ b/docs/getting-started/cli.rst
@@ -123,19 +123,25 @@ Call the algorithm scripts like this:
 Utility Scripts
 ===============
 
-+--------------------------------+----------+--------------------------+
-| Functionality                  | Script   |  command line arguments  |
-+================================+==========+==========================+
-|Reinforcement Learning          |  xxx     |  xxx                     |
-+--------------------------------+----------+--------------------------+
-| Evaluating a Policy            |  xxx     |  xxx                     |
-+--------------------------------+----------+--------------------------+
-| Parallel Execution             |  xxx     |  xxx                     |
-+--------------------------------+----------+--------------------------+
-| Converting Trajectory Formats  |  xxx     |  xxx                     |
-+--------------------------------+----------+--------------------------+
-| Analyzing Experimental Results |  xxx     |  xxx                     |
-+--------------------------------+----------+--------------------------+
+Call the utility scripts like this:
+
+.. code-block:: bash
+
+    python -m imitation.scripts.<script>
+
++-----------------------------------------+-----------------------------------------------------------+
+| Functionality                           | Script                                                    |
++=========================================+===========================================================+
+| Reinforcement Learning                  | :py:mod:`train_rl <imitation.scripts.train_rl>`           |
++-----------------------------------------+-----------------------------------------------------------+
+| Evaluating a Policy                     | :py:mod:`eval_policy <imitation.scripts.eval_policy>`     |
++-----------------------------------------+-----------------------------------------------------------+
+| Parallel Execution of Algorithm Scripts | :py:mod:`parallel <imitation.scripts.parallel>`           |
++-----------------------------------------+-----------------------------------------------------------+
+| Converting Trajectory Formats           | :py:mod:`convert_trajs <imitation.scripts.convert_trajs>` |
++-----------------------------------------+-----------------------------------------------------------+
+| Analyzing Experimental Results          | :py:mod:`analyze <imitation.scripts.analyze>`             |
++-----------------------------------------+-----------------------------------------------------------+
 
 
 Output Directories

From f47d6e7358de525a55f46205c269840e2d19dfd2 Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Fri, 7 Jul 2023 17:23:53 +0200
Subject: [PATCH 6/9] Improve ingredient documentation.

---
 src/imitation/scripts/ingredients/__init__.py    |  2 +-
 src/imitation/scripts/ingredients/bc.py          |  2 +-
 .../scripts/ingredients/demonstrations.py        | 16 ++++++++++++----
 src/imitation/scripts/ingredients/environment.py |  2 +-
 src/imitation/scripts/ingredients/expert.py      | 14 +++++++++++++-
 src/imitation/scripts/ingredients/logging.py     |  6 +++++-
 src/imitation/scripts/ingredients/policy.py      |  2 +-
 .../scripts/ingredients/policy_evaluation.py     |  5 ++++-
 src/imitation/scripts/ingredients/reward.py      |  2 +-
 src/imitation/scripts/ingredients/rl.py          |  5 ++++-
 src/imitation/scripts/ingredients/wb.py          |  2 +-
 11 files changed, 44 insertions(+), 14 deletions(-)

diff --git a/src/imitation/scripts/ingredients/__init__.py b/src/imitation/scripts/ingredients/__init__.py
index 282ca2058..832d6178c 100644
--- a/src/imitation/scripts/ingredients/__init__.py
+++ b/src/imitation/scripts/ingredients/__init__.py
@@ -1 +1 @@
-"""Ingredients for scripts."""
+"""Ingredients for Sacred experiments."""
diff --git a/src/imitation/scripts/ingredients/bc.py b/src/imitation/scripts/ingredients/bc.py
index 65105633c..10ec27854 100644
--- a/src/imitation/scripts/ingredients/bc.py
+++ b/src/imitation/scripts/ingredients/bc.py
@@ -1,4 +1,4 @@
-"""Ingredients for training a BC policy."""
+"""This ingredient provides BC algorithm instance by either loading it from disk or constructing it from scratch."""
 import warnings
 from typing import Optional, Sequence
 
diff --git a/src/imitation/scripts/ingredients/demonstrations.py b/src/imitation/scripts/ingredients/demonstrations.py
index a8d8e9ced..0744e9194 100644
--- a/src/imitation/scripts/ingredients/demonstrations.py
+++ b/src/imitation/scripts/ingredients/demonstrations.py
@@ -1,4 +1,8 @@
-"""Ingredient for scripts learning from demonstrations."""
+"""This ingredient provides (expert) demonstrations to learn from.
+
+The demonstrations are either loaded from disk, from the HuggingFace Dataset Hub, or
+sampled from the expert policy provided by the expert ingredient.
+"""
 
 import logging
 import pathlib
@@ -25,11 +29,15 @@
 
 @demonstrations_ingredient.config
 def config():
+    # Either "local" or "{algo}-huggingface" to load them from the HuggingFace Dataset Hub.
     rollout_type = "local"
-    # path to file containing rollouts. If rollout_path is None
-    # and rollout_type is local, they are sampled from the expert.
+
+    # If none, they are sampled from the expert policy.
     rollout_path = None
-    n_expert_demos = None  # Num demos used or sampled. None loads every demo possible.
+
+    # Num demos used or sampled. None loads every demo possible.
+    n_expert_demos = None
+
     locals()  # quieten flake8
 
 
diff --git a/src/imitation/scripts/ingredients/environment.py b/src/imitation/scripts/ingredients/environment.py
index 9ddaa971c..40f70d5c2 100644
--- a/src/imitation/scripts/ingredients/environment.py
+++ b/src/imitation/scripts/ingredients/environment.py
@@ -1,4 +1,4 @@
-"""Environment Ingredient for sacred experiments."""
+"""This ingredient provides a vectorized gym environment."""
 import contextlib
 from typing import Any, Generator, Mapping
 
diff --git a/src/imitation/scripts/ingredients/expert.py b/src/imitation/scripts/ingredients/expert.py
index 14073e420..318d5c20f 100644
--- a/src/imitation/scripts/ingredients/expert.py
+++ b/src/imitation/scripts/ingredients/expert.py
@@ -1,4 +1,16 @@
-"""Common configuration elements for loading of expert policies."""
+"""This ingredient provides an expert policy.
+
+The expert policy is either loaded from disk or from the HuggingFace Model Hub or is
+a test policy (e.g., random or zero).
+The supported policy types are:
+
+- `ppo` and `sac`: A policy trained with SB3. Needs a `path` in the `loader_kwargs`.
+- `<algo>-huggingface` (algo can be `ppo` or `sac`): A policy trained with SB3 and uploaded to the HuggingFace Model
+    Hub. Will load the model from the repo `<organization>/<algo>-<env_name>`.
+    You can set the organization with the `organization` key in `loader_kwargs`. The default is `HumanCompatibleAI`.
+- `random`: A policy that takes random actions.
+- `zero`: A policy that takes zero actions.
+"""
 import sacred
 
 from imitation.policies import serialize
diff --git a/src/imitation/scripts/ingredients/logging.py b/src/imitation/scripts/ingredients/logging.py
index 80fab628e..30e7825b8 100644
--- a/src/imitation/scripts/ingredients/logging.py
+++ b/src/imitation/scripts/ingredients/logging.py
@@ -1,4 +1,8 @@
-"""Logging ingredient for scripts."""
+"""This ingredient provides a number of logging utilities.
+
+It is responsible for logging to WandB, TensorBoard, and stdout.
+It will also create a symlink to the sacred logging directory in the log directory.
+"""
 
 import logging
 import pathlib
diff --git a/src/imitation/scripts/ingredients/policy.py b/src/imitation/scripts/ingredients/policy.py
index 27c9a8319..ab2f972ac 100644
--- a/src/imitation/scripts/ingredients/policy.py
+++ b/src/imitation/scripts/ingredients/policy.py
@@ -1,4 +1,4 @@
-"""Ingredient implementation for a SB3 policy."""
+"""This ingredient provides a newly constructed stable-baselines3 policy."""
 
 import logging
 from typing import Any, Mapping, Type
diff --git a/src/imitation/scripts/ingredients/policy_evaluation.py b/src/imitation/scripts/ingredients/policy_evaluation.py
index 80083f361..2cad2958c 100644
--- a/src/imitation/scripts/ingredients/policy_evaluation.py
+++ b/src/imitation/scripts/ingredients/policy_evaluation.py
@@ -1,4 +1,7 @@
-"""Sacred ingredient for evaluating a policy on a VecEnv."""
+"""This ingredient performs evaluation of learned policy.
+
+It takes care of the right wrappers, does some rollouts and computes statistics of the rollouts.
+"""
 
 from typing import Mapping, Union
 
diff --git a/src/imitation/scripts/ingredients/reward.py b/src/imitation/scripts/ingredients/reward.py
index c40d3751f..2e2b67022 100644
--- a/src/imitation/scripts/ingredients/reward.py
+++ b/src/imitation/scripts/ingredients/reward.py
@@ -1,4 +1,4 @@
-"""Common configuration elements for reward network training."""
+"""This ingredient provides a reward network."""
 
 import logging
 import typing
diff --git a/src/imitation/scripts/ingredients/rl.py b/src/imitation/scripts/ingredients/rl.py
index b97b4211a..10e19a152 100644
--- a/src/imitation/scripts/ingredients/rl.py
+++ b/src/imitation/scripts/ingredients/rl.py
@@ -1,4 +1,7 @@
-"""Common configuration elements for reinforcement learning."""
+"""This ingredient provides a reinforcement learning algorithm from stable-baselines3.
+
+The algorithm instance is either freshly constructed or loaded from a file.
+"""
 
 import logging
 import warnings
diff --git a/src/imitation/scripts/ingredients/wb.py b/src/imitation/scripts/ingredients/wb.py
index 0ac62ffb5..ebc15317f 100644
--- a/src/imitation/scripts/ingredients/wb.py
+++ b/src/imitation/scripts/ingredients/wb.py
@@ -1,4 +1,4 @@
-"""Weights & Biases configuration elements for scripts."""
+"""This ingredient provides Weights & Biases logging."""
 
 import logging
 from typing import Any, Mapping, Optional

From b5cc7b65f4b87bd804f18a01adf0f38e6c78dabb Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Fri, 7 Jul 2023 17:46:13 +0200
Subject: [PATCH 7/9] Fix formatting issues.

---
 src/imitation/scripts/ingredients/bc.py           |  5 ++++-
 .../scripts/ingredients/demonstrations.py         |  2 +-
 src/imitation/scripts/ingredients/expert.py       | 15 +++++++++------
 .../scripts/ingredients/policy_evaluation.py      |  3 ++-
 4 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/src/imitation/scripts/ingredients/bc.py b/src/imitation/scripts/ingredients/bc.py
index 10ec27854..70d16bfe1 100644
--- a/src/imitation/scripts/ingredients/bc.py
+++ b/src/imitation/scripts/ingredients/bc.py
@@ -1,4 +1,7 @@
-"""This ingredient provides BC algorithm instance by either loading it from disk or constructing it from scratch."""
+"""This ingredient provides BC algorithm instance.
+
+It is either loaded from disk or constructed from scratch.
+"""
 import warnings
 from typing import Optional, Sequence
 
diff --git a/src/imitation/scripts/ingredients/demonstrations.py b/src/imitation/scripts/ingredients/demonstrations.py
index 0744e9194..433ddfa6a 100644
--- a/src/imitation/scripts/ingredients/demonstrations.py
+++ b/src/imitation/scripts/ingredients/demonstrations.py
@@ -29,7 +29,7 @@
 
 @demonstrations_ingredient.config
 def config():
-    # Either "local" or "{algo}-huggingface" to load them from the HuggingFace Dataset Hub.
+    # Either "local" or "{algo}-huggingface" to load them from the HF Dataset Hub.
     rollout_type = "local"
 
     # If none, they are sampled from the expert policy.
diff --git a/src/imitation/scripts/ingredients/expert.py b/src/imitation/scripts/ingredients/expert.py
index 318d5c20f..012d96d2c 100644
--- a/src/imitation/scripts/ingredients/expert.py
+++ b/src/imitation/scripts/ingredients/expert.py
@@ -4,12 +4,15 @@
 a test policy (e.g., random or zero).
 The supported policy types are:
 
-- `ppo` and `sac`: A policy trained with SB3. Needs a `path` in the `loader_kwargs`.
-- `<algo>-huggingface` (algo can be `ppo` or `sac`): A policy trained with SB3 and uploaded to the HuggingFace Model
-    Hub. Will load the model from the repo `<organization>/<algo>-<env_name>`.
-    You can set the organization with the `organization` key in `loader_kwargs`. The default is `HumanCompatibleAI`.
-- `random`: A policy that takes random actions.
-- `zero`: A policy that takes zero actions.
+- :code:`ppo` and :code:`sac`: A policy trained with SB3.
+    Needs a `path` in the `loader_kwargs`.
+- :code:`<algo>-huggingface` (algo can be `ppo` or `sac`):
+    A policy trained with SB3 and uploaded to the HuggingFace Model Hub.
+    Will load the model from the repo :code:`<organization>/<algo>-<env_name>`.
+    You can set the organization with the `organization` key in :code:`loader_kwargs`.
+    The default is `HumanCompatibleAI`.
+- :code:`random`: A policy that takes random actions.
+- :code:`zero`: A policy that takes zero actions.
 """
 import sacred
 
diff --git a/src/imitation/scripts/ingredients/policy_evaluation.py b/src/imitation/scripts/ingredients/policy_evaluation.py
index 2cad2958c..c836cfa1f 100644
--- a/src/imitation/scripts/ingredients/policy_evaluation.py
+++ b/src/imitation/scripts/ingredients/policy_evaluation.py
@@ -1,6 +1,7 @@
 """This ingredient performs evaluation of learned policy.
 
-It takes care of the right wrappers, does some rollouts and computes statistics of the rollouts.
+It takes care of the right wrappers, does some rollouts
+and computes statistics of the rollouts.
 """
 
 from typing import Mapping, Union

From c021627fdc9b13813b31a2e3ce338c758c4e1800 Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Mon, 10 Jul 2023 10:38:58 +0200
Subject: [PATCH 8/9] Use .. note:: sections where appropriate.

---
 docs/getting-started/cli.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
index 23285451c..ffe4fd380 100644
--- a/docs/getting-started/cli.rst
+++ b/docs/getting-started/cli.rst
@@ -35,7 +35,7 @@ You can always find out all the configurable values by running:
 Run BC on the `CartPole-v1` environment with a pre-trained PPO policy as expert:
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Note: the cartpole environment is specified via a named configuration
+.. note:: Here the cartpole environment is specified via a named configuration.
 
 .. code-block:: bash
 
@@ -74,7 +74,7 @@ Run AIRL on the `MountainCar-v0` environment with a expert from the HuggingFace
         expert.policy_type=ppo-huggingface \
         demonstrations.n_expert_demos=500
 
-Note: the small number of total timesteps is only for demonstration purposes and will not produce a good policy.
+.. note:: The small number of total timesteps is only for demonstration purposes and will not produce a good policy.
 
 
 Run GAIL on the `seals/Swimmer-v0` environment
@@ -83,7 +83,7 @@ Run GAIL on the `seals/Swimmer-v0` environment
 Here we do not use the named configuration for the seals environment, but instead specify the gym_id directly.
 The :code:`seals:` prefix ensures that the seals package is imported and the environment is registered.
 
-Note that the Swimmer environment needs `mujoco_py` to be installed.
+.. note:: The Swimmer environment needs `mujoco_py` to be installed.
 
 .. code-block:: bash
 

From 1d2c18857bc3267b9b6e399bff0e1461bf035d93 Mon Sep 17 00:00:00 2001
From: Maximilian Ernestus <maximilian@ernestus.de>
Date: Mon, 10 Jul 2023 10:42:24 +0200
Subject: [PATCH 9/9] Use double backticks for literals.

---
 docs/getting-started/cli.rst | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/docs/getting-started/cli.rst b/docs/getting-started/cli.rst
index ffe4fd380..cb9ef7cab 100644
--- a/docs/getting-started/cli.rst
+++ b/docs/getting-started/cli.rst
@@ -32,8 +32,8 @@ You can always find out all the configurable values by running:
 
     python -m imitation.scripts.<script> print_config
 
-Run BC on the `CartPole-v1` environment with a pre-trained PPO policy as expert:
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Run BC on the ``CartPole-v1`` environment with a pre-trained PPO policy as expert:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. note:: Here the cartpole environment is specified via a named configuration.
 
@@ -49,8 +49,8 @@ Run BC on the `CartPole-v1` environment with a pre-trained PPO policy as expert:
 50 expert demonstrations are sampled from the PPO policy that is included in the testdata folder.
 2000 batches are enough to train a good policy.
 
-Run DAgger on the `CartPole-v0` environment with a random policy as expert:
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Run DAgger on the ``CartPole-v0`` environment with a random policy as expert:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: bash
 
@@ -63,8 +63,8 @@ Run DAgger on the `CartPole-v0` environment with a random policy as expert:
 This will not produce any meaningful results, since a random policy is not a good expert.
 
 
-Run AIRL on the `MountainCar-v0` environment with a expert from the HuggingFace model hub:
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Run AIRL on the ``MountainCar-v0`` environment with a expert from the HuggingFace model hub:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: bash
 
@@ -77,11 +77,11 @@ Run AIRL on the `MountainCar-v0` environment with a expert from the HuggingFace
 .. note:: The small number of total timesteps is only for demonstration purposes and will not produce a good policy.
 
 
-Run GAIL on the `seals/Swimmer-v0` environment
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Run GAIL on the ``seals/Swimmer-v0`` environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Here we do not use the named configuration for the seals environment, but instead specify the gym_id directly.
-The :code:`seals:` prefix ensures that the seals package is imported and the environment is registered.
+The ``seals:`` prefix ensures that the seals package is imported and the environment is registered.
 
 .. note:: The Swimmer environment needs `mujoco_py` to be installed.