T2.1_deep_expert

Expert Agent python package exploiting the expert knowledge in two ways:

1. Focus the exploration phase of an RL agent (DeepQ) on specific zones of a power grid
2. Reduce the action space to most relevant ones and improve the scalability or RL agents. There are two variants of this approach:
   • Heuristic-based: A set of well known heuristics with some greedy search over the reduced action space is used to solve the overload and congestion problems;
   • Learning-based: A PPO tries to learn the effective topological manipulations on the grid, by considering the reduced action space. Its combination with some heuristics helps to remedy most of the overload and congestion problems.

Credits

• Credits for Javaness winning solution at the L2RPN 2023 IDF AI challenge which has inspired the heuristic part of this work and most of the code is adapted and reused. The adapted code is on a forked repository which could be found here.
• Credits for CurriculumAgent, which has inspired the search for reduced action space. Herein, we have replaced the greedy search over all the action space, by those suggested using expert knowlege.
• The action suggested by expert knowledge uses the ExpertOp4Grid package.

Installation guide

To be able to run the experiments in this repository, the following steps show how to install this package and its dependencies from source.

Requirements

• Python >= 3.6
• ExpertOp4Grid package (customized)
• LJN agent package (customized)

Setup a Virtualenv (optional)

Create a Conda env (recommended)

conda create -n expert_agent python=3.10
conda activate venv_gnn

Create a virtual environment

cd my-project-folder
pip3 install -U virtualenv
python3 -m virtualenv venv_expert_agent
source venv_expert_agent/bin/activate

Install the prerequisites

Important

These steps are mandatory to be able to use the package and its different functionalities

ExpertOp4Grid package

git clone git@github.com:Mleyliabadi/ExpertOp4Grid.git
cd ExpertOp4Grid
pip install -U .

LJN Agent package

git clone git@github.com:Mleyliabadi/l2rpn-2023-ljn-agent.git
cd l2rpn-2023-ljn-agent
pip install -U .

Prepare the environment

git clone git@github.com:AI4REALNET/grid2op-scenario.git
cd grid2op-scenario
cp -r ai4realnet_small /home/<USERNAME>/data_grid2op/.

Install the current package from source

git clone git@github.com:AI4REALNET/T2.1_deep_expert.git
cd T2.1_deep_expert
pip3 install -U .[recommended]

To contribute

pip3 install -e .[recommended]

Overview of code structure

📂 ExpertAgent

├── 📂 configs

│ └── ...

├── 📂 getting_started

│     └── 0_extract_actions.ipynb

│     └── 1_apply_deepqexpert.ipynb

│     └── 2_apply_expert_agent_heuristic.ipynb

│     └── 3_apply_expert_agent_rl.ipynb

├── 📂 ExpertAgent

│ └── 📂 assets

│     └── ...

│ └── 📂 DeepQExpert

│     └── ...

│ └── 📂 ExpertAgent

│     └── agentHeuristic.py

│     └── agentRL.py

│ └── 📂 utils

│     └── extractExpertActions.py

│     └── extractAttackingExpertActions.py

│     └── ...

├── setup.py

How to use

A set of jupyter notebooks are provided to ease the use of the package for the users. Here is the list of notebooks:

1. 01_deep_q_expert.ipynb : It shows how to use the extended DeepQ agent
2. 02_expert_agent_heuistics.ipynb : It shows how to use the heuristic agent using the expert knowledge
3. 03_expert_agent_RL.ipynb : It show how to use the RL based agent harnessing expert knowledge to reduce the action space.

Reproducibility

1. DeepQExpert Agent

---

This agent applies an extended DeepQ algorithm for power grids and specifically works good with l2rpn_case14_sandbox environment.

Train

To train this agent, the following command could be executed from root and CLI:

python ExpertAgent/DeepQExpert/train.py \ 
    --save_path="l2rpn_case14_sandbox" \
    --num_train_steps=1000000 \
    --name="DeepQExpert" \
    --logs_dir="l2rpn_case14_sandbox/logs"

At the end of the training, the weights of the model and some information concerning the neural network architecture are saved and logged.

Evaluate

To evaluate an already trained version of it, the following command could be executed from root and using CLI:

python ExpertAgent/DeepQExpert/evaluate.py

At the end of the evaluation, a graphic representing the performance (reward/alive time) of the agent is visualized to the user.

2. ExpertAgent Heuristics

---

The heuristic version of the ExpertAgent does not require any training and the evaluation could be run using a main function included in the root of the package. This agent is already provided to work for ai4realnet_small scenario of AI4REALNET project and power grid usecase (first).

python main_expert_heuristic.py --nb_episode=15 --nb_process=1 --max_step=2016 --verbose=True 

At the end of the evaluation a graphic representing the performance (reward/alive time) of the agent is visualized to the user.

3. ExpertAgent RL

---

The RL based agent learning the reduced action space obtained using Expert Knowledge should be trained. The training could be launched using the corresponding main file provided in the root of the repository as:

python main_expert_rl_train.py

An already trained agent is also provided in the root of repository which can be loaded easily as (see notebook for a full example):

from ExpertAgent.utils import get_package_root
from ExpertAgent.ExpertAgent import ExpertAgentRL

env, env_gym = creat_env(...)

nn_kwargs = {...}

load_path = os.path.join(get_package_root(), "..", name, "model", "PPO_SB3")
agent = ExpertAgentRL(name="PPO_SB3",
                      env=env,
                      action_space=env.action_space,
                      gymenv=env_gym,
                      gym_act_space=env_gym.action_space,
                      gym_obs_space=env_gym.observation_space,
                      nn_kwargs=nn_kwargs
                      )

agent.load(load_path)

One the agent is trained or loaded, the evaluation could be done using the following command and the main file main_expert_rl_eval.py:

python main_expert_rl_eval.py --nb_episode=15 --nb_process=1 --max_step=2016 --verbose=True 

At the end of the evaluation a graphic representing the performance (reward/alive time) of the agent is visualized to the user.