{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "# Training using REINFORCE for Mujoco\n",
        "\n",
        "<img src=\"file://_static/img/tutorials/reinforce_invpend_gym_v26_fig1.gif\" width=\"400\" alt=\"agent-environment-diagram\">\n",
        "\n",
        "This tutorial serves 2 purposes:\n",
        " 1. To understand how to implement REINFORCE [1] from scratch to solve Mujoco's InvertedPendulum-v4\n",
        " 2. Implementation a deep reinforcement learning algorithm with Gymnasium's v0.26+ `step()` function\n",
        "\n",
        "We will be using **REINFORCE**, one of the earliest policy gradient methods. Unlike going under the burden of learning a value function first and then deriving a policy out of it,\n",
        "REINFORCE optimizes the policy directly. In other words, it is trained to maximize the probability of Monte-Carlo returns. More on that later.\n",
        "\n",
        "**Inverted Pendulum** is Mujoco's cartpole but now powered by the Mujoco physics simulator -\n",
        "which allows more complex experiments (such as varying the effects of gravity).\n",
        "This environment involves a cart that can moved linearly, with a pole fixed on it at one end and having another end free.\n",
        "The cart can be pushed left or right, and the goal is to balance the pole on the top of the cart by applying forces on the cart.\n",
        "More information on the environment could be found at https://gymnasium.farama.org/environments/mujoco/inverted_pendulum/\n",
        "\n",
        "**Training Objectives**: To balance the pole (inverted pendulum) on top of the cart\n",
        "\n",
        "**Actions**: The agent takes a 1D vector for actions. The action space is a continuous ``(action)`` in ``[-3, 3]``,\n",
        "where action represents the numerical force applied to the cart\n",
        "(with magnitude representing the amount of force and sign representing the direction)\n",
        "\n",
        "**Approach**: We use PyTorch to code REINFORCE from scratch to train a Neural Network policy to master Inverted Pendulum.\n",
        "\n",
        "An explanation of the Gymnasium v0.26+ `Env.step()` function\n",
        "\n",
        "``env.step(A)`` allows us to take an action 'A' in the current environment 'env'. The environment then executes the action\n",
        "and returns five variables:\n",
        "\n",
        "-  ``next_obs``: This is the observation that the agent will receive after taking the action.\n",
        "-  ``reward``: This is the reward that the agent will receive after taking the action.\n",
        "-  ``terminated``: This is a boolean variable that indicates whether or not the environment has terminated.\n",
        "-  ``truncated``: This is a boolean variable that also indicates whether the episode ended by early truncation, i.e., a time limit is reached.\n",
        "-  ``info``: This is a dictionary that might contain additional information about the environment.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from __future__ import annotations\n",
        "\n",
        "import random\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.distributions.normal import Normal\n",
        "\n",
        "import gymnasium as gym\n",
        "\n",
        "plt.rcParams[\"figure.figsize\"] = (10, 5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Policy Network\n",
        "\n",
        "<img src=\"file://_static/img/tutorials/reinforce_invpend_gym_v26_fig2.png\">\n",
        "\n",
        "We start by building a policy that the agent will learn using REINFORCE.\n",
        "A policy is a mapping from the current environment observation to a probability distribution of the actions to be taken.\n",
        "The policy used in the tutorial is parameterized by a neural network. It consists of 2 linear layers that are shared between both the predicted mean and standard deviation.\n",
        "Further, the single individual linear layers are used to estimate the mean and the standard deviation. ``nn.Tanh`` is used as a non-linearity between the hidden layers.\n",
        "The following function estimates a mean and standard deviation of a normal distribution from which an action is sampled. Hence it is expected for the policy to learn\n",
        "appropriate weights to output means and standard deviation based on the current observation.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "class Policy_Network(nn.Module):\n",
        "    \"\"\"Parametrized Policy Network.\"\"\"\n",
        "\n",
        "    def __init__(self, obs_space_dims: int, action_space_dims: int):\n",
        "        \"\"\"Initializes a neural network that estimates the mean and standard deviation\n",
        "         of a normal distribution from which an action is sampled from.\n",
        "\n",
        "        Args:\n",
        "            obs_space_dims: Dimension of the observation space\n",
        "            action_space_dims: Dimension of the action space\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "\n",
        "        hidden_space1 = 16  # Nothing special with 16, feel free to change\n",
        "        hidden_space2 = 32  # Nothing special with 32, feel free to change\n",
        "\n",
        "        # Shared Network\n",
        "        self.shared_net = nn.Sequential(\n",
        "            nn.Linear(obs_space_dims, hidden_space1),\n",
        "            nn.Tanh(),\n",
        "            nn.Linear(hidden_space1, hidden_space2),\n",
        "            nn.Tanh(),\n",
        "        )\n",
        "\n",
        "        # Policy Mean specific Linear Layer\n",
        "        self.policy_mean_net = nn.Sequential(\n",
        "            nn.Linear(hidden_space2, action_space_dims)\n",
        "        )\n",
        "\n",
        "        # Policy Std Dev specific Linear Layer\n",
        "        self.policy_stddev_net = nn.Sequential(\n",
        "            nn.Linear(hidden_space2, action_space_dims)\n",
        "        )\n",
        "\n",
        "    def forward(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:\n",
        "        \"\"\"Conditioned on the observation, returns the mean and standard deviation\n",
        "         of a normal distribution from which an action is sampled from.\n",
        "\n",
        "        Args:\n",
        "            x: Observation from the environment\n",
        "\n",
        "        Returns:\n",
        "            action_means: predicted mean of the normal distribution\n",
        "            action_stddevs: predicted standard deviation of the normal distribution\n",
        "        \"\"\"\n",
        "        shared_features = self.shared_net(x.float())\n",
        "\n",
        "        action_means = self.policy_mean_net(shared_features)\n",
        "        action_stddevs = torch.log(\n",
        "            1 + torch.exp(self.policy_stddev_net(shared_features))\n",
        "        )\n",
        "\n",
        "        return action_means, action_stddevs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Building an agent\n",
        "\n",
        "<img src=\"file://_static/img/tutorials/reinforce_invpend_gym_v26_fig3.jpeg\">\n",
        "\n",
        "Now that we are done building the policy, let us develop **REINFORCE** which gives life to the policy network.\n",
        "The algorithm of REINFORCE could be found above. As mentioned before, REINFORCE aims to maximize the Monte-Carlo returns.\n",
        "\n",
        "Fun Fact: REINFROCE is an acronym for \" 'RE'ward 'I'ncrement 'N'on-negative 'F'actor times 'O'ffset 'R'einforcement times 'C'haracteristic 'E'ligibility\n",
        "\n",
        "Note: The choice of hyperparameters is to train a decently performing agent. No extensive hyperparameter\n",
        "tuning was done.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "class REINFORCE:\n",
        "    \"\"\"REINFORCE algorithm.\"\"\"\n",
        "\n",
        "    def __init__(self, obs_space_dims: int, action_space_dims: int):\n",
        "        \"\"\"Initializes an agent that learns a policy via REINFORCE algorithm [1]\n",
        "        to solve the task at hand (Inverted Pendulum v4).\n",
        "\n",
        "        Args:\n",
        "            obs_space_dims: Dimension of the observation space\n",
        "            action_space_dims: Dimension of the action space\n",
        "        \"\"\"\n",
        "\n",
        "        # Hyperparameters\n",
        "        self.learning_rate = 1e-4  # Learning rate for policy optimization\n",
        "        self.gamma = 0.99  # Discount factor\n",
        "        self.eps = 1e-6  # small number for mathematical stability\n",
        "\n",
        "        self.probs = []  # Stores probability values of the sampled action\n",
        "        self.rewards = []  # Stores the corresponding rewards\n",
        "\n",
        "        self.net = Policy_Network(obs_space_dims, action_space_dims)\n",
        "        self.optimizer = torch.optim.AdamW(self.net.parameters(), lr=self.learning_rate)\n",
        "\n",
        "    def sample_action(self, state: np.ndarray) -> float:\n",
        "        \"\"\"Returns an action, conditioned on the policy and observation.\n",
        "\n",
        "        Args:\n",
        "            state: Observation from the environment\n",
        "\n",
        "        Returns:\n",
        "            action: Action to be performed\n",
        "        \"\"\"\n",
        "        state = torch.tensor(np.array([state]))\n",
        "        action_means, action_stddevs = self.net(state)\n",
        "\n",
        "        # create a normal distribution from the predicted\n",
        "        #   mean and standard deviation and sample an action\n",
        "        distrib = Normal(action_means[0] + self.eps, action_stddevs[0] + self.eps)\n",
        "        action = distrib.sample()\n",
        "        prob = distrib.log_prob(action)\n",
        "\n",
        "        action = action.numpy()\n",
        "\n",
        "        self.probs.append(prob)\n",
        "\n",
        "        return action\n",
        "\n",
        "    def update(self):\n",
        "        \"\"\"Updates the policy network's weights.\"\"\"\n",
        "        running_g = 0\n",
        "        gs = []\n",
        "\n",
        "        # Discounted return (backwards) - [::-1] will return an array in reverse\n",
        "        for R in self.rewards[::-1]:\n",
        "            running_g = R + self.gamma * running_g\n",
        "            gs.insert(0, running_g)\n",
        "\n",
        "        deltas = torch.tensor(gs)\n",
        "\n",
        "        loss = 0\n",
        "        # minimize -1 * prob * reward obtained\n",
        "        for log_prob, delta in zip(self.probs, deltas):\n",
        "            loss += log_prob.mean() * delta * (-1)\n",
        "\n",
        "        # Update the policy network\n",
        "        self.optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        self.optimizer.step()\n",
        "\n",
        "        # Empty / zero out all episode-centric/related variables\n",
        "        self.probs = []\n",
        "        self.rewards = []"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now lets train the policy using REINFORCE to master the task of Inverted Pendulum.\n",
        "\n",
        "Following is the overview of the training procedure\n",
        "\n",
        "   for seed in random seeds\n",
        "       reinitialize agent\n",
        "\n",
        "       for episode in range of max number of episodes\n",
        "           until episode is done\n",
        "               sample action based on current observation\n",
        "\n",
        "               take action and receive reward and next observation\n",
        "\n",
        "               store action take, its probability, and the observed reward\n",
        "           update the policy\n",
        "\n",
        "Note: Deep RL is fairly brittle concerning random seed in a lot of common use cases (https://spinningup.openai.com/en/latest/spinningup/spinningup.html).\n",
        "Hence it is important to test out various seeds, which we will be doing.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Episode: 0 Average Reward: 8\n"
          ]
        },
        {
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "Cell \u001b[0;32mIn[5], line 28\u001b[0m\n\u001b[1;32m     26\u001b[0m done \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m     27\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m done:\n\u001b[0;32m---> 28\u001b[0m     action \u001b[38;5;241m=\u001b[39m \u001b[43magent\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msample_action\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     30\u001b[0m     \u001b[38;5;66;03m# Step return type - `tuple[ObsType, SupportsFloat, bool, bool, dict[str, Any]]`\u001b[39;00m\n\u001b[1;32m     31\u001b[0m     \u001b[38;5;66;03m# These represent the next observation, the reward from the step,\u001b[39;00m\n\u001b[1;32m     32\u001b[0m     \u001b[38;5;66;03m# if the episode is terminated, if the episode is truncated and\u001b[39;00m\n\u001b[1;32m     33\u001b[0m     \u001b[38;5;66;03m# additional info from the step\u001b[39;00m\n\u001b[1;32m     34\u001b[0m     obs, reward, terminated, truncated, info \u001b[38;5;241m=\u001b[39m wrapped_env\u001b[38;5;241m.\u001b[39mstep(action)\n",
            "Cell \u001b[0;32mIn[4], line 40\u001b[0m, in \u001b[0;36mREINFORCE.sample_action\u001b[0;34m(self, state)\u001b[0m\n\u001b[1;32m     38\u001b[0m distrib \u001b[38;5;241m=\u001b[39m Normal(action_means[\u001b[38;5;241m0\u001b[39m] \u001b[38;5;241m+\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39meps, action_stddevs[\u001b[38;5;241m0\u001b[39m] \u001b[38;5;241m+\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39meps)\n\u001b[1;32m     39\u001b[0m action \u001b[38;5;241m=\u001b[39m distrib\u001b[38;5;241m.\u001b[39msample()\n\u001b[0;32m---> 40\u001b[0m prob \u001b[38;5;241m=\u001b[39m \u001b[43mdistrib\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlog_prob\u001b[49m\u001b[43m(\u001b[49m\u001b[43maction\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     42\u001b[0m action \u001b[38;5;241m=\u001b[39m action\u001b[38;5;241m.\u001b[39mnumpy()\n\u001b[1;32m     44\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprobs\u001b[38;5;241m.\u001b[39mappend(prob)\n",
            "File \u001b[0;32m~/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/torch/distributions/normal.py:83\u001b[0m, in \u001b[0;36mNormal.log_prob\u001b[0;34m(self, value)\u001b[0m\n\u001b[1;32m     80\u001b[0m \u001b[38;5;66;03m# compute the variance\u001b[39;00m\n\u001b[1;32m     81\u001b[0m var \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscale\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m2\u001b[39m\n\u001b[1;32m     82\u001b[0m log_scale \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m---> 83\u001b[0m     math\u001b[38;5;241m.\u001b[39mlog(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscale) \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28;43misinstance\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscale\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mReal\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscale\u001b[38;5;241m.\u001b[39mlog()\n\u001b[1;32m     84\u001b[0m )\n\u001b[1;32m     85\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m (\n\u001b[1;32m     86\u001b[0m     \u001b[38;5;241m-\u001b[39m((value \u001b[38;5;241m-\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mloc) \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39m \u001b[38;5;241m2\u001b[39m) \u001b[38;5;241m/\u001b[39m (\u001b[38;5;241m2\u001b[39m \u001b[38;5;241m*\u001b[39m var)\n\u001b[1;32m     87\u001b[0m     \u001b[38;5;241m-\u001b[39m log_scale\n\u001b[1;32m     88\u001b[0m     \u001b[38;5;241m-\u001b[39m math\u001b[38;5;241m.\u001b[39mlog(math\u001b[38;5;241m.\u001b[39msqrt(\u001b[38;5;241m2\u001b[39m \u001b[38;5;241m*\u001b[39m math\u001b[38;5;241m.\u001b[39mpi))\n\u001b[1;32m     89\u001b[0m )\n",
            "File \u001b[0;32m~/anaconda3/envs/pycuda_3_10/lib/python3.10/abc.py:117\u001b[0m, in \u001b[0;36mABCMeta.__instancecheck__\u001b[0;34m(cls, instance)\u001b[0m\n\u001b[1;32m    111\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Register a virtual subclass of an ABC.\u001b[39;00m\n\u001b[1;32m    112\u001b[0m \n\u001b[1;32m    113\u001b[0m \u001b[38;5;124;03m    Returns the subclass, to allow usage as a class decorator.\u001b[39;00m\n\u001b[1;32m    114\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m    115\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m _abc_register(\u001b[38;5;28mcls\u001b[39m, subclass)\n\u001b[0;32m--> 117\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__instancecheck__\u001b[39m(\u001b[38;5;28mcls\u001b[39m, instance):\n\u001b[1;32m    118\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Override for isinstance(instance, cls).\"\"\"\u001b[39;00m\n\u001b[1;32m    119\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m _abc_instancecheck(\u001b[38;5;28mcls\u001b[39m, instance)\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "# Create and wrap the environment\n",
        "env = gym.make(\"InvertedPendulum-v4\")\n",
        "wrapped_env = gym.wrappers.RecordEpisodeStatistics(env, 50)  # Records episode-reward\n",
        "\n",
        "total_num_episodes = int(5e3)  # Total number of episodes\n",
        "# Observation-space of InvertedPendulum-v4 (4)\n",
        "obs_space_dims = env.observation_space.shape[0]\n",
        "# Action-space of InvertedPendulum-v4 (1)\n",
        "action_space_dims = env.action_space.shape[0]\n",
        "rewards_over_seeds = []\n",
        "\n",
        "for seed in [1, 2, 3, 5, 8]:  # Fibonacci seeds\n",
        "    # set seed\n",
        "    torch.manual_seed(seed)\n",
        "    random.seed(seed)\n",
        "    np.random.seed(seed)\n",
        "\n",
        "    # Reinitialize agent every seed\n",
        "    agent = REINFORCE(obs_space_dims, action_space_dims)\n",
        "    reward_over_episodes = []\n",
        "\n",
        "    for episode in range(total_num_episodes):\n",
        "        # gymnasium v26 requires users to set seed while resetting the environment\n",
        "        obs, info = wrapped_env.reset(seed=seed)\n",
        "\n",
        "        done = False\n",
        "        while not done:\n",
        "            action = agent.sample_action(obs)\n",
        "\n",
        "            # Step return type - `tuple[ObsType, SupportsFloat, bool, bool, dict[str, Any]]`\n",
        "            # These represent the next observation, the reward from the step,\n",
        "            # if the episode is terminated, if the episode is truncated and\n",
        "            # additional info from the step\n",
        "            obs, reward, terminated, truncated, info = wrapped_env.step(action)\n",
        "            agent.rewards.append(reward)\n",
        "\n",
        "            # End the episode when either truncated or terminated is true\n",
        "            #  - truncated: The episode duration reaches max number of timesteps\n",
        "            #  - terminated: Any of the state space values is no longer finite.\n",
        "            done = terminated or truncated\n",
        "\n",
        "        reward_over_episodes.append(wrapped_env.return_queue[-1])\n",
        "        agent.update()\n",
        "\n",
        "        if episode % 1000 == 0:\n",
        "            avg_reward = int(np.mean(wrapped_env.return_queue))\n",
        "            print(\"Episode:\", episode, \"Average Reward:\", avg_reward)\n",
        "\n",
        "    rewards_over_seeds.append(reward_over_episodes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Plot learning curve\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/frank14f/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.\n",
            "  with pd.option_context('mode.use_inf_as_na', True):\n",
            "/home/frank14f/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.\n",
            "  with pd.option_context('mode.use_inf_as_na', True):\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 99916 characters]",
            "text/plain": [
              "<Figure size 1000x500 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "rewards_to_plot = [[reward[0] for reward in rewards] for rewards in rewards_over_seeds]\n",
        "df1 = pd.DataFrame(rewards_to_plot).melt()\n",
        "df1.rename(columns={\"variable\": \"episodes\", \"value\": \"reward\"}, inplace=True)\n",
        "sns.set(style=\"darkgrid\", context=\"talk\", palette=\"rainbow\")\n",
        "sns.lineplot(x=\"episodes\", y=\"reward\", data=df1).set(\n",
        "    title=\"InvertedPendulum\"\n",
        ")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<img src=\"file://_static/img/tutorials/reinforce_invpend_gym_v26_fig4.png\">\n",
        "\n",
        "Author: Siddarth Chandrasekar\n",
        "\n",
        "License: MIT License\n",
        "\n",
        "## References\n",
        "\n",
        "[1] Williams, Ronald J.. “Simple statistical gradient-following\n",
        "algorithms for connectionist reinforcement learning.” Machine Learning 8\n",
        "(2004): 229-256.\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
