Actor-Critic and PPO Explained: Advanced Deep Reinforcement Learning Algorithms
Learn actor-critic methods, the advantage function, A2C and proximal policy optimization (PPO), with the clipped objective explained and tested SB3 code.
The previous article ended on a problem. REINFORCE learns a policy directly, but it judges every action by the full return of one noisy episode. A lucky episode makes bad actions look good, an unlucky one does the opposite, and the learning signal bounces around. Worse, a single oversized update can wreck a policy that was working.
This article fixes both problems in two steps. First, actor-critic methods add a second network, the critic, that estimates how good states are, so the policy can be judged against a sensible expectation instead of raw luck. Second, Proximal Policy Optimization (PPO) limits how far each update can move the policy, which keeps training stable. PPO is one of the most widely used deep reinforcement learning algorithms. You will see the intuition for each idea before the equations, work through numerical examples, and train a PPO agent on CartPole with Stable-Baselines3.
It assumes you have read Policy Gradient Methods Explained: Learning Policies Directly, and we compare against Deep Q-Networks (DQN) Explained: From Q-Learning to Deep Reinforcement Learning near the end. For the foundations, see What Is Reinforcement Learning? A Complete Guide.
Why Policy Gradient Methods Have High Variance
In REINFORCE, the weight on each action’s gradient is the return Gt that followed it. That return adds up every reward until the episode ends, and each of those rewards depends on many random things: the environment’s randomness, the policy’s own random action choices, and every later decision. Two episodes that start identically can end with very different returns, so the weight on the same action can differ wildly from one episode to the next.
The cost is noisy updates, slow learning, and the need for many episodes to average the noise away. The previous article introduced a baseline, a number subtracted from the return so the question becomes “was this better than usual?”. The obvious follow-up is where the baseline should come from. Actor-critic methods answer by learning it, as a value function, alongside the policy.
What Is an Actor-Critic Architecture?
An actor-critic agent has two learned components that train together:
- The actor is the policy πθ(a | s). It decides what to do.
- The critic is a value function Vφ(s) that estimates how good a state is: the return the agent expects from that state onward. It evaluates how things are going.
Actor-critic is an architecture, a way of organizing an agent, not a single algorithm. Many algorithms use it, including A2C, A3C and PPO (below), and DDPG and SAC for continuous control. Keep the distinction in mind: actor-critic describes the structure, while PPO describes a specific way of updating the actor.
The role of the actor
The actor takes a state and outputs action probabilities, exactly like the policy network from the previous article. Its training signal comes from the critic’s feedback instead of raw episode returns: actions that turned out better than expected become more likely, and worse ones become less likely.
The role of the critic
The critic takes a state and outputs one number, Vφ(s). It is trained by regression to match a target built from observed rewards. Its estimates serve two purposes: a baseline that reduces variance, and a way to bootstrap, meaning the agent can estimate the future with the critic’s prediction instead of waiting for the episode to finish.
How the actor and critic work together
At each step the actor chooses an action and the environment returns a reward and a new state. The critic compares what happened (the reward plus its estimate of the new state’s value) with what it expected (its estimate of the old state’s value). That difference, the TD error, says whether the step went better or worse than expected. The actor uses it to adjust the policy, and the critic uses the same number to correct itself.
The Advantage Function
The signal the actor needs is the advantage: how much better an action is than the agent’s average behavior in that state. Two value functions define it.
- The state value Vπ(s) is the expected return from state s if the agent follows policy π from there. It scores the situation.
- The action value Qπ(s, a) is the expected return if the agent takes action a in s and follows π afterwards. It scores a specific choice.
The advantage is their difference: Aπ(s, a) = Qπ(s, a) − Vπ(s).
| Quantity | Question it answers | Values |
|---|---|---|
| Value V(s) | How good is it to be in this state, on average? | Any number, such as 10.2 |
| Action value Q(s, a) | How good is it to take this action here? | Any number |
| Advantage A(s, a) | How much better is this action than the average action here? | Positive means better than average, negative means worse. It averages to zero across the policy’s own action choices. |
The advantage is the “better than usual?” signal in its cleanest form. Because it is centered around zero, it removes the shared offset that made every action look good when all rewards were positive.
Temporal Difference Learning in Actor-Critic Methods
We cannot compute the advantage exactly, but the critic gives a cheap estimate. If its values were perfect, Qπ(s, a) would equal the expected value of r + γ V(s′). So one observed step gives the TD error:
If you have read the Q-learning update rule, this is the same TD idea applied to state values. Here rt+1 is the reward, γ the discount factor, Vφ(st+1) the critic’s estimate of the next state, and Vφ(st) its estimate for the current one. When the critic is accurate, the expected value of δt equals the advantage, so δt is a usable stand-in.
A worked example. Let γ = 0.99, the reward be 1, the critic’s value for the current state be V(s) = 10.0 and for the next state V(s′) = 9.5. Then δ = 1 + 0.99 × 9.5 − 10.0 = 1 + 9.405 − 10.0 = 0.405. The step went better than the critic expected, so the actor makes the action just taken a little more likely. The critic, with a learning rate of 0.1, moves its estimate toward the new target 10.405: from 10.0 to 10.0 + 0.1 × 0.405 = 10.0405. One number teaches both networks.
The bias-variance trade-off and GAE
Using the critic’s estimate instead of the real return trades one problem for another. The Monte Carlo return used by REINFORCE is unbiased (it is the true outcome) but has high variance. The one-step TD error has low variance (it depends on a single reward) but is biased, because it leans on the critic’s imperfect estimate. The best choice usually sits in between.
Generalized Advantage Estimation (GAE), introduced by Schulman et al. in 2016, blends TD errors from many steps with exponentially shrinking weights: Ât = δt + (γλ) δt+1 + (γλ)2 δt+2 + …. Setting λ = 0 gives the one-step TD error (low variance, more bias), and λ = 1 recovers the Monte Carlo advantage (no critic bias, high variance). For example, with γ = 0.99 and λ = 0.95 (so γλ = 0.9405) and three consecutive TD errors of 0.405, 0.2 and −0.1, the estimate is 0.405 + 0.9405 × 0.2 + 0.94052 × (−0.1) = 0.405 + 0.1881 − 0.0885 = 0.5046. PPO implementations, including the one below, typically use GAE.
Advantage Actor-Critic (A2C)
Advantage Actor-Critic (A2C) is the straightforward algorithm built from these pieces: collect a short batch of experience, estimate advantages with the critic, and update both networks. The name comes from A3C, the Asynchronous Advantage Actor-Critic of Mnih et al. (2016), where many workers ran in parallel and updated shared weights independently. A2C is the synchronous version: it waits for all workers to finish a batch and makes one combined update. Its loss has three parts:
- Actor loss: −log πθ(at | st) · Ât, the policy gradient from the previous article with the advantage in place of the raw return.
- Critic loss: the squared error (R̂t − Vφ(st))2 between the critic’s prediction and a return target.
- Entropy bonus: a small reward for keeping the action distribution spread out, which discourages premature determinism and encourages exploration.
A2C works, but it still has the second problem from the introduction: each batch is used for one gradient step, and a badly sized step can damage the policy.
Why Was PPO Developed?
With policy gradients, step size is delicate. Too small and learning crawls. Too large and one update can push the policy somewhere bad, after which the data it collects is poor and recovery is slow. The root issue is that a change in parameters does not map predictably onto a change in behavior: a small parameter change can sometimes alter action probabilities enormously.
Trust Region Policy Optimization (TRPO, Schulman et al., 2015) addressed this by limiting how much the policy’s behavior may change per update, using a constraint on the KL divergence (a measure of how different two probability distributions are). It works well but is complicated, because it needs second-order optimization. Proximal Policy Optimization, introduced by Schulman et al. in 2017, keeps the “do not change the policy too much” idea but achieves it through a simple change to the loss that ordinary first-order optimizers can handle. That simplicity and reliability are a large part of why PPO became a default choice.
PPO’s Clipping Mechanism: The Intuition
Here is the idea in words, before any formula. Suppose you collected a batch with the current policy, which we call the old policy, and you want to improve the policy using that batch. Ideally you would take many gradient steps on it, because data is expensive. But the further the new policy drifts from the old one, the less the batch tells you about it, since the old policy generated that batch. Past some point, more steps are not trustworthy.
PPO builds a guardrail. For each action in the batch it asks: how much more or less likely is this action now than under the old policy? If the policy has already moved that action’s probability by about 20% in the helpful direction, PPO stops rewarding further movement on that sample. Think of a leash: the policy is free to improve, but only within a short distance of where the data came from.
The probability ratio
The leash is measured by the probability ratio. For a stored action at in state st:
rt(θ) = πθ(at | st) / πθ_old(at | st)
Read it as “new probability divided by old probability”. A ratio of 1 means the action is exactly as likely as before, 1.5 means 50% more likely, and 0.6 means 40% less likely. At the start of each update the new policy equals the old one, so every ratio is exactly 1.
The PPO Clipped Objective Function
Now the formula. PPO multiplies the ratio by the advantage, clips the ratio to the interval [1 − ε, 1 + ε], and takes the smaller of the clipped and unclipped values:
| Symbol | Meaning |
|---|---|
| rt(θ) | The probability ratio: how much more or less likely the action is under the policy being trained than under the old policy that collected the data. |
| Ât | The estimated advantage of the action, typically from GAE. The hat marks it as an estimate. |
| ε | The clip range, such as 0.2. The allowed ratio interval is [1 − ε, 1 + ε] = [0.8, 1.2]. |
| clip(r, 1 − ε, 1 + ε) | Forces r into the interval: values below 0.8 become 0.8 and values above 1.2 become 1.2. |
| min( … ) | Takes the smaller of the unclipped and clipped terms, making the objective a pessimistic estimate. |
| Vφ(st) − R̂t | The critic’s error against its return target, squared to give the value loss. |
| H[πθ] | The policy’s entropy at that state, which rewards spread-out action probabilities. |
| c1, c2 | Coefficients weighting the value loss and the entropy bonus. |
Why the clipping and the min matter
For a good action (positive advantage), raising its probability raises the objective, so the optimizer pushes the ratio up. Once the ratio passes 1 + ε, the clipped term takes over and the objective goes flat: no gradient, nothing more to gain. For a bad action (negative advantage), lowering its probability helps, until the ratio drops below 1 − ε, where the objective goes flat for the same reason.
The min adds an asymmetry that is easy to miss. If a bad action’s probability went up a lot (ratio well above 1.2), the unclipped term is the smaller one, so it is kept and the penalty remains. The clip only removes the incentive to keep moving in a helpful direction, never the penalty for moving in a harmful one.
A numerical example
Take ε = 0.2, an advantage of +2 for a good action and −2 for a bad one.
| Case | Old → new probability | Ratio | Advantage | Unclipped r · A | Clipped objective | What happens |
|---|---|---|---|---|---|---|
| A | 0.40 → 0.55 | 1.375 | +2 | 2.750 | 2.400 | Ratio above 1.2 is clipped to 1.2: no gradient, no extra reward. |
| B | 0.40 → 0.44 | 1.100 | +2 | 2.200 | 2.200 | Inside the range: the gradient is active. |
| C | 0.50 → 0.30 | 0.600 | −2 | −1.200 | −1.600 | Ratio below 0.8 is clipped to 0.8: no extra push down. |
| D | 0.40 → 0.60 | 1.500 | −2 | −3.000 | −3.000 | A bad action became much more likely: the penalty is kept. |
You can verify the table with a few lines of Python. This function is the clipped objective for one sample, and the output below is what it printed when I ran it.
def ppo_clip_objective(ratio, advantage, epsilon=0.2):
clipped_ratio = max(1 - epsilon, min(ratio, 1 + epsilon))
return min(ratio * advantage, clipped_ratio * advantage)
cases = [
("A: good action, probability up a lot", 0.40, 0.55, +2.0),
("B: good action, probability up a little", 0.40, 0.44, +2.0),
("C: bad action, probability down a lot", 0.50, 0.30, -2.0),
("D: bad action, probability up a lot", 0.40, 0.60, -2.0),
]
for name, p_old, p_new, adv in cases:
r = p_new / p_old
print(f"{name:42s} ratio = {r:5.3f} | unclipped = {r * adv:6.3f} | "
f"clipped objective = {ppo_clip_objective(r, adv):6.3f}")
A: good action, probability up a lot ratio = 1.375 | unclipped = 2.750 | clipped objective = 2.400
B: good action, probability up a little ratio = 1.100 | unclipped = 2.200 | clipped objective = 2.200
C: bad action, probability down a lot ratio = 0.600 | unclipped = -1.200 | clipped objective = -1.600
D: bad action, probability up a lot ratio = 1.500 | unclipped = -3.000 | clipped objective = -3.000
One honest caveat: clipping is a heuristic, not a hard guarantee. It removes the incentive to move the policy far, but nothing physically stops it from drifting somewhat past the range, especially over many epochs. Practical implementations therefore also monitor quantities such as the approximate KL divergence.
How PPO Training Works
PPO alternates between collecting data and improving the policy with that data, and the second part is where it differs most from REINFORCE. Instead of one gradient step per batch, PPO makes many passes over the same batch, which is safe because the clipping keeps the policy near where the data came from.
- Collect. Run the current policy in the environment for a fixed number of steps. Record states, actions, rewards, and the probability of each action under this policy, which becomes πθ_old.
- Estimate advantages. Use the critic and GAE to compute an advantage and a return target for every step.
- Optimize for several epochs. Shuffle the batch into minibatches and loop over it K times. For each minibatch, compute the ratio rt, the clipped policy objective, the critic’s value loss and the entropy bonus, and take a gradient step.
- Replace and repeat. The updated policy becomes the new old policy, the batch is discarded, and the loop restarts with fresh data.
PPO pseudocode
initialize actor pi(a | s; theta) and critic V(s; phi)
repeat for each iteration:
run the current policy for T steps, store (s, a, r, log_prob_old, done)
compute advantages A_hat and return targets R_hat with the critic and GAE
for epoch in 1..K:
for each minibatch of the stored data:
ratio = exp(log_prob_new(a | s) - log_prob_old)
policy_obj = mean( min(ratio * A_hat, clip(ratio, 1 - eps, 1 + eps) * A_hat) )
value_loss = mean( (V(s; phi) - R_hat)^2 )
entropy = mean( entropy of pi(. | s; theta) )
loss = -policy_obj + c1 * value_loss - c2 * entropy
take a gradient step on theta and phi using loss
Implementing PPO with Stable-Baselines3
Writing PPO from scratch is educational but fiddly, and small implementation details matter. For the practical part we use Stable-Baselines3, a widely used library of reinforcement learning algorithms built on PyTorch. We train on CartPole-v1 again so the results sit alongside the earlier articles. The agent earns +1 per step, and the maximum return is 500.
Installation. Run pip install stable-baselines3 gymnasium numpy (this also installs PyTorch if it is missing). I ran the script below with Python 3.12.3, Stable-Baselines3 2.9.2, PyTorch 2.15.0, Gymnasium 1.4.0 and NumPy 2.5.4 on one CPU core. A run takes roughly a minute and a half. Copy the stages in order into one file named ppo_cartpole.py.
Stage 1: Imports and settings
import gymnasium as gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.monitor import Monitor
SEED = 0
ENV_NAME = "CartPole-v1"
N_STEPS = 2048 # steps collected per update
UPDATES_PER_CHUNK = 10 # report progress every 10 updates
CHUNKS = 5 # 5 * 10 * 2048 = 102,400 timesteps in total
PPO is the algorithm class and evaluate_policy is a helper for testing a trained agent. Monitor records episode statistics. The constants define a run of 102,400 timesteps split into five chunks so we can print progress along the way.
Stage 2: Build the model
def build_model(seed=SEED):
env = gym.make(ENV_NAME)
return PPO(
"MlpPolicy", # an actor network and a critic network over the state
env,
learning_rate=3e-4,
n_steps=N_STEPS,
batch_size=64, # minibatch size inside each update
n_epochs=10, # passes over each collected batch
gamma=0.99, # discount factor
gae_lambda=0.95, # advantage estimation smoothing
clip_range=0.2, # the epsilon in the clipped objective
seed=seed,
device="cpu",
verbose=0,
)
This one constructor holds every concept in the article. "MlpPolicy" builds the actor-critic networks. n_steps=2048 is how much data is collected before each update; with batch_size=64 and n_epochs=10, those 2,048 steps are split into 32 minibatches and passed over 10 times, giving 320 gradient steps per update. gamma is γ, gae_lambda is the λ in GAE, and clip_range=0.2 is ε. These values are the Stable-Baselines3 defaults for PPO, written out so you can see them. I did not tune them for CartPole.
The library creates two separate networks inside the policy. Printing model.policy shows them, and the relevant part looked like this (the comments are mine):
(mlp_extractor): MlpExtractor(
(policy_net): Sequential( # the actor body
(0): Linear(in_features=4, out_features=64, bias=True)
(1): Tanh()
(2): Linear(in_features=64, out_features=64, bias=True)
(3): Tanh()
)
(value_net): Sequential( # the critic body
(0): Linear(in_features=4, out_features=64, bias=True)
(1): Tanh()
(2): Linear(in_features=64, out_features=64, bias=True)
(3): Tanh()
)
)
(action_net): Linear(in_features=64, out_features=2, bias=True) # actor output: 2 action logits
(value_net): Linear(in_features=64, out_features=1, bias=True) # critic output: one state value
This is Figure 1 in code: an actor that maps the 4 state numbers to 2 action scores, and a separate critic that maps the same state to one value. The same defaults set the value-loss coefficient c1 to 0.5 and the entropy coefficient c2 to 0.0, so no entropy bonus is applied by default.
Stage 3: Train
def train(seed=SEED, log=True):
model = build_model(seed)
for chunk in range(1, CHUNKS + 1):
model.learn(total_timesteps=UPDATES_PER_CHUNK * N_STEPS, reset_num_timesteps=False)
if log:
recent = np.mean([ep["r"] for ep in model.ep_info_buffer])
print(f"timesteps {model.num_timesteps:7d} | avg return of recent episodes {recent:6.1f}")
return model
model.learn runs the whole loop from Figure 5: collect, compute advantages, optimize for several epochs, repeat. We call it five times for 20,480 steps each, with reset_num_timesteps=False so training continues rather than restarting. After each chunk we print the average return of the most recent episodes, taken from the model’s internal episode buffer.
Stage 4: Evaluate the trained agent
def evaluate(model, episodes=20, seed=1234):
eval_env = Monitor(gym.make(ENV_NAME))
eval_env.reset(seed=seed)
rewards, _ = evaluate_policy(
model, eval_env, n_eval_episodes=episodes,
deterministic=True, return_episode_rewards=True,
)
return rewards
if __name__ == "__main__":
model = train()
scores = evaluate(model)
print(f"\nDeterministic evaluation over {len(scores)} episodes: "
f"mean = {np.mean(scores):.1f}, min = {np.min(scores):.0f}, max = {np.max(scores):.0f}")
Evaluation uses a fresh environment with its own starting conditions and no learning. deterministic=True makes the agent always take its most probable action, which turns the stochastic policy into a deterministic one, as discussed in the previous article. Running the script with seed 0 printed:
timesteps 20480 | avg return of recent episodes 145.6
timesteps 40960 | avg return of recent episodes 325.6
timesteps 61440 | avg return of recent episodes 458.9
timesteps 81920 | avg return of recent episodes 500.0
timesteps 102400 | avg return of recent episodes 500.0
Deterministic evaluation over 20 episodes: mean = 500.0, min = 500, max = 500
Reading the Results Honestly
- It learned quickly. The average return rose from about 146 to the maximum of 500, and the deterministic agent balanced the pole for all 500 steps in all 20 evaluation episodes.
- Seeds. I ran seeds 0 to 4 with this exact script. Every seed scored a mean of 500.0 over 20 evaluation episodes, with no episode below 500.
- No tuning. I used the library defaults and did not adjust them after seeing results, so there is no hidden selection of settings here, unlike the hand-tuned runs in the DQN article.
- Limits of this evidence. CartPole is a very easy task and five seeds is a small sample. These numbers show that PPO works smoothly on a simple problem. They do not show how it performs on hard ones, and they are not a benchmark. Your numbers may differ with other library versions or hardware.
It is tempting to read this as “PPO beats DQN”, but the series’ runs are not a fair head-to-head: the three algorithms used different training budgets, update schedules and tuning effort. What the results do support is the reputation PPO has earned for being forgiving about hyperparameters.
Actor-Critic vs REINFORCE
| Aspect | REINFORCE | Actor-critic (A2C, PPO) |
|---|---|---|
| Learned components | A policy only | A policy (actor) and a value function (critic) |
| Learning signal | The full return of the episode | An advantage estimated with the critic |
| When it can update | Only after an episode ends | After short batches, even mid-episode |
| Variance | High | Lower |
| Bias | Unbiased gradient estimate | Some bias from an imperfect critic |
| Extra cost | None | A second network to train and tune |
PPO vs DQN
| Aspect | DQN | PPO |
|---|---|---|
| Family | Value-based | Policy optimization with an actor-critic architecture |
| What the network outputs | A Q-value per action | Action probabilities (actor) and a state value (critic) |
| Action spaces | Discrete only | Discrete or continuous |
| Data use | Off-policy: a replay buffer reuses old experience | On-policy: data from the current policy, reused for a few epochs and then discarded |
| Stability mechanism | Target network and replay | Clipped objective that limits each policy change |
| Sample efficiency | Often better, since experience is stored and replayed | Often worse, since data is used only briefly |
| Exploration | Epsilon-greedy | Sampling from the stochastic policy |
PPO and DQN solve different problems well. If your actions are continuous, you need a stochastic policy, or you value forgiving hyperparameters, PPO is the usual first choice. If actions are few and discrete and environment samples are expensive, an off-policy value-based method such as DQN can use data more efficiently.
Advantages of PPO and Actor-Critic Methods
- Stable updates. Clipping limits how far one batch can move the policy, so training is less likely to collapse.
- Simple to implement. It needs only first-order optimization, unlike the trust-region method it simplified.
- Handles continuous and discrete actions. The same algorithm covers both through the policy’s output distribution.
- Reuses each batch. Several epochs per batch squeeze more learning out of costly experience than plain policy gradients.
- Lower variance than REINFORCE. The critic and GAE give a steadier learning signal.
- Widely supported. Mature libraries make it easy to try.
Limitations of PPO
- Still on-policy. It discards data after each update, so it can need many more environment steps than off-policy methods.
- Clipping is a heuristic. It does not guarantee a true trust region, as noted above.
- Implementation details matter. Research such as Engstrom et al. (2020) found that code-level choices like advantage normalization and gradient clipping can account for a large share of performance differences between implementations, so a from-scratch version may behave differently from a library one.
- The critic can mislead. If the value estimates are poor, the advantages are biased and learning suffers.
- Hyperparameters still matter. PPO is forgiving but not immune to bad settings, especially on hard tasks.
- Heavier than REINFORCE. There are two networks and several epochs per update.
Real-World Applications of PPO
- Games. OpenAI reported training its Dota 2 agents, OpenAI Five, with PPO at large scale.
- Robotics. PPO has been used in research on robot control, including OpenAI’s work on dexterous in-hand manipulation.
- Language models. PPO has been widely used for reinforcement learning from human feedback, where a model is tuned toward responses people prefer.
- General research baseline. Its reliability makes it a common default for new reinforcement learning experiments.
For a broader tour, see Applications of Reinforcement Learning: 10 Real-World Examples.
Frequently Asked Questions
What is the difference between actor-critic and PPO?
Actor-critic is an architecture with a policy (actor) and a value function (critic). PPO is a specific algorithm for updating the policy, and in practice it is usually run with an actor-critic architecture.
What does the critic do in actor-critic methods?
It estimates how good each state is. The actor uses the critic’s estimates to judge whether an action was better or worse than expected, which reduces the noise in the learning signal.
What is the advantage function?
The advantage A(s, a) = Q(s, a) − V(s) measures how much better an action is than the average action in that state. Positive means better than average.
What is the PPO clipped objective?
It multiplies the probability ratio by the advantage, clips the ratio to a range such as 0.8 to 1.2, and takes the smaller value. This removes the incentive to change the policy too far from the one that collected the data.
Is PPO better than DQN?
Neither is universally better. PPO handles continuous actions and is forgiving to tune, while DQN is limited to discrete actions but can reuse past experience efficiently.
Is PPO on-policy or off-policy?
On-policy. It learns from data collected by the current policy, reuses each batch for a few epochs, and then discards it.
Conclusion and Next Steps
Actor-critic methods fix REINFORCE’s noise by adding a critic that provides a baseline and a bootstrapped advantage. PPO fixes unstable updates by clipping the probability ratio so the policy cannot stray far from the data that trained it. Keep the distinction clear: actor-critic is the architecture, PPO is the policy-optimization algorithm, and the two are usually used together. You have seen the ideas with worked numbers and a PPO agent solving CartPole in a few lines of Stable-Baselines3.
To practice, try changing clip_range to 0.1 or 0.3, set n_epochs to 1, or train on another Gymnasium environment, and compare several seeds each time. To review the series, start with What Is Reinforcement Learning? A Complete Guide, then Q-learning, SARSA, DQN and policy gradient methods.
For further reading, the original sources are the PPO paper by Schulman et al. (2017), the GAE paper (2016) and the TRPO paper (2015). Natural next topics are Double DQN and prioritized replay on the value-based side, and off-policy actor-critic algorithms such as SAC and TD3 for continuous control.
