JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD

Course Objectives

Knowledge on fundamentals of reinforcement learning and the methods used to

create agents that can solve a variety of complex tasks.

Course Outcomes

• 1. Understand basics of RL
• 2. Understand RL Framework and Markov Decision Process
• 3. Analyzing ning through the use of Dynamic Programming and Monte Carlo
• 4. Understand TD(0) algorithm, TD(λ) algorithm

UNIT - I

Basics of probability and linear algebra, Definition of a stochastic multi-armed bandit, Definition of regret, Achieving sublinear regret, UCB algorithm, KL-UCB, Thompson Sampling.

UNIT - II

Markov Decision Problem, policy, and value function, Reward models (infinite discounted, total, finite horizon, and average), Episodic & continuing tasks, Bellman's optimality operator, and Value iteration & policy iteration

UNIT - III

The Reinforcement Learning problem, prediction and control problems, Model-based algorithm, Monte Carlo methods for prediction, and Online implementation of Monte Carlo policy evaluation

UNIT - IV

Bootstrapping; TD(0) algorithm; Convergence of Monte Carlo and batch TD(0) algorithms; Model-free control: Q-learning, Sarsa, Expected Sarsa.

UNIT - V

n-step returns; TD(λ) algorithm; Need for generalization in practice; Linear function approximation and geometric view; Linear TD(λ). Tile coding; Control with function approximation; Policy search; Policy gradient methods; Experience replay; Fitted Q Iteration; Case studies.

TEXT BOOKS:

• 1. “Reinforcement learning: An introduction,” First Edition, Sutton, Richard S., and Andrew G. Barto, MIT press 2020
• 2. “Statistical reinforcement learning: modern machine learning approaches,” First Edition, Sugiyama, Masashi. CRC Press 2015

REFERENCES:

• 1. “Bandit algorithms,” First Edition, Lattimore, T. and C. Szepesvári. Cambridge University Press. 2020
• 2. “Reinforcement Learning Algorithms: Analysis and Applications,” Boris Belousov, Hany Abdulsamad, Pascal Klink, Simone Parisi, and Jan Peters First Edition, Springer 2021
• 3. Alexander Zai and Brandon Brown “Deep Reinforcement Learning in Action,” First Edition, Manning Publications 2020