Examples of Markov Decision Processes (MDPs) in AI

📚 Quick Study Guide

• 🤖 Markov Property: The future state depends only on the current state, not the past.
• 📐 MDP Components: Defined by a tuple (S, A, P, R, $\gamma$), where:
  • 🗺️ S = Set of states
  • 🕹️ A = Set of actions
  • 🎲 P = Transition probability $P(s'|s, a)$ (probability of transitioning to state s' from state s by taking action a)
  • 💰 R = Reward function $R(s, a)$ (reward received after taking action a in state s)
  • 📉 $\gamma$ = Discount factor (between 0 and 1)
• 🧮 Bellman Equation: A fundamental equation for finding the optimal value function: $V(s) = \max_{a} [R(s, a) + \gamma \sum_{s'} P(s'|s, a)V(s')]$
• 🎯 Goal: To find an optimal policy $\pi^*(s)$ that maximizes the expected cumulative reward.

Practice Quiz

1. Which of the following best describes the Markov property?
   1. The future state depends on the entire history of past states and actions.
   2. The future state depends only on the current state.
   3. The current state depends on the future state.
   4. The past state depends on the future state.
2. In an MDP, what does 'S' represent?
   1. Set of actions
   2. Set of rewards
   3. Set of states
   4. Set of policies
3. What does the discount factor ($\gamma$) in an MDP represent?
   1. The probability of transitioning to the next state.
   2. The importance of future rewards compared to immediate rewards.
   3. The expected cumulative reward.
   4. The learning rate.
4. Which equation is used to find the optimal value function in MDPs?
   1. Markov Equation
   2. Bellman Equation
   3. Shannon Equation
   4. Euler Equation
5. In the context of a self-driving car, which of the following could be considered an 'action' in an MDP?
   1. The car's current speed
   2. The traffic light color
   3. Steering left
   4. The car's GPS coordinates
6. Which of the following is NOT a core component of an MDP?
   1. States
   2. Actions
   3. Rewards
   4. Initial State Distribution
7. What is the primary goal when solving an MDP?
   1. To minimize the number of states.
   2. To find an optimal policy that maximizes expected cumulative reward.
   3. To maximize the immediate reward.
   4. To minimize the transition probabilities.