[Probabilistic Robotics] Planning and Control: Partially Observable Markov Decision Processes

이 글은 Probabilistic Robotics  Chapter 16. Partially Observable Markov Decision Processes를 읽고 정리한 글입니다. 

To choose the right action, we need accurate state estimation. We need information-gathering tasks, such as robot exploration, for precise state estimation (i.e. reduce uncertainty). There are two types of uncertainty: uncertainty in action, and uncertainty in perception.

1. Uncertainty in action) Deterministic versus stochastic action effects: Uncertainty arising from the stochastic nature of the robot and its environments mandate that the robot senses at execution time, and reacts to unanticipated situations -even if the environment state is fully observable.
2. Uncertainty in perception) Fully observable versus partially observable systems: Classical robotics often assumes that sensors can measure the full state of the environment, which is an unrealistic assumption.

Partailly Observable Markov Decision Processes (aka POMDP) is more general case than MDP, where the state is not fully observable. Lack of observability requires the robot to estimate a posterior distribution over possible world state, which we refer to as the belief $b$.

The POMDP algorithm can anticipate uncertainty, actively gather information, and explore optimally for performance goal, besides with computational complexity. 

POMDP uses algorithmic prior to update its state, and this process is called as belief state estimation. We need to construct belief state $b(s_t)$ from history of partial observations $o_{0:t}$ and actions $a_{0:t}$. Formally, this can be expressed as $b(s_t) = p(s_t|o_{0:t}, a_{0:t})$, and it can be recursively expressed as,

belief state update

Here, observation model $p(o_t|s_t)$, transition model $p(s_t|s_{t-1}, a_{t-1})$ is given. To update the belief state, Kalman filter and Particle filter can be used.