Summary

Theorem 3.2 and theorem 4.1 lead to

$$V_N^*(s) = \max_{ \pi \in {\Pi}^{HR}} \{V_N^{\pi}(s) = \max_{\pi \in {\Pi}^{MD}} \{V_N^{\pi}(s)\}$$

Namely, the optimal policy can always be chosen out of the group of Markovian deterministic policies.