(updated 22 January 2002)
| No. | Introduction; Title/pages | Errata/links | Updated |
| 1. | Introduction; pages 1--17. | ||
| 2. | Finite State and Action MDPs; pages 21--88. | Page 73 Reference [66]: the authors are E.B. Dynkin and A.A. Yushkevich |
12/09/01 |
| 3. | Bias Optimality; pages 89--112. | ||
| 4. | Singular Perturbations of Markov Chains and Decision Processes; pages 113--150. | 11/27/01 | |
| 5. | Average Reward Optimization Theory for Denumerable State Spaces; pages 153--172. | Page 159 formula 5.17--omit the "*" |
12/04/01 |
| 6. | Total Reward Criteria; pages 173--208. | ||
| 7. | Mixed Criteria; pages 209--230. | ||
| 8. | Blackwell Optimality; pages 231--268. | ||
| 9. | The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations; pages 269--304. | ||
| 10. | Stability, Performance Evaluation, and Optimization; pages 305--346. | ||
| 11. | Convex Analytic Methods in Markov Decision Processes; pages 347--376. | ||
| 12. | The Linear Programming Approach; pages 377--408. | ||
| 13. | Invariant Gambling Problems and Markov Decision Processes; pages 409--430. | ||
| 14. | Neuro-Dynamic Programming: Overview and Recent Trends; pages 431--460. | ||
| 15. | Markov Decision Processes in Finance and Dynamic Options; pages 461--488. | 1/22/02 | |
| 16. | Applications of Markov Decision Processes in Communication Networks; pages 489--536. | ||
| 17. | Water Reservoir Applications of Markov Decision Processes; pages 537--558. |