The main contribution of this thesis is two-fold....[ Read more ]

The main contribution of this thesis is two-fold.

First, a sample-path-based construction of the sensitivity formulas for the continuous-time Markov systems is given in this thesis. The major complexity involved in the optimization of the continuous-time Markov systems is the mixed effect due to change of the transition rates, in addition to the change of the transition probabilities. By using the sample-path-based construction, we decompose this effect into two parts, each due to a type of perturbations which can be measured by the performance potentials.

Second, by using the GSMP model, we provide a general framework of an event-driving optimization approach for the continuous-time Markov systems. Each event is defined as a subset of active events evolved in GSMP. It is associated with a lifetime, and may trigger a transition at the end of that. The control on the transition rates of states is decomposed into the control on the lifetime of different events, and the control on the transition probabilities of states is decomposed into the control on the transitions triggered by different events. Independent actions are applied to different events, instead of states. Potentials, aggregated with events, are used to construct two types of performance sensitivity formulas which lead to gradient-based optimizations and, with some conditions, event-based policy iterations. This approach provides a clear physical meaning and structure insights for the optimization of the continuous-time systems.

The main results in this thesis can be applied to many problems that do not fit the standard MDP formulation, and therefore opens up new research directions for many practical problems.