Decision-making processes often involve selecting the simulated alternative with the largest (smallest) mean performance from a finite set of alternatives. This thesis studies two topics on designing frequentist procedures that help select the best alternative.

In the literature, many frequentist selection-of-the-best procedures have been proposed and they are typically designed under either the subset-selection (SS) formulation or the indifference-zone (IZ) formulation. However, both formulations may encounter problems when the goal is to select the unique best alternative for any configuration of means. In particular, SS procedures may return a subset that contains more than one alternative, and the outcome of IZ procedures hinges upon the relationship between the chosen IZ pa...[ Read more ]

Decision-making processes often involve selecting the simulated alternative with the largest (smallest) mean performance from a finite set of alternatives. This thesis studies two topics on designing frequentist procedures that help select the best alternative.

In the literature, many frequentist selection-of-the-best procedures have been proposed and they are typically designed under either the subset-selection (SS) formulation or the indifference-zone (IZ) formulation. However, both formulations may encounter problems when the goal is to select the unique best alternative for any configuration of means. In particular, SS procedures may return a subset that contains more than one alternative, and the outcome of IZ procedures hinges upon the relationship between the chosen IZ parameter and the true mean differences that is unknown to decision makers a priori. In light of this, our first topic is devoted to overcoming the drawbacks of both formulations. In this topic, we propose an indifference-zone-free formulation that guarantees to select the unique best alternative with a user-specified probability of correct selection (PCS), regardless of the configuration of means, and we design a class of sequential procedures under this formulation. These procedures are only parameterized by the PCS value, and their continuation boundaries are determined based on the Law of the Iterated Logarithm. Furthermore, we show that the users can add a stopping criterion to these procedures to convert them into IZ procedures, and we argue that these procedures have several advantages over existing IZ procedures. Lastly, we conduct an extensive numerical study to show the performance of our procedures and compare their performance with existing procedures.

Moreover, the outcome of selection-of-the-best procedures may critically depend on the underlying simulation model that drives the simulation and selection processes. In practical situations, ambiguity may exist in the specification of simulation models and ignoring the ambiguity may lead to an undesirable alternative. In light of this, our second topic is devoted to addressing the issue of selecting the best alternative in the presence of ambiguity. In this topic, we propose a robust selection-of-the-best (RSB) formulation which compares alternatives based on their worst-case performances over a group of possible distributions contained in the ambiguity set and selects the alternative with the “best” worst-case performance. In the robust framework, we design two types of RSB procedures that are either two-stage or fully sequential, and both of them contain two layers. Particularly, they identify the worst-case distribution (or estimate the worst-case performance) in the inner layer and select the best alternative in the second. These procedures guarantee to select an alternative whose worst-case mean performance is within δ of the best with at least a prescribed PCS, where the IZ parameter δ denotes the smallest difference worth detecting among alternatives. Last, we investigate their small-sample performances numerically and demonstrate the benefits of using the RSB framework via a numerical example of queueing service system.