This thesis works on the evaluation assignment and consolidation problems that often appear in various contexts in academia. The studies of evaluations are dedicated to the development of techniques that enable decision makers to cope with the increasing complexity of our world. Applications of evaluation assignment can be seen in problems such as the conference paper-reviewer assignment, the research proposal review assignment, the journal impact survey assignment, etc. Here we propose a new variant of assignment problem that to our knowledge has not been addressed yet, which we name as "the non-dispersed assignment problem for evaluation purposes". In the problem, the assignees are research papers, manuscripts, proposals, or surveys, and the assigners are peer reviewers or experts. Th...[ Read more ]

This thesis works on the evaluation assignment and consolidation problems that often appear in various contexts in academia. The studies of evaluations are dedicated to the development of techniques that enable decision makers to cope with the increasing complexity of our world. Applications of evaluation assignment can be seen in problems such as the conference paper-reviewer assignment, the research proposal review assignment, the journal impact survey assignment, etc. Here we propose a new variant of assignment problem that to our knowledge has not been addressed yet, which we name as "the non-dispersed assignment problem for evaluation purposes". In the problem, the assignees are research papers, manuscripts, proposals, or surveys, and the assigners are peer reviewers or experts. This new variant not only considers the "expertise matching" feature that requires each assignee is sent to assigners who are "as expert as possible", but also addresses the "non-dispersion" feature that facilitate meaningful consolidation of partial rankings of assignees by multiple assigners into a final composite ranking for further decision making. The problem is shown to be strongly NP-Hard. To solve it practically, we formulate it as an Integer Programming (IP) model that has an objective to maximize the sum of the assignment score and evaluation score of candidate assignments. We find that as the problem size grows, the IP model quickly expands to an extent that the ILOG CPLEX Solver can hardly manage. Therefore, two meta-heuristic approaches, Tabu Search (TS) and Genetic algorithm (GA), are proposed. To measure the performance of different algorithms, extensive computational experiments are conducted, showing that while small problem instances can be solved ILOG CPLEX, the tabu search meta-heuristic is ideal for solving instances in the order of magnitude of hundreds of assigners and assignees under realistic problem settings.

We next shift our focus on the evaluation consolidation problem by presenting a case study for ranking academic journals for the Information System (IS) area. We demonstrate the flaws of the currently popular approach - the average-score method, and develop a more rigorous model for ranking consolidation, the minimum ranking reversal IP model. An expedient replacement to the average-score method that is based on the group-AHP methodology is also proposed. As the problem is proven to be strongly NP-Hard, we further conduct intensive experiments on the evaluation consolidation problem instances. Empirical hardness model is developed step-by-step by a collection of DOE tools, revealing an important instance feature, the density of consistency, which strongly affects the empirical hardness of problem instances. Tabu search is selected from a collections of meta-heuristics based on experimental results that examine the trade-offs between solution quality and runtime. An empirical hardness model is developed to predict of the run-time distribution of CPLEX and helps algorithm selection between tabu search and CPLEX in a portfolio approach. The portfolio, which is based on the empirical hardness model, is shown to be a stronger ensemble that outperforms all individual algorithms, and can be integrated into a decision support framework in various practical contexts.