Large-scale network of information sources and digital libraries amplifies the information overload problem constantly. A popular resource discovery tool on the World Wide Web nowadays is centralized index server, which attempts to be a 'know-it-all' of online resources. It suffers from many shortcomings resulted from the centralization of metadata extraction, index building and query processing. A better approach is to use disparate index servers for individual resource repository. A user query is only routed to a site if the repository is likely to contain relevant resources. This task is often called collection selection, and the server to perform this is usually called meta-indexer. Nevertheless, most research projects assume a centralized meta-indexer which will eventually be degra...[ Read more ]

Large-scale network of information sources and digital libraries amplifies the information overload problem constantly. A popular resource discovery tool on the World Wide Web nowadays is centralized index server, which attempts to be a 'know-it-all' of online resources. It suffers from many shortcomings resulted from the centralization of metadata extraction, index building and query processing. A better approach is to use disparate index servers for individual resource repository. A user query is only routed to a site if the repository is likely to contain relevant resources. This task is often called collection selection, and the server to perform this is usually called meta-indexer. Nevertheless, most research projects assume a centralized meta-indexer which will eventually be degraded by centralization.

In this work, we look into issues of distributing the collection selection subsystem itself. An abstract model for distributed resource discovery systems is introduced as a common model to describe different architectures. We then study some algorithms in collection ranking used in a centralized meta-indexer, and show how they can be scaled up for distributed architectures. Two basic architectures, the hierarchical and flat architectures, are first considered. The features of these architectures are examined and mixed to form two new hybrid architectures. Finally, we consider the interoperation of the various architectures, each of which is suitable for different scenarios.