The confluence of ubiquitous, high-performance networking and increased availability of online information has led to the emergence of a new class of large-scale stream processing and dissemination applications. These applications often exhibit diverse logic and performance requirements, yet they all require common facilities, which include construction of an overlay network, routing and processing logic, and membership management. In contrast to existing approaches that provide custom, point solutions to point applications, we introduce a general-purpose infrastructure that provides these core functionalities and can be easily extended for a broad spectrum of target applications. Extensibility is the central design consideration for our infrastructure, which can be customized to support diverse processing logic, stream types, and performance targets through a set of methods that encapsulate application-specific behavior and a cost model for defining the desired QoS and resource-utilization metrics and constraints. Given these specifications, the system automatically creates and optimizes a data stream acquisition, processing and dissemination overlay network. Its optimization is driven by metric-independent operations, which can refine the structure of the overlay network as well as efficiently distribute processing across the network. In this dissertation, we first describe the basic concepts and models used by our infrastructure and present its generic optimization framework. We study different types of optimization techniques we have developed for supporting stream dissemination and stream processing applications. Finally, we evaluate the performance of our framework through results we have obtained by a prototype deployment on the PlanetLab testbed as well as LAN emulation and simulation experiments.
Thesis (Ph.D.) -- Brown University (2008)
"An Extensible Overlay Infrastructure for Wide-Area Stream Processing and Dissemination"
Computer Science Theses and Dissertations.
Brown Digital Repository. Brown University Library.