Efficient Data Analytics Using Speculative Compilation TechniquesdissertationsSpiegelberg, Leonhard FranzcreatorSchwarzkopf, MalteAdvisorKraska, TimAdvisorKrishnamurthi, ShriramReaderBrown University. Department of Computer Sciencesponsor2023, None p.born digitalThesis (Ph. D.)--Brown University, 2023thesesOver the last decade, Python emerged as the language of choice for data processing and model building. Yet, the developer productivity of Python comes at the cost of slow execution speed and high demand for resources. Compiling Python to efficient, optimized machine code could help, but writing a general-purpose Python compiler is inherently difficult. Python’s dynamic typing, dynamic dispatch, and object format all impede standard compiler optimizations, and efforts to write compilers over two decades have failed to produce substantial speedups for practical data science workloads. This dissertation introduces two specialized compilers that leverage the high-level structure of a query to compile Python to efficient code for parallel data science workloads. The first exploits the domain-specific program structure of data science pipelines and generates code specialized to a sample of the input data, which allows making assumptions about types and common-case behavior. This results in a novel data-driven compilation approach that produces highly efficient machine code. Our prototype data analytics system, Tuplex, demonstrates that data-driven compilation beats state-of-the-art systems by a factor of 5× to 91× on realistic workloads. We then explore the idea of hyperspecialization, which generates custom code for disjoint data partitions to avoid sampling errors and improve overall efficiency for heterogeneous datasets. In a distributed setting using serverless functions to avoid sampling errors and improve overall efficiency. Building on Tuplex, we show that this next-generation system, Viton, benefits from hyperspecialization for realistic workloads and reduces cost and runtime by a factor of 2× to 3×. Viton uses serverless Lambda functions to achieve massive parallelism at low latencies together with a more advanced data-driven compilation approach backed by hyperspecialization.Big dataQuery Processingdata scienceData analyticsCompilers (Computer programs)EnglishRPB20230602