Efficient solvers and preconditioners for the implicit time integration of discontinuous Galerkin methods by Will E. Pazner B.Sc., University of Toronto; Toronto, ON, 2011 Sc.M., Brown University; Providence, RI, 2015 A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Division of Applied Mathematics at Brown University PROVIDENCE, RHODE ISLAND May 2018 c Copyright 2018 by Will E. Pazner This dissertation by Will E. Pazner is accepted in its present form by the Division of Applied Mathematics as satisfying the dissertation requirement for the degree of Doctor of Philosophy. Date Chi-Wang Shu, Ph.D., Advisor Recommended to the Graduate Council Date Per-Olof Persson, Ph.D., Reader Date Johnny Guzm´an, Ph.D., Reader Approved by the Graduate Council Date Andrew G. Campbell, Dean of the Graduate School iii Vitae Will Pazner received his B.Sc. in Mathematics from the University of Toronto in 2011. He joined Brown University’s Division of Applied Mathematics in 2014, from where he received his Sc.M. in 2015. Publications: W. Pazner and P.-O. Persson. “Approximate tensor-product preconditioners for very high order discontinuous Galerkin methods”. In: Journal of Compu- tational Physics, 354 (2018). doi: 10.1016/j.jcp.2017.10.030. W. Pazner and P.-O. Persson. “Interior penalty tensor-product preconditioners for high-order discontinuous Galerkin discretizations”. In: Proceedings of the 2018 AIAA Aerospace Sciences Meeting. American Institute of Aeronautics and Astronautics, 2018. doi: 10.2514/6.2018-1093. W. Pazner and P.-O. Persson. “On the convergence of iterative solvers for polygonal discontinuous Galerkin discretizations”. In: Communications in Applied Mathematics and Computational Science, 13.1 (2018). doi: 10.2140/ camcos.2018.13.27. W. Pazner and P.-O. Persson. “Stage-parallel fully implicit Runge-Kutta solvers for discontinuous Galerkin fluid simulations”. In: Journal of Com- putational Physics, 335 (2017). doi: 10.1016/j.jcp.2017.01.050. W. Pazner and P.-O. Persson. “High-order DNS and LES simulations using an implicit tensor-product discontinuous Galerkin method”. In: Proceedings of the 23rd AIAA Computational Fluid Dynamics Conference. American Institute of Aeronautics and Astronautics, 2017. First Place, Student Paper Competition. doi: 10.2514/6.2017-3948. W. Pazner, A. Nonaka, J. Bell, M. Day, and M. Minion. “A high-order spectral deferred correction strategy for low Mach number flow with com- plex chemistry”. In: Combustion Theory and Modeling, 20.3 (2016). doi: 10.1080/13647830.2016.1150519. iv ‫להילה‬ ‫‪v‬‬ Acknowledgements I must begin by acknowledging the invaluable contributions of my advisors, Pro- fessors Per-Olof Persson and Chi-Wang Shu. Professor Persson introduced me to the world of research, and guided me closely throughout my doctoral studies. His knowledge, advice, insights, and suggestions have proven, time and time again, to be essential to the work that went into this thesis. I could not have asked for a better advisor, mentor, or trusted friend. I gratefully acknowledge Professor Shu for affording me the freedom to pursue a very independent route to my Ph.D. I sincerely thank him for always making himself available to me, sharing his expertise, and for providing valuable support and input during my time at Brown. To my friends and colleagues from the Division of Applied Mathematics at Brown University, the Department of Mathematics at UC Berkeley, and the Mathematics Group at Lawrence Berkeley National Laboratory, I would like to express my grat- itude. I had the privilege of looking forward to interesting and insightful conversa- tions, both mathematical in nature and not, on a daily basis. My graduate studies have been a great experience, in no small part because of these friendships. Finally, I wish to acknowledge the unwavering support that I received from my entire family, and in particular, my parents, Micha and Nancy, and brother, Tom. Most of all, I give my deepest thanks to my beloved and inspirational wife, Hila, for her endless love, unfailing support, and steadfast encouragement. vi Abstract of “Efficient solvers and preconditioners for the implicit time integration of discontinuous Galerkin methods” by Will E. Pazner, Ph.D., Brown University, May 2018 In this work, we develop and analyze solvers and preconditioners designed for the implicit time integration of discontinuous Galerkin (DG) discretizations. The dis- continuous Galerkin method is a high-order accurate finite element method for the numerical solution of partial differential equations on unstructured meshes. The temporal integration of such discretizations by means of implicit methods has the im- portant advantage of avoiding restrictive stability conditions on the size of the time step. Because of the large number of degrees of freedom and potentially poorly- conditioned nature of the resulting algebraic systems of equations, sophisticated solvers and preconditioners are a requirement for good performance. This thesis focuses on the iterative solution of the resulting linear systems by means of Krylov subspace solvers. We develop and study preconditioners that allow of the efficient use of fully-implicit Runge-Kutta methods, which have previously been considered prohibitively expensive. These methods have the additional advantage that they allow for parallelism in the temporal dimension. Additionally, we develop an implicit tensor-product solver that makes use of a construction known as the Kronecker-product singular value decomposition to obtain asymptotically-improved computational complexities on quadrilateral and hexahedral meshes. Finally, we study the effect of polygonal mesh geometries on the convergence of iterative linear solvers. The applicability of these methods is demonstrated on a wide range of large-scale, two- and three-dimensional test cases. Contents Vitae iv Dedication v Acknowledgments vi Abstract vii Contents viii 1 Introduction 1 1.1 The discontinuous Galerkin method . . . . . . . . . . . . . . . . . . . 4 1.2 Treatment of second-order terms . . . . . . . . . . . . . . . . . . . . . 10 1.2.1 Choice of viscous fluxes . . . . . . . . . . . . . . . . . . . . . . 13 1.2.2 Generalized interior penalty formulation . . . . . . . . . . . . 15 1.3 Semi-discrete system of equations . . . . . . . . . . . . . . . . . . . . 16 1.3.1 Implicit time integration . . . . . . . . . . . . . . . . . . . . . 18 1.4 Governing equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4.1 Linear advection equation . . . . . . . . . . . . . . . . . . . . 20 1.4.2 Convection-diffusion equation . . . . . . . . . . . . . . . . . . 21 1.4.3 Compressible Euler equations . . . . . . . . . . . . . . . . . . 22 1.4.4 Compressible Navier-Stokes equations . . . . . . . . . . . . . . 24 2 Fully-implicit Runge-Kutta methods 27 2.1 Runge-Kutta methods . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.2 Efficient solution of implicit Runge-Kutta systems . . . . . . . . . . . 33 2.2.1 Block structure of the Jacobian . . . . . . . . . . . . . . . . . 33 2.2.2 Transformation of the system of equations . . . . . . . . . . . 34 2.2.3 Preconditioning . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2.4 Preconditioning the large system . . . . . . . . . . . . . . . . 38 2.2.5 Computational cost and memory requirements . . . . . . . . . 41 2.2.6 Stage-parallelism and partitioned ILU . . . . . . . . . . . . . . 43 2.3 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.3.1 Euler vortex . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 viii 2.3.2 High Reynolds number flow over 2D NACA airfoil . . . . . . . 52 2.3.3 Parallel large eddy simulation of 3D NACA airfoil . . . . . . . 60 2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3 Tensor-product preconditioners 65 3.1 The sum-factorization approach . . . . . . . . . . . . . . . . . . . . . 68 3.1.1 Tensor-product elements . . . . . . . . . . . . . . . . . . . . . 69 3.1.2 Kronecker-product structure . . . . . . . . . . . . . . . . . . . 71 3.1.3 Explicit time integration . . . . . . . . . . . . . . . . . . . . . 73 3.1.4 Implicit time integration . . . . . . . . . . . . . . . . . . . . . 76 3.2 Tensor-product preconditioners . . . . . . . . . . . . . . . . . . . . . 81 3.2.1 Kronecker-product singular value decomposition . . . . . . . . 83 3.2.2 Two spatial dimensions . . . . . . . . . . . . . . . . . . . . . . 86 3.2.3 Three spatial dimensions . . . . . . . . . . . . . . . . . . . . . 90 3.2.4 Algorithm overview . . . . . . . . . . . . . . . . . . . . . . . . 94 3.3 Extension to second-order operators . . . . . . . . . . . . . . . . . . . 95 3.3.1 Exact representations . . . . . . . . . . . . . . . . . . . . . . . 98 3.4 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 3.4.1 2D linear advection equation . . . . . . . . . . . . . . . . . . . 102 3.4.2 2D convection-diffusion equation . . . . . . . . . . . . . . . . 109 3.4.3 2D Euler vortex . . . . . . . . . . . . . . . . . . . . . . . . . . 114 3.4.4 2D Kelvin-Helmholtz instability . . . . . . . . . . . . . . . . . 119 3.4.5 2D NACA airfoil . . . . . . . . . . . . . . . . . . . . . . . . . 122 3.4.6 2D viscous flow over a circular cylinder . . . . . . . . . . . . . 124 3.4.7 3D periodic Euler problem . . . . . . . . . . . . . . . . . . . . 126 3.4.8 3D compressible Taylor-Green Vortex at Re = 1600 . . . . . . 129 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4 Iterative convergence of polygonal elements 138 4.1 Numerical methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 4.1.1 Temporal integration and linear solvers . . . . . . . . . . . . . 141 4.2 Jacobi analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 4.2.1 von Neumann analysis . . . . . . . . . . . . . . . . . . . . . . 144 4.2.2 1D analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.2.3 2D analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 4.3 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 4.3.1 Advection with variable velocity field . . . . . . . . . . . . . . 154 4.3.2 Compressible Euler equations . . . . . . . . . . . . . . . . . . 159 4.3.3 Inviscid flow problems . . . . . . . . . . . . . . . . . . . . . . 163 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 5 Case studies 171 5.1 Benchmark problem: transitional flow over the SD7003 airfoil . . . . 173 5.1.1 Performance comparison . . . . . . . . . . . . . . . . . . . . . 177 5.1.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 ix 5.2 High-order simulation of vertical-axis wind turbines . . . . . . . . . . 181 5.2.1 Arbitrary Lagrangian-Eulerian formulation . . . . . . . . . . . 181 5.2.2 Problem configuration and results . . . . . . . . . . . . . . . . 192 5.2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 6 Conclusion 197 A Butcher tableaux 200 A.1 Radau IIA methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 A.2 DIRK methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Bibliography 204 x List of Tables 2.1 L-stable implicit Runge-Kutta schemes . . . . . . . . . . . . . . . . . 31 2.2 Per-iteration computational costs for solving implicit Runge-Kutta systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.3 Computational cost of computing the IRK preconditioners . . . . . . 42 2.4 Memory requirements for Jacobian matrices and preconditioners . . . 43 2.5 L∞ error and runtime for the Euler vortex, DIRK and Radau IIA methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.6 Equivalent multiplications and wall-clock time for NACA LES test case 55 2.7 L∞ error for NACA LES test cases . . . . . . . . . . . . . . . . . . . 56 2.8 L∞ error for NACA LES test cases (order verification) . . . . . . . . 58 3.1 Kronecker-product form of DG operations . . . . . . . . . . . . . . . 74 3.2 GMRES iterations for Jacobi and Kronecker-product preconditioners, advection equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.3 GMRES iterations for Jacobi and Kronecker-product preconditioners, convection diffusion test case 1 . . . . . . . . . . . . . . . . . . . . . . 112 3.4 Largest allowable time step for RK4, convection-diffusion test case 2 . 113 3.5 GMRES iterations for Jacobi and Kronecker-product preconditioners, convection diffusion test case 2 . . . . . . . . . . . . . . . . . . . . . . 115 3.6 GMRES iterations for Jacobi and Kronecker-product preconditioners, Euler equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 3.7 GMRES iterations for Jacobi and Kronecker-product preconditioners, Kelvin-Helmholtz instability . . . . . . . . . . . . . . . . . . . . . . . 122 3.8 Runtime for NACA test-case, comparing block Jacobi and Kronecker- product preconditioners with explicit time integration . . . . . . . . . 124 3.9 GMRES iterations for Jacobi and Kronecker-product preconditioners, flow over a circular cylinder . . . . . . . . . . . . . . . . . . . . . . . 126 3.10 GMRES iterations for three-dimensional Euler test-case . . . . . . . . 129 3.11 Grid configurations for Taylor-Green problem . . . . . . . . . . . . . 131 3.12 Average number of GMRES iterations per Newton solve for Taylor- Green test case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 4.1 Ratios of the logarithm of eigenvalues for polygonal generating patterns152 4.2 Block Jacobi iterations for variable-coefficient advection on polygonal meshes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.3 Block Jacobi iterations on randomized polygonal meshes . . . . . . . 157 xi 4.4 GMRES iterations with block Jacobi preconditioner for advection on polygonal meshes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 4.5 GMRES iterations with ILU(0) preconditioner for advection on polyg- onal meshes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 4.6 Block Jacobi iterations for Euler equations on polygonal meshes . . . 161 4.7 GMRES iterations with block Jacobi preconditioner for Euler equa- tions on polygonal meshes . . . . . . . . . . . . . . . . . . . . . . . . 162 4.8 GMRES iterations with ILU(0) preconditioner for Euler equations on polygonal meshes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 4.9 GMRES iterations for inviscid flow over a circular cylinder . . . . . . 165 4.10 Block Jacobi iterations for inviscid flow over a circular cylinder . . . . 165 4.11 GMRES iterations for supersonic flow over a circular cylinder . . . . 168 5.1 Comparison of experimental and computational results for lift, drag, separation location, and reattachment location for SD7003 airfoil . . . 178 5.2 Comparison of static and moving-domain variants of laminar flow over a circular cylinder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 5.3 Comparison of static and moving-domain variants of laminar flow over NACA 0012 airfoil . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.4 GMRES iteration counts for vertical-axis wind turbine . . . . . . . . 196 xii List of Figures 1.1 Nodal interpolation points on the unit simplex . . . . . . . . . . . . . 6 1.2 Nodal interpolation points on the unit cube . . . . . . . . . . . . . . 7 1.3 Compact and non-compact stencils for second-order operators . . . . 14 1.4 Spectral radius of first derivative and second derivative DG operators 19 2.1 Schematic of stage-parallel IRK solution strategy . . . . . . . . . . . 46 2.2 Stage-parallel partitioning of the IRK Jacobian matrix . . . . . . . . 46 2.3 Compressible Euler vortex . . . . . . . . . . . . . . . . . . . . . . . . 47 2.4 Comparison of IRK preconditioners, number of average number of equivalent multiplications vs. ∆t . . . . . . . . . . . . . . . . . . . . . 49 2.5 L∞ error vs. time step and wall-clock time for Euler vortex test case . 52 2.6 Triangular mesh of NACA airfoil . . . . . . . . . . . . . . . . . . . . 52 2.7 Density plots for NACA Re = 40k test case . . . . . . . . . . . . . . . 54 2.8 Average number of equivalent multiplications and wall-clock time vs. ∆t for the 2D NACA LES test case . . . . . . . . . . . . . . . . . . . 55 2.9 L∞ errors vs. wall-clock time for the NACA LES test case . . . . . . 57 2.10 Number of equivalent multiplications vs. number of processes for NACA LES test case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 2.11 Three-dimensional NACA LES test case . . . . . . . . . . . . . . . . 61 2.12 Linear solve time vs. number of processes for 3D NACA LES test case 62 3.1 Meshes used for the advection equation . . . . . . . . . . . . . . . . . 104 3.2 Velocity fields used for the advection equation . . . . . . . . . . . . . 104 3.3 Meshes used for anisotropic test case . . . . . . . . . . . . . . . . . . 107 3.4 Wall-clock times for scalar advection on anisotropic meshes . . . . . . 108 3.5 Wall-clock time required to form and apply the block Jacobi and KSVD preconditioners, advection equation . . . . . . . . . . . . . . . 108 3.6 Convection-diffusion test problem . . . . . . . . . . . . . . . . . . . . 110 3.7 Computed outflow profiles for test problem 1 . . . . . . . . . . . . . . 110 3.8 Convection-diffusion test problem 2 . . . . . . . . . . . . . . . . . . . 113 3.9 Initial conditions (density) for Euler vortex on unstructured mesh. . . 116 3.10 Runtime performance comparison of Kronecker-product preconditioner with exact block Jacobi preconditioner for 2D Euler equations . . . . 119 3.11 Initial conditions for the Kelvin-Helmholtz instability. . . . . . . . . . 121 3.12 Solution (density) of Kelvin-Helmholtz instability . . . . . . . . . . . 121 3.13 NACA 0012 mesh, with zoom-in around the surface of the airfoil, and solution (density) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 xiii 3.14 Mesh and computed solution for viscous compressible flow over a cir- cular cylinder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 3.15 Initial conditions (density) for smooth 3D Euler test case. . . . . . . . 128 3.16 Wall-clock time for three-dimensional periodic test-case . . . . . . . . 129 3.17 Time evolution of kinetic energy, kinetic energy dissipation rate (KEDR), and enstrophy, for Taylor-Green test case . . . . . . . . . . . . . . . . 132 3.18 Compressible Taylor-Green vortex at Re = 1600, isosurfaces of vor- ticity norm, colored by helicity . . . . . . . . . . . . . . . . . . . . . . 136 4.1 Generating patterns for polygonal meshes . . . . . . . . . . . . . . . . 144 4.2 Ratios of the logarithm of eigenvalues for polygonal generating patterns153 4.3 Velocity field β(x, y) = (2y − 1, −2x + 1) . . . . . . . . . . . . . . . . 155 4.4 Randomized polygonal and triangular meshes . . . . . . . . . . . . . 157 4.5 Illustration of the natural ordering of mesh elements . . . . . . . . . . 159 4.6 Overview of the coarse polygonal mesh with 15,404 elements . . . . . 164 4.7 Overview of the coarse polygonal mesh with 31,162 elements . . . . . 167 5.1 Two-dimensional meshes used for SD7003 test case . . . . . . . . . . 173 5.2 Two-dimensional solution for flow over SD7003 airfoil at t = 10 tc . . 176 5.3 Snapshot of three-dimensional solution at t = 15 tc , showing isosur- faces of Q-criterion, colored by velocity magnitude . . . . . . . . . . . 176 5.4 Distribution of pressure coefficient on suction and pressures sides of SD7003 airfoil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 5.5 Distribution of skin friction coefficient on the suction side of SD7003 airfoil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 5.6 Log-log plots showing the performance and parallel scaling of IRK and DIRK solvers for the SD7003 test case . . . . . . . . . . . . . . . . . 180 5.7 Mesh used for laminar flow over a circular cylinder . . . . . . . . . . 188 5.8 Solution (Mach number) for laminar flow over a circular cylinder . . . 191 5.9 Coefficient of pressure and coefficient of skin friction for laminar flow over circular cylinder . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 5.10 Coarse and fine meshes used for laminar NACA test case . . . . . . . 192 5.11 Pressure contours of steady, laminar flow over NACA 0012 airfoil . . 193 5.12 Coefficient of pressure and coefficient of skin friction for laminar flow over NACA 0012 airfoil . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.13 Mesh of three-blade vertical-axis wind turbine . . . . . . . . . . . . . 194 5.14 Flow over a three-blade vertical-axis wind turbine . . . . . . . . . . . 195 xiv List of Algorithms 2.1 Block ILU(0) algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2 Simplified block ILU(0) algorithm . . . . . . . . . . . . . . . . . . . . 38 2.3 ILU(0) algorithm for IRK systems . . . . . . . . . . . . . . . . . . . . 39 3.1 Matrix-free computation of Jv in 2D and 3D . . . . . . . . . . . . . . 81 3.2 Lanczos singular value decomposition . . . . . . . . . . . . . . . . . . 85 3.3 Computation of the shuffled matrix-vector product . . . . . . . . . . 86 3.4 Computation of the shuffled, transposed matrix-vector product . . . . 86 3.5 Form 2D preconditioner A ≈ P = A1 ⊗ B2 + A2 ⊗ B2 . . . . . . . . . 94 3.6 Apply 2D preconditioner to solve P x = b . . . . . . . . . . . . . . . . 94 3.7 Form 3D preconditioner A ≈ P = A1 ⊗ B1 ⊗ C1 + A1 ⊗ B2 ⊗ C2 . . . 95 3.8 Apply 3D preconditioner to solve P x = b . . . . . . . . . . . . . . . . 95 A.1 Mathematica code to compute the coefficients of Radau IIA schemes . 201 xv Chapter One Introduction 2 The discontinuous Galerkin (DG) method was originally introduced in 1973 by Reed and Hill for solving the neutron transport equation [108]. In a series of papers in the 1990s, Cockburn, Shu, and others extended the DG method to general systems of hyperbolic conservation laws using a total variation diminishing Runge-Kutta time discretization [28, 29, 32, 33, 34]. Several different extensions to elliptic and parabolic problems were subsequently developed and applied to a wide range of equations [2, 9, 21, 31]. In 2002, Arnold et al. presented these methods in a unified framework [3]. The DG method has many attractive features, including high order of accuracy, geometric flexibility, and efficient parallelization [30]. For these reasons there has been much interest in recent years in applying DG methods to a wide range of applications, including compressible and incompressible fluid flow, atmospheric modeling, and magnetohydrodynamics, among others [84, 85, 99]. The Runge-Kutta discontinuous Galerkin (RKDG) methods introduced and de- veloped in [28, 29, 32, 33, 34] are specifically designed to use explicit Runge-Kutta time integration methods. In particular, the RKDG method was designed to use total variation diminishing Runge-Kutta schemes developed by Shu and Osher [119, 120, 121], and studied extensively by Gottlieb, Shu, and Tadmor under the name strongly-stability preserving methods [58, 59]. The use of explicit time integration means that each time step can be computed without requiring large-scale linear solvers, which can prove to be beneficial both in terms of efficiency and ease of im- plementation. However, explicit time integration is not well-suited for all problems, prompting considerable interest in the use of implicit time integration and steady- state solvers together with discontinuous Galerkin spatial discretizations [8, 10, 48]. Significant attention has been devoted to the efficient solution and preconditioning of these systems [49, 56, 78, 86, 107, 137]. The topic of this thesis is the development and analysis of efficient solvers and preconditioners designed for the implicit time 3 integration of discontinuous Galerkin methods, which are required in order to reduce the potentially daunting computational cost associated with these systems. In the remainder of this chapter, we will define the discontinuous Galerkin method, together with some common notation that will be used throughout this thesis. We will formulate the DG method for systems of hyperbolic conservation laws, and then discuss several possible extensions of the method to elliptic and parabolic problems. We will then discuss the semi-discrete system that results from discretizing the spa- tial derivatives, and motivate the use of implicit time integration methods. Finally, we will discuss several classes of governing partial differential equations which we will consider in this thesis. The structure of this thesis is as follows. The second chapter concerns the use of the so-called “fully-implicit” Runge-Kutta methods, and the development of stage- parallel incomplete LU preconditioners for the solution of the resulting large linear systems. The third chapter describes the development of approximate tensor-product preconditioners that allow of the efficient solution of systems that arise from very high-order discretizations, with asymptotic improvements to the computational com- plexity. The fourth chapter provides an analysis of the convergence of iterative linear solvers for DG methods on polygonal meshes. The fifth chapter describes two de- tailed case studies, highlighting the methods developed in this thesis. Finally, the sixth chapter gives a conclusion and outlines potential future research directions. 4 1.1 The discontinuous Galerkin method In this section, we will formulate the discontinuous Galerkin method, and lay out some common notation that will be used throughout the remainder of the thesis. We will first formulate the DG method for systems of hyperbolic conservation laws, and then in the following section we will describe extensions to elliptic and parabolic equations. Let Ω ⊂ Rd be a given domain. We decompose the boundary of the domain as ∂Ω = ΓD ∪ ΓN . We consider a system of hyperbolic conservation laws given by ∂u + ∇ · F (u) = f in Ω, (1.1) ∂t u = gD on ΓD , (1.2) ∂u = gN on ΓN . (1.3) ∂n Here u(x, t) is a vector of nc unknown functions, for x ∈ Rd , and t ∈ R. F = (F1 , F2 , . . . , Fd ), Fi : Rnc → Rnc is a given flux function, f (x, t) is a forcing func- tion, and gD (x, t) and gN (x, t) are Dirichlet and Neumann boundary conditions, respectively. n(x) is the vector normal to ∂Ω facing outwards from Ω. We consider the nc ×nc Jacobian matrices Ji = ∂Fi /∂u. In order for the system to be hyperbolic, we require that the matrix α1 J1 + α2 J2 + · · · + αd Jd be diagonalizable with only real eigenvalues for all coefficients αi ∈ R. To formulate the discontinuous Galerkin method, we begin by discretizing the geometry. We define Th = {Ki }ni=1 t to be a triangulation of the domain Ω. By that we S t mean that ni=1 Ki = Ω, and Ki ∩Kj has measure zero for i 6= j. We refer to Ki as an element of the triangulation, which we take to be the image of the so-called reference 5 element under an invertible transformation map Ti . That is, we write Ki = Ti (R), where R is a fixed domain. We generally consider two classes of meshes: simplicial meshes and block meshes. Simplicial meshes correspond to the case where R is the unit simplex in Rd , and block meshes correspond to the case where R is the unit cube [0, 1]d . It will be clear from context which of these two cases apply. In Chapter 4, we consider more general polygonal meshes. If we consider simplicial meshes, then we take the reference element to be the unit  simplex, R = S d = x = (x1 , x2 , . . . , xd ) ∈ Rd : xi > 0 and x1 + x2 + · · · + xd ≤ 1 . We define the space of polynomials of total degree at most p on R as ( ) X P p (S d ) = p : S d → R : p(x) = cα x α , (1.4) α where the summation is taken over all multi-indices α = (α1 , α2 , . . . , αd ) with α1 + α2 + · · · αd ≤ p. Here we define the notation xα = xα1 1 xα2 2 · · · xαd d . The dimension of this vector space is     p + d  (p + d)! dim P p (S d ) =  = . (1.5) p p!d! We define a nodal basis for this space, meaning we choose (p + d)!/(p!d!) nodal interpolation points inside S d , with (p+d−1)!/(p!(d−1)!) of these nodes lying on each face of the simplex. For each node xi , we define a basis function φi (x) corresponding to the Lagrange interpolating polynomial satisfying φi (xi ) = 1, and φi (xj ) = 0 for all j 6= i. The specific choice of nodal interpolation points is determined by numerical conditioning considerations. Figure 1.1 shows the p = 3 nodal points for d = 1, 2, 3. As a consequence of this choice of basis, the value of a given function in this space at any point on a face of the simplex is determined only by its nodal values for nodes 6 Figure 1.1: Nodal interpolation points on the unit simplex in d = 1, 2, 3 for polyno- mial degree p = 3. which lie on the same face. In the case of block meshes, we consider the space of polynomials of degree at most p in each variable, ( )  X Qp [0, 1]d = p : [0, 1]d → R : p(x) = cβ x β , (1.6) β where the summation is taken over all multi-indices β = (β1 , β2 , . . . , βd ) such that  βi ≤ p for all 1 ≤ i ≤ d. The dimension of this vector space is dim Qp ([0, 1]d ) = (p + 1)d . Again, we define a nodal basis for this space. In particular, we choose a tensor-product basis. We let B 1 be the set of p + 1 Gauss-Lobatto points on the unit interval [0, 1]. Then, we define the nodal points to be given by the d-fold Cartesian product, B d = B × B × · · · × B. These points are shown in Figure 1.2. The induced Lagrange interpolation polynomials inherit the tensor-product structure from the nodal points. This tensor-product structure, and the associated notation, will be discussed in greater detail in Chapter 3. We will choose a fixed polynomial degree p, and define the function space Vh (R) to be either P p (S d ) or Qp ([0, 1]d ) according to whether the mesh is simplicial or block. This allows us to define a local function space for each element,   Vh (Ki ) = v : Ki → R : v(x) = p Ti−1 (x) for some p ∈ Vh (R) . (1.7) 7 Figure 1.2: Nodal interpolation points on the unit cube in d = 1, 2, 3 for polynomial degree p = 3. Thus, for each basis function φj of the space Vh (R), we obtain a transformed basis  function φej of the space Vh (Ki ), defined by φej (x) = φj Ti−1 (x) . If we denote the node corresponding to φj by ξj ∈ R, then the transformed basis functions φej interpolate the transformed nodes xj = Ti (ξj ). Having defined the local function spaces for each element, we can now define the broken function space on the entirety of the mesh. We define the space Vh = {v : Ω → R : v|Ki ∈ Vh (Ki ) for each element Ki ∈ Th } . (1.8) Thus, each function in this space, when restricted to one element of the mesh, is a member of the corresponding local function space. Here we emphasize that no conti- nuity between elements is enforced for these functions, leading to the discontinuous nature of the DG method. The basis functions φei for the space Vh (Ki ) are easily extended to basis functions for the entire space Vh by defining these functions to be identically zero outside of the element Ki . We also define the corresponding space of vector-valued functions. For any given nc ∈ N, we define [Vh ]nc = {v = (v1 , v2 , . . . , vn ) : Ω → Rnc : vi ∈ Vh for all 1 ≤ i ≤ nc } . (1.9) For the remainder of this work, we choose the element transformation mapping Ti 8 to be isoparametric, meaning that Ti ∈ [Vh (R)]d . Thus, the transformation can be entirely defined by the location of the nodal interpolation points on each element Ki . If the transformation mapping lies in the subspace of [Vh (R)]d corresponding to polynomial degree p = 1, then the resulting element is straight-sided. Otherwise, we say that the mesh is curved. The transformation mappings must be chosen such S t that the mapped elements satisfy ni=1 Ki = Ω and neighboring elements Ki and Kj intersect only at their common edge. Having defined the relevant function spaces, we are now ready to formulate the spatial discretization. For the sake of simplicity, the presentation here is restricted to a scalar hyperbolic conservation law. The discretization of second-order terms is discussed in the subsequent section. The extension to multidimensional systems can be found in [32]. We consider the equation ∂u + ∇ · F (u) = f in Ω, (1.10) ∂t where appropriate boundary conditions are enforced on ∂Ω. We look for a numerical solution uh ∈ Vh , multiply equation (1.10) by a test function vh ∈ Vh , and integrate over Ω to obtain Z Z Z ∂uh vh dx + ∇ · F (uh )vh dx = f vh dx. (1.11) Ω ∂t Ω Ω We consider the second integral on the left-hand side of (1.11). If we integrate by parts element-by-element, we obtain Z X Z ∇ · F (uh )vh dx = ∇ · F (uh )vh dx Ω Ki ∈Th Ki X Z Z  (1.12) = (F (uh ) · n) vh ds − F (uh ) · ∇vh dx . Ki ∈Th ∂Ki Ki 9 Since the function uh is discontinuous across element interfaces, we must replace the flux function F on ∂Ki with a numerical flux function Fb that depends on the trace of uh on both sides of a given element interface. To make this notion precise, we consider two neighboring elements, denoted K − and K + . These elements share a face, e = ∂K − ∩ ∂K + . We define u− h to be the trace of uh on e taken within K − , and likewise we define u+ h to be the trace of uh on e taken within K + . Similarly, n− is the normal vector to e facing outwards from K − , and n+ is the normal vector to e facing outwards from K + . We can now define the jump Juh K and average {uh } of a scalar function uh ∈ Vh as  Juh K = u− − + + h n + uh n , {uh } = 1 2 u− + h + uh . (1.13) Similarly, we define the jump Jτh K and average {τh } of a vector-valued function τh ∈ [Vh ]d as  Jτh K = τh− · n− + τh+ · n+ , {τh } = 1 2 τh− + τh+ . (1.14) We note that Juh K is a vector parallel to the normal vector, and Jτh K is a scalar. It S t will also be useful to define the notation Γ = ni=1 ∂Ki to be the set of all interior and exterior element boundaries. We then choose Fb (·, ·) to be a consistent numerical flux function, meaning that Fb (u, u) = F (u). (1.15) This numerical flux function is usually obtained by finding the exact or approximate solution to the one-dimensional Riemann problem in the normal direction, with dis- continuous initial data given by u− + h and uh . On the domain boundary ∂Ω, boundary 10 conditions are enforced through an appropriately modified numerical flux function FbΓ (·). Then, we obtain the DG formulation for equation (1.10): find uh ∈ Vh , such that, for all vh ∈ Vh , we have X Z ∂uh Z vh dx − F (uh ) · ∇vh dx Ki ∈Th Ki ∂t Ki Z Z  (1.16) + Fb (u− + − − h , uh ) · n vh ds − f vh dx = 0, ∂Ki Ki or, equivalently, Z Z Z Z ∂t uh vh dx − F (uh ) · ∇h vh dx + Fb (u− + h , uh ) · Jvh K ds − f vh dx = 0, (1.17) Ω Ω Γ Ω where ∇h , known as the broken gradient operator, denotes the gradient taken element- by-element. 1.2 Treatment of second-order terms In this section, we extend the discretization presented in the previous section to parabolic and elliptic equations. Many such discretizations have been developed, in- cluding the interior penalty (IP) method [2], the local discontinuous Galerkin (LDG) method [31], the compact discontinuous Galerkin (CDG) method [98], the first and second methods of Bassi and Rebay [9, 11], and others. A comprehensive study of these methods was performed by Arnold and colleagues in 2002 [3]; in this section, we roughly follow the presentation developed therein. As a model problem, we consider the Poisson problem with homogeneous Dirich- 11 let boundary conditions on a fixed domain Ω ⊆ Rd , −∆u = f in Ω, (1.18) u=0 on ∂Ω. (1.19) We transform this equation into a system of first-order equations by introducing the gradient σ, obtaining σ = ∇u in Ω, (1.20) −∇ · σ = f in Ω, (1.21) u=0 on ∂Ω. (1.22) We look for approximate solutions to the above equation, σh ∈ [Vh ]d , and uh ∈ Vh . To obtain the discontinuous Galerkin discretization, we multiply the above equations by test functions τh ∈ [Vh ]d , and vh ∈ Vh , and then repeat the elementwise integration by parts procedure described in Section 1.1. Thus, we obtain the weak form Z Z Z σh · τh dx = − uh ∇ · τh dx + bh Jτh K ds, u (1.23) Ω Ω Γ Z Z Z σh · ∇vh dx − b h · Jvh K ds = σ f vh dx, (1.24) Ω Γ Ω b h and u where σ bh are yet-to-be-defined numerical fluxes. The choice of these numeri- cal flux functions defines the DG discretization. The formulation given by (1.23–1.24) is known as the system flux formulation. It will be useful to eliminate the auxiliary variable σh to obtain a reduced system of equations. To this end, we perform integration by parts element by element to 12 obtain the identity Z Z Z Z − uh ∇ · τh dx = ∇h uh · τh dx − Juh K · {τh } ds − {uh }Jτh K ds. (1.25) Ω Ω Γ Γ\∂Ω We can then replace the first term on the right-hand-side of (1.23) by the expression on the right-hand-side of (1.25) to obtain Z Z Z Z σh · τh dx = ∇h uh · τh dx − Juh K · {τh } ds − {uh − u bh }Jτh K ds. (1.26) Ω Ω Γ Γ\∂Ω At this point we define the lifting operators r : [L2 (Γ)]d → [Vh ]d and ` : L2 (Γ \ ∂Ω) → [Vh ]d by Z Z Z Z r(q) · τ dx = − q · {τ } ds, `(v) · τ dx = − vJτ K ds. (1.27) Ω Γ Ω Γ\∂Ω Given these definitions, we see that equation (1.26) implies that σh = ∇h (uh ) + r(Juh K) + `({uh − u ch }). (1.28) Thus, having expressed σh explicitly in terms of uh , we can rewrite the system of equations (1.23–1.24) as Z B(uh , vh ) = f vh dx, (1.29) Ω where the bilinear form B(·, ·), called the primal form, is given by Z Z B(uh , vh ) = ∇h (uh ) · ∇h vh dx − Juh K · {∇h vh } ds Ω Z Γ Z (1.30) − {uh − u bh }J∇h vh K ds − σb h · Jvh K ds. Γ\∂Ω Γ 13 1.2.1 Choice of viscous fluxes b h and u As mentioned in the preceding section, the choice of numerical fluxes σ bh entirely determine the resulting numerical method. The simple choices of symmetric b h = {σh } gives rise to the first method of Bassi and Rebay. bh = {uh } and σ fluxes u b h = {∇h uh } − bh = {uh } and σ The interior penalty method results from choosing u ηE Juh K, where we recall that ∇h is the broken gradient, and ηe is a stability parameter local to each edge that depends on the mesh size. An explicit formula for this penalty parameter has been derived by Shabazi [116]. This method has the particularly simple primal form given by Z Z B(uh , vh ) = ∇h uh · ∇h vh dx − Juh K · {∇h vh } ds Ω Z Γ Z (1.31) − {∇h uh } · Jvh K ds + ηe Juh K · Jvh K ds, Γ Γ which also has the important property that it does not include either of the lifting operators ` or r. Additionally, this method has a compact stencil, meaning that the residual corresponding to a test function whose support lies entirely in a given element depends only on the value of the trial function in the immediately adjacent elements. b h = {σh } + C11 Juh K + C12 Jσh K for bh = {uh } − C12 · Juh K and σ The choices u parameters C11 ∈ R, C12 ∈ Rd give rise to the local DG method. The role of the parameter C11 is to stabilize the method, and can be thought of as adding an artificial viscosity. If C11 = 0 and |C12 ·n| = 12 , the resulting method has an upwind-downwind structure, and is referred to as the minimal dissipation LDG method [27]. The primal 14 Figure 1.3: Illustration of compact (left) and non-compact (right) stencils for DG discretizations of second-order operators. The stencil corresponding to a particular triangle (in black) is shown in dark gray. form associated with the LDG method is given by Z Z Z B(uh , vh ) = ∇h uh · ∇h vh dx − Juh K · {∇h vh } ds + C11 Juh K · Jvh K ds Ω Γ Γ Z Z − {∇h uh } · Jvh K ds + (C12 · Juh KJ∇h vh K + Juh KC12 · Jvh K) ds (1.32) Γ Z Γ\∂Ω + (r(Juh K) + `(C12 · Juh K)) · (r(Jvh K) + `(C12 · Jvh K)) dx. Ω The local DG method does not, in general, give rise to a compact stencil. The residual for a given element can depend on trial function in both the neighboring elements, and the elements neighboring the neighboring elements. An illustration of compact and non-compact stencils is shown in Figure 1.3. The compact DG method [98] is a modification of the LDG method that results in a compact stencil while retaining many of the same properties. In the remainder of this work, we will only consider the IP and CDG discretizations. 15 1.2.2 Generalized interior penalty formulation In this section, we follow the presentation from [62] in order to generalize the interior penalty formulation from the previous section to equations of the form ∂u  + ∇ · F I (u) + F V (u, ∇u) = f . (1.33) ∂t We refer to F I and F V as the inviscid and viscous fluxes, respectively. We make the assumption that the viscous flux is linear in the gradient of u, allowing us to write d X ∂u FiV (u, ∇u) = Hij (u) , (1.34) j=1 ∂xj where Hij (u) is a nc × nc matrix whose entries are given by arbitrary functions of u. We then transform (1.33) into the system ut + ∇ · (F I (u) + σ) = f , (1.35) d X V ∂u σ = F (u, ∇u) = Hij (u) . (1.36) j=1 ∂xj As in the case of the Poisson problem, we obtain the flux formulation by multiplying by test functions vh and τh , and integrating by parts Z Z Z ∂t uh · vh dx − I F (uh ) : ∇h vh dx + F cI (u− , u+ ) : Jvh K ds h h Ω Ω Z ZΓ Z (1.37) − σh : ∇h vh dx + σ b h : Jvh K ds = f · vh , Ω Γ Ω Z Z d X d X ∂ σh : τh dx = uh HijT (uh )(τh )i dx Ω Ω j=1 ∂xi i=1 Z d d (1.38) X X − bh u nj HijT (uh )(τh )i ds, Γ j=1 i=1 16 cI can be chosen as any where the numerical flux functions remain to be defined. F b h and σ standard numerical flux for hyperbolic problems, and we define u b h as  b h = {uh }, u b h = F V (uh , ∇h uh ) − ηe Juh K, σ (1.39) where ηe is a penalty parameter as in the Poisson case. Boundary conditions are enforced by appropriate modification of these numerical flux functions. By setting τh = ∇h vh in equation (1.38) and inserting the resulting expression into (1.37), and subsequently integrating by parts again, we can eliminate σh to obtain the primal formulation Z Z Z I ∂t uh · vh dx − F (uh ) : ∇vh dx + F cI (u− , u+ ) : Jvh K ds h h Ω Z X d Ω Z X( Γ d ) ∂uh ∂uh − Hij (uh ) : ∇vh dx + Hij (uh ) : Jvh K ds Ω j=1 ∂xj Γ ∂xj (1.40) j=1 Z Z ( d ) Z X ∂v h T + ηe Juh K : Jvh K ds + Juh K : Hij (uh ) ds = f · vh . Γ Γ i=1 ∂x i Ω 1.3 Semi-discrete system of equations Once we apply the discontinuous Galerkin discretization, we obtain a system of equations of the form (1.17) or (1.40). When considering this system of equations, it will be useful to introduce some additional notation. For the sake of simplicity, we describe here the scalar case. The extension to multidimensional systems of size nc is straightforward. Let N denote the dimension of the space Vh . Thus, any function uh ∈ Vh can be expanded in terms of the basis functions φei , N X uh (x) = ui φi (x), (1.41) i=1 17 where we slightly abuse notation when using ui to refer to the coefficients of this expansion, and define the vector u = (ui )N i=1 . Equation (1.17) can be stated equiva- lently as: find uh ∈ Vh such that, for all basis functions φei , we have Z Z Z Z ∂t uh φei dx − F (uh ) · ∇h φei dx + Fb (u− + e h , uh ) · Jφi K ds − f φei dx = 0. (1.42) Ω Ω Γ Ω We then define a vector-valued residual r : RN → RN by Z Z Z (r(u))i = F (uh ) · ∇h φei dx − Fb (u− + e h , uh ) · Jφi K ds + f φei dx. (1.43) Ω Γ Ω Similarly, we define the N × N mass matrix M by Z Mij = φei φej dx, (1.44) Ω and thus, Z ∂t uh φei dx = M ut . (1.45) Ω It is easy to see that the mass matrix is symmetric positive definite. Due to the def- inition of the discontinuous basis functions in Section 1.1, if φei and φej correspond to distinct elements of the mesh Th , then their supports are disjoint, and the correspond- ing entry of the mass matrix is zero. Thus, M has an element-wise block-diagonal structure, making its inversion both computationally efficient and local. We rewrite the DG formulation (1.42) as a coupled system of ordinary differential equations, M ut = r(u), (1.46) where the unknowns are the time-dependent coefficients ui (t). We refer to this system of ODEs as the semi-discrete system of equations. We use the method of lines to 18 integrate this system of equations in time using standard numerical methods. The main focus of this thesis is the efficient solution of the linear and nonlinear algebraic systems of equations that arise from the implicit integration of these equations. 1.3.1 Implicit time integration Although the block-diagonal structure of the mass matrix allows for efficient imple- mentation of explicit time integration, there are situations in which implicit methods are preferred over explicit methods. The main motivation for the use of implicit time integration is the time step restriction that arises from the use of explicit methods. Analysis of the DG method applied to the advection equation and wave equation show that the spectrum of the semi-discrete operator grows slightly slower than p2 /h, where p is the polynomial degree, and h is the mesh size [26, 72, 133]. In practice, this means that for a fixed-order explicit time integration scheme, the time step must satisfy (approximately) h ∆t ≤ C , (1.47) p2 for some problem-dependent constant C. It has been shown that if one uses an explicit Runge-Kutta method of order p+1 (such that the overall order of convergence is p + 1), then the time step restriction is approximately ∆t ≤ Ch/(2p + 1) due to the enlarged regions of stability [30]. This restriction becomes even more severe in the case of second-order equations. The spectrum of the Laplacian operator grows slightly slower than p4 /h2 , and so, explicit integration of a DG discretization of the heat equation would require approximately (for fixed temporal order) ∆t < Ch2 /p4 . These estimates illustrate to us two factors that can result in a prohibitively restrictive stability condition. The first is the mesh size h. In order to adequately 19 1010 First derivative Second derivative 108 3.88 max{|λ|} 1 6 10 104 1.78 2 1 10 100 100 101 102 p Figure 1.4: Spectral radius of first derivative and second derivative DG operators resolve small-scale features, for example the boundary layer that results from viscous fluid flow over a wall, one must introduce very small elements. The time step condi- tion is determined by the smallest element in the mesh, and can become prohibitively restrictive in the case of high Reynolds numbers. The second factor is the degree of polynomial approximation. A central advantage of the discontinuous Galerkin method is its arbitrarily high order of accuracy. However, the use of high-degree polynomials comes at a very severe cost in terms of the spectrum of the operator. Implicit time integration allows us to circumvent these severe time step restric- tions. A-stable methods, whose regions of stability include the entire left half-plane, are stable for any choice of time step when applied to the advection equation. Even when applied to nonlinear problems, in practice one can use time steps that are sev- eral order of magnitude larger than the largest stable explicit time step. The main challenge associated with such methods is the efficient solution of the resulting system of equations. This problem reduces to the solution of the linear system of equations obtained by linearizing the DG residual. This linear system has a large number of degrees of freedom, and possesses a natural block structure. For three-dimensional or large two-dimensional problems, the size of this system renders direct solvers im- 20 practical. Therefore, iterative solvers are typically employed. The remainder of this thesis will be devoted to the study and development of effective preconditioners for the iterative solution of these systems of equations. 1.4 Governing equations In the subsequent chapters of this thesis, we will consider several sets of governing equations. For reference, we will list these equations in the following sections. 1.4.1 Linear advection equation The simplest equation we consider is the linear advection equation. Let the velocity β(x) be a given vector field in Ω ⊆ Rd . Then, this scalar hyperbolic equation is given by ∂u (x, t) + ∇ · (β(x)u(x, t)) = 0, (1.48) ∂t u(x, 0) = u0 (x), (1.49) for some given initial condition u0 . We must supplement (1.48–1.49) by appro- priate boundary conditions on ∂Ω. We define the inflow boundary to be Γ− = {x ∈ ∂Ω : β(x) · n < 0}. Then, we enforce the inflow boundary condition u(x, t) = g(x, t) on Γ− , (1.50) where g(x, t) is a specified function. No boundary condition is required on ∂Ω \ Γ− . 21 We remark that in the special case that β is spatially-constant and Ω = Rd , then the exact solution to this equation is given by u(x, t) = u0 (x − tβ). The numerical flux function for this equation can be chosen by simple upwinding,    β(x)u−h if β · n− ≥ 0, Fb (u− + h , uh ) = (1.51)   β(x)u+ h otherwise. The boundary condition is enforced by modifying this numerical flux function on the domain boundary,    β(x)u− h if β · n− ≥ 0, FbΓ (u− h) = on ∂Ω. (1.52)   β(x)g otherwise 1.4.2 Convection-diffusion equation We modify the advection equation by adding a diffusion term, ∂u (x, t) + ∇ · (β(x)u(x, t) + (x)∇u(x, t)) = 0, (1.53) ∂t u(x, 0) = u0 (x), (1.54) with appropriate boundary conditions specified on ∂Ω. As in the previous case, β is the advection velocity.  is known as the diffusion coefficient. Due to the presence of the viscous flux F V (u, ∇u) = (x)∇u, we make use of the discretizations described in Section 1.2 to solve this equation. The P´eclet number Pe = kβkL/, where L is a characteristic length, can be used to classify the equation as either convection- dominated or diffusion-dominated. 22 1.4.3 Compressible Euler equations The previous two sets of equations can be thought of as simple model problems for the Euler and Navier-Stokes equations. The compressible Euler equations of gas dynamics are a system of hyperbolic equations given by ∂u  + ∇ · F I (u) = 0. (1.55) ∂t Here the vector of unknowns is u = (ρ, ρv, ρE)T , where ρ is the density, v = (ui )di=1 is the fluid velocity, and E is total energy per unit mass, for d = 1, 2, 3. For an ideal gas, we define the pressure p through the equation of state  p = (γ − 1)ρ E − kvk2 /2 , (1.56) where the constant γ is the ratio of specific heats, which we take as γ = 1.4 for the remainder of this thesis. The flux function is defined by    ρv    F I (u) =  T  ρv ⊗ v + pI ,  (1.57)   ρHv where I is the d × d identity matrix, and H = E + p/ρ is the stagnation enthalpy. For instance, for the specific case of d = 3, we have       ρu1 ρu2 ρu3        2       ρu1 + p   ρu2 u1   ρu3 u1          I   I   F1 (u) =  I  ρu1 u2  , F2 (u) =  ρu2 + p   2  , F3 (u) =  ρu u  .   3 2  (1.58)            2   ρu1 u3   ρu2 u3   ρu3 + p        ρHv1 ρHu2 ρHu3 23 We define the flux Jacobian matrices Ai (u) = ∂FiI /∂u. Due to the hyperbolicity of P the Euler equations, we have that B(u) = di=1 ni Ai (u) is diagonalizable with only real eigenvalues for any vector of coefficients n ∈ Rd . In particular, the eigenvalues of B(u) are given by λ1 = v · n − c, λ2 = · · · = λd+1 = v · n, λd+2 = v · n + c, (1.59) p where c is the speed of sound, given by c = γp/ρ. The signs of the eigenvalues on the domain boundary ∂Ω determine the number of boundary conditions that are required. In order to define the discontinuous Galerkin discretization, we must provide a numerical flux function. A simple choice is the local Lax-Friedrichs flux, given by cI (u− , u+ ) = 1 F I (u− ) + F I (u+ ) + α u− n− + u+ n+  F h h h h 2 2 h h (1.60)  α = F I (uu ) + Juh K, 2   where α is the maximum absolute eigenvalue of the two matrices B u− + h and B uh . A more sophisticated numerical flux function can be obtained using the Roe approximate Riemann solver [110, 111, 126]. The idea behind this numerical flux function is to solve the Riemann problem for a linearized problem, which is obtained by linearizing the flux function F I around a Roe averaged state, given by p − − p + + p − − p + + ρh vh + ρh vh b h = ρp h Hh + ρ H bh = v p − p + , H − p h+ h , (1.61) ρh + ρh ρh + ρh r   b ch = (γ − 1) Hh − kb which gives rise to the averaged sound speed b 2 vh k /2 . Then, the linearized Riemann problem can be solved efficiently, avoiding the computation- 24 ally expensive solution of the true nonlinear Riemann problem. The numerical flux function is then given by evaluating the Euler flux function at the solution of the linearized Riemann problem. 1.4.4 Compressible Navier-Stokes equations The Navier-Stokes equations describe viscous fluid flow, whereas the Euler equations describe inviscid fluid flow. The compressible Navier-Stokes equations are given by ∂u  + ∇ · F I (u) + F V (u, ∇u) = 0, (1.62) ∂t where the inviscid flux is the same as the Euler flux given by (1.57). This viscous flux is defined by    0    F V =  −τ ,  (1.63)   qj − ui τij where the viscous stress tensor τ and heat flux q are given by     ∂ui ∂uj 2 ∂uk µ ∂ p 1 τij = µ + − δij , qj = − E + − uk uk , (1.64) ∂xj ∂xi 3 ∂xk Pr ∂xj ρ 2 where we use the convention that repeated indices are implicitly summed. Here µ is the coefficient of viscosity, and Pr is the Prandtl number, which we take as Pr = 0.72. The advective numerical flux can be chosen in the same way as for the Euler equations. In order to apply the interior penalty method for the viscous terms, we 25 must write F V in the form d X ∂u FiV (u, ∇u) = Hij (u) (1.65) j=1 ∂xj for nc × nc matrices Hij (u) whose entries are given as functions of u. We consider d = 2 for simplicity, and following the derivation of Hartmann [62, 63], we define the matrices    0 0 0 0    −µ  − 43 u1 0  4  3 0  H11 =  , ρ  −u2 0 1 0      4    4 2 γ γ γ γ − u 3 1 + u22 + Pr (E− |v|2 ) 3 − Pr u1 1 − Pr u2 Pr      0 0 0 0   0 0 0 0      −µ  − 23 0  −µ  0  2  u 3 2 0   −u2 0 1  H12 =   , H21 =  , ρ  −u1 1 0 0  ρ  2u − 23 0 0     3 1      − 13 u1 u2 u2 − 23 u1 0 − 13 u1 u2 − 32 u2 u1 0    0 0 0 0    −µ   −u1 1 0 0   H22 =   ρ  − 43 u2 0 4 0   3       γ γ γ γ − u21 + 43 u22 + Pr (E − |v|2 ) 1− Pr u1 4 3 − Pr u2 Pr such that 2 X ∂u FiV (u, ∇u) = Hij (u) , (1.66) j=1 ∂xj allowing us to use the interior penalty formulation given by (1.40). For simplicity, we choose the penalty parameter to be a constant multiple of the viscosity µ. 26 Isentropic assumption In some cases, we modify the above equations by introducing an isentropic assump- tion of the form p = Kργ for a given constant K, as described in [70]. This additional simplification can be thought of as an artificial compressibility model for the incom- pressible flows. We prefer to solve the compressible equations because they result in a system of ordinary differential equations rather than differential-algebraic equations, and thus do not need specialized projection-type solvers. This model decouples the equation for ρE from the equations for ρ and ρv. As a result, it is therefore possible to solve for one fewer component. Chapter Two Fully-implicit Runge-Kutta methods 28 Implicit time integration methods for DG have been much studied. Multi-step backward differentiation formulas (BDF) and single-step diagonally implicit Runge- Kutta (DIRK) methods have been applied to discontinuous Galerkin discretizations for fluid flow problems [102, 105]. Nigro et al. have seen success applying multi-stage, multi-step modified extended BDF (MEBDF) and two implicit advanced step-point (TIAS) schemes to the compressible Euler and Navier-Stokes equations [87, 88]. Additionally, in [12], Bassi et al. have used linearly implicit Rosenbrock-type to integrate DG discretizations for various fluid flow problems. The BDF and DIRK methods have some limitations: BDF schemes can be A-stable only up to second- order (the famous second Dahlquist barrier) [37], a severe limitation when used in conjunction with a high-order spatial discretization. On the other hand, there exist high-order A-stable (and even L-stable) DIRK schemes, but these methods have a low stage-order, often resulting in order reduction when applied to stiff problems [50]. The Radau IIA methods, one class of the so-called fully implicit Runge-Kutta (IRK) methods, are high-order, L-stable, and have relatively high stage order. Con- sequently, these methods suffer less from order reduction than the corresponding DIRK methods when applied to stiff problems. Furthermore, these methods require only a small number of stages s, with the order of accuracy given by 2s − 1. These methods have the drawback that each step involves the solution of large, coupled linear systems of equations. The difficulty in efficiently implementing such methods has caused them to remain not widely used or studied for practical applications [24, 25]. There has been previous work on improving the efficiency of solving these large, coupled systems. In [66], Jay and Braconnier develop a parallelizable preconditioner for IRK methods by means of Hairer and Wanner’s W -transformation. In [124], De Swart et al. have developed a parallel software package for the four-stage Radau IIA 29 method, and Burrage et at. have developed a matrix-free, parallel implementation of the fifth-order Radau IIA method in [22]. In this chapter, we develop a new strategy for efficiently solving the resulting large linear systems by means of the iterative preconditioned GMRES algorithm. A simple transformation of the linear system results in a significant reduction of the cost per GMRES iteration. Furthermore, the block ILU(0) preconditioner, used successfully with implicit time-integrators for the discontinuous Galerkin method in [107], proves to be effective also for these large systems. A shifted, uncoupled, block ILU(0) factorization is also found to be an effective preconditioner, with the advantage of allowing parallelism in time by computing the stage solutions simultaneously. The structure of this chapter is as follows. In Section 2.1, we discuss the Runge- Kutta time integration schemes used. Then, in Section 2.2, we introduce the trans- formation used to reduce the solution cost, and discuss the preconditioners used for the GMRES method. Finally, in Section 2.3, we perform numerical experiments on a variety of test cases, in two and three spatial dimensions. 2.1 Runge-Kutta methods In this chapter, we will focus on the one-step, multi-stage Runge-Kutta methods. We consider the initial-value problem defined by (1.46), M ut (t) = r(t, u), t > t0 (2.1) u(t0 ) = u0 . (2.2) 30 A general s-stage, pth-order Runge-Kutta method for advancing the solution to u1 = u(t0 + ∆t) + O(∆tp+1 ) can be written as s ! X M ki = r t0 + ∆tci , u0 + ∆t aij kj , (2.3) j=1 s X u1 = u0 + ∆t bi ki , (2.4) i=1 where the coefficients aij , bi , and ci can be expressed compactly in the form of the Butcher tableau, c1 a11 · · · a1s .. .. . . . . . . .. c A = . cs as1 · · · ass b T b1 ··· bs If the matrix of coefficients A is strictly lower-triangular, then each stage ki only depends on the preceding stages, and the method is called an explicit Runge- Kutta method. In this case, each stage may be computed by simply evaluating the function r, and solving a linear system with the mass matrix M , which, as described in Section 1.3, can be accomplished in a straightforward manner. If A is not strictly lower-triangular, the method is called an implicit Runge-Kutta method (IRK). A particular class of implicit Runge-Kutta methods is those for which the matrix A is lower-triangular. Such methods are called diagonally-implicit Runge- Kutta (DIRK) methods [1]. Implicit Runge-Kutta methods enjoy high accuracy and very favorable stability properties, but computing the stages requires the solution of (in general nonlinear) systems of equations. In the case of DIRK methods, since A is lower-triangular, each stage ki depends only on those stages kj , j ≤ i, requiring the sequential solution of s systems, each of size N . In contrast, general IRK methods couple all of the stages, resulting in one nonlinear system of equations of size s × N . 31 Table 2.1: L-stable implicit Runge-Kutta schemes Scheme Order Stages Implicit stages Stage order Leading error coeff. RADAU23 3 2 2 2 1.39 × 10−2 DIRK33 3 3 3 1 2.59 × 10−2 RADAU35 5 3 3 3 1.39 × 10−4 ESDIRK65 5 6 5 2 5.30 × 10−4 RADAU47 7 4 4 4 7.09 × 10−7 RADAU59 9 5 5 5 2.19 × 10−9 For the solution of stiff systems of equations, we are interested in those methods that are L-stable, meaning that their stability region includes the entire left half- place (A-stability), together with the additional criterion that the stability function R(z) satisfies limz→∞ R(z) = 0. In the present study, we compare the efficiency and effectiveness of several L-stable IRK and DIRK schemes. The methods considered are listed in Table 2.1. The IRK schemes considered are the Radau IIA schemes, which are L-stable, s-stage schemes of order 2s − 1 based on the Radau right quadra- ture. The construction of these schemes can be found in [60, 61]. The two-stage and three-stage Radau IIA methods are listed as RADAU23 and RADAU35, respectively. The DIRK schemes considered are the three-stage, third-order L-stable scheme de- noted DIRK33, which is derived in detail in [1], and the six-stage, fifth-order scheme constructed in [18], and denoted ESDIRK65. The latter scheme is an explicit singly diagonally implicit Runge-Kutta (ESDIRK) method, meaning that the first diagonal entry of the Butcher matrix is zero, and the remaining diagonally entries are nonzero and equal. In addition to the third- and fifth-order methods, we also consider the seventh- and ninth-order Radau IIA methods, although we do not compare these methods to equal-order DIRK methods. The Butcher tableaux for the methods considered are given in Appendix A. Also of interest is the phenomenon of order reduction, whereby, when applied to stiff problems, the overall order of accuracy is reduced from p to the stage order of 32 the method (denoted q) [50]. The stage order q is defined as q = min{p, qi }, for i = 1, . . . , s, where qi is defined by s X u(t0 + ∆tci ) = u0 + ∆t aij kj + O(∆tqi +1 ). (2.5) j=1 It can be shown that the maximum stage order for any DIRK method is 2, whereas the stage order for the Radau IIA methods is given by the number of stages, q = s [61]. The DIRK33 has stage order of q = 1. An advantage of the ESDIRK methods such as ESDIRK65 is that they have stage order of q = 2 [75]. The Radau IIA methods are very attractive because of their high order of ac- curacy, small number of stages, high stage order, and L-stability, but solving the coupled system of s × N equations is computationally expensive. Supposing that we solve the nonlinear system of equations for the stages ki by means of Newton’s method, then at each iteration we must solve a linear system of equations by invert- ing the Jacobian matrix of the right-hand side, r(t, u). Assuming a dense Jacobian matrix, and solution via Gaussian elimination (or LU factorization), then the cost of performing a linear solve scales as the cube of the number of unknowns. Therefore, the cost per linear solve for a general IRK method is O(s3 N 3 ), whereas the cost per solve for a DIRK method scales like O(sN 3 ). In [23], Butcher describes how to transform the resulting set of linear equations to reduce the computational work for solving the IRK systems to O(2sN 3 ). Despite this reduction in computational complexity, the cost of solving the large systems of equations has proven in practice to be prohibitive [24]. On the other hand, DIRK methods have proven be popular and effective for solving computational fluid dynamics problems [17], at the cost of lower stage order and an increased number of stages. 33 2.2 Efficient solution of implicit Runge-Kutta sys- tems In this section we describe a method for efficiently solving the systems arising from general IRK methods when applied to discontinuous Galerkin discretizations. The Jacobian matrices of the function r are sparse, block-structured matrices, which lend themselves to solution via iterative Krylov subspace methods. In particular, we consider the solution of these systems by means of the GMRES method with a zero fill-in block ILU(0) preconditioner, as in [107]. In this case, each iteration of the GMRES method requires one matrix-vector multiplication, and one application of the ILU(0) preconditioner. In order to efficiently solve the linear systems resulting from IRK methods, we will rewrite the system of equations in such a way so as to reduce the cost of a matrix-vector multiplication from s2 N 2 to order sN 2 . 2.2.1 Block structure of the Jacobian We consider the vector of unknowns u to be ordered such that the m degrees of freedom associated with one element of the triangulation appear consecutively. We suppose that there are a total of nt elements, such that there are a total of N = mnt degrees of freedom. Then, the Jacobian matrix J = ∂r/∂u can be seen as a nt × nt block matrix, with blocks of size m × m. The ith row consists of blocks on the diagonal, and in columns j, where elements Ki and Kj share a common face, such that the total number of off-diagonal blocks in the ith row is equal to the number of neighbors of element Ki . We note that the off-diagonal blocks of size m × m are themselves sparse, but for the sake of simplicity we will consider them as dense matrices. The mass matrix M is a nt × nt block diagonal matrix, with blocks of size 34 m × m, and therefore matrices of the form αM − βJ have the same sparsity pattern as the Jacobian. 2.2.2 Transformation of the system of equations Recalling equation that the stages ki are given by the equation s ! X M ki = r t0 + ∆tci , u0 + ∆t aij kj , (2.6) j=1 we define K to be the concatenation of the vectors ki , U0 to be the concatenation of s copies of u0 , and R the function r applied component-wise on these vectors. Then, we rewrite equation (2.6) in vector form as (Is ⊗ M )K = R (t0 + ∆tc, U0 + ∆t(A ⊗ IN )K) , (2.7) where ⊗ is the Kronecker product, and Is and IN are the s × s and N × N identity matrices, respectively. This nonlinear system can be solved by means of Newton’s method, which will require solving at each step a linear system of the form          M 0   a11 J1 · · · a1s J1   k1   b1           ...  − ∆t  ... ... .. .   .. . = .. .  (2.8)                 0 M as1 Js · · · ass Js , ks bs for the residual vectors (b1 , . . . , bs )T , where the matrices on the left-hand side are s × s block matrices blocks, with each block of size N × N . We use the following 35 notation for the Jacobian matrix of r, s ! X Ji = Jr t0 + ∆tci , u0 + ∆t aij kj . j=1 We can rewrite equation (2.8) in the following form,      a11 J1 · · · a1s J1     Is ⊗ M − ∆t  ... ... .. .  K = B, (2.9)       as1 Js · · · ass Js The sparsity pattern of the matrix in (2.9) is simply that of the Jacobian matrix J repeated s×s times, and we can conclude that the cost of computing a matrix-vector product with this matrix is s2 times that of computing the matrix-vector product of one Jacobian matrix. In order to reduce the cost of the matrix-vector multiplication, we perform a simple transformation in to rewrite (2.9) in a slightly modified form. We begin by defining s X wi = aij kj , j=1 and similarly, we let W denote the vectors wi stacked, such that W = (A ⊗ In )K. Then, we rewrite the nonlinear system of equations (2.6) in terms of the variables wi to obtain M ki = r(t0 + ∆tci , u0 + ∆twi ), 36 or, equivalently, in the case that the Butcher matrix A is invertible, (A−1 ⊗ M )W = R(t0 + ∆tc, U0 + ∆tW ). (2.10) In the transformed variables, the new solution u1 can be written as u1 = u0 + ∆t(bT A−1 ⊗ In )W , In the case of the Radau IIA methods, bT A−1 = (0, . . . , 0, 1), and so this further simplifies to u1 = u0 + ∆tws . Solving equation (2.10) with Newton’s method gives rise to the linear system of equations      J1 0 ··· 0       0 J2 ··· 0    −1   A ⊗ M − ∆t  . . .  W = R. (2.11)   .. .. ..   . ..      0 0 · · · Js The advantage of this formulation is that the resulting system enjoys greater sparsity. The resulting matrix is a s × s block matrix, with multiples of the mass matrix in every off-diagonal block, and with matrices of the form (A−1 )ii M − ∆tJi along the diagonal. Computing a matrix-vector product with this s × s block matrix requires performing s matrix-vector multiplications with the mass matrix, and s matrix- vector multiplications with a Jacobian Ji . Therefore the cost of computing such products scales as s times the cost of computing one matrix-vector product with the Jacobian matrix. We additionally remark that the fully-implicit IRK methods requiring storing each of the s Jacobian matrices Ji , resulting in memory usage that is s-times that 37 of the DIRK methods. A further advantage of the transformed system of equations is that the memory requirements for the ILU-based preconditioners are reduced, as discussed in the following sections. 2.2.3 Preconditioning The use of an appropriate preconditioner is essential in accelerating the convergence of a Krylov subspace method such as GMRES. We briefly describe the block ILU(0) preconditioner from [107]. Block ILU(0) preconditioner The block ILU(0) (or zero fill-in) preconditioner is a method for obtaining block ˜ and U˜ given a block sparse matrix B. These lower- and upper-triangular matrices L matrices are obtaining by computing the standard block LU factorization, but dis- carding any blocks which do not appear in the sparsity pattern of B. We denote the block in the ith row and jth column as Bij . The ILU(0) algorithm can be written as shown in Algorithm 2.1. Algorithm 2.1 Block ILU(0) algorithm 1: for i = 1 to nt do 2: for neighbors j of i with j > i do 3: Bji ← Bji Bii−1 4: Bjj ← Bjj − Bji Bij 5: for neighbors k of j and i with k > j do 6: Bjk ← Bjk − Bji Bik ˜ ← I + strict block lower triangle of B 7: L ˜ ← block upper triangle of B 8: U If we impose the condition on the triangulation of our domain that, if elements j 38 and k both neighbor element i, then elements j and k are not neighbors of each other, then the ILU(0) algorithm has the particularly simple form, show in Algorithm 2.2. In practice, most well-shaped meshes satisfy this condition and henceforth we will use this simpler algorithm. Algorithm 2.2 Simplified block ILU(0) algorithm 1: for i = 1 to nt do 2: for neighbors j of i with j > i do 3: Bji ← Bji Bii−1 4: Bjj ← Bjj − Bji Bij ˜ ← I + strict block lower triangle of B 5: L ˜ ← block upper triangle of B 6: U 2.2.4 Preconditioning the large system In the case of the general IRK methods, we must solve systems of the form      J1 0 ··· 0       0 J2 ··· 0    −1   BW = R, B = A ⊗ M − ∆t  .. .. .  . (2.12)   ..   . . . ..       0 0 · · · Js We remark that the matrix B can now be considered as a s × s block matrix, with blocks of size N × N . Each N × N block is itself an nt × nt block matrix, with subblocks of size m × m. We introduce the notation Bk`,ij to denote the (i, j) subblock of the (k, `) block of B. That is to say, Bk`,ij is the (i, j) block of the matrix (A)−1 k` M − δk` ∆tJk , where δk` is the Kronecker delta. 39 Stage-coupled block ILU(0) preconditioner We consider two preconditioners for this large sN × sN system. The first is the standard (stage-coupled) block ILU(0) preconditioner, which can be computed using Algorithm 2.3. We note that this preconditioner couples all s stages of the method. This preconditioner requires storing s Jacobian-sized diagonal blocks, and s2 − s off-diagonal blocks, each the same size as the mass matrix. Algorithm 2.3 ILU(0) algorithm for IRK systems of the form (2.12) 1: for k = 1 to s do 2: for i = 1 to nt do 3: for neighbors j of i with j > i do −1 4: Bkk,ji ← Bkk,ji Bkk,ii 5: Bkk,jj ← Bkk,jj − Bkk,ji Bkk,ij 6: for ` = k + 1 to s do −1 7: B`k,ii ← B`k,ii Bkk,ii 8: B``,ii ← B``,ii − B`k,ii Bkk,ii 9: L˜ ← I + strict block lower triangle of B 10: U˜ ← block upper triangle of B Stage-uncoupled, shifted ILU(0) preconditioner In order to avoid the above coupling of the stages, we can compute a simplified preconditioner in the form of the following block matrix,   ˜ ˜ ···  L1 U1 0 0     0 ˜ 2 U˜2 L ··· 0     . .. .. , (2.13)  .. ...   . .    0 0 ˜ s U˜s ··· L 40  ˜ i U˜i is the block ILU(0) factorization of a matrix of the form A−1 + αi M − where L ii ∆tJi . We let αi denote a shift, so that the standard unshifted factorization corre- sponds to αi = 0. The so-called shifted ILU factorization, described by Manteuffel in [81], can result in eigenvalues clustered away from the origin, and hence faster con- vergence in GMRES. Indeed, our experience shows that the unshifted preconditioner underperforms certain other choices of shift. The preconditioner has several advantages over the stage-coupled block ILU(0) preconditioner. The first is that it is easily constructed using an already-implemented block ILU(0) factorization of the Jacobian matrix. The second is that none of the stages are coupled, allowing for both efficient computation and application. In par- ticular, this has implications for the parallelization of the preconditioner, which we discuss in Section 2.2.6. Finally, as this preconditioner does not include any off diag- onal blocks, the memory requirements are exactly s times that of the standard block ILU(0) used for the DIRK methods. ˜ i U˜i As mentioned, the unshifted preconditioner, with αi = 0 for all i, such that L is the ILU(0) factorization of the ith diagonal block of the matrix B, is a natural choice. This choice of coefficients ignores all the off-diagonal mass matrices. By making certain judicious choices of the coefficients αi , we can attempt to compen- sate for the off-diagonal blocks by adding multiples of the mass matrix back to the diagonal entries. In particular, our numerical experiments have shown that setting P αi = j6=i A−1 ji results in a more efficient preconditioner, requiring fewer GMRES iterations in order to converge to a given desired tolerance. 41 2.2.5 Computational cost and memory requirements In order to compare the computational cost of the transformed IRK implementation described above with both that of the untransformed formulation, and with the usual DIRK methods, we summarize the computational cost associated with solving the resulting linear systems. We note that the IRK methods require the solution of one large, coupled system, whereas the DIRK methods require the solution of s smaller systems. In Table 2.2 we record the leading terms of the computational cost of operations required to be performed every iteration. We recall that s is the number of Runge-Kutta stages, m is the number of degrees of freedom per mesh element, nt is the total number of elements in the mesh, and r is the number of neighbors per element. Here we assume that the m×m blocks of the Jacobian matrix are dense, and hence require 2m2 floating point operations per matrix-vector multiply. Computing the preconditioner requires the LU factorization of the diagonal blocks, which incurs a cost of 32 m3 floating point operations per block. Each iteration in Newton’s method requires re-evaluation of the Jacobian matrix, and therefore also the re-computation of the preconditioner. Hence, the preconditioner must be computed once per linear solve. The costs associated with these calculations are listed in Table 2.3. 42 Table 2.2: Per-iteration computational costs for solving implicit Runge-Kutta sys- tems Operation Cost (leading term) Untransformed IRK matrix-vector product s2 m2 (r + 1)nt Transformed IRK matrix-vector product sm2 (r + s)nt DIRK matrix-vector product m2 (r + 1)nt Coupled preconditioner application (IRK) sm2 (r + s)nt Uncoupled preconditioner application (IRK) sm2 (r + 1)nt Preconditioner application (DIRK) m2 (r + 1)nt Table 2.3: Computational cost of computing the IRK preconditioners (once per solve) Operation Cost (leading term) Computing coupled block ILU(0) preconditioner (IRK) s(m3 + (r + s)m2 )nt Computing uncoupled ILU(0) preconditioner (IRK) s(m3 + rm2 )nt Computing block ILU(0) preconditioner (DIRK) (m3 + rm2 )nt We remark that each GMRES iteration using the formulation described in Section 2.2 requires a factor of s fewer floating-point operations per iteration than the na¨ıve IRK implementation. The stage-uncoupled IRK preconditioner is less expensive to both compute and apply than the stage-coupled preconditioner. We also note that for equal order of accuracy, the Radau IIA IRK methods require fewer implicit stages than do the DIRK methods. Each such implicit stage incurs the cost of assembling the Jacobian matrix. This cost is problem-dependent, but is in general non-trivial, and in the model problems considered in this chapter, it scales like O(m3 nt ). 43 Finally, we present the memory requirements for the IRK and DIRK methods, and the stage-coupled and uncoupled preconditioners in Table 2.4. We note that for the transformed IRK methods, only the s Jacobian matrices Ji need to be stored. The stage-coupled block ILU(0) preconditioner requires an additional s2 − s off-diagonal blocks, which have the same block-diagonal structure as the mass matrix, each having m2 nt nonzero entries. The stage-uncoupled preconditioner does not require these off-diagonal blocks, and therefore its memory requirements are exactly s times that of the DIRK block ILU(0) preconditioner. The block ILU(0) preconditioner for the untransformed system of equations would require storing an additional (s2 −s)rm2 nt nonzero entries in the off-diagonal blocks. Table 2.4: Memory requirements for Jacobian matrices and preconditioners Method Memory required IRK Jacobian matrix s(r + 1)m2 nt DIRK Jacobian matrix (r + 1)m2 nt Coupled block ILU(0) preconditioner (IRK) sm2 (s + r)nt Uncoupled ILU(0) preconditioner (IRK) sm2 (r + 1)nt Block ILU(0) preconditioner (DIRK) m2 (r + 1)nt 2.2.6 Stage-parallelism and partitioned ILU In order to parallelize the computations, the spatial domain is decomposed into several subdomains. The compact stencil of the CDG and IP schemes allows for very low communication costs between processes for residual evaluation and Jacobian assembly operations. The matrix-vector multiplications, which constitute the bulk of the work for the linear solve, also scale well in terms of communication for the same 44 reason. In order to parallelize the ILU(0) preconditioner, the contributions between elements in different partitions are ignored, allowing each process to compute the ILU(0) factorization independently. Because these contributions are ignored, we find that the number of GMRES iterations required to converge increases as the number of domain partitions increases [104]. In fact, it is easy to see that when the number of partitions is equal to the number of mesh elements, the preconditioner simply reduces to the block Jacobi preconditioner. However, in general the partitioned ILU(0) preconditioner is found to be superior to the block Jacobi preconditioner. These considerations apply equally to both the DIRK and general IRK meth- ods, using the stage-coupled ILU(0) preconditioner. If we use the stage-uncoupled ILU(0) preconditioner, then we are able to decompose the domain into a factor of s fewer partitions. We then assume that the number of processes is equal to the number of mesh partitions times the number of stages. The processes are first di- vided into groups according to the mesh partitioning such that each group consists of s processes. Within each group, each process is then assigned to one stage of the IRK method. Thus, when assembling the block matrix of the form (2.12), the Jacobian matrices for all of the stages are computed in parallel. This does not re- quire any communication between the groups. Then, since the stage-uncoupled block ILU(0) preconditioner does not take into account any of the off-diagonal blocks, the preconditioner can also be computed without any inter-stage communication. Sim- ilarly, each application of the preconditioner can be computed in parallel over all the stages without any communication. When computing matrix-vector products, the products with the Jacobian blocks for each stage are computed in parallel, and the products with the mass matrix blocks must be communicated within the stages. It is possible to overlap the communication with the computation of the matrix- vector product with the stage-Jacobian block, such that the cost of communication 45 is negligible. The main advantage of this parallelization scheme is that the mesh is decomposed into a factor of s fewer partitions. Therefore, the effect of ignoring the coupling between regions in the ILU(0) factorization is lessened, and the result is a more efficient preconditioner. In Sections 2.3.2 and 2.3.3, we numerically study the effects of parallelizing these preconditioners. In the context of hybrid shared-distributed memory systems, it is possible to further reduce the communication cost of the stage-parallel IRK algorithm. On such a system, groups of compute cores called nodes have access to the same shared memory. As a consequence, intranode communication is much faster than internode communication. If each stage-group of s processes described above is located on one node, and none of the groups are split across nodes, then the solutions are only communicated within a node, resulting in negligible internode communication costs. An illustration of such an arrangement is shown in Figures 2.1 and 2.2. The example matrix shown is a scalar problem on a mesh with eight elements, decomposed into four partitions. The hypothetical architecture consists of four compute nodes, each with two CPUs with shared memory. For a two-stage IRK method, each partition of the mesh belongs to a different node, and each of the stages for a given partition belong to different CPUs within one node. 2.3 Numerical results 2.3.1 Euler vortex We solve the compressible Euler equations of gas dynamics, which are given in Section 1.4.3. We consider the model problem of an unsteady compressible vortex in a 46 Mesh Partition 1 Partition 2 Partition 3 Partition 4 Core 1 Core 1 Core 1 Core 1 Stage 1 Core 2 Core 2 Core 2 Core 2 Stage 2 Node 1 Node 2 Node 3 Node 4 Figure 2.1: Schematic of stage-parallel IRK solution strategy Node 1, CPU 1 Node 1, CPU 2 Node 2, CPU 1 Node 2, CPU 2 Node 3, CPU 1 Node 3, CPU 2 Node 4, CPU 1 Node 4, CPU 2 | {z }| {z } Stage 1 Stage 2 CPU 1 CPU 2 Nodes 1–4 Nodes 1–4  Figure 2.2: Stage-parallel partitioning of the matrix B = A−1 ⊗ M − ∆t J01 J02 . The color of the matrix entries indicates the node to which it belongs. The entries in the first block-column all belong to the first CPU of each node, and the entries the second block-column all belong to the second CPU of each node. 47 −6.6 6 4 −6.8 2 −7.0 0 −2 −7.2 −4 −7.4 −6 0.0 0.2 0.4 0.6 0.8 0 5 10 15 20 (a) Bottom-left corner of mesh, with p = 4 (b) Density contours for the initial conditions Figure 2.3: Compressible Euler vortex rectangular domain [132]. The domain is taken to be the rectangle [0, 20]×[−7.5, 7.5], and the vortex is initially centered at (x0 , y0 ) = (5, −2.5). The vortex is moving with the free-stream at an angle of θ. This problem is particularly useful as a benchmark because the exact solution is given by the following analytic formulas, allowing for convenient computation of the numerical accuracy. The exact solution at (x, y, t) is given by    ((y − y0 ) − vt) f (x, y, t) u = u∞ cos(θ) − exp , (2.14) 2πrc 2    ((x − x0 ) − ut) f (x, y, t) u = u∞ sin(θ) − exp , (2.15) 2πrc 2   γ−1 1 2 (γ − 1)M∞ 2 ρ = ρ∞ 1 − exp((f (x, y, t)) , (2.16) 8π 2   γ−1 γ 2 (γ − 1)M∞ 2 p = p∞ 1 − exp((f (x, y, t)) , (2.17) 8π 2 where f (x, y, t) = (1−((x−x0 )−ut)2 −((y −y0 )−vt)2 )/rc2 , M∞ is the Mach number, u∞ , ρ∞ , and p∞ are the free-stream velocity, density, and pressure, respectively. The free-stream velocity is given by (u, v) = u∞ (cos(θ), sin(θ)). The strength of the vortex is given by , and its size is rc . 48 For our test case, we choose parameters  = 15, rc = 1.5, M0 = 0.5, θ = arctan(1/2). In order for the temporal error to dominate the spatial error, we com- pute the solution on a fine mesh consisting of 6144 regular right-triangular elements depicted in Figure 2.3a. The finite element space is chosen to be piecewise polyno- mials of degree 4, corresponding to 15 nodes per element. The solution consists of four components, for a total of 368,640 degrees of freedom. The initial conditions are shown in Figure 2.3b. Comparison of preconditioners We first use this test case to compare the effectiveness of the preconditioners dis- cussed in Section 2.2.4. As a baseline, we will consider the DIRK solver using the block ILU(0) preconditioner. We will then compare the stage-coupled block ILU(0) and both shifted and unshifted stage-uncoupled ILU(0) preconditioners for the IRK solver. In all cases, we require Newton’s method to converge to an infinity-norm tol- erance of 10−8 . We compare the number of GMRES iterations required to converge to a relative, preconditioned tolerance of 10−5 . In order to make a fair comparison between the methods, we will compute the number of N × N matrix-vector mul- tiplications required per iteration of Newton’s method, across all the stages. We will refer to this quantity as the number of equivalent multiplications. For the DIRK methods, this number is equal to the number of GMRES iterations times the number of implicit stages. For the IRK methods, we recall that each multiplication by the large block matrix of the form (2.12) essentially consists of s N × N matrix-vector multiplications. We compute 5 time steps in serial using representative time steps of ∆t = 0.4, 0.3, 0.2, and 0.1. We then average the number of GMRES iterations required 49 Equivalent multiplications 160 RADAU23 (coupled) RADAU23 (uncoupled, shifted) 80 RADAU23 (uncoupled, unshifted) DIRK33 RADAU35 (coupled) 40 RADAU35 (uncoupled, shifted) RADAU35 (uncoupled, unshifted) ESDIRK65 20 0.1 0.2 0.3 0.4 ∆t Figure 2.4: Log-log plot of number of average number of equivalent multiplications vs. ∆t. Coupled preconditioners are shown in solid lines, uncoupled shifted precon- ditioners in dashed lines, and uncoupled, unshifted preconditioners in dotted lines. per linear solve, and multiply this number by the number of implicit stages. For the shifted stage-uncoupled block ILU(0) preconditioner, we choose a shift of αi = P −1 j6=i Aji , which has in our experience resulted in the fastest convergence. We present the results in the log-log plot shown in Figure 2.4. We notice that the third order Radau IIA method with the stage-coupled ILU(0) preconditioner re- quires fewer matrix-vector multiplications than the corresponding third-order DIRK method, while the stage-uncoupled, shifted ILU(0) preconditioner requires roughly the same number of multiplications. For the fifth-order methods, both the cou- pled and uncoupled preconditioners require more matrix-vector multiplications than the corresponding ESDIRK method. For both third- and fifth-order methods, the stage-uncoupled, unshifted preconditioner requires greatly more matrix-vector mul- tiplications than the other methods, and therefore in the further test cases we will only consider the shifted preconditioner. 50 Temporal accuracy Since the analytical solution to this test case is known, it is particularly convenient to compare the accuracy of the time discretization schemes. The solution is integrated until a final time of t = 60. For the third-order methods, we choose time steps of ∆t = 0.4, 0.3, 0.2, 0.1, 0.075, 0.05. Because of the increased accuracy of the fifth-order methods, we choose larger time steps of ∆t = 0.8, 0.6, 0.5, 0.4, 0.3 for the RADAU35 and ESDIRK65 methods. Time steps between 0.5 and 1.2 are chosen for the seventh- and ninth-order Radau IIA methods. We approximate the L∞ error by comparing the numerical solution with the known analytic solution at the DG nodes. We also estimate the order of accuracy by comparing successive choices of ∆t and computing the rate of convergence log(L∞ ∞ i+1 /Li ) ratei = , log(∆ti+1 /∆ti ) where L∞ i denotes the L ∞ error of the numerical solution computed using time step ∆ti . For each method, we present the wall-clock time required to compute the solution in parallel on 16 cores. The results for both the third-order and fifth-order methods are presented in Table 2.5. The theoretical order of accuracy is observed for all of the methods used up to seventh order. Also listed is the ratio of the DIRK error to the Radau IIA error. Comparing the coefficients presented in Table 2.1, it can easily be shown that the leading coefficient of the truncation error for the DIRK33 method is about 1.86 times larger than the leading coefficient for the RADAU23 method. We see that the ratio of the errors approaches this value as ∆t tends to zero. The leading coefficient of the truncation error for the ESDIRK65 methods is about 3.82 times larger than the leading coefficient for the RADAU35 method. In the fifth-order test case, the ratio of the numerical errors is found to be closer to about 1.5, likely because of the additional contribution of spatial discretization error. 51 Table 2.5: L∞ error and runtime for the Euler vortex, DIRK and Radau IIA methods (wall-clock time presented for stage-coupled/uncoupled preconditioners). RADAU23 DIRK33 ∆t L∞ error Order Time C/UC (s) L∞ error Order Time (s) Ratio 0.2 7.29 × 10−2 - 820/829 1.06 × 10−1 - 1096 1.452 0.1 9.71 × 10−3 2.91 965/956 1.61 × 10−2 2.72 1268 1.660 0.075 4.11 × 10−3 2.98 1227/1223 7.19 × 10−3 2.81 1618 1.747 0.05 1.22 × 10−3 2.99 1734/1746 2.22 × 10−3 2.90 2291 1.811 0.025 1.78 × 10−4 2.78 3214/3322 3.01 × 10−4 2.88 4212 1.691 RADAU35 ESDIRK65 ∆t L∞ error Order Time C/UC (s) L∞ error Order Time (s) Ratio 0.6 5.86 × 10−2 - 843/775 8.33 × 10−2 - 697 1.421 0.5 2.42 × 10−2 4.84 974/810 3.61 × 10−2 4.59 802 1.488 0.4 7.89 × 10−3 5.03 862/905 1.26 × 10−2 4.71 947 1.598 0.3 1.82 × 10−3 5.11 1079/1030 3.07 × 10−3 4.91 1197 1.692 0.2 2.73 × 10−4 4.67 1523/1478 4.13 × 10−4 4.95 1121 1.512 RADAU47 RADAU59 ∆t L∞ error Order Time C/UC (s) ∆t L∞ error Order Time C/UC (s) −2 −3 1.0 1.87 × 10 - 1222/1029 1.2 5.04 × 10 - 2117/1533 0.8 4.54 × 10−3 6.34 1158/1016 1.0 1.73 × 10−3 5.87 2094/1567 0.6 8.73 × 10−4 5.73 1463/1307 0.8 4.70 × 10−4 5.84 1988/1554 0.5 3.03 × 10−4 5.81 1732/1541 0.6 8.87 × 10−5 5.79 2544/2040 A log-log plot of the L∞ error vs. ∆t is shown in Figure 2.5a. A log-log plot of the L∞ error vs. wall-clock time is shown in Figure 2.5b. We remark that the RADAU23 method achieved the same accuracy as the DIRK33 method in faster runtime for all of the cases considered. Among the fifth-order methods, we did not observe one method to be clearly more efficient than the others. The difference between the stage-coupled and stage-uncoupled preconditioners was found to be insignificant for the third- and fifth-order methods, and the stage-uncoupled preconditioner resulted in faster performance for the seventh- and ninth-order methods. 52 10−1 10−1 9 8 10−2 10−2 L∞ error L∞ error 9 8 10−3 10−3 98 3 5 9 98 10−4 1 1 1 10−4 10−1 100 700 1,000 1,500 2,250 3,370 ∆t Wall-clock time (s) RADAU23 DIRK33 RADAU23 (C) RADAU35 (C) RADAU35 ESDIRK65 RADAU23 (UC) RADAU35 (UC) RADAU47 RADAU59 DIRK33 ESDIRK65 8 RADAU47 (C) RADAU59 (C) (a) Log-log plot of L∞ error vs. 1/∆t 9 RADAU47 (UC) RADAU59 (UC) (b) Log-log plot of L∞ error vs. wall-clock time Figure 2.5: Log-log plots of L∞ error vs. time step and wall-clock time for Euler vor- tex test case. Stage-coupled preconditioners are shown in solid lines, stage-uncoupled in dashed lines. 2 0.035 0.030 1 0.025 0 0.020 0.015 −1 0.010 −2 0.005 0.000 −2 0 2 4 −0.51 −0.50 −0.49 −0.48 −0.47 −0.46 (a) Mesh of NACA airfoil with 3154 elements (b) Boundary layer elements with p = 3 Figure 2.6: Triangular mesh of NACA airfoil 2.3.2 High Reynolds number flow over 2D NACA airfoil In this test case, we consider the two-dimensional viscous flow around a NACA airfoil with an angle of attack of 30◦ . The fluid domain is the rectangle [−2.5, 4]×[−2.5, 2.5] 53 with the airfoil removed. The governing equations are the compressible Navier-Stokes equations given by (1.62) with the isentropic assumption discussed in Section 1.4.4. For this case, the Mach number is taken to be 0.1, the gas constant 1.4, and the Reynolds number 40,000. The wing is centered vertically, and placed closer to the inlet boundary. The mesh consists of 3154 triangular elements, with finer elements close to the airfoil and in the wake. The mesh and the boundary-layer elements with DG nodes are depicted in Figure 2.6, and are rotated by 30◦ to correspond with the desired angle of attack. The triangles near the boundary of the wing are refined in the transverse direction, resulting in highly anisotropic elements. These stretched elements give rise to a highly restrictive CFL condition, suggesting that this problem is particularly well suited to implicit methods. We have found that the CFL condition renders explicit methods impractical for this problem, with the fourth-order explicit Runge-Kutta method exhibiting instability for time steps greater than 7 × 10−8 . On the other hand, the implicit Runge-Kutta methods remain stable for time steps many orders of magnitude larger. A no-slip condition is enforced on the boundary of the airfoil, and far-field con- ditions are enforced on all other boundaries. At time t = 0 the solution is set to be free-stream everywhere. The solution is then integrated until time t = 5, at which point vortices have developed in the wake of the wing. This solution (shown in Figure 2.7a) is taken to be the initial condition for our test case. The finite element space is taken to be piecewise degree 3 polynomials, with 10 nodes per element, resulting in 94,620 degrees of freedom. 54 2 2 1 1 0 0 −1 −1 −2 −2 −2 0 2 4 −2 0 2 4 (a) Initial condition, t = 5 (density) (b) Solution, t = 5.75 (density) Figure 2.7: Density plots for NACA Re = 40k test case Solver efficiency We study the effectiveness of the DIRK and Radau IIA IRK methods by compar- ing both the average number of equivalent multiplications per linear solve and the total wall-clock time. As in the previous case, the methods RADAU23, RADAU35, DIRK33, and ESDIRK65 are used. As in the case of the Euler vortex, a tol- erance of 10−8 is used for the Newton solver. We integrate the equations from time t = 5 until t = 5.75. For the third-order methods, time steps of ∆t = 1.25 × 10−2 , 7.50 × 10−3 , 6.25 × 10−3 , 5.00 × 10−3 , 2.50 × 10−3 , 1.25 × 10−3 , are used. For the fifth-order methods, we use the same time steps, in addition to the larger time steps of ∆t = 5.00 × 10−2 , and ∆t = 2.50 × 10−2 . In Table 2.6 we present the runtime and number of equivalent multiplications for all of the methods considered. As in Section 2.3.1, we compute the number of N × N matrix-vector products per- formed per Newton iteration (over all the stages) by multiplying the average number of GMRES iterations by the number of implicit stages. In Figure 2.8a we present a log-log plot of the average number of equivalent multiplications vs. ∆t. In Figure 2.8b we present the total wall-clock time required to run the simulation until the 55 Table 2.6: Equivalent multiplications and wall-clock time for NACA LES test case RADAU23 (Coupled) RADAU23 (Uncoupled) DIRK33 ∆t Mult. Time (s) Mult. Time (s) Mult. Time (s) 1.25 × 10−2 70.2 75.4 72.0 72.1 100.2 92.1 7.50 × 10−3 53.4 96.5 56.0 93.1 81.3 133.0 6.25 × 10−3 48.8 105.2 52.4 105.5 73.2 145.7 5.00 × 10−3 43.0 116.5 46.2 117.1 65.1 162.9 2.50 × 10−3 34.0 208.9 36.8 202.7 42.6 238.0 1.25 × 10−3 27.2 389.8 30.8 386.7 33.6 437.5 RADAU35 (Coupled) RADAU35 (Uncoupled) ESDIRK65 ∆t Mult. Time (s) Mult. Time (s) Mult. Time (s) 5.00 × 10−2 189.2 59.7 178.5 49.4 216.0 51.3 2.50 × 10−2 132.6 79.8 130.2 69.2 173.5 81.0 1.25 × 10−2 107.7 141.0 107.1 113.4 137.0 136.6 7.50 × 10−3 96.0 189.2 96.0 169.8 107.5 184.6 6.25 × 10−3 89.1 222.0 93.3 200.7 95.0 197.2 5.00 × 10−3 78.6 230.4 88.8 263.2 81.5 217.8 2.50 × 10−3 54.9 340.2 73.5 461.7 60.0 375.9 1.25 × 10−3 42.6 619.4 60.6 836.3 47.0 720.1 230 800 Equivalent multiplications 153 Wall-clock time (s) 400 102 200 68 45 100 30 50 0.00125 0.0025 0.00625 0.0125 0.025 0.05 0.00125 0.0025 0.00625 0.0125 0.025 0.05 ∆t ∆t RADAU23 (Coupled) RADAU35 (Coupled) RADAU23 (Coupled) RADAU35 (Coupled) RADAU23 (Uncoupled) RADAU35 (Uncoupled) RADAU23 (Uncoupled) RADAU35 (Uncoupled) DIRK33 ESDIRK65 DIRK33 ESDIRK65 (a) Matrix-vector multiplications vs. ∆t (b) Wall-clock time vs. ∆t Figure 2.8: Log-log plots of average number of equivalent multiplications and wall- clock time vs. ∆t for the 2D NACA LES test case final time. For the third-order methods, the RADAU23 IRK method required fewer matrix- 56 Table 2.7: L∞ error for NACA LES test cases RADAU23 DIRK33 ∆t L∞ Order L∞ Order Ratio 1.25 × 10−2 6.006 × 10−1 - 4.514 × 10−1 - 1.668 7.50 × 10−3 1.743 × 10−1 2.42 1.615 × 10−1 2.01 1.706 6.25 × 10−3 1.207 × 10−1 2.02 1.423 × 10−1 0.70 1.717 5.00 × 10−3 8.503 × 10−2 1.57 1.216 × 10−1 0.70 1.735 2.50 × 10−3 1.347 × 10−2 2.66 2.022 × 10−2 2.59 1.839 1.25 × 10−3 1.342 × 10−3 3.33 2.708 × 10−3 2.90 1.823 RADAU35 ESDIRK65 ∆t L∞ Order L∞ Order Ratio 5.00 × 10−2 4.601 × 10−1 - 6.239 × 10−1 - 2.308 2.50 × 10−2 3.158 × 10−1 0.54 5.853 × 10−1 0.092 1.896 1.25 × 10−2 6.739 × 10−2 2.23 1.127 × 10−1 2.377 2.414 7.50 × 10−3 5.124 × 10−3 5.04 3.989 × 10−3 6.540 4.441 6.25 × 10−3 5.329 × 10−3 -0.22 2.147 × 10−3 3.397 3.263 5.00 × 10−3 2.616 × 10−3 3.19 3.279 × 10−3 -1.897 3.200 2.50 × 10−3 8.135 × 10−4 1.68 2.222 × 10−3 0.561 1.044 1.25 × 10−3 2.059 × 10−4 1.98 1.727 × 10−4 3.686 0.896 vector multiplications than the DIRK33 method, for the same choice of ∆t, resulting in a shorter run time. The uncoupled preconditioner resulted in a somewhat larger number of GMRES iterations, and hence more matrix-vector multiplications, but the difference in run time was found to be negligible. For the fifth-order methods, the RADAU35 method with stage-coupled preconditioner resulted in a smaller number of matrix-vector multiplications per linear solve than the ESDIRK65 method. For larger ∆t, the stage-uncoupled preconditioner proved to be effective, resulting in faster run times than both the DIRK method and the stage-coupled preconditioner. For smaller ∆t, the stage-uncoupled preconditioner required a greater number of GMRES iterations and hence longer run times. 57 100 100 10−1 10−1 L∞ error L∞ error 10−2 10−2 10−3 10−3 10−4 80 120 180 270 405 60 120 240 480 960 Wall-clock time (s) Wall-clock time (s) RADAU23 (Coupled) DIRK33 RADAU35 (Coupled) ESDIRK65 RADAU23 (Uncoupled) RADAU35 (Uncoupled) Figure 2.9: Log-log plots of L∞ errors vs. wall-clock time for the NACA LES test case Accuracy We note that the above comparisons were made for equal choices of ∆t. Because the Radau IIA method enjoy a smaller leading coefficient of the truncation error, we expect to achieve better accuracy for the same time step. We therefore study the accuracy of the methods applied to the above problem by considering the semidiscrete system of equations purely as a system of ODEs. We compute a reference solution by numerically integrating the equations for 6000 time steps with ∆t = 1.25 × 10−4 using the fifth-order ESDIRK method. Then, we take this solution to be the “exact” solution, with respect to which the L∞ norm of the error is computed, and perform a grid convergence study. For each choice of time step, we compute the L∞ norm of the error, the approximate rate of convergence of the method, and the ratio of the DIRK error to the Radau IIA error in Table 2.7. In Figure 2.9, we present log-log plots of the wall-clock time vs. L∞ error. We note that we do not observe the formal order of temporal accuracy for this test problem, possibly due to the choice of time step, which is about five orders of magnitude larger than the explicit CFL, together with 58 Table 2.8: L∞ error for NACA LES test cases (order verification) RADAU23 DIRK33 ∆t L∞ Order L∞ Order 6.25 × 10−5 5.694 × 10−6 - 3.580 × 10−6 - 3.125 × 10−5 8.765 × 10−7 2.70 5.004 × 10−7 2.84 1.5625 × 10−5 1.177 × 10−7 2.90 6.401 × 10−8 2.97 7.8125 × 10−6 1.494 × 10−8 2.98 8.033 × 10−9 2.99 RADAU35 ESDIRK65 ∆t L∞ Order L∞ Order 1.25 × 10−4 3.438 × 10−6 - 1.231 × 10−6 - 6.25 × 10−5 1.621 × 10−7 4.41 4.758 × 10−8 4.69 3.125 × 10−5 5.434 × 10−9 4.90 1.602 × 10−9 4.89 1.5625 × 10−5 1.964 × 10−10 4.79 7.913 × 10−11 4.34 the stiff and turbulent nature of the problem. In order to verify the formal order of accuracy, we also run this test problem with time steps that are two to three orders of magnitude smaller than those of the above comparison (for a shorted total time of integration). The results presented in Table 2.8 confirm that the expected theoretical orders of accuracy are attained for all methods considered. We remark that for the third-order methods, we can achieve the same accuracy as the DIRK33 method with a faster run time using the RADAU23 method, with both the stage-coupled or stage-uncoupled preconditioner. The differences in run time between the two preconditioners were negligible. For the fifth-order methods, our results indicate that the RADAU35 method outperformed the ESDIRK for a majority of the test cases considered. For this test problem, we found the stage- uncoupled preconditioner to perform better for the larger choices of ∆t. 59 Equivalent multiplications 120 RADAU23 (SP) RADAU35 (SP) RADAU23 (C) RADAU35 (C) 60 RADAU23 (UC) RADAU35 (UC) DIRK33 ESDIRK65 30 1 2 4 8 16 32 64 128 256 512 1024 Number of processes Figure 2.10: Log-log plot of average number of equivalent multiplications vs. number of processes. Dashed lines indicate stage-uncoupled block ILU(0) preconditioner, and dotted lines indicate stage-parallel solver. Parallel performance on NACA airfoil We study the parallel performance of the IRK and DIRK solvers applied to the two- dimensional NACA airfoil. We choose a representative time step of ∆t = 1.25×10−2 , and integrate the system for five time steps until t = 5.0625, using as before the solution at t = 5 as the initial condition. We perform this test using both the Radau IIA IRK and the DIRK solvers. For the Radau methods, we use both the stage-coupled and stage-uncoupled ILU(0) preconditioners. Using the method described in Section 2.2.6, we decompose the domain into a set number of partitions according to the number of processes. For the DIRK and stage- coupled IRK solvers, the number of partitions is equal to the number of processes. For the stage-uncoupled IRK solver, we can choose the number of partitions to be a factor of s smaller than the number of processes. We consider the mesh decomposed into 1, 2, 4, 8, 16, 32, 64, 128, 256, and 512 partitions. Then, we compute the average number of GMRES iterations required per solve. In the case of the DIRK methods, we multiply the number of iterations by the number of implicit stages to obtain 60 the number of equivalent multiplications performed assuming one Newton iteration. Similarly, in the case of the Radau IIA IRK methods, we multiply the number of iterations by the number of stages to obtain the number of N × N matrix-vector multiplications. In Figure 2.10 we show a log-log plot of the average number of equivalent multiplications vs. number of processes. As the number of processes (and hence number of mesh partitions) increases, we observe an increase in the number of GMRES iterations required to converge. This is because the contributions between different mesh partitions are ignored in the block ILU(0) factorization, rendering the preconditioner less effective. Since the stage-uncoupled ILU(0) preconditioner can be parallelized for the same number of processes with a factor of s fewer partitions, we notice that approximately 15–20% fewer matrix-vector multiplications are required when compared with the standard uncoupled solver. This suggests a substantial benefit to the stage-uncoupled block ILU(0) preconditioner when run in a massively-parallel environment. This difference in performance is numerically validated in the following three-dimensional NACA test case. 2.3.3 Parallel large eddy simulation of 3D NACA airfoil For our final test case, we consider the three-dimensional viscous flow over a NACA airfoil with angle of attack of 30◦ . The Reynolds number is taken to be 5,300 and the Mach number 0.1. The governing equations are the compressible Navier-Stokes equations given by (1.62) with the isentropic assumption discussed in Section 1.4.4. The mesh consists of 151,392 tetrahedral elements. The local basis consists of degree 3 polynomials, for a total of 20 nodes per element. We consider the fluid to be isentropic, and hence the solution consists of 4 components, resulting in a total of 61 (a) Boundaries of three-dimensional NACA (b) Isosurfaces of Q-criterion, Q = 25, colored mesh. by velocity magnitude. Figure 2.11: Three-dimensional NACA LES test case 12,111,360 degrees of freedom. The mesh is shown in Figure 2.11a. A no-slip condition is enforced on the boundary of the airfoil, periodic conditions are enforced in the span-wise direction, and far-field conditions are enforced on all other boundaries. The solution is initialized to freestream conditions, and then integrated numerically until time t = 4 using ∆t = 0.02. At this point, vortices have developed in the wake of the wing. In Figure 2.11b, isosurfaces of the Q-criterion, for Q = 25, are shown. The Q-criterion, proposed by Hunt et al. in [65] is often used to identify vortical structures, and is defined as the difference of the symmetric and antisymmetric components of the velocity gradient, 1 Q= (Ωij Ωij − Sij Sij ) , (2.18) 2 where Ωij = 21 (ui,j − uj,i ), and Sij = 12 (ui,j + uj,i ). We then integrate the equations for 30 time steps using the third- and fifth-order DIRK methods, as well as the Radau IIA methods of order 3, 5, 7, and 9. Due to 62 32 Linear solve time (s) RADAU23 (SP) RADAU35 (SP) 16 8 RADAU23 (C) RADAU35 (C) 8 RADAU23 (UC) RADAU35 (UC) 9 DIRK33 ESDIRK65 8 9 8 9 9 RADAU47 (SP) RADAU59 (SP) 9 9 8 9 8 RADAU47 (C) RADAU59 (C) 4 9 9 9 RADAU47 (UC) RADAU59 (UC) 1 2 1 360 540 720 1,080 Number of processes Figure 2.12: Log-log plot of linear solve time in seconds vs. number of processes. Dashed lines indicate stage-uncoupled block ILU(0) preconditioner, and dotted lines indicate stage-parallel solver. A reference triangle for perfect speedup is shown. the turbulent nature of this problem, we do not study the accuracy of the numerical solutions, but rather the efficiency of the solvers for fixed ∆t. We also consider the parallel scaling of the solvers by running our test case on 360, 540, 720, and 1080 processes. For the Radau IIA methods, we consider three preconditioners: stage- coupled, stage-uncoupled, and stage-parallel. As with the DIRK methods, for the stage-coupled and stage-uncoupled preconditioners we decompose the mesh into a number of partitions equal to the number of processes. For the stage-parallel ILU(0) preconditioner we decompose the mesh into a factor of s fewer partitions. The preconditioner then exploits the stage parallelism using the methodology described in Section 2.2.6. We record the average wall-clock time required per linear solve using each of the methods, and present the results in Figure 2.12. For each order of accuracy, the Radau IIA method with the stage-parallel pre- conditioner resulted in the fastest runtime. For orders three and five, for which we compare against equal-order DIRK methods, the Radau IIA methods resulted in faster performance with all of the preconditioners considered. Since the number of stages is greater for the higher order methods, and the stage-parallel preconditioner 63 allows for a factor of s fewer mesh partitions, we would expect that the reduction in the number of GMRES iterations due to the improved ILU preconditioner would be greater than for the lower order methods. Indeed, we observe that the relative perfor- mance gain of the stage-parallel preconditioner compared with the stage-uncoupled preconditioner increases as the the order of the method increases. For example, while the third-order, stage-parallel preconditioner is only about 5–10% faster than the stage-uncoupled preconditioner, and about 20% faster than the stage-coupled pre- conditioner, the ninth-order stage-parallel preconditioner is between 20–30% faster than the stage-uncoupled preconditioner, and approximately twice as fast as the stage-coupled preconditioner. Furthermore, as the number of processes approaches the strong scaling limit, we anticipate that the additional factor of s processes allowed by the stage-parallel preconditioner will result in even greater benefit, especially for those methods with a large number of stages. 2.4 Summary In this chapter, we have developed a new strategy for efficiently solving the large, cou- pled linear systems arising from fully implicit Runge-Kutta time discretizations. By transforming the system of equations, the computational work required per GMRES iterations is reduced significantly. This new method makes it feasible to use high- order, L-stable implicit Runge-Kutta methods, such as the Radau IIA methods, as time integrators for discontinuous Galerkin discretizations. We additionally develop new preconditioners for these methods, included a parallel-in-time ILU precondi- tioner that allows for the computation of the Runge-Kutta stages simultaneously. Numerical experiments on both two- and three-dimensional fluid flow problems 64 are performed using the Radau IIA methods of up to ninth order. These results indi- cate that using the transformed system of equations, the fully implicit IRK methods are competitive with, and in our experience, often preferable to the more standard DIRK methods, both in terms of efficiency and accuracy. In a parallel computing environment, the stage-parallel ILU preconditioner results in additional performance gains. Chapter Three Tensor-product preconditioners 66 One of the most attractive features of the discontinuous Galerkin method is the straightforward generalization to arbitrarily high formal order of accuracy together with the ability to use general, unstructured meshes with complex geometry. In particular, the promise of a high-order method for fluid flow problems has spurred much recent interest in the DG method [99]. Higher-order methods promise highly- accurate solutions for less computational cost than traditional low-order methods. Additionally, high-order methods are more computationally intensive per degree of freedom than corresponding low-order methods, resulting in a higher computation- to-communication ratio, and thus rendering these method more amenable to paral- lelization [5]. High-order accuracy is achieved with the DG method by using a high-degree local polynomial basis on each element in the mesh. There are several challenges that can prevent the use of very high-degree polynomials as basis functions. The number of degrees of freedom per element scales as O(pd ), where p is the degree of polynomial approximation, and d is the spatial dimension, resulting in very computationally ex- pensive methods. Using tensor-product evaluations and sum factorizations [91], it is possible to reduce the computational cost of these methods, however, the spectrum of the semi-discrete operator grows at a rate bounded above by (p + 1)(p + 2)/h, and well approximated by (p + 1)1.78 /h where p is the degree of polynomial ap- proximation, and h is the element size [57, 133]. As a result, when using explicit time integration schemes, the time step must satisfy a restrictive stability condition given by (approximately) ∆t ≤ Ch/(p + 1)1.78 [72]. On the other hand, the DG method couples all the degrees of freedom within each element, so that implicit time integration methods result in block-structured systems of equations, with blocks of size pd × pd . Strategies for solving these large linear systems include Newton-Krylov iterative solvers coupled with an appropriate preconditioner [107]. 67 Many of the standard preconditioners require the inversion of large the pd × pd blocks corresponding to each element. Using dense linear algebra, this requires O(p3d )operations, which quickly becomes intractable. One approach to reduce the computational complexity of implicit methods is to combine Kronecker and sum- factorization techniques with a matrix-free approach. Matrix-free approaches for the DG method have been considered in e.g. [35] and [71]. Past work on efficiently pre- conditioning these systems includes the use of alternating-direction-implicit (ADI) and fast diagonalization method (FDM) preconditioners [41]. Kronecker-product ap- proaches have been studied in the context of spectral methods [117], and applications to the Navier-Stokes equations were considered in [47]. In this chapter, we describe a new approximate Kronecker-product preconditioner that, when combined with a matrix-free tensor product evaluation approach, allows for efficient solution of the linear systems that arise from implicit time discretizations for high polynomial degree DG methods. This preconditioner requires tensor-product bases on quadrilateral or hexahedral elements. Then, the pd × pd blocks that arise in these systems can be well-approximated by certain Kronecker products of one dimensional p × p matrices. Using a shuffled singular value decomposition introduced by Van Loan in [128], it is possible to compute decompositions into tensor products of one-dimensional terms that are optimal in the Frobenius norm. Using these techniques, it is possible to construct an approximate tensor-product version of the standard block Jacobi pre- conditioner, that avoids inverting, or even storing, the large diagonal blocks of the Jacobian matrix. In the remainder of this chapter, we will consider a quadrilateral or hexahedral mesh, with a tensor-product basis, as described in Section 1.1. In Section 3.1, we outline the sum-factorization approach, and describe equivalent Kronecker-product representations. Then, in Section 3.2 we develop the approximate Kronecker-product 68 preconditioners, and provide a new set of algorithms that can be used to efficiently compute and apply these preconditioners. In Section 3.3, we discuss the gener- alization of these preconditioners to second-order elliptic and parabolic problems. Finally, in Section 3.4, we apply these preconditioners to several test problems, in- cluding the scalar advection equation, compressible Navier-Stokes equations, and the Euler equations of gas dynamics, in two and three spatial dimensions. 3.1 The sum-factorization approach As described in Section 1.3, we represent the numerical solution uh as a vector u, whose entries represent the coefficients of basis functions local to each element. In Rd , the number of degrees of freedom m per element thus scales as O(pd ). In this work, we will make the assumption that the number of quadrature points required to compute integrals over elements Kj ∈ Th , denoted µ, is given by a constant multiple of m, and thus also O(pd ). We first note that in order to approximate integrals of the form Z f (vh ) dx, (3.1) Kj for arbitrary f , we must evaluate the given function vh ∈ Vh at each of the quadrature nodes in the element Kj . Recalling the notation from Section 1.3, we let the basis for the local function space Vh (Kj ) be given by {φe1 , . . . , φem }. Then, for any x ∈ Kj , we can expand vh in terms of its coefficients vi , m X vh (x) = vi φei (x). (3.2) i=1 It is important to note that each computation of vh (x) thus requires n evaluations of the basis functions for any specified point x. Performing this computation for 69 each quadrature point therefore requires a total of O(p2d ) evaluations. In order to reduce the computational cost of this, and other operations, we describe the sum- factorization approach, first introduced in [91], and extended to the DG method in e.g. [131]. 3.1.1 Tensor-product elements The key to the sum-factorization approach is the use of tensor-product elements. We consider a block mesh, where each element Kj in the triangulation Th is given as the image of the cartesian product [0, 1]d under a transformation mapping. Then, the local basis functions for each element Kj are given as the product of one-dimensional basis functions. To be precise, we let {φi (x)}pi=0 be a basis for P p ([0, 1]), the space of polynomials of degree at most p on the unit interval. These basis functions are given by the Lagrange interpolating polynomials for p + 1 Gauss-Lobatto points in the unit interval, which we denote as B = (b0 , b1 , . . . , bp ). Since the nodal interpolation points for the reference element R = [0, 1]d are given by the d-fold cartesian product B × · · · × B, the corresponding basis functions for Vh (R) can be written as a tensor- product of the corresponding one-dimensional basis functions. That is to say, we define basis functions Φi1 i2 ···id (x1 , x2 , . . . , xd ) = φi1 ⊗ φi2 ⊗ · · · ⊗ φid (x1 , x2 , . . . , xd ) (3.3) = φi1 (x1 )φi2 (x2 ) · · · φid (xd ), and thus can write Vh (R) = span{Φi1 i2 ···id : 0 ≤ ik ≤ p}. (3.4) 70 Each basis function Φi1 i2 ···id is the unique multivariate polynomial in Vh (R) satisfying Φi1 i2 ···id (bj1 , bj2 , . . . , bjd ) = δi1 j1 δi2 j2 · · · δid jd . (3.5) Thus, the basis functions for the transformed element Kj = Tj (R) are given by e i1 i2 ···i = Φi1 i2 ···i ◦ T −1 . Φ d d j Given this choice of basis, we also define the quadrature nodes on the d-dimensional unit cube to be the d-fold cartesian product of given one-dimensional quadrature nodes, whose weights are the corresponding products of the one-dimensional weights. Equipped with these choices, we return to the calculation of the quantity (3.2). For the sake of concreteness, we consider the case where d = 3, for which the cal- culation above should naively require O(p6 ) evaluations. We suppose that the one- dimensional quadrature points are given as x1 , x2 , . . . , xµ , and hence we can write the three-dimensional quadrature points as xα,β,γ = (xα , xβ , xγ ), for all 1 ≤ α, β, γ ≤ µ. We then factor the summation in (3.2) to obtain p+1 X vh (xα,β,γ ) = vijk Φijk (xα , xβ , xγ ) i,j,k=1 p+1 p+1 p+1 X XX = vijk φi (xα )φj (xβ )φk (xγ ) (3.6) k=1 j=1 i=1 p+1 p+1 p+1 X X X = φk (xγ ) φj (xβ ) vijk φi (xα ). k=1 j=1 i=1 We notice that the index of each summation ranges over p + 1 values, and there are three free indices in each sum. Thus, the total number of operations required to evaluate a function vh at each of the quadrature points is O(p4 ). For general dimension d, this computation requires O(pd+1 ) basis function evaluations. Thus, the computational work per degree of freedom is linear in the degree p of polynomial 71 basis, in contrast to the original estimate of O(pd ) computational work per degree of freedom, which is exponential in spatial dimension d. In a similar fashion, the tensor- product structure of this function space can be exploited in order to compute the integrals in the discontinuous Galerkin formulation (1.17) in linear time per degree of freedom. 3.1.2 Kronecker-product structure The sum-factorization procedure shown in (3.6) can be described simply and ele- gantly as a linear-algebraic Kronecker product structure. We recall that the Kro- necker product of matrix Ak×` and B m×n (whose dimensions are indicated by the superscripts), is the km × `n matrix C defined by    a11 B a12 B · · · a1` B     a21 B a22 B · · · a2` B    A⊗B =C = . .. ..  . (3.7)  .. ..   . . .    ak1 B ak2 B · · · ak` B The Kronecker product has many desirable and useful properties, enumerated in Van Loan’s exposition [129]. We can define the one-dimensional Gauss point evaluation matrix as the µ×(p+1) Vandermonde-type matrix obtained by evaluating each of the one-dimensional basis functions at all of the quadrature points, Gαj = φj (xα ), (3.8) and, in a similar fashion, it is also convenient to define the one-dimensional differ- 72 entiation matrix, D, whose entries are given by Dαj = φ0j (xα ). We now describe the Kronecker-product structure of a general d-dimensional DG method, for arbitrary d. Let i = (i1 , i2 , . . . , id ) and α = (α1 , α2 , . . . , αd ) be multi-indices of length d. Then, we can define a vector v of length (p + 1)d whose entries are given by concatenating the entries of the dth-order tensor vi . Thus, we obtain the values of vh evaluated at the quadrature points xα by computing the d-fold Kronecker product vh (xα ) = (G ⊗ G ⊗ · · · ⊗ G)v. (3.9) This Kronecker-product representation is computationally equivalent to the sum- factorized version from the preceding section. Indeed, many of the operations needed for the computation of the discontinuous Galerkin method are amenable to being written in the form of Kronecker products. For instance, it is often useful to approximate quantities of the form Z f vh dx, (3.10) K where f is an arbitrary function whose value is known at the appropriate quadrature nodes. This requires approximating the integrals Z e i dx fΦ (3.11) K e i . We consider the element K to be the image under for each of the basis functions Φ the isoparametric transformation map T of the reference element R = [0, 1]d . In this notation, for all x ∈ K, x = T (ξ), where ξ ∈ R. We then write (3.11) as an integral 73 over the reference element, Z Z e i (x) dx = f (x)Φ f (T (ξ))Φi (ξ)| det(T 0 (ξ))| dξ. (3.12) K R To this end, we define a diagonal weight matrix W by whose entries along the diagonal are given by wα , where wα is the quadrature weight associated with the point xα . Additionally, we define the µd × µd diagonal matrix JT whose entries are equal to the absolute Jacobian determinant of the isoparametric mapping | det(T 0 (ξ))| at each of the quadrature points. Then, the (p + 1)d integrals of the form (3.12) can be found as the entries of the vector    GT W ⊗ GT W ⊗ · · · ⊗ GT W JT f (xα ). (3.13) In a similar fashion, the computation of all of the quantities needed to formulate a DG method can be written in Kronecker form. In Table 3.1, we summarize the Kronecker-product formulation of several other important operations needed for the DG method, for the special cases of d = 2 and d = 3. 3.1.3 Explicit time integration It is important to note that all of the above operations have a computational com- plexity of at most O(pd+1 ). In other words, the cost of these operations scales linearly in p per degree of freedom. We now return to the semi-discrete system of equations (1.17), which we rewrite as M ut = r(u). (3.14) Then, using the operations described above, it is possible to compute all the inte- grals required to form the residual vector r. The final remaining operation required 74 Table 3.1: Kronecker-product form of DG operations Operation 2D 3D Evaluate solution at (G ⊗ G) u (G ⊗ G ⊗ G) u quadrature points   Integrate function f GT W ⊗ GT W JT f GT W ⊗ GT W ⊗ GT W JT f (known at quadrature points) against test functions   Integrate function GT W ⊗ DT W JT f1 GT W ⊗ GT W ⊗ DT W JT f1   f = (f1 , . . . , fd ) DT W ⊗ GT W JT f2 GT W ⊗ DT W ⊗ GT W JT f2 against gradient of  DT W ⊗ GT W ⊗ GT W JT f3 test functions in order to integrate this semi-discrete equation explicitly in time is to invert the mass matrix M . We remark that, with a tensor-product basis, we can compute the element-wise mass matrix (on, e.g., element K) as  Mj = (GT W ) ⊗ · · · ⊗ (GT W ) JT (G ⊗ · · · ⊗ G) (3.15) where G is the Gauss point evaluation matrix defined above, W is the diagonal matrix with the one-dimensional quadrature weights on the diagonal, and JT is the (µd ) × (µd ) diagonal matrix whose entries are equal to the absolute Jacobian determinant of the element transformation map, evaluated at each of the quadrature points. One strategy, proposed in [73], is to use the same number of quadrature points as DG nodes, such that µ = p + 1. In that case, all of the matrices appearing in 75 (3.15) are square, and we can compute   Mj−1 = G−1 ⊗ · · · ⊗ G−1 JT−1 (GT W )−1 ⊗ · · · ⊗ (GT W )−1 . (3.16) Since G and GT W are both (p + 1) × (p + 1) matrices, these operations can be performed in O(p3 ) time. Additionally, JT is a (p + 1)d × (p + 1)d diagonal matrix, and thus can be inverted in (p + 1)d operations. (On a practical note, in this case we would avoid explicitly forming the inverse matrices G−1 and (GT W )−1 , and would instead opt to form their LU factorizations). Thus, the linear system (3.14) can be solved in the same complexity as multiplying by the expression on the right-hand side of (3.16), i.e. O(pd+1 ). A serious drawback to this approach is that using the same number of quadrature points as DG nodes does not, in general, allow for exact integration of the quantity Z uh vh dx, uh , vh ∈ Vh , (3.17) K because of the use of isoparametric elements, where the Jacobian determinant of the transformation mapping may itself be a high-degree polynomial. In order to address this issue, we introduce a new strategy for solving the system (3.14). We first note that the global mass matrix has a natural element-wise block-diagonal structure, where, furthermore, each block Mj is a symmetric positive-definite matrix. Thus, we can solve this system of equations element-by-element, using the preconditioned conjugate gradient (PCG) method [64]. As a preconditioner, we use the under- integration method described above. Thus, each iteration in the PCG solver requires a multiplication by the exact mass matrix, and a linear solve using the approximate, under-integrated mass ma- 76 trix, given by equation (3.16), for the purposes of preconditioning. Both of these operations are performed in O(pd+1 ) time by exploiting their tensor-product struc- ture. This has the consequence that if the element transformation mapping is bilinear (and so the corresponding element is straight-sided) then its Jacobian determinant is linear, and with the appropriate choice of quadrature points, the integral (3.17) can be computed exactly, and the PCG method will converge within one iteration. In practice, we observe that the number of PCG iterations required to converge is very small even on curved, isoparametric meshes, and does not grow with p. The techniques described above are sufficient to implement an explicit discontin- uous Galerkin method with tensor-product elements, requiring O(pd+1 ) operations per time step. The main restriction to using such explicit methods with very high polynomial degree p is the restrictive CFL condition. It has been shown that the rate of growth of the spectral radius of the semi-discrete DG operator is bounded above by (p + 1)(p + 2)/h [57, 133], and well-approximated by (p + 1)1.78 [72]. This requires that the time step satisfy approximately approximately ∆t ≤ Ch/(p + 1)1.78 , which can prove to be prohibitively expensive as the number of time steps needed increases. For this reason, we are interested in applying some of the same tensor-product tech- niques to efficiently integrate in time implicitly, and thus avoid the overly-restrictive CFL condition. 3.1.4 Implicit time integration Instead of an explicit time integration method, we now consider an implicit schemes such as backward differentiation formulas (BDF) or diagonally-implicit Runge-Kutta (DIRK) methods. The main advantage of such methods is that they remain stable for larger time steps, even in the presence of highly anisotropic elements. Addi- 77 tionally, these methods avoid the restrictive p-dependent explicit stability condition mentioned above. Such implicit methods typically require the solution of algebraic systems of the form M u − ∆tr(u) = f , (3.18) which, when solved by means of Newton’s method, give rise to linear systems of the form (M − ∆tJ)x = b, (3.19) where the matrix J is the Jacobian of the residual function r. The most immediate challenge towards efficiently implementing an implicit method for high polynomial degree on tensor product elements is forming the Jacobian ma- trix. In general, all the degrees of freedom within one element are coupled, and thus the diagonal blocks of the Jacobian matrix corresponding to a single element are dense (p + 1)d × (p + 1)d matrices. Therefore, it is impossible to explicitly form this matrix in less than O(p2d ) time. To circumvent this, using the techniques described in the preceding sections, it is possible to solve the linear systems arising from im- plicit time integration by means of an iterative method such as GMRES [107]. Each iteration requires performing a matrix-vector multiplication by the mass matrix and the Jacobian matrix. If we avoid explicitly forming these matrices, then the multipli- cations can be performed in O(pd+1 ) time, using methods similar to those described in the preceding section. 78 Matrix-free tensor-product Jacobians In order to efficiently implement the implicit method described above, we first apply the sum-factorization technique to efficiently evaluate the matrix-vector product (M − ∆tJ)v (3.20) for a given vector v. As described in Section 3.1.2, the mass matrix is a block diagonal matrix whose jth block can be written in the Kronecker form given by equation (3.15). Multiplying by the Kronecker products can be performed with O(pd+1 ) operations, and multiplying by JT requires exactly (p + 1)d operations. Thus, the product M v requires O(pd+1 ) operations. We now describe our algorithm for also computing the product Jv with the same complexity. For simplicity of presentation, we will take d = 2, but the algorithm is immediately generalizable to arbitrary dimension d. We first consider an element- wise blocking of the matrix J. Each block is a (p + 1)2 × (p + 1)2 matrix, with blocks along the diagonal corresponding to each element in the triangulation, and blocks off the diagonal corresponding to the coupling between neighboring elements through their common face. We consider the residual vector r, and restrict our attention to one element. We define the indices 1 ≤ i, j, k, ` ≤ p + 1, such that the entries of the diagonal block of the Jacobian can be written as ∂rij . (3.21) ∂uk` We define rij by Z Z rij = ˜ ij dx − F (uh ) · ∇Φ Fb (u− + −˜ h , uh ) · n Φij dA (3.22) K ∂K 79 which we evaluate using the quadrature rule µ µ X X rij = wα wβ F (uh (xα , xβ )) · ∇ (φi (xα )φj (xβ )) α=1 β=1 µ XX − wα Fb (u− e e + e e − e e e e h (xα , yα ), uh (xα , yα )) · n (xα , yα ))φi (xα )φj (yα ), (3.23) e∈∂K α=1 where the notation (xeα , yαe ) represents the coordinates of the αth quadrature node alone the face e of ∂K. We also recall that the function uh is evaluated by expanding in terms of the local basis functions, e.g. p+1 p+1 X X uh (xα , xβ ) = uk` φk (xα )φ` (xβ ), (3.24) k=1 `=1 which can be evaluated efficiently as (G ⊗ G) u. (3.25) Thus, the entries of the Jacobian can be written as µ µ ∂rij XX ∂F = wα wβ φk (xα )φ` (xβ ) (uh (xα , xβ )) · ∇ (φi (xα )φj (xβ )) ∂uk` α=1 β=1 ∂u h µ XX ∂ Fb − e e − wα φk (xeα )φ` (yαe ) + e e (3.26) − (u (xα , yα ), u (xα , yα )) e∈∂K α=1 ∂uh · n− (xeα , yαe ))φi (xeα )φj (yαe ). Since there are (p + 1)2d such entries, we avoid explicitly computing the entries of this matrix, and instead describe how to compute the matrix-vector product Jv. As a pre-computation step, we compute the flux derivatives ∂F /∂uh and numerical flux derivatives ∂ Fb /∂u− h at each of the quadrature nodes (xα , xβ ) in the element K. 80 For simplicity, we introduce the notation ∂F ∂F (xα , xβ ) = (uh (xα , xβ )), (3.27) ∂uh ∂uh ∂ Fb e e ∂ Fb − e e + e e − e e (xα , yα ) = − (uh (xα , yα ), uh (xα , yα )) · n (xα , yα )). (3.28) ∂u−h ∂uh Then, applying the sum-factorization technique, the terms of the product of Jv for a given vector v corresponding to the diagonal block takes the form   p+1 p+1 µ X XXX µ  ∂rij ∂F vk` = wα wβ φk (xα )φ` (xβ ) (xα , xβ ) ∂uk` k=1 `=1 α=1 β=1 ∂uh  · ∇ (φi (xα )φj (xβ )) vk` (3.29) p+1 p+1 µ X XXX ∂ Fb e e − wα φk (xeα )φ` (yαe ) (xα , yα )φi (xeα )φj (yαe )vk` k=1 `=1 e∈∂K α=1 ∂u−h µ µ p+1 p+1 X X ∂F1 X X = wα φ0i (xα ) wβ (xα , xβ )φj (xβ ) φ` (xβ ) φk (xα )vk` α=1 β=1 ∂uh `=1 k=1 µ µ p+1 p+1 X X ∂F2 X X 0 + wα φi (xα ) wβ (xα , xβ )φj (xβ ) φ` (xβ ) φk (xα )vk` (3.30) α=1 β=1 ∂uh `=1 k=1 µ p+1 p+1 XX ∂ Fb e e X X e e e + wα − (xα , yα )φi (xα )φj (yα ) φ` (yα ) φk (xeα )vk` , e∈∂K α=1 ∂u h `=1 k=1 where F1 and F2 are the x and y components of the flux function F , respectively. We notice that in each of the above summations, there are at most two free indices, and therefore each sum can be computed in O(p3 ) time, achieving linear time in p per degree of freedom. The terms of the product corresponding to the off-diagonal blocks have a similar form to the face integral in the above equations, and can similarly be computed in O(p3 ) time. To summarize, we describe the algorithm for computing the matrix-products of the form Jx in Algorithm 3.1. The first two operations can be performed as a 81 pre-computation step, and only the third step need be repeated when successively multiplying the same Jacobian matrix by different vectors (as in the case of an iterative linear solver). Algorithm 3.1 Matrix-free computation of Jv in 2D and 3D 1: Pre-computation: 2: Evaluate the solution at quadrature points: (in 2D, compute (G ⊗ G) u, and in 3D, compute (G ⊗ G ⊗ G) u) Complexity: O(pd+1 ) 3: Evaluate the flux Jacobians ∂F ∂u ∂ Fb and ∂u − at quadrature points Complexity: O(pd ) 4: Compute the matrix-vector product using the sum-factorized form Complexity: O(pd+1 ) 3.2 Tensor-product preconditioners One of the main challenges in successfully applying such a matrix-free method is preconditioning [114]. Common preconditioners typically used for implicit DG meth- ods include block Jacobi, block Gauss-Seidel, and block ILU preconditioners [107]. Computing these preconditioners first requires forming the matrix, and additionally requires the inversion of certain blocks. Typically, this would incur a cost of O(p3d ), which quickly grows to be prohibitive as we take p to be large. To remedy this issue, we develop a preconditioner for two and three spatial dimensions that takes a similar Kronecker product form to those seen in the previous section. We draw inspiration from the tensor-product structure often seen in finite-difference and spectral approximations to, e.g. the Laplacian operator on a nd cartesian grid, 82 which can be written in one, two, and three spatial dimensions, respectively, as L1D = Tn , (3.31) L2D = I ⊗ Tn + Tn ⊗ I, (3.32) L3D = I ⊗ I ⊗ Tn + I ⊗ Tn ⊗ I + Tn ⊗ I ⊗ I, (3.33) where Tn is the standard one-dimensional approximation to the Laplacian. Given a general conservation law of the form (1.1), the flux function F is not required to possess any particular structure, and thus the DG discretization of such a function will not be exactly expressible in a similar tensor-product form. That being said, many of the key operations in DG, listed in Table 3.1, are expressible in a similar form. Therefore, in order to precondition the implicit systems of the form (M − ∆tJ)u = b, (3.34) we look for tensor-product approximations to the diagonal blocks A of the matrix M − ∆tJ. Specifically, we are interested in finding preconditioners P of the form r X A≈P = Aj ⊗ Bj in 2D, (3.35) j=1 r X A≈P = Aj ⊗ Bj ⊗ Cj in 3D, (3.36) j=1 for a fixed number of terms r, where each of the matrices Aj , Bj , and Cj are of size (p + 1) × (p + 1). Given r, it is possible to find the best possible approximation (in the Frobenius norm) of the form (3.35) to an arbitrary given matrix by means of the Kronecker-product singular value decomposition (KSVD). 83 3.2.1 Kronecker-product singular value decomposition In [128], Van Loan posed the nearest Kronecker product problem (NKP): given a matrix A ∈ Rm×n (with m = m1 m2 and n = n1 n2 ), and given a fixed number r, find matrices Aj ∈ Rm1 ×n1 , Bj ∈ Rm2 ×n2 that minimize the Frobenius norm r X A − Aj ⊗ Bj . (3.37) j=1 F The solution to the NKP given by Van Loan is as follows. We first consider the “blocking” of A:    A11 A12 ··· A1,n1     A21 A22 · · · A2,n1    A= . .. .. , (3.38)  .. ...   . .    Am1 ,1 Am1 ,2 · · · Am1 ,n1 where each block is a m2 × n2 matrix. We then define a rearranged (or shuffled ) e of the matrix A, which is a m1 n1 × m2 n2 matrix given by version A     e T  A1   vec(A1j )       Ae2   vec(A2j )T  e   e e   A =  .  , where Aj is a block of rows given by Aj =  .. ,  ..   .          en1 A vec(Am1 ,j )T (3.39) where the vec operator is defined so that vec(Aij ) is the column vector of length m2 n2 obtained by “stacking” the columns of Aij . This rearranged matrix has the property that, given matrices Aj , Bj r X r X e T A − Aj ⊗ Bj = A − vec(Aj ) vec(Bj ) , (3.40) j=1 F j=1 F 84 and therefore the NKP problem (3.37) has been reduced the finding the closest rank- e This approximation can be found by computing the singular r approximation to A. e value decomposition (SVD) of A, e = U ΣV T , A (3.41) then the solution to (3.37) is given by reshaping the columns of U and V , such that √ √ vec(Aj ) = σj Uj , vec(Bj ) = σj Vj . (3.42) This construction is referred to as the Kronecker product SVD (KSVD). Efficient computation of the KSVD In general, computing the singular value decomposition of a matrix is an expensive process, with cubic complexity. However, if the number r of desired terms in the e then it is possible summation (3.35) is much smaller than the rank r˜ of the matrix A, to well-approximate the largest singular values and associated left and right singular vectors by means of a Lanczos algorithm [55]. This algorithm has the additional e is not required, rather only advantage that an explicit representation of the matrix A e and A the ability to multiply vectors by the shuffled matrices A eT . In this section, we follow the presentation from [128]. The Lanczos SVD procedure is described in Algorithm 3.2. We remark that there are many variations on the orthogonalization procedure referred to in lines 6 and 10 of Algorithm 3.2, including partial or full orthogonal- ization. In this work, we perform full orthogonalization of the vectors uj and vj at each iteration of the Lanczos algorithm. 85 Algorithm 3.2 Lanczos singular value decomposition 1: v0 ← random vector with kv0 k2 = 1 2: p0 ← v0 , β0 ← 1, u0 ← 0 3: for j = 0 to J (maximum number of iterations) do 4: vj ← pj /βj 5: rj ← Av e j − βj uj 6: Orthogonalize rj . 7: αj ← krj k2 8: uj+1 ← rj /αj 9: pj+1 ← A eT uj − αj vj 10: Orthogonalize pj+1 . 11: βj+1 ← kpj+1 k2 12: if |βj+1 | < tolerance then 13: break 14: U ← (u1 , u2 , . . . , uj+1 ) 15: V ← (v0 , v1 , . . . , vj ) 16: Construct bidiagonal matrix B, with diagonal α0 , . . . , αj , and superdiagonal β1 , . . . , β j . 17: Compute r largest singular values σk (and corresponding left and right singular vectors, u0k , vk0 ) of B 18: Singular values of A e are σj , singular vectors are U u0 and V v 0 . k k As mentioned previously, one of the key advantages of the Lanczos algorithm is e can be foregone, since only matrix- that an explicit representation of the matrix A e and A vector products of the form Av eT v are required. As described in [128], we can compute these matrix-vector products according to Algorithms 3.3 and 3.4. Taking advantage of the specific tensor-product form of the matrix A, and using techniques similar to those used for the matrix-free Jacobian evaluation from Section 3.1.4, it is possible to efficiently compute the matrix-vector products. Specialized kernels are required for two and three spatial dimensions, and the details of this process are described in the following sections. 86 e Algorithm 3.3 Computation of the shuffled matrix-vector product u = Av 1: u ← 0 2: for i = 1 to n1 do 3: rows ← (i − 1)m1 + 1, . . . , im1 4: for j = 1 to n2 do 5: Define Z ∈ Rm2 ×m1 by vec(Z) = A( : , (i − 1)n2 + j) 6: u(rows) ← u(rows) + Z T v((j − 1)m2 + 1:jm2 ) eT v Algorithm 3.4 Computation of the transposed matrix-vector product u = A 1: u ← 0 2: for i = 1 to n2 do 3: rows ← (i − 1)m2 + 1, . . . , im2 4: for j = 1 to n1 do 5: Define Z ∈ Rm2 ×m1 by vec(Z) = A( : , (j − 1)n2 + i) 6: u(rows) ← u(rows) + Zv((j − 1)m1 + 1:jm1 ) 3.2.2 Two spatial dimensions Having shown that, given the number of terms r in the sum, it is possible to find the best approximation of the form (3.35), we now address the issue of solving linear systems of equations with such a matrix. In the case that r = 1, we have P = A1 ⊗B1 , and it is clear that P −1 = A−1 −1 2 2 1 ⊗ B1 , and thus the (p + 1) × (p + 1) problem is reduced to two problems of size (p+1)×(p+1). Our experience has shown that r = 1 is not sufficient to accurately approximate the Jacobian matrix, and the resulting preconditioners are not very effective. For this reason, we choose r = 2, and obtain a linear system of the form P x = (A1 ⊗ B1 + A2 ⊗ B2 )x = b. (3.43) Because of the additional term in this sum, it is not possible to invert this matrix factor-wise. Instead, we follow the matrix diagonalization technique described in [79, 87 117]. We multiply the system of equations on the left by (A−1 −1 2 ⊗ B1 ) to obtain (A−1 −1 −1 −1 2 A1 ⊗ I + I ⊗ B1 B2 )x = (A2 ⊗ B1 )b. (3.44) We let C1 = A−1 −1 2 A1 and C2 = B1 B2 . We then remark that if C1 and C2 are diagonalizable matrices, the sum C1 ⊗ I + I ⊗ C2 can be simultaneously diagonalized by means of the eigendecomposition. More generally, the Schur factorization of the matrices C1 and C2 is guaranteed to exist, and thus the summation C1 ⊗ I + I ⊗ C2 can be simultaneously (quasi)-triangularized by of the (real) Schur decomposition. That is to say, we find orthogonal transformation matrices Q1 and Q2 such that C1 = Q1 T1 QT1 , (3.45) C2 = Q2 T2 QT2 , (3.46) where T1 and T2 are quasi-triangular matrices. Our numerical experiments have indicated that the Schur factorizations results in better numerical conditioning than the eigendecomposition, and thus we elect to triangularlize the matrix rather than diagonalize. Therefore, we can reformulate the linear system as  (Q1 ⊗ Q2 )(T1 ⊗ I + I ⊗ T2 )(QT1 ⊗ QT2 )x = Q1 T1 QT1 ⊗ I + I ⊗ Q2 T2 QT2 x = (C1 ⊗ I + I ⊗ C2 )x (3.47) = (A−1 −1 2 ⊗ B1 )b. Since the matrices Q1 and Q2 are orthogonal, the inverse of their Kronecker product Q1 ⊗ Q2 is trivially given by QT1 ⊗ QT2 . Thus, solving the system (3.47) is reduced to solving a system of the form T1 ⊗ I + I ⊗ T2 . Well-known solution techniques exist for this Sylvester-type system of equations, which can be solved in O(p3 ) operations. Thus, once the approximate preconditioner P = A1 ⊗B1 +A2 ⊗B2 has been computed, 88 solving linear systems of the form P x = b can be performed in linear time per degree of freedom. e and A Efficient computation of Av eT v in two dimensions One of the key operations in efficiently computing the approximate Kronecker- product preconditioner is the fast, shuffled matrix-vector product operation used in the Lanczos algorithm. Since our algorithm avoids the explicit construction and evaluation of the entries of the matrix A, we present a matrix-free algorithm to com- pute the shuffled product in O(p3 ) time. Setting A = M − ∆tJ, we apply Algorithm e for a given vector u. We first write 3.3 to compute the shuffled product Au e=M A f − ∆t(Jev + Jef ), (3.48) where Jv and Jf are the volume and face contributions to the Jacobian matrix, fv. Recall that respectively. We first demonstrate the computation of the product M the entries of M are given by Z Mij,k` = ˜ ij (x, y)Φ Φ ˜ k` (x, y) dxdy. (3.49) K fv Then, Algorithm 3.3 allows us to write u = M X u:i = (M::,ji )T v:j . (3.50) j 89 Writing out the matrix-vector product explicitly, and expanding the integral in (3.49) as a sum over quadrature nodes, we have XXXX uki = φ` (xα )φk (xβ )φj (xα )φi (xβ )| det(JT (xα , xβ ))|wα wβ v`j . (3.51) j ` α β This sum can be factorized as X X X X uki = wβ φk (xβ )φi (xβ ) wα | det(JT (xα , xβ ))| φj (xα ) φ` (xα )v`j , (3.52) β α j ` where we notice that each summation in this expression involves no more than two free indices, and therefore the expression can be computed in O(p3 ) time. Following the same procedure, and recalling the representation for Jv and Jf given in (3.26), we can evaluate the shuffled product u = Jev v as X X ∂F1 X X uki = wβ φk (xβ )φi (xβ ) (xα , xβ ) wα φj (xα ) φ0` (xα )v`j β α ∂uh j ` X X ∂F2 X X (3.53) + wβ φ0k (xβ )φi (xβ ) wα (xα , xβ ) φj (xα ) φ` (xα )v`j . β α ∂uh j ` Finally, we consider the face integral terms, and write out the factorized form of the shuffled product u = Jef v, which takes the form XX ∂ Fb e e X X uki = − wα φi (yαe )φk (yαe ) (x , α αy ) φ j (x e α ) φ` (xeα )v`j . (3.54) e∈∂K α ∂u−h j ` We further remark that many of the terms in this sum can be eliminated by using the fact that many of the basis functions are identically zero along a given face e of the element K. eT v can be performed Computation of the transpose of the shuffled product A 90 using a very similar matrix-free approach, following the framework of Algorithm 3.4. These two procedures allow for the computation of steps 5 and 10 in the Lanczos algorithm in O(p3 ) time. 3.2.3 Three spatial dimensions In the case of three spatial dimensions, it would be natural to consider a precondi- tioner matrix P of the form A ≈ P = A1 ⊗ B1 ⊗ C1 + A2 ⊗ B2 ⊗ C2 + A3 ⊗ B3 ⊗ C3 . (3.55) Unfortunately, it is not readily apparent how to solve a general system of the form (3.55). Therefore, we instead look for a preconditioner that has the simplified form A ≈ P = A1 ⊗ B1 ⊗ C1 + A1 ⊗ B2 ⊗ C2 , (3.56) where we emphasize that the same matrix A1 appears in both terms on the right- hand side. This has the advantage that the system P x = b can be transformed by multiplying on the left by A−1 −1 −1 1 ⊗ B2 ⊗ C1 to obtain (I ⊗ B2−1 B1 ⊗ I + I ⊗ I ⊗ C1−1 C2 )x = (A−1 −1 −1 1 ⊗ B2 ⊗ C1 )b. (3.57) Applying the same technique as in the two-dimensional case allows us to simultane- ously quasi-triangularize both terms on the left-hand side, which then results in a system of the form (I ⊗ Q1 ⊗ Q2 )(I ⊗ T1 ⊗ I + I ⊗ I ⊗ T2 )(I ⊗ QT1 ⊗ QT2 )x = (A−1 −1 −1 1 ⊗ B2 ⊗ C1 )b, (3.58) 91 which, as in the case of the two-dimensional system, is a Sylvester-type system that can be efficiently solved in O(p3 ) time (constant time in p per degree of freedom). Forming the three-dimensional preconditioner We now address how to generate an effective preconditioner of the form (3.56) using the KSVD. First, we recall that the element Jacobian will be a (p + 1)3 × (p + 1)3 matrix. We wish to approximate this matrix by a Kronecker product A1 ⊗ D1 , 2 ×(p+1)2 where A1 ∈ R(p+1)×(p+1) and D1 ∈ R(p+1) . We find such matrices A1 and D1 by finding the largest singular value and corresponding singular vectors of the e obtaining permuted matrix A, A ≈ A1 ⊗ D1 . (3.59) In order to find the singular values using the Lanczos algorithm, we must compute e and A the matrix-vector product Av eT v. In the following section, we describe how to perform Algorithms 3.3 and 3.4 efficiently by taking advantage of the tensor-product structure of the Jacobian. Once the matrices A1 and D1 have been obtained, we can then repeat the KSVD process to find the best two-term approximation D1 ≈ B1 ⊗ C1 + B2 ⊗ C2 . (3.60) This too involves the Lanczos algorithm, but since the matrix D1 has dimensions T f1 v and D (p + 1)2 × (p + 1)2 , computing the permuted products D f1 v using standard dense linear algebra requires O(p4 ) operations, and thus is linear in p per degree of freedom. Combining (3.59) and (3.60), we obtain an approximation of the form A ≈ A1 ⊗ B1 ⊗ C1 + A1 ⊗ B2 ⊗ C2 (3.61) 92 as desired. e and A Efficient computation of Av eT v in three dimensions As in Section 3.2.2, we describe the matrix-free procedure for computing the shuffled e and A matrix-vector products Av eT v. The general approach to this method is the same as in the two-dimensional case, but there are several key differences that in- crease the complexity of this problem. First, we recall that since our approximation takes the form A ≈ A1 ⊗ D1 where A1 is (p + 1) × (p + 1) and D is (p + 1)2 × (p + 1)2 , the matrix A˜ is rectangular, with dimensions (p + 1)2 × (p + 1)4 . The algorithm we describe has linear complexity per degree of freedom of the vector v, which results e unfortunately not meeting our overall goal in O(p5 ) operations for the product Av, of linear time per degree of freedom in the solution vector. e=M As before, we decompose the matrix A f − ∆t(Jev + Jef ). First, we describe the method for the mass matrix. Recall that the entries of M are given by X Mijk,`mn = wα wβ wγ | det(JT (xα , xβ , xγ ))| α,β,γ (3.62) φi (xα )φj (xβ )φk (xγ )φ` (xα )φm (xβ )φn (xγ ). fv, where u is a vector of length Then, following Algorithm 3.3, we write u = M (p + 1)2 and v is a vector of length (p + 1)4 , XX u:i = (M:::,jki )T v::jk . (3.63) j k Following the same sum factorization procedure as in the two-dimensional case, we 93 can write X XX u`i = φi (xγ )φ` (xγ ) wα wβ wγ | det(JT (xα , xβ , xγ ))| γ β α X X X X (3.64) φk (xβ ) φj (xα ) φn (xβ ) φm (xα )vmnjk . k j n m The O(p5 ) complexity is clear from this form, as, for example, the right-most sum- mation has four free indices. The shuffled product with the Jacobian of the volume integral takes a similar form, where u = Jev v can be written as X XX u`i = φi (xγ )φ` (xγ ) wα wβ wγ γ β α  X X X X ∂F1 (xα , xβ , xγ ) φk (xβ ) φj (xα ) φn (xβ ) φ0m (xα )vmnjk ∂uh k j n m X X X X  ∂F2 0 + (xα , xβ , xγ ) φk (xβ ) φj (xα ) φn (xβ ) φm (xα )vmnjk (3.65) ∂uh k j n m X XX ∂F3 + φi (xγ )φ0` (xγ ) wα wβ wγ (xα , xβ , xγ ) γ β α ∂u h X X X X φk (xβ ) φj (xα ) φn (xβ ) φm (xα )vmnjk . k j n m Finally, we write the product corresponding to the face integral Jacobian, u = Jef v, as X XX ∂ Fb e e e e e u`i = wα wβ φi (zαβ )φ` (zαβ − (xαβ , yαβ , zαβ ) ) e∈∂K β α ∂u h X X X X (3.66) e φk (yαβ ) φj (xeαβ ) e φn (yαβ ) φm (xeαβ )vmnjk , k j n m where (xeαβ , yαβ e e , zαβ ) represents the coordinates of the quadrature nodes on the face e of element K indexed by (α, β). We remark that for each face e, two of the indices in the above expression can be eliminated. This simplification depends on the orientation of the face, and therefore we leave the full expression for the sake of 94 generality. 3.2.4 Algorithm overview Here we describe the overall algorithms used to form and apply the tensor prod- uct preconditioner. We present the algorithm for both the cases of two and three spatial dimensions. Forming the preconditioner requires the Lanczos SVD, given by Algorithm 3.2, and the two permuted matrix-vector multiplication kernels, given by Algorithms 3.3 and 3.4, and described in the preceding section. Computational com- plexities are indicated for each step of the algorithm. We note that in the 2D case, we obtain an overall complexity of O(pd+1 ). In the 3D case, all the operations have complexity at most O(pd+1 ), except the first Lanczos SVD, which requires O(p5 ) operations. Algorithm 3.5 Form 2D preconditioner A ≈ P = A1 ⊗ B2 + A2 ⊗ B2 1: Compute A ≈ A1 ⊗B1 +A2 ⊗B2 using Lanczos iteration and matrix-free products e and A Av eT v Complexity: O(p3 ) 2: Precompute LU factorizations of A2 and B1 Complexity: O(p3 ) 3: Precompute Schur factorizations Q1 T1 QT1 , Q2 T2 QT2 of A−1 −1 2 A1 and B1 B2 , respec- tively Complexity: O(p3 ) Algorithm 3.6 Apply 2D preconditioner to solve P x = b ˜ ← A−1 ⊗ B −1 b 1: b Complexity: O(p3 ) 2 1 2: Solve the Sylvester system (T1 ⊗ I + I ⊗ T2 )x ˜ ˜ = (QT1 ⊗ QT2 )b Complexity: O(p3 ) 3: x ← (Q1 ⊗ Q2 )x ˜ Complexity: O(p3 ) 95 Algorithm 3.7 Form 3D preconditioner A ≈ P = A1 ⊗ B1 ⊗ C1 + A1 ⊗ B2 ⊗ C2 e and 1: Compute A ≈ A1 ⊗ D1 using Lanczos iteration and matrix-free products Av eT v A Complexity: O(p5 ) 2: Compute D1 ≈ B1 ⊗ C1 + B2 ⊗ C2 using Lanczos iteration and dense permuted products Complexity: O(p4 ) 3: Precompute LU factorizations of A1 , B2 , and C1 Complexity: O(p3 ) 4: Precompute Schur factorizations Q1 T1 QT1 , Q2 T2 QT2 of B2−1 B1 and C1−1 C2 , respec- tively Complexity: O(p3 ) Algorithm 3.8 Apply 3D preconditioner to solve P x = b ˜ ← A−1 ⊗ B −1 ⊗ C −1 b 1: b Complexity: O(p4 ) 1 2 1 2: Solve the Sylvester system (I ⊗ T1 ⊗ I + I ⊗ I ⊗ T2 )x ˜ ˜ = (I ⊗ QT1 ⊗ QT2 )b Complexity: O(p4 ) 3: x ← (I ⊗ Q1 ⊗ Q2 )x ˜ Complexity: O(p4 ) 3.3 Extension to second-order operators The construction of the approximate Kronecker-product preconditioners described in the preceding sections is applicable in the case of first-order hyperbolic conservation laws. It is of interest to extend this preconditioners to systems of second-order partial differential equations, including elliptic and parabolic problems. In Section 1.2, the DG discretization of such operators is discussed, with particular emphasis on the local discontinuous Galerkin (LDG) method and the interior penalty (IP) method. We begin by considering the model Poisson problem with homogeneous Dirichlet 96 boundary conditions, −∆u = f in Ω, (3.67) u=0 on ∂Ω. (3.68) By introducing the unknown σ = ∇u, we obtain the equivalent system of first-order equations given by (1.20–1.22). Directly discretizing this first-order system gives rise to what is known as the DG system flux formulation, Z Z Z σh · τh dx = − uh ∇ · τh dx + bh Jτh K ds, u (3.69) Ω Z ZΩ ZΓ σh · ∇vh dx − b h · Jvh K ds = σ f vh dx, (3.70) Ω Γ Ω b . We can write this sytem of b and σ for appropriately defined numerical fluxes u equations as       M −G   σ   0    = , (3.71) D E u f where M is the mass matrix, G is a discretization of the gradient operator, and D is a discretization of the divergence. The matrix E contains stabilization and boundary terms. Since both the LDG and IP methods are symmetric, we obtain the saddle-point system,      T  M D  σ   0    = . (3.72) D E u f Generally, we prefer not to solve the system of equations in this form, both because it requires for solving for a large number of unknowns (both u and its gradient, σ), and because such saddle-point systems are typically difficult to solve efficiently by means of iterative solvers [14]. Instead, we prefer to solve the system of equations 97 that results from the primal form. The primal form is obtained by eliminating the discrete gradient from equations (3.69) and (3.70), and is given by Z Z B(uh , vh ) = ∇h (uh ) · ∇h vh dx − Juh K · {∇h vh } ds Ω Z Γ Z (3.73) − {uh − u bh }J∇h vh K ds − σb h · Jvh K ds. Γ\∂Ω Γ This system is equivalent to the Schur complement system that arises from perform- ing a block-LU decomposition of the above saddle-point system,      T −1 T  M D   M 0  I M D   =    . (3.74) D E D S 0 I The primal form corresponding to the local DG method is given by Z Z BLDG (uh , vh ) = ∇h uh · ∇h vh dx − (Juh K · {∇h vh } + C11 Juh K · Jvh K) ds Ω Γ Z Z − {∇h uh } · Jvh K ds + (C12 · Juh KJ∇h vh K + Juh KC12 · Jvh K) ds (3.75) Γ Z Γ\∂Ω + (r(Juh K) + `(C12 · Juh K)) · (r(Jvh K) + `(C12 · Jvh K)) dx, Ω where we recall the lifting operators r : [L2 (Γ)]d → [Vh ]d and ` : L2 (Γ \ ∂Ω) → [Vh ]d defined by, Z Z Z Z r(q) · τ dx = − q · {τ } ds, `(v) · τ dx = − vJτ K ds. (3.76) Ω Γ Ω Γ\∂Ω The primal form corresponding to the interior penalty methods is given by Z Z BIP (uh , vh ) = ∇h uh · ∇h vh dx − Juh K · {∇h vh } ds Ω Z Γ Z (3.77) − {∇h uh } · Jvh K ds + ηe Juh K · Jvh K ds. Γ Γ 98 In order to efficiently form the approximate Kronecker-product preconditioners for these operators, we must be able to perform efficient shuffled matrix-vector products, similar to those given by (3.53) and (3.65). The presence of the lifting operators r and ` in the LDG primal form (3.75) poses an immediate obstacle. These opera- tors are computed using local inverse mass matrices. As noted in Section 3.1.3, the inverse mass matrix does not necessarily posses a Kronecker-product structure that is amenable to sum-factorization, specifically in the case of isoparametric elements. This makes the construction of efficient, sum-factorized shuffled matrix-vector prod- ucts infeasible in the case of the local DG method. On the other hand, the primal form for the interior penalty method does not require the computation of lifting operators. The algorithms for computing the efficient shuffled matrix-vector products described in Sections 3.2.2 and 3.2.3 can be modified in a straightforward manner to account for the gradient terms that appear in the IP primal form (3.77). This allows us to construct approximations to the block Jacobi preconditioner, making use of the KSVD framework described in the preceding sections. The construction used for the Poisson problem immediately generalizes to the interior penalty formulation for more general classes of equations, which is described in Section 1.2.2. 3.3.1 Exact representations In order to motivate the use of this preconditioner for second-order systems, we consider the cases in which the approximate Kronecker-product preconditioner is able to exactly reproduce the block Jacobi preconditioner. We consider the Poisson problem − ∆u = f (3.78) 99 and the convection diffusion equation ut + ∇ · (βu − ∇u) = 0, (3.79) both with homogeneous Dirichlet boundary conditions. We restrict the analysis here to two spatial dimensions. First we consider the Poisson problem. We consider the diagonal block A of the matrix associated with the bilinear form BIP (·, ·), corresponding to an element K. We restrict ourselves to the case of a Cartesian grid, and so K is given by a translation of the unit square. Thus, the transformation Jacobian is equal to the identity matrix, and it suffices to consider the simple case of K = R. The entries e k` , Φ of A are then given by B(Φ e ij ). We can see that the terms corresponding to the volume integral can be written in the form Z  e k` · ∇h Φ ∇h Φ e ij dx = GT W G ⊗ DT W D + DT W D ⊗ GT W G, (3.80) Ω using the notation from Section 3.1.2. In a similar fashion, the penalty boundary terms can be written as Z  e k` K · JΦ ηe JΦ e ij K = ηe GT W G ⊗ GT G0 + GT W G ⊗ GT G1 0 1 Γ (3.81)  + GT0 G0 T ⊗ G WG + GT1 G1 T ⊗ G WG , where (G0 )i = φi (0) and (G1 )i = φi (1) are 1 × (p + 1) end-point evaluation matrices. The end-point differentiation matrices D0 and D1 are defined similarly. The remain- ing boundary terms can be treated similarly, resulting in the following form for the 100 diagonal blocks, GT W G ⊗ − DT W D − ηe GT0 G0 − ηe GT1 G1  − D0T G0 − GT0 D0 + D1T G1 + GT1 D1 (3.82) T + − D WD − ηe GT0 G0 − ηe GT1 G1  − D0T G0 − GT0 D0 + D1T G1 + GT1 D1 ⊗ GT W G, which, in particular, demonstrates that the diagonal blocks can be written as the sum of two Kronecker products. Therefore, the tensor-product preconditioner is able to exactly reproduce the diagonal blocks of the DG discretization in the case of the Poisson problem on a Cartesian grid. It can also be shown that the fully discrete system for the scalar advection equa- tion on a Cartesian grid with constant velocity field β = (βx , βy ) gives rise to diagonal blocks of the form GT W G ⊗ GT W G − α∆t βx GT W G ⊗ DT W G + βy DT W G ⊗ GT W G  (3.83) T − βx G W G ⊗ GT1 G1 − βy GT1 G1 T ⊗ G WG , where we assume for simplicity that βx , βy > 0, though this assumption is not nec- essary. Combining (3.82) and (3.83), we see that the diagonal blocks corresponding to the DG discretization of the convection-diffusion equation (3.79) on a Cartesian grid with constant velocity field can be written as   G W G ⊗ GT W G − αβx ∆t DT W G − GT1 G1 − DT W D T  − ηe GT0 G0 − ηe GT1 G1 − D0T G0 − GT0 D0 + D1T G1 + GT1 D1   (3.84) + − αβy ∆t DT W G − GT1 G1 − DT W D − ηe GT0 G0  − ηe GT1 G1 − D0T G0 − GT0 D0 + D1T G1 + GT1 D1 ⊗ GT W G, 101 which shows that the approximate tensor-product preconditioner exactly reproduces the block Jacobi preconditioner in this case. In more general cases, we cannot expect the preconditioner to exactly reproduces the diagonal blocks. However, the KSVD construction guarantees the optimal (in Frobenius norm) such approximation, and the effectiveness of the preconditioner is demonstrated on a variety of test cases in the following sections. In particular, in Section 3.4.2 we consider several convection- diffusion test cases for which the preconditioner is not exact, yet results in favorable performance. 3.4 Numerical results In the following sections, we present numerical results that demonstrate several im- portant features of the preconditioner and its performance when applied to a variety of equations and test cases. We consider both two-dimensional and three-dimensional problems, and solve the scalar advection equation, the Euler equations, and the Navier-Stokes equations. The nonlinear systems of equations resulting from im- plicit time integration are solved using Newton’s method, with a relative tolerance of 10−8 . Within each Newton iteration, the linear system is solved using precon- ditioned, restarted GMRES, with a relative tolerance of 10−5 . The parameters of Newton tolerance, GMRES tolerance, and restart iterations are chosen according to performance study found in [137]. Although these parameters can have an effect on overall solution time, the relationships are often neither simple nor well-understood, and these issues are not considered in depth in this work. 102 3.4.1 2D linear advection equation The simplest example we consider is that of the two-dimensional scalar advection equation, described in Section 1.4.1. The equation is given by ut + ∇ · (α, β) u = 0, (3.85) where (α, β) is a space-dependent velocity field. Because of the particularly sim- ple structure of this equation, it is possible to see how the approximate Kronecker preconditioner, given by A1 ⊗ B1 + A2 ⊗ B2 relates to the true diagonal blocks A of the matrix M − ∆tJ. In this case, the properties of the velocity field (α, β) can determine how well the discontinuous Galerkin discretization can be approximated by a tensor-product structure. Neglecting for now the face integral terms, the diagonal blocks of M − ∆tJ can be written as   GT W ⊗ GT W JT (G ⊗ G) − ∆t GT W ⊗ DT W F1   (3.86) + DT W ⊗ GT W F2 (G ⊗ G) , where JT , F1 , and F2 are µ2 × µ2 diagonal matrices. If the matrices JT , F1 , and F2 additionally posses a Kronecker-product structure, then it is possible to rewrite (3.86) exactly in the form A1 ⊗ B1 + A2 ⊗ B2 . The Kronecker structure of JT is determined by the geometry of the mesh, and the structure of F1 and F2 is determined by the form of the velocity field. For example, we first suppose that the mesh is a cartesian grid with grid size h, and thus the Jacobian determinant of the transformation map is equal to h2 . Hence, JT is equal to h2 times the identity matrix. If we further suppose that the 103 velocity field is separable, in the sense that, each component depends only on the corresponding spatial variable, i.e. α(x, y) = α(x), β(x, y) = β(y), then the flux derivatives can be written as F1 = I ⊗ F1x , and F2 = F2y ⊗ I. Therefore, we can rewrite (3.86) in the form   h2 GT W G − ∆tDT W F2y G ⊗ GT W G − ∆t GT W G ⊗ DT W F1x G , (3.87) and we see that the diagonal blocks of M − ∆tJ are exactly representable by our Kronecker-product approximation. If, on the other hand, we allow straight-sided, non-cartesian meshes, then the transformation mapping is a bilinear function, and its Jacobian determinant is a linear function in the variables x and y. Thus, JT = JTy ⊗ I + I ⊗ JTx . If the velocity field is constant in space, then we obtain the following representation of the diagonal blocks  GT W G ⊗ GT W JTx G − ∆tαDT W G +  (3.88) GT W JTy G − ∆tβDT W G ⊗ GT W G, and we see that our Kronecker-product approximation is again exact. If, in contrast to the previous two cases, the transformation mapping is given by a higher degree polynomial, or if the velocity field is not separable, then the approximate preconditioner will not yield the exact diagonal blocks. However, if the deformation of the mesh is not too large, if the velocity field is well approximated by one that is separable, or if the time step ∆t is relatively small, then we expect the Kronecker product preconditioner to compare favorably with the exact block Jacobi preconditioner. 104 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 (a) Cartesian mesh (b) Unstructured mesh Figure 3.1: Meshes used for the advection equation 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.5 1.0 0.0 0.5 1.0 0.0 0.5 1.0 1 1 1 1 (a) α = 1, β = 2 (b) α = x − 2, β = 2 −y (c) α = y − 2, β = 2 −x Figure 3.2: Velocity fields used for the advection equation We remark that this numerical experiment is designed to highlight two main features of the Kronecker-product preconditioner. The first is that if any exact representation of the diagonal blocks in the form A1 ⊗ B1 + A2 ⊗ B2 exists, such as those given by equations (3.87) and (3.88), then the KSVD algorithm provides an automatic, and purely algebraic method to identity this decomposition. No special structure of the flux functions is required to be known a priori in order for the KSVD to exactly reproduce this tensor-product structure. Secondly, even in a case where it is impossible to write such an expression exactly, the KSVD method will identify the best possible approximation of this form. Thus, in cases where the velocity field is close to constant (e.g. when the mesh size is very small), or where the mesh deformation is small, we expect this approximation to be very accurate. 105 In order to compare the performance of these two preconditioners, we solve equa- tion (3.85) on both regular and irregular meshes, with constant, separable, and non- separable velocity fields. The meshes are shown in Figure 3.1, and the velocity fields in Figure 3.2. We choose a representative time step of ∆t = 0.5, and consider poly- nomial degrees p = 1, 2, . . . , 10. In the case of the regular cartesian mesh, we expect identical performance for the exact block Jacobi and approximate Kronecker-product preconditioners for the velocity fields shown in Figures 3.2a and 3.2b, since the di- agonal blocks can be reproduced exactly. In the case of the unstructured mesh, we expect to see identical performance for the constant velocity field in Figure 3.2a. Indeed, the numerical results corroborate our expectations, and the number of iter- ations is identical between the two preconditioners in those test cases. Additionally, even in cases where the Kronecker-product approximation cannot reproduce the ex- act blocks, such for for velocity field 3.2c or 3.2b on the unstructured mesh, the performance is, in most cases, extremely similar to that of exact block Jacobi. The number of GMRES iterations required to converge with each preconditioner is shown in Table 3.2. For very large values of polynomial degree p on the unstructured mesh, with non-separable velocity field, we begin to see a degradation in the performance of the Kronecker-product preconditioner. Anisotropic grids One main motivation for the use of implicit time integration methods is the presence of stretched or highly anisotropic elements, for instance in the vicinity of a shock, or at a boundary layer [135]. In order to investigate the performance of the Kronecker- product preconditioner for this important class of problems, we consider the scalar advection equation on two anisotropic grids, shown in Figure 3.3. The first mesh consists entirely of rectangular elements, refined around the center line x = 1/2, 106 Table 3.2: Number of GMRES iterations for Jacobi (J ) and Kronecker-product (K ) preconditioners, advection equation on cartesian and unstructured grids, velocity fields (a), (b), and (c) from Figure 3.2. (a) Cartesian grid (b) Unstructured mesh (a) (b) (c) (a) (b) (c) p J K J K J K p J K J K J K 1 12 12 5 5 29 29 1 14 14 10 11 29 29 2 14 14 7 7 29 29 2 15 15 11 10 29 29 3 13 13 7 7 29 29 3 14 14 11 12 28 28 4 14 14 7 7 29 29 4 15 15 9 12 28 31 5 13 13 7 7 29 30 5 14 14 10 12 28 34 6 17 17 7 7 29 31 6 14 14 10 12 28 39 7 13 13 7 7 30 28 7 13 13 10 12 28 46 8 14 14 7 7 30 29 8 13 13 12 13 28 53 9 12 12 7 7 27 30 9 13 13 12 14 28 62 10 14 14 7 7 27 30 10 13 13 12 15 28 69 such that the thinnest elements have an aspect ratio of about 77. The second mesh is similar, with the main difference being that the quadrilateral elements no longer posses right-angles. In accordance with the analysis from the preceding section, we can expect the Kronecker-product preconditioner to exactly reproduce the diagonal blocks in the rectangular case for separable velocity fields. However, in the case of the skewed quadrilaterals, the Kronecker preconditioner is only exact for constant velocity fields, and provides an approximation to the diagonal blocks of the Jacobian in other cases. In the interest of generality, we consider the non-separable velocity field (c) shown in Figure 3.2, for which the Kronecker-product preconditioner is approximate for both the rectangular and skewed meshes. We use this test case to compare the runtime performance of the Kronecker- product preconditioner both with explicit time integration methods, and with the exact block Jacobi preconditioner. To this end, we choose an implicit time step of ∆t = 5 × 10−2 . We then compute one time step using a third-order L-stable DIRK method [1]. Additionally, we integrate until t = 5 × 10−2 using the standard 107 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Figure 3.3: Meshes used for anisotropic test case fourth-order explicit Runge-Kutta method, with the largest possible stable explicit time step. The explicit time step restriction becomes more severe as the polynomial degree p increases [72], resulting in a large increase in the number of time steps required. We choose polynomial degrees p = 1, 2, . . . , 30, and measure the runtime required to integrate until t = 5 × 10−2 . Due to the excessive runtimes, we use only p = 1, 2, . . . , 15 for the explicit method. We display the runtimes for both rectangular and general quadrilateral meshes in Figure 3.4. For p > 1, the explicit RK4 method is not competitive for this problem. For both meshes, the KSVD preconditioner results in faster runtimes than the exact block Jacobi preconditioner starting at about p = 4 or p = 5. In the rectangular case, we see a noticeable asymptotic improvement in the runtime in this case. For p = 30, the Kronecker-product preconditioner results in runtimes close to 20 times faster than block Jacobi. In the case of the skewed quadrilateral mesh, we observe an increase in the number of GMRES iterations required per time step, similar to what was observed in column (c) of Table 3.2b. Despite this increase in iteration count, the Kronecker-product preconditioner still resulted in runtimes about three times shorter than the exact block Jacobi. Additionally, we measure the average wall-clock time required to both form and 108 RK4 RK4 102 KSVD 102 KSVD Jacobi Jacobi Runtime (s) Runtime (s) 101 101 100 100 10−1 10−1 100 101 100 101 p p (a) Rectangular anisotropic mesh (b) Skewed anisotropic mesh Figure 3.4: Wall-clock times for scalar advection on anisotropic meshes 101 Form KSVD 6 Form Jacobi 0 Apply KSVD 1 10 Apply Jacobi Runtime (s) 10−1 4 1 10−2 3 1 10−3 10−4 10−5 100 101 p Figure 3.5: Wall-clock time required to form (solid lines) and apply (dashed lines) the exact block Jacobi and approximate Kronecker-product preconditioners for the scalar advection equation apply the Kronecker and block Jacobi preconditioners, for all polynomial degrees considered. We see that the cost of forming the Jacobi preconditioner quickly domi- nates the runtime. For large p, we begin to see the asymptotic O(p6 ) complexity for this operation. Applying the Jacobi preconditioner requires O(p4 ) operations, while both forming and applying the Kronecker preconditioner require O(p3 ) operations. These computational complexities are evident from the measured wall-clock times, shown in Figure 3.5. 109 3.4.2 2D convection-diffusion equation We consider the time-dependent scalar convection-diffusion equation, ut + ∇ · (βu − ∇u) = 0 in Ω, u = gD on ∂ΩD , (3.89) ∂u/∂n = gN on ∂ΩN , where β(x, y) is a given velocity field, and  > 0 is a constant diffusion coefficient. Also of interest to us is the steady version of this problem, where the first equation in (3.89) is replaced by ∇ · (βu − ∇u) = 0. (3.90) In this section, we consider two benchmark convection-diffusion test cases studied in detail by Mackenzie and Morton [80]. The domain for both test cases is the rectangle Ω = [−1, 1] × [0, 1]. A prescribed velocity field (shown in Figure 3.6a) given by β(x, y) = (2y(1 − x2 ), −2x(1 − y 2 )) is used. Test case 1 In the first test case, [122] we partition the boundary of the domain ∂Ω = Γ1 ∪Γ2 ∪Γ3 . On Γ1 = {−1 ≤ x ≤ 0, y = 0}, we specify a Dirichlet condition with a steep gradient given by gD (x) = 1 + tanh(20x + 10). On Γ2 = {0 ≤ x ≤ 1, y = 0}, we specific a homogeneous Neumann (outflow) condition. On the remaining tangential boundaries, Γ3 , we use a compatible Dirichlet condition gD = 1 − tanh(10). We consider a wide range of diffusion coefficients  = 10−6 , 2 × 10−3 , 10−2 , and 110 1.0 2 gD 0.5 1 0.0 0 −1.0 −0.5 0.0 0.5 1.0 −1 −0.5 0 x (a) Prescribed velocity field β(x, y) (b) Inflow condition Figure 3.6: Convection-diffusion test problem  = 10−6  = 2 × 10−3 2 n = 10, p = 1 2 n = 4, p = 4 n = 2, p = 9 Reference u(x, 0) u(x, 0) 1 1 0 0 0 0.5 1 0 0.5 1 x x  = 10−1  = 10−1 2 2 u(x, 0) u(x, 0) 1 1 0 0 0 0.5 1 0 0.5 1 x x Figure 3.7: Computed outflow profiles for test problem 1 10−1 . The P´eclet number ranges from 20 to 2×106 . An important quantity of interest to study for this problem is the steady-state outflow profile u(x, 0) for 0 ≤ x ≤ 1. In order to motivate the use of high-order methods, we compare the computed outflow profiles for a variety of grids and polynomial degrees. The mesh is taken to be a 111 regular grid of size 2n × n. The number of degrees of freedom is fixed at 800, and we consider three configurations: (n = 10, p = 1), (n = 4, p = 4), and (n = 2, p = 9). The outflow profiles for all choices of diffusion coefficient  are shown in Figure 3.7. We note that for the convection-dominated case of  = 10−6 , the lower-order methods severely underperform the higher-order methods. For  = 10−2 this effect is much less dramatic, however for the most diffusive case of  = 10−1 , the second-order method does not accurately capture the steep gradient observed near the origin. This suggests that there is a benefit to using moderate to high polynomial degrees rather than low degree polynomials on h-refined meshes. Now we turn our attention to solver and preconditioner performance. To study the effectiveness of the preconditioner, we compute the number of GMRES iterations required per linear solve. We consider four choices of diffusion coefficient . We also vary the time step ∆t for the unsteady version of this problem. As a baseline for our comparisons, we use the exact block Jacobi preconditioner. We then compare both the approximate Kronecker-product preconditioner which incorporates second-order terms through the sum-factorized interior penalty method (which we denote KSVD- IP) and the Kronecker-product preconditioner that did not incorporate the diffusion terms (which we denote KSVD). The iteration counts are presented in Table 3.3. We observe that the block Jacobi and Kronecker-product preconditioners exhibit extremely similar convergence properties. However, for large diffusion coefficient, and in particular for high degree p, we see that Kronecker-product preconditioner that does not include the diffusion term does not result in fast convergence. In particular, for  = 10−1 and p = 4, p = 9, GMRES did not converge to the steady- state solution in fewer than 2000 iterations. These results indicate a significant advantage to incorporating the diffusion terms using the interior penalty method. 112 Table 3.3: Number of GMRES iterations for Jacobi/KSVD-IP/KSVD preconditioner for convection-diffusion test case 1. A dash indicates no convergence in less than 2000 iterations.  = 10−6 ∆t p=1 p=4 p=9 1 × 10−2 7/7/7 6/6/6 6/8/8 2 × 10−2 7/7/7 6/8/8 6/9/9 4 × 10−2 8/8/8 6/8/8 5/10/10 8 × 10−2 10/10/10 6/8/8 5/12/12 1.6 × 10−1 15/15/15 7/9/9 6/14/14 Steady 25/25/25 12/12/12 9/9/9  = 2 × 10−3 ∆t p=1 p=4 p=9 1 × 10−2 7/7/7 9/10/10 12/14/14 2 × 10−2 9/9/9 11/12/12 15/17/19 4 × 10−2 10/10/10 15/15/15 19/23/25 8 × 10−2 14/14/14 19/20/21 23/30/33 1.6 × 10−1 21/21/21 25/26/27 28/35/41 Steady 59/59/59 53/55/64 49/57/82  = 10−2 ∆t p=1 p=4 p=9 1 × 10−2 10/9/10 14/14/15 18/23/24 2 × 10−2 12/12/12 18/18/18 22/26/29 4 × 10−2 16/16/16 22/22/24 26/30/39 8 × 10−2 22/22/22 28/28/34 31/36/58 1.6 × 10−1 33/33/33 36/36/52 39/46/94 Steady 96/98/97 64/69/171 67/84/382  = 10−1 ∆t p=1 p=4 p=9 1× 10−2 19/19/18 24/24/34 29/30/61 2 × 10−2 24/24/24 29/29/48 34/34/86 4 × 10−2 33/33/33 36/36/77 42/43/148 8 × 10−2 48/48/48 46/46/138 51/52/296 1.6 × 10−1 67/67/67 59/59/288 61/63/656 Steady 156/160/158 121/122/– 98/103/– 113 Test case 2 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 0.6 0.8 1.0 0.6 0.8 1.0 (a) Anisotropic mesh with (b) Solution contours show- p = 4 nodes ing boundary layer,  = 2 × 10−3 Figure 3.8: Convection-diffusion test problem 2 Table 3.4: Largest allowable time step for RK4, convection-diffusion test case 2  p=1 p=4 p=9 1× 10−6 1× 10−2 1× 10−2 4 × 10−3 2 × 10−3 2 × 10−4 8 × 10−5 2 × 10−5 1 × 10−2 6 × 10−5 1 × 10−5 4 × 10−6 1 × 10−1 6 × 10−6 1 × 10−6 4 × 10−7 The second test case is a slight modification of the above problem. We decompose the boundary ∂Ω = Γ1 ∪ Γ2 ∪ Γ3 . On the right boundary Γ1 = {x = 1, 0 ≤ y ≤ 1} we enforce a Dirichlet condition of gD = 100. On the outflow boundary, Γ2 = {0 ≤ x ≤ 1, y = 0}, we enforce a homogeneous Neumann condition. On the remaining boundaries, Γ3 , we enforce a homogeneous Dirichlet condition. The most important feature of this test case is the boundary layer that forms on the right boundary. In order to properly resolve this feature, we use an anisotropic mesh that is refined in the vicinity of the right boundary, see Figure 3.8b. We consider three mesh config- 114 urations, each with 3200 degrees of freedom, using degree 1, 4, and 9 polynomials. The anisotropy of the mesh, combined with the CFL condition resulting from the diffusion term, results in a severe time step restriction for explicit methods. In Table 3.4 we show the maximum stable time step for each configuration using the standard fourth-order explicit Runge-Kutta method. We measure the number of GMRES iterations required per linear solve for both the time-dependent case (as a function of ∆t), and for the steady case. The results are shown in Table 3.5. The number of iterations required with interior penalty Kronecker-product preconditioner is almost identical to the number of iterations required with the exact block Jacobi preconditioner for all cases considered. When comparing against the Kronecker-product preconditioner that did not include the diffusion term, the results are quite similar for either small diffusion coefficient, or for low polynomial degree. However, for the more diffusive cases, and for higher degree polynomials, we see a dramatic difference in the number of iterations required. For  = 10−2 or  = 10−1 and p = 4 or p = 9, the solver did not converge in under 2000 iterations for most of the test cases, unless the viscous terms were properly incorporated, demonstrating a marked improvement in preconditioner performance. 3.4.3 2D Euler vortex In this example, we consider the compressible Euler equations of gas dynamics in two dimensions, given in Section 1.4.3. We consider the model problem of an unsteady compressible vortex in a rectangular domain [132]. The domain is taken to be a 20 × 15 rectangle and the vortex is initially centered at (x0 , y0 ) = (5, 5). The vortex 115 Table 3.5: Number of GMRES iterations for Jacobi/KSVD-IP/KSVD preconditioner for convection-diffusion test case 2. A dash indicates no convergence in less than 2000 iterations.  = 10−6 ∆t p=1 p=4 p=9 10−2 6/6/6 6/6/6 8/8/7 2 × 10−2 9/9/9 8/8/8 10/10/9 4 × 10−2 15/15/15 11/11/10 12/12/12 8 × 10−2 23/23/23 14/14/14 14/14/14 1.6 × 10−1 31/31/31 18/18/18 15/15/16 Steady 45/44/45 24/24/24 18/19/19  = 2 × 10−3 ∆t p=1 p=4 p=9 10−2 28/28/28 45/45/106 52/53/280 2 × 10−2 36/36/36 61/61/163 74/74/673 4 × 10−2 49/49/49 84/84/281 102/103/– 8 × 10−2 70/70/71 110/110/503 136/137/– 1.6 × 10−1 103/103/103 136/136/755 178/180/– Steady 178/178/179 202/202/– 279/283/–  = 10−2 ∆t p=1 p=4 p=9 10−2 42/42/42 61/61/258 78/78/– 2 × 10−2 54/54/54 75/76/533 100/100/– 4 × 10−2 72/72/73 91/91/– 151/160/– 8 × 10−2 105/105/105 111/111/– 178/179/– 1.6 × 10−1 141/141/142 134/134/– 204/206/– Steady 230/230/231 197/197/– 312/305/–  = 10−1 ∆t p=1 p=4 p=9 10−2 83/83/85 89/89/– 120/120/– 2× 10−2 106/106/108 104/104/– 137/138/– 4 × 10−2 137/137/139 126/126/– 167/168/– 8 × 10−2 160/160/163 151/151/– 178/180/– 1.6 × 10−1 193/193/196 176/176/– 194/194/– Steady 310/310/313 259/258/– 290/299/– 116 1.0 15.0 0.9 12.5 0.8 10.0 0.7 7.5 0.6 5.0 0.5 2.5 0.4 0.3 0.0 0 5 10 15 20 0.2 ρ Figure 3.9: Initial conditions (density) for Euler vortex on unstructured mesh. is moving with the free-stream at an angle of θ. The exact solution is given by    ((y − y0 ) − vt) f (x, y, t) u = u∞ cos(θ) − exp , (3.91) 2πrc 2    ((x − x0 ) − ut) f (x, y, t) u = u∞ sin(θ) − exp , (3.92) 2πrc 2   γ−1 1 2 (γ − 1)M∞ 2 ρ = ρ∞ 1 − exp(f (x, y, t)) , (3.93) 8π 2   γ−1 γ 2 (γ − 1)M∞ 2 p = p∞ 1 − exp(f (x, y, t)) , (3.94) 8π 2 where f (x, y, t) = (1−((x−x0 )−ut)2 −((y −y0 )−vt)2 )/rc2 , M∞ is the Mach number, u∞ , ρ∞ , and p∞ are the free-stream velocity, density, and pressure, respectively. The free-stream velocity is given by (u, v) = u∞ (cos(θ), sin(θ)). The strength of the vortex is given by , and its size is rc . We choose the parameters to be M∞ = 0.5, u∞ = 1, θ = arctan(1/2),  = 0.3, and rc = 1.5. As in the case of the linear advection equation, we consider both a regular nx ×ny cartesian grid, and an unstructured mesh. The unstructured mesh is obtained by scaling the mesh in Figure 3.1b by 20 in the x-direction and 15 in the y-direction. Density contours of the initial conditions are shown in Figure 3.9. As opposed to the scalar advection equation, the solution to the Euler equations consists of multiple 117 components. Thus, the blocks of the Jacobian matrix can be considered to be of size nc (p + 1)2 × nc (p + 1)2 , where nc is the number of solution components (in the case of the 2D Euler equations, nc = 4). The exact block Jacobi preconditioner computes the inverses of these large blocks. The approximate Kronecker-product preconditioner find optimal approximation of the form A1 ⊗ B1 + A2 ⊗ B2 , where A1 and A2 are nc (p + 1) × nc (p + 1) matrices, and B1 and B2 are (p + 1) × (p + 1) matrices. We choose two representative time steps of ∆t = 0.1 and ∆t = 0.01, and com- pute the average number of GMRES iterations per linear solve required to perform one backward Euler time step. We choose the polynomial degree p = 3, 4, . . . , 15, and consider cartesian and unstructured meshes, both with 160 quadrilateral ele- ments. We present the results in Table 3.6. Very similar results are observed for the structured and unstructured results. We note that for the smaller time step, the approximate Kronecker-product preconditioner requires a very similar number of it- erations when compared with the exact block Jacobi preconditioner, even for high polynomial degree p. For the larger time step, the number of iterations required for the KSVD preconditioner increases with p at a faster rate when compared with the block Jacobi preconditioner, suggesting that the Kronecker-product preconditioner is most effective for moderate time steps ∆t. Performance comparison In this section we compare the runtime performance of the Kronecker-product pre- conditioner with the exact block Jacobi preconditioner. Although we have observed that for large time steps ∆t or polynomial degrees p, the KSVD preconditioner re- quires more iterations to converge, it is also possible to compute and apply this 118 Table 3.6: Number of GMRES iterations for Jacobi (J ) and Kronecker-product (K ) preconditioners, Euler equations on cartesian and unstructured grids, with ∆t = 0.1 and ∆t = 0.01. (a) Cartesian grid (b) Unstructured mesh ∆t = 0.01 ∆t = 0.1 ∆t = 0.01 ∆t = 0.1 p J K J K p J K J K 3 5 6 11 18 3 6 7 12 19 4 6 7 12 23 4 6 7 14 26 5 6 8 13 30 5 7 9 16 32 6 7 9 15 38 6 7 10 17 42 7 7 10 17 47 7 8 11 18 51 8 8 11 18 59 8 8 12 20 64 9 8 13 20 71 9 9 13 21 74 10 9 15 21 88 10 9 15 23 90 11 9 17 23 103 11 9 17 24 110 12 10 19 25 121 12 10 19 25 125 13 11 20 24 123 13 10 21 25 142 14 11 23 25 157 14 11 24 26 164 15 12 25 26 196 15 11 26 27 245 preconditioner much more efficiently. Here, we compare the wall-clock time required to compute and apply the preconditioner, according to Algorithms 3.5 and 3.6, re- spectively, for p = 3, 4, . . . , 15. The block Jacobi preconditioner is computed by first assembling the diagonal block of the Jacobian matrix using the sum-factorized form of expression (3.26), and then computing its LU factorization. The wall-clock times for these operations are shown in Figure 3.10a. We remark that we observe the expected asymptotic computational complexities for each of these operations, where forming the Jacobi preconditioner requires O(p6 ) operations, and applying the Jacobi preconditioner requires O(p4 ) operations. Both forming and applying the approximate Kronecker-product preconditioner can be done in O(p3 ) time. The total runtime observed per backward Euler step for ∆t = 0.1 and ∆t = 0.01 is shown in Figure 3.10b. We see that for p ≥ 7 and ∆t = 0.01, the Kronecker-product precon- ditioner results in overall faster runtime, while for ∆t = 0.1, because of the large number of iterations required per solve, the Jacobi preconditioner results in overall 119 100 Form KSVD KSVD, ∆t = 10−2 Form Jacobi 6 101 Jacobi, ∆t = 10−2 1 −1 10 Apply KSVD KSVD, ∆t = 10−1 3 Apply Jacobi Jacobi, ∆t = 10−1 Runtime (s) Runtime (s) 1 10−2 100 4 1 10−3 3 10−1 1 10−4 10−2 3 4 5 6 7 8 9 10 15 3 4 5 6 7 8 9 10 15 p p (a) Wall-clock time required to form and (b) Total wall-clock time required per back- apply the KSVD and Jacobi precondition- ward Euler step ers Figure 3.10: Runtime performance comparison of Kronecker-product preconditioner with exact block Jacobi preconditioner for 2D Euler equations faster performance. 3.4.4 2D Kelvin-Helmholtz instability For a more sophisticated test case, we consider a two-dimensional Kelvin-Helmholtz instability. This important fluid instability occurs in shear flows of fluids with differ- ent densities. The domain is taken to be the periodic unit square [0, 1]2 . We define the function 1 f (x) = (erf(α(x − 0.25)) + 1)(erf(α(0.75 − x)) + 1), (3.95) 4 where α = 100, as a smooth approximation to the discontinuous characteristic func- tion    1, 0.25 ≤ x ≤ 0.75 χ(x) = (3.96)   0, otherwise. 120 Following [123], we define the initial conditions by ρ(x, y) = f (y) + 1, u(x, y) = f (y) − 1/2, p(x, y) = 2.5, (3.97) where the vertical velocity is given by      1 (y − 0.25)2 (y − 0.75)2 v(x, y) = sin(4πx) exp − + exp − . (3.98) 10 2σ 2 2σ 2 Thus, the fluid density is equal to 2 inside the strip y ∈ [0.25, 0.75], and 1 outside the strip. The fluid is moving to the right with horizontal velocity 0.5 inside the strip, and is moving to the left with equal speed outside of the strip. A small perturbation in the vertical velocity, localized around the discontinuity, determines the large-scale behavior of the instability. The initial conditions are shown in Figure 3.11. We use a 128 × 128 cartesian grid, with polynomial bases of degree 3, 7, and 10. For 10th degree polynomials, the total number of degrees of freedom is 7,929,856. The Euler equations are integrated for 1.5 s using a fourth-order explicit method with a time step of ∆t = 2.5 × 10−5 on the NERSC Edison supercomputer, using 480 cores. At this point, the solution has developed sophisticated large- and small-scale features, as shown in Figure 3.12. We then linearize the Euler equations around this solution in order to test the preconditioner performance. Because of the varied scale of the features in this solu- tion, we believe that the resulting linearization is a representative of the DG systems we are interested in solving. Using this solution, we then solve one backward Eu- ler step using both the Jacobi and approximate Kronecker-product preconditioners. For the implicit solve, we choose a range of time steps, from the explicit step size of ∆t = 2.5 × 10−5 , to a larger step of ∆t = 10−3 . The number of GMRES iterations 121 1.0 2.0 1.0 0.100 0.075 0.8 1.8 0.8 0.050 0.6 1.6 0.6 0.025 0.000 0.4 1.4 0.4 −0.025 −0.050 0.2 1.2 0.2 −0.075 0.0 1.0 0.0 −0.100 0.0 0.2 0.4 0.6 0.8 1.0 ρ 0.0 0.2 0.4 0.6 0.8 1.0 v (a) Density (b) Vertical velocity Figure 3.11: Initial conditions for the Kelvin-Helmholtz instability. 1.0 2.0 0.90 2.0 0.8 1.8 1.8 0.85 0.6 1.6 1.6 0.80 0.4 1.4 1.4 0.75 0.2 1.2 1.2 0.0 1.0 0.70 1.0 0.0 0.2 0.4 0.6 0.8 1.0 ρ 0.00 0.05 0.10 0.15 0.20 ρ Figure 3.12: Solution (density) of Kelvin-Helmholtz instability at t = 1.5, with zoomed-in subregion to show small-scale features. required per linear solve are show in Table 3.7. We observe that for the explicit-scale time step, the exact block Jacobi and approximation Kronecker-product precondi- tioner exhibit very similar performance for all choices of p. For the largest time step, ∆t = 10−3 , the Kronecker-product preconditioner required about twice as many it- erations for p = 3, three times as many for p = 7, and four times as many iterations for p = 10. 122 Table 3.7: Number of GMRES iterations for Jacobi (J ) and Kronecker-product (K ) preconditioners, Euler equations for 2D Kelvin-Helmholtz instability p=3 p=7 p = 10 ∆t J K ∆t J K ∆t J K 2.5 × 10−5 4 4 2.5 × 10−5 5 6 2.5 × 10−5 6 8 5.0 × 10−5 5 5 5.0 × 10−5 6 8 5.0 × 10−5 8 11 1.0 × 10−4 6 7 1.0 × 10−4 8 12 1.0 × 10−4 10 16 2.5 × 10−4 8 10 2.5 × 10−4 12 20 2.5 × 10−4 14 31 5.0 × 10−4 10 14 5.0 × 10−4 15 35 5.0 × 10−4 19 55 1.0 × 10−3 13 22 1.0 × 10−3 21 62 1.0 × 10−3 24 106 3.4.5 2D NACA airfoil In this test case, we consider the viscous flow over a NACA 0012 airfoil with angle of attack 30◦ at Reynolds number 16000, with Mach number M0 = 0.2. We take the domain to be a disk of radius 10, centered at (0,0). The leading edge of the airfoil is placed at the origin. A no-slip wall condition is enforced at the surface of the airfoil, and far-field conditions are enforced at all other domain boundaries. The far-field velocity is set to unity in the freestream direction. The domain is discretized using an unstructured quadrilateral mesh, refined in the vicinity of the wing and in its wake. Isoparametric mappings are used to curve the elements on the airfoil surface. This flow is characterized by the thin boundary layer that develops on the airfoil. In order to resolve this boundary layer, we introduce stretched, anisotropic boundary- layer elements at the surface of the airfoil. These small elements result in a CFL condition that requires the use of very small time steps when using an explicit time integration method. The mesh and density contours are shown in Figure 3.13. This test case differs from the preceding two test cases because instead of the Euler equations we solve the compressible Navier-Stokes equations, given in Section 1.4.4. We integrate the equations until t = 2.5 in order to obtain a representative 123 10.0 1.02 4 1.00 7.5 5.0 0.98 2 2.5 0.96 0 0.94 0.0 0.92 −2.5 −2 0.90 −5.0 0.88 −7.5 −4 0.86 −10.0 −10 −5 0 5 10 −4 −2 0 2 4 ρ Figure 3.13: NACA 0012 mesh, with zoom-in around the surface of the airfoil showing anisotropic boundary-layer elements (top), and solution (density) initial condition about which to linearize the equations. We consider polynomial degrees p = 1, 3, 7, 10, 15, and compare the efficiency of explicit and implicit time in- tegration methods, using both the Kronecker-product preconditioner and exact block Jacobi. For this test case, we omit the viscous terms from the Kronecker-product preconditioner. In order to make this comparison, we determine experimentally the largest explicit timestep for which the system is stable. Then, we measure the wall- clock time required to integrate the system from t = 2.5 until t = 3.5. Similarly, for the implicit methods, we experimentally choose an appropriate ∆t measure the wall-clock time required to advance the simulation until t = 3.5 using a three-stage, third-order, L-stable DIRK scheme. We present these results in Table 3.8. We note that for p > 3, the Kronecker-product preconditioner results in the shortest runtimes. For large p, the exact block Jacobi preconditioner becomes impractical due to the large p-dependence of the computational complexity. For p = 15, the high degree 124 10.0 3 7.5 2 5.0 1 2.5 0.0 0 −2.5 −1 −5.0 −2 −7.5 −10.0 −3 −10 −5 0 5 10 −3 −2 −1 0 1 2 3 4 5 (a) Mesh of annulus A(1, 10) with p = (b) Solution (velocity magnitude) at t = 50 for 9 curved isoparametric elements flow over a cylinder at Re = 200 Figure 3.14: Mesh and computed solution for viscous compressible flow over a circular cylinder polynomials considered for this test case, the Kronecker preconditioner resulted in runtimes that were about a factor of two faster than explicit, and a factor of ten faster than exact block Jacobi. Table 3.8: Runtime results for NACA test-case, comparing explicit (fourth-order Runge-Kutta) with implicit (three-stage DIRK), using Kronecker and exact block Jacobi preconditioners. Runtime in seconds per simulation second is presented. Runtime (s) p Explicit ∆t Implicit ∆t RK4 K J 1 1.0 × 10−4 2.0 × 10−1 8.86 × 101 5.23 × 101 5.85 × 101 3 2.5 × 10−5 5.0 × 10−2 6.38 × 102 8.41 × 102 4.63 × 102 7 1.0 × 10−6 2.0 × 10−3 5.44 × 104 3.06 × 104 6.77 × 104 10 5.0 × 10−7 2.5 × 10−4 2.22 × 105 1.88 × 105 1.95 × 106 15 1.0 × 10−7 2.0 × 10−4 2.37 × 106 1.18 × 106 1.22 × 107 3.4.6 2D viscous flow over a circular cylinder We consider viscous compressible flow over a circular cylinder. The domain is given by, Ω = A(1, 10), the annulus with inner radius 1 and outer radius 10. As in the 125 previous examples, we consider three mesh configurations, corresponding to poly- nomial degrees p = 1, p = 4, p = 9. Each mesh has 6400 degrees of freedom per solution component. The coarsest mesh is shown in Figure 3.14a. A no-slip bound- ary condition is enforced at the inner boundary, and far-field conditions are enforced at the outer boundary. The Mach number is chosen to be M = 0.2, and we con- sider a range of Reynolds numbers: Re = 10, Re = 200, and Re = 1000. We start from freestream conditions and integrate in time until t = 1 in order to obtain a representative solution. At this point, we measure the number of GMRES iterations required per linear solve. We consider the time steps ∆t = 10−2 , 2 × 10−2 , 4 × 10−2 , and 8 × 10−2 . As before, we compare the effectiveness of three preconditioners: exact block Jacobi, the interior penalty approximate Kronecker-product preconditioner (KSVD-IP), and the Kronecker-product preconditioner that does not include viscous terms (KSVD). Each of these preconditioners is applied component-wise to the Jacobian matrix (i.e. with block size (p + 1)d × (p + 1)d , with nc diagonal blocks per element). A comprehensive comparison of iteration counts in shown in Table 3.9. We observe that the interior penalty Kronecker-product preconditioner is able to match the performance of the exact block Jacobi preconditioner for all polynomial de- grees considered, and at all choices of Reynolds number. In contrast, the Kronecker- product preconditioner that did not include viscous terms results in highly decreased performance at both low Reynolds numbers and high polynomial degrees. For the low-degree case of p = 1, the effect was extremely modest for all Reynolds numbers. This suggests that the proper incorporation of second-order terms is important for good preconditioner performance at high degrees. For the convection-dominated case of Re = 1000, the effect was modest except for at the largest time step ∆t = 8×10−2 . However, for the viscous-dominated case of Re = 10, the increase in iterations was 126 Table 3.9: Number of GMRES iterations for Jacobi/KSVD-IP/KSVD preconditioner for flow over a circular cylinder. A dash indicates no convergence in less than 2000 iterations. Re = 10 ∆t p=1 p=4 p=9 1 × 10−2 16/16/14 28/28/47 46/46/125 2 × 10−2 20/20/22 45/44/83 81/81/229 4 × 10−2 35/35/40 73/74/160 146/145/652 8 × 10−2 61/61/72 118/118/388 247/250/- Re = 200 ∆t p=1 p=4 p=9 1 × 10−2 17/17/13 21/21/18 27/30/35 2 × 10−2 21/21/21 32/33/37 56/60/78 4 × 10−2 35/35/36 70/71/95 128/134/257 8 × 10−2 65/65/67 176/179/383 371/379/1610 Re = 1000 ∆t p=1 p=4 p=9 1 × 10−2 17/17/14 22/22/19 29/31/31 2 × 10−2 22/22/22 37/37/39 61/66/73 4 × 10−2 36/36/36 90/91/100 213/218/384 8 × 10−2 66/66/66 293/297/509 1344/1554/- sizable for all time steps for degree p ≥ 4. 3.4.7 3D periodic Euler problem In this example, we provide a test case for the three-dimensional preconditioner. We solve the Euler equations on the cube [0, 2]3 with periodic boundary conditions, and 127 use the initial conditions from [67], given by ρ = 1 + 0.2 sin(π(x + y + z)), (3.99) u = 1, v = −1/2, w = 1, (3.100) p = 1. (3.101) The exact solution to the Euler equations is known analytically in this case. Velocity and pressure remain constant in time, and the density at time t is given by ρ = 1 + 0.2 sin(π(x + y + z − t(u + v + w))). (3.102) The initial conditions are shown in Figure 3.15. The mesh is taken to be a regular 6 × 6 × 6 hexahedral grid, and we consider polynomial degrees of p = 1, 2, . . . , 12. We choose a representative time step of ∆t = 2.5 × 10−3 . This time step can be used for explicit methods with low-degree polynomials, but because of the p-dependency of the CFL condition, we observe that for p ≥ 9, explicit methods become unstable, motivating the use of implicit methods. In order to compare the efficiency of the approximate Kronecker-product preconditioner, we compute one backward Euler step and compare the number of GMRES iterations required per linear solve using the exact block Jacobi preconditioner and the KSVD preconditioner. In this test case, we also consider two variations each of these preconditioners. Since the solution to the Euler equations consists of five components, we can consider the diagonal blocks of the Jacobian to either be large 5(p + 1)3 × 5(p + 1)3 blocks coupling all solution components, or as smaller (p + 1)3 × (p + 1)3 blocks, which do not couple the solution components. We expect that using the smaller blocks will require more GMRES iterations per linear solve, because each block captures less information. The larger blocks, on the other hand, are much more computationally 128 Figure 3.15: Initial conditions (density) for smooth 3D Euler test case. expensive to compute. We present the number of iterations required to converge for each of the preconditioners in Table 3.10, where J and K stand for the block Jacobi and Kronecker-product preconditioners, respectively, and the subscripts “full” and “small” refer to the block size used. We observe that for small polynomial degree p, the KSVD preconditioner results in close-to-identical number of iterations, when compared with exact block Jacobi. For this test case, for p closer to 12, we observe that the number of iterations grows faster for the KSVD preconditioner than for the block Jacobi preconditioner, but the difference remains relatively small. Additionally, as expected, the small-block preconditioner requires a greater number of iterations to converge when compared with the full-block preconditioner. We note that due to the very large memory requirements, the full Jacobi preconditioner did not complete for p = 12. We additionally measure the wall-clock time required per backward Euler step for each of the preconditioners, and present the results in Figure 3.16. For reference, we also include the wall-clock time required to integrate the system of equations for an equivalent time using the explicit RK4 method with a stable time step. This 129 p Jfull Kfull Jsmall Ksmall 103 1 4 4 5 5 KSVD (full) 2 4 5 6 6 Jacobi (full) 102 KSVD (small) 3 5 5 7 7 Runtime (s) 101 Jacobi (small) 4 5 5 8 8 RK4 5 5 6 9 10 100 6 5 6 11 12 7 5 6 12 13 10−1 8 5 7 15 16 9 5 7 16 17 10−2 10 5 8 19 21 1 2 3 4 5 6 7 8 9 12 11 5 8 20 22 p 12 - 9 23 26 Figure 3.16: Wall-clock time required per Table 3.10: GMRES iterations re- backward Euler solve for three-dimensional quired per linear solve for three- periodic Euler test case. For reference, dimensional periodic Euler test- wall-clock time required for explicit RK4 is case shown. problem is well-suited for explicit solvers, and thus the RK4 method is more effi- cient than the implicit methods. From these measurements, it is possible to draw several conclusions. Firstly, as p grows, it becomes possible to observe the O(p9 ) complexity of the exact block Jacobi preconditioner, which becomes prohibitively expensive. This is in contrast to the Kronecker-product preconditioner, whose O(p5 ) complexity results in reasonable runtimes for all p considered. The full-block KSVD preconditioner results in faster runtimes starting at about p = 5, and the small- block preconditioner at about p = 4. We also see that, despite the larger number of iterations required, the small-block preconditioner results in faster overall runtime. 3.4.8 3D compressible Taylor-Green Vortex at Re = 1600 The direct numerical simulation of the compressible Taylor-Green vortex at Re = 1600 was selected as a difficult benchmark test problem for the first International 130 Workshop on High-Order CFD Methods [132]. This problem has been much studied [20, 38, 109, 134], and fully-resolved reference data are available, making this test case particularly useful. The spatial domain is taken to be the periodic cube −π ≤ x, y, z ≤ π. The initial conditions are given by u(x, y, z) = u0 sin(x) cos(y) cos(z) (3.103) v(x, y, z) = −u0 cos(x) sin(y) cos(z) (3.104) w(x, y, z) = 0 (3.105) ρ0 u20 p(x, y, z) = p0 + (cos(2x) + cos(2y)) (cos(2z) + 2) , (3.106) 16 where we take the parameters to be γ = 1.4, Pr = 0.71, u0 = 1, ρ0 = 1, with Mach number M0 = u0 /c0 = 0.10, where c0 is the speed of sound computed in accordance with the pressure p0 . The initial density distribution is then given by ρ = pρ0 /p0 . The characteristic convective time is given by tc = 1, and the final time is tf = 20 tc . We discretize the geometry using a sequence of regular hexahedral grids, and consider polynomial degrees varying from p = 3 to p = 15. The number of numerical degrees of freedom per solution component ranges from 643 to 2403 . A table of configurations, including grid size (number of elements per dimension), polynomial degree, and total number of numerical degrees of freedom per solution component is listed in Table 3.11. We compare our computed solution at several resolutions with a known reference solution computed using a pseudo-spectral method with 5123 degrees of freedom per solution component. In order to compare the results, we consider several quantities 131 Table 3.11: Grid configurations for Taylor-Green problem Grid size nx p Numerical DOFs 8 7 643 8 15 1283 16 3 643 16 7 1283 32 3 1283 48 4 2403 integrated over the domain. First, we consider the mean kinetic energy, Z 1 u·u Ek (t) = ρ dx, (3.107) ρ0 |Ω| Ω 2 where |Ω| = (2π)3 is the volume of the domain. From the time-evolution of this quantity, we can also compute the kinetic energy dissipation rate (KEDR), given by dEk (t) = − (t). (3.108) dt Additionally, we consider the mean enstrophy over the domain, Z 1 ω·ω E(t) = ρ dx. (3.109) ρ0 |Ω| Ω 2 For incompressible flow, the relationship  = 2µE/ρ0 holds, and for low-Mach compressible flow, this relationship holds approximately. Indeed, we notice that the profiles of E and  are close to indistinguishable, indicating that compressible effects do not play a large role in this test problem. For each of the configurations listed above, we compare the results with the pseudo-spectral reference solution in Figure 3.17a. In Figure 3.17b, we present plots of the same quantities, zoomed in around the point of peak dissipation at about t = 9 tc . Additionally, the isosurfaces of the vorticity norm |ω| at times t = 0, 2, 4, 6, 8, 10 tc are shown in Figure 3.18, illustrating 132  Reference 5123 p = 3, nx = 16 0.10 p = 3, nx = 32 p = 4, nx = 48 Ek p = 7, nx = 8 0.05 p = 7, nx = 16 p = 15, nx = 8 0 2 4 6 8 10 12 14 16 18 20 t  = −dEk /dt 0.01 0.01 0.00 0 2 4 6 8 10 12 14 16 18 20 t 10.00 E 5.00 0.00 0 2 4 6 8 10 12 14 16 18 20 t (a) Time evolution of diagnostics for t ∈ [0, 20tc ]  0.10 Reference 5123 p = 3, nx = 16 Ek 0.08 p = 3, nx = 32 p = 4, nx = 48 0.06 p = 7, nx = 8 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 p = 7, nx = 16 t p = 15, nx = 8  = −dEk /dt 0.01 0.01 0.01 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 t 10.00 8.00 E 6.00 4.00 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 t (b) Zoom-in around point of peak dissipation. Figure 3.17: Time evolution of kinetic energy Ek , kinetic energy dissipation rate (KEDR) , and enstrophy E, for Taylor-Green test case. Comparison of various DG configurations with reference pseudo-spectral solution. 133 the transition to turbulence, and subsequent decay. The simulation with 2403 degrees of freedom per component (p = 4, nx = 48) reproduces almost exactly all three of the profiles. We note that even for the severely under-resolved cases with 643 degrees of freedom, the kinetic energy profiles closely match the reference data. However, the low-order p = 3 discretizations tend to under- predict the enstrophy and KEDR. With an equal number of degrees of freedom, the p = 7, nx = 8 discretization more closely matches the reference profiles than the p = 3, nx = 16 configuration, providing accuracy comparable to the p = 3, nx = 32 case, with one eighth the number of degrees of freedom. Similarly, despite the exceed- ingly coarse mesh, the p = 15, nx = 8 configuration captured the diagnostics more accurately than all other configurations considered with equal number of degrees of freedom. This p = 15 discretization successfully captured the peak enstrophy and energy dissipation rate, as shown in Figure 3.17b, and, in fact, matches the reference data comparably to the finest discretization with nx = 48. The improved accuracy per degree of freedom for higher polynomial degree p motivates the use of very high order methods. We now examine the efficiency of the approximate tensor-product preconditioner for the Taylor-Green test case, for each of the configurations listed in Table 3.11. For each configuration, we choose a range of timesteps, the smallest timestep cor- responding to ∆t = 10−4 . We then increase the timestep four times in multiples of two, until we obtain a largest timestep of ∆t = 1.6 × 10−3 . We measure the number of iterations required to perform one backward Euler step (i.e. Newton solve) to find the average number of iterations per linear solve. We also compute the average number of iterations per linear solve using the exact block Jacobi preconditioner, for all configurations except for p = 15, nx = 8, for which the test did not complete because of its excessive memory and runtime requirements. The iteration counts are 134 displayed in Table 3.12. Using this methodology, we can compare the effectiveness of the approximate Kronecker-product preconditioner with exact block Jacobi. As expected, for both the Jacobi and Kronecker preconditioners, we observe that the number of iterations per linear solve increases with ∆t. We also observe that in both cases, this dependence is sublinear, with the exception of the case p = 15, nx = 8, ∆t = 1.6 × 10−3 . For a majority of cases, the Kronecker-product preconditioner resulted in very similar (or sometimes identical) iteration counts. We conclude that the Kronecker-product preconditioners are able to achieve iteration counts very similar to the exact block Jacobi preconditioner, but with large savings in terms of computational complexity and memory requirements. For the block Jacobi preconditioner, storing the diagonal block requires (nc (p + 1)3 )2 memory per block, and computing the LU factorization requires O(p9 ) operations. On the other hand, the memory requirements for the Kronecker-product preconditioner grows linearly in p with the number of degrees of freedom, while the computational complexity scales as O(p5 ), indicating that a large savings in computational complexity can be achieved at the cost of a modest increase in iteration count. 3.5 Summary In this chapter, we have developed new approximate tensor-product based precondi- tioners for very high-order discontinuous Galerkin methods. These preconditioners are computed using an algebraic singular value-based algorithm, and compare favor- ably with the traditional block Jacobi preconditioner. The computational complexity is reduced from O(p6 ) to O(p3 ) in two spatial dimensions and O(p9 ) to O(p5 ) in three spatial dimensions. Numerical results in two and three dimensions for the advection 135 Table 3.12: Average number of GMRES iterations per Newton solve for Taylor-Green test case (a) p = 3, nx = 16 (b) p = 3, nx = 32 ∆t Jacobi KSVD ∆t Jacobi KSVD 1 × 10−4 4 4 1 × 10−4 5 5 2 × 10−4 5 7 2 × 10−4 6 6 4 × 10−4 6 7 4 × 10−4 7 8 8 × 10−4 8 9 8 × 10−4 10 12 1.6 × 10−3 11 12 1.6 × 10−3 16 18 (c) p = 4, nx = 48 (d) p = 7, nx = 8 ∆t Jacobi KSVD ∆t Jacobi KSVD 1 × 10−4 7 7 1 × 10−4 5 6 2 × 10−4 9 10 2 × 10−4 6 6 4 × 10−4 12 16 4 × 10−4 8 8 8 × 10−4 19 27 8 × 10−4 10 11 1.6 × 10−3 30 40 1.6 × 10−3 15 16 (e) p = 7, nx = 16 (f) p = 15, nx = 8 ∆t Jacobi KSVD ∆t Jacobi KSVD 1 × 10−4 6 6 1 × 10−4 – 8 2 × 10−4 7 9 2 × 10−4 – 11 4 × 10−4 10 13 4 × 10−4 – 15 8 × 10−4 15 17 8 × 10−4 – 26 1.6 × 10−3 25 29 1.6 × 10−3 – 88 136 (a) t = 0 tc (b) t = 2 tc (c) t = 4 tc (d) t = 6 tc (c) t = 8 tc (d) t = 10 tc Figure 3.18: Compressible Taylor-Green vortex at Re = 1600, isosurfaces of vorticity norm |ω|, colored by helicity H = ω · u 137 and Euler equations, using polynomial degrees up to p = 30, confirm the expected computational complexities, and demonstrate significant reductions in runtimes for certain test problems. Additionally, we developed an extension of this preconditioner to second-order equations based on an interior penalty discretization. Incorporat- ing second-order terms into the preconditioner formulation was shown to result in a significant benefit to preconditioner performance. Chapter Four Iterative convergence of polygonal elements 139 Most work on DG methods has been based on meshes of either simplex elements (triangles and tetrahedra), block elements (quadrilaterals and hexahedra), or com- binations of these such as prism elements. This is likely because of the availability of excellent automatic unstructured mesh generators, at least for the simplex case [100, 112, 118], and also because of the advantages with the outer-product struc- ture of block elements, as described in detail in Chapter 3 of this thesis. However, it is well known that since no continuity is enforced between the elements, it is straightforward to apply the DG methods to meshes with elements of any shapes (even non-conforming ones). For example, vertex-centered DG methods based on the polygonal dual meshes were studied in [16, 77]. This is a major advantage over standard continuous FEM methods, which need significant developments for the extension to arbitrary polygonal and polyhedral elements [82]. In the finite volume CFD community, there has recently been considerable inter- est in meshes of arbitrary polygonal and polyhedral elements. In fact, the popular vertex-centered finite volume method applied to a tetrahedral mesh can be seen as a cell-centered method on the dual polyhedral mesh. Because of this, a number of methods have been proposed for generation of polyhedral meshes, which in many cases have advantages over traditional simplex meshes [53, 89]. Although it is still unclear exactly what benefits these elements provide, they have been reported to be both more accurate per degree of freedom and to have better convergence prop- erties in the numerical solvers than for a corresponding tetrahedral mesh [7, 101]. There have also been studies showing that vertex-centered schemes are preferred over cell-centered [42, 43], again indicating the benefits of polyhedral elements. Inspired by the promising results for the polyhedral finite volume method, and since DG is a natural higher-order extension of these schemes, in this chapter we study some of the properties of DG discretizations on polygonal meshes. To limit 140 the scope, we only investigate the convergence properties of iterative solvers for the discrete systems, assuming an equal number of degrees of freedom per unit area for all element shapes. We first consider the iterative block-Jacobi method applied to a pure convection problem, which in the constant coefficient case can be solved ana- lytically using von Neumann analysis. Next we apply the solver to Euler’s equations of gas dynamics for relevant model flow problems, to obtain numerical results for the convergence of the various element shapes. We consider regular meshes of hexagons, squares, and two different configurations of triangles, as well as the dual of fully unstructured triangular Delaunay refinement meshes. We also perform numerical experiments with the GMRES Krylov subspace solver and a block-ILU precondi- tioner. Although the results are not entirely conclusive, most of the results indicate a clear benefit with the hexagonal and quadrilateral elements over the triangular ones. This chapter is organized as follows. In Section 4.1, we describe the implicit temporal discretization, and introduce the iterative solvers. In Section 4.2 we perform the von Neumann analysis of the constant coefficient advection problem, in 1D and for several mesh configurations in 2D. In Section 4.3 we show numerical results for more general advection fields, for more general meshes, as well as for the Euler equations and the GMRES solver. We conclude with a summary of our findings as well as directions for future work. 4.1 Numerical methods In this chapter, we retain the same discontinuous Galerkin spatial discretization described in Section 1.1, with the distinction that we allow the computational mesh 141 Th to consist of arbitrary straight-sided polygonal elements in R2 . Instead of the nodal bases considered earlier in this thesis, we choose to use basis functions defined by the orthogonal Legendre polynomials defined on the bounding box of each mesh element. We consider hyperbolic conservation laws of the form (1.1), and obtain a semi-discrete system of equations of the form (1.17). We begin by considering the scalar advection equation (1.48), which, once dis- cretized, results in a system of linear ODEs M ut + Lu = 0, (4.1) where L is the linear operator corresponding to the DG residual. 4.1.1 Temporal integration and linear solvers We consider the solution of (4.1) by means of implicit time integration schemes, the simplest of which is the standard backward Euler scheme, (M + kL)un+1 = M un (4.2) Furthermore, each stage of a higher-order scheme, such as a diagonally-implicit Runge-Kutta (DIRK) scheme (cf. Chapter 2 of this thesis), can be written as a similar equation. The block sparse system can be solved efficiently by means of an iterative linear solver. In this chapter, we consider two solvers: the simple block Jacobi method, and the preconditioned GMRES method. 142 Block Jacobi method A popular and simple iterative solver is the block Jacobi method [39], defined as follows. Each iteration of the method for solving the linear system Ax = b is given by x(n+1) = D−1 b + RJ x(n) , (4.3) where D is the block-diagonal part of A, and RJ = I − D−1 A. This simple method has the advantage that it is possible to analyze the convergence properties of the method simply by examining the eigenvalues of the matrix RJ . An upper bound of 1 for the absolute value of the eigenvalues of the matrix RJ is a necessary and sufficient condition in order for Jacobi’s method to converge (for any choice of initial vector x(0) ). The spectral radius of RJ determines the speed of convergence. Preconditioned GMRES method Another popular and oftentimes more efficient [8, 104] method for solving large, sparse linear systems is the GMRES (generalized minimal residual) method [113]. As with most Krylov subspace methods, the choice of preconditioner has great impact on the efficiency of the solver [107]. A simple and popular choice of preconditioner is the block Jacobi preconditioner. Each application of this preconditioner is performed by multiplying by the inverse of the block-diagonal part of the matrix. Another, often more effective choice of preconditioner is the block ILU(0) preconditioner [40]. This preconditioner produces an approximate block-wise LU factorization, whose sparsity pattern is enforced to be the same as that of the original matrix. This factorization can be performed in-place, and requires no more storage that the original matrix. Unlike the block Jacobi method, the block ILU(0) preconditioner can be highly 143 sensitive to the ordering of the mesh elements [15, 46]. Because of this property, it is common to combine the use of ILU preconditioners with certain orderings of the mesh elements designed to increase efficiency, such as reverse Cuthill-McKee [36], minimum degree [83], nested dissection [54], or minimum discarded fill [104]. In this paper, we focus our study on the block Jacobi method, which is simpler and more amenable to analysis. We then perform numerical experiments using both the block Jacobi method and the preconditioned GMRES method using ILU(0) and block Jacobi preconditioning. 4.2 Jacobi analysis We compare tessellations of the plane by four sets of generating patterns, each con- sisting of one or more polygons. We consider tessellations consisting of squares, regular hexagons, two right triangles, and two equilateral triangles. The generating patterns considered are shown in Figure 4.1. Each generating pattern Gj consists of fj . We will refer to these generating patterns one or two elements, labeled Kj and K as S, H, R, and E for squares, hexagons, right triangles, and equilateral triangles, respectively. We are interested in computing the spectral radius of the Jacobi matrix RJ that arises from the discontinuous Galerkin discretization on the mesh resulting from tessellating the plane by each of the four generating patterns. For the sake of comparison, we choose the elements from each of the generating patters to have the same area. Therefore, if the side length of the equilateral triangle is hE = h, then √ 4 the two equal sides of the isosceles right triangle have side length hR = √ 3 hE , the 2 144 Kj Kj (a) Square Cartesian grid (b) Regular hexagons Kj fj K fj K Kj (c) Isosceles right triangles (d) Equilateral triangles Figure 4.1: Examples of generating patterns Gj shown with bolded lines. Neighbor- ing elements are shown unbolded. √ 4 √1 hE , 3 hexagon has side length hH = 6 and the square has side length hS = 2 hE . Then, the global system will have the same number of degrees of freedom regardless of choice of generating pattern. 4.2.1 von Neumann analysis First, we compare the efficiency of each of the four types of generating patterns when used to solve the advection equation (1.48) with the discontinuous Galerkin spatial discretization and implicit time integration. We compute the spectral radius of the matrix RJ using the classical von Neumann analysis for each of the generating patterns, in a manner similar to [74]. 145 Let u denote the solution vector. We introduce the notation Uj , which is itself a vector, denote the degrees of freedom of u in Gj , the jth generating pattern. We remark that in the case of squares and hexagons, this corresponds exactly to the degrees of freedom in the element Kj , but in the case of the triangular generating patters, this corresponds to the degrees of freedom from both of the elements Kj and fj . In order to determine the eigenvalues of RJ , we consider the planar wave with K wavenumber (nx , ny ) defined by b, Uj = ei(nx xj +ny yj ) U (4.4) where (xj , yj ) are fixed coordinates in Gj . Then, we let ` index the generating patterns neighboring Gj , and we let δ` = (δx` , δy` ) = (xj − x` , yj − y` ) be the offsets satisfying Gj + δ` = G` . We can then write the solution in each of the neighboring generating patterns as U` = ei(nx δx` +ny δy` ) Uj . (4.5) In this case we write the semi-discrete equations (4.1) in the following form X Mj (∂t Uj ) + ei(nx δx` +ny δy` ) Lj` Uj = 0, (4.6) ` where the summation over ` ranges over all neighboring generating patterns, Mj denotes the diagonal block of M corresponding to the jth generating pattern, and Lj` denotes the block of L in the jth row and `th column. We can write X bj = L ei(nx δx` +ny δy` ) Lj` (4.7) ` to further simplify and obtain b) + L Mj (∂t U b = 0. bj U (4.8) 146 In order to solve equation (4.8) using an implicit method, we consider the backward Euler-type equation b n+1 = Mj U b j )U (Mj + k L b n. (4.9) The Jacobi iteration matrix RJ can then be written as cJ j = I − D−1 (Mj + k L R bj ), (4.10) where the matrix D = Mj + kLjj consists of the jth diagonal block of M + kL. The cJ j control the speed of convergence of Jacobi’s method. eigenvalues of the matrix R In the simple cases of piecewise constant functions (p = 0), or in the case of a one-dimensional domain, the eigenvalues can be computed explicitly. In the more complicated case of p ≥ 1 in a two-dimensional domain, we study these eigenvalues numerically. 4.2.2 1D analysis To illustrate the von Neumann analysis, we consider the one-dimensional scalar ad- vection equation ut + ux = 0 (4.11) on the interval [0, 2π] with periodic boundary conditions. We divide the domain into nt subintervals Kj , each of length h. Let u denote the solution vector, and let Uj denote the degrees of freedom for the jth interval Kj . For example, if piecewise constants are used, the method is identical to the upwind finite volume method, and each Uj represents the average of the solution over the interval. If piecewise polynomials of degree p are used, each Uj is a vector of length p + 1. 147 For the purposes of illustration, we choose p = 1, and let Uj = (uj,1 , uj,2 ) rep- resent the value of the solution at the left and right endpoints of the interval Kj . Then, the local basis on the interval Kj consists of the functions φj,1 (x) = j − x/h, φj,2 (x) = x/h − j + 1. (4.12) We remark that the upwind flux in this case is always equal to the value of the function immediately to the left of the boundary point: h ijh Fb(u− h , u+ h )v h (x) = uj,2 vj,2 − uj−1,2 vj,1 . (4.13) (j−1)h The entries of the jth block of the mass matrix M are given by Z jh (Mj )i` = φj,i (x)φj,` (x) dx. (4.14) (j−1)h Additionally, we remark that the diagonal blocks of L consist of the volume integrals and right boundary terms given by Z jh (Ljj )i` = φj,i (jh)φi,` (jh) − φ0j,i (x)φj,` (x) dx. (4.15) (j−1)h We let A denote the backward Euler-type operator defined by A = M + kL, (4.16) and, solving the equation Ax = b by means of Jacobi iterations, we define the Jacobi matrix RJ by RJ = I − D−1 A, (4.17) where D is the matrix consisting of the diagonal blocks of A. The entries of the 148 diagonal blocks Mj and Ljj can be computed explicitly using (4.12) to obtain       h h 1 1 h k h k  3 6   2 2   3 + 2 6 + 2  Mj =  , Ljj =  , Dj =  . (4.18) h h 6 3 − 21 1 2 h 6 − k 2 h 3 + k 2 In order to perform the von Neumann analysis, we seek solutions of the form b , which allows us to explicitly compute the form of the matrix L Uj = einhj U bj . Recalling the compact form from (4.8), we obtain   1 1 − e−ihn  bj =  L  2 2 . (4.19) − 21 1 2 cJ j is given by Then, the Jacobi matrix R   2e−ihn k(2h+3k) 0 cJ j =  R   h2 +4kh+6k2 , (4.20) 2e−ihn (h−3k)k 0 − h2 +4kh+6k2 whose eigenvalues λ1 and λ2 are given by 2k(3k − h)e−ihn λ1 = 0, λ2 = . (4.21) h2 + 4hk + 6k 2 Therefore, each wavenumber n from 0 to 2π/h corresponds to an eigenvalue of the Jacobi matrix RJ , and the magnitude of these eigenvalues determine the speed of convergence of Jacobi’s method. In this case, the expression 2k |h − 3k| λmax = (4.22) h2 + 4hk + 6k 2 determines the speed of convergence of Jacobi’s method. This expression can easily be seen to be bounded above by 1 for all positive values of h and k, therefore indicat- 149 ing that Jacobi’s method is guaranteed to converge regardless of spatial resolution or time step. 4.2.3 2D analysis We now turn to the analysis of the four generating patterns shown in Figure 4.1. The analysis proceeds along the same lines as in the one-dimensional example from Section 4.2.2. As an example, we present the case of piecewise constants, for which it is possible to explicitly compute the eigenvalues of the Jacobi matrix RJ . In this case the discontinuous Galerkin formulation simplifies to the upwind finite volume method Z I ∂t uh dx + Fb (u+ , u− ) · n− ds = 0. (4.23) Kj ∂Kj For the sake of concreteness, we assume without loss of generality that the velocity vector β = (α, β) satisfies α, β ≥ 0. In order to explicitly write the upwind flux on the meshes consisting of hexagons and equilateral triangles, we further assume that √ 3α−β ≥ 0, and on the mesh consisting of right triangles we assume that α−β ≥ 0. In the case of the square and hexagonal meshes, there is only one degree of freedom per generating pattern, and we will write uj to represent the average value of the solution over the generating pattern Gj . We then consider the planar wave with wavenumber (nx , ny ) given by uj = ei(nx xj +ny yj ) u b. In the case of the square mesh √ 4 3 with side length hS = 2 hE , the method can be written as  b) = −hS α(1 − e−inx hS ) + β(1 − e−iny hS ) u h2S (∂t u b. (4.24) In this case, the mass matrix M is a diagonal matrix with h2S along the diagonal, and the diagonal entries of the matrix L are given by hS (α + β). Therefore, the 150 eigenvalues of the Jacobi matrix RJS = I − D−1 (M + kL) are given by 1  λ(RJS ) = 1 − h2 + hS k α(1 − e−inx hS ) + β(1 − e−iny hS ) hS + hS k(α + β) S 2 −inx hS −iny hS  k αe + βe = . hS + k(α + β) (4.25) In the case of the hexagonal mesh with side length hH = √1 hE , the method is 6 √ √   √  √ 3 3 2   3 β ihH − 32 nx + 23 ny b) = −hH h (∂t u 3α + β + − 2 α + 2 e 2 H ! (4.26)  √   √  √ β ihH − 32 nx − 23 ny + − 2 3 α − 2 e − βe−ihH 3ny b. u A similar analysis shows that the eigenvalues of the matrix RJH are given by 1 √ √  1 √ √  √  ke− 2 ihH (3nx + 3ny ) 3β 2e 2 ihH (3nx − 3ny ) −ei 3hH ny +1 +3α(1+ei 3hH ny ) λ(RJH ) = √ . (4.27) 9hH +6αk+2 3βk In the case of the two triangular meshes, there are two degrees of freedom per fj in the generating generating pattern, corresponding to the elements Kj and K pattern Gj . We write Uj = (uj,1 , uj,2 ), where uj,1 is the average of the solution over fj . The planar wave the element Kj , and uj,2 is the average of the solution over K b , for U solution is then given by Uj = ei(nx xj +ny yj ) U b = (b b2 ). We consider the case u1 , u of a right-triangular mesh, where the two equal sides of the isosceles right triangles √ 4 have length hR = √ 3 hE . The method then reads: 2     b1   u 2  αbu1 − e−ihR nx αb u2  ∂t   = −  . (4.28) b2 u hR u2 + (β − α)b αb u1 − e −ihR ny βb u1 In the case of the mesh consisting of equilateral triangles, each with side length hE , 151 the method reads:    √    √    3 1 3 1 b1   u  2 α + 2β ub1 + e−ihE nx − 2 α + 2 β − e−ihE ny β ub2  ∂t  = √−4   √  √  . 3hE 3 3 1 1 b2 u − 2 α − 2β u b1 + 2 α + 2 β u b2 (4.29) Computing the eigenvalues of the corresponding Jacobi matrices RJR and RJE , we obtain 1 √ p 2ke− 2 ihR (nx +ny ) α β + (α − β)eihR ny λ(RJR ) =± , (4.30) hR + 2αk √ q √  2k 3α + 3β 2βeihE nx + 3α − β eihE ny λ(RJE ) = ± √ q √  . (4.31) 3hE + 6αk + 2 3βk 3α + β e ih E (nx +n y ) Then, equations (4.25), (4.27), (4.30), and (4.31) completely determine the speed of convergence for Jacobi’s method of each of the four generating patterns considered. In the case of a higher-order discontinuous Galerkin method with basis consisting of piecewise polynomials of degree p > 0, we obtain a Jacobi matrix given by equation cJ j , D, Mj , and L (4.10), where the matrices R bj are (p+1)(p+2) × (p+1)(p+2) blocks. In 2 2 this case, we do not obtain closed-form expressions for the eigenvalues, but rather compute them numerically. We normalize the velocity magnitude and consider β = (cos(θ), sin(θ)). On the square mesh, θ can range from 0 to π/2. On the hexagonal and equilateral triangle meshes, θ ranges from 0 to π/3, and on the right-triangular mesh θ ranges from 0 to π/4. We consider a fixed spatial resolution h, and compare the efficiency of the four patterns for three choices of temporal resolution. We first consider an “explicit” time step, satisfying the CFL-type condition h kexp = . (4.32) |β| 152  Table 4.1: Ratio of logarithm of eigenvalues log λmax (RJmin ) / log (λmax (RJ∗ )) ranging over angle θ and wavenumber (nx , ny ), for piecewise polynomials of degree 0, 1, 2, and 3, for varying choices of time step k. The smallest eigenvalue in each column is highlighted. p=0 p=1 k1 k2 k3 k1 k2 k3 Hexagons 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 Squares 1.128939 1.133989 1.136772 1.058098 1.118222 1.130101 Right triangles 1.128939 1.133989 1.136772 1.084223 1.132326 1.137313 Equilateral triangles 1.207328 1.215467 1.219948 1.137267 1.201638 1.214376 p=2 p=3 k1 k2 k3 k1 k2 k3 Hexagons 1.000000 1.000000 1.000000 1.077183 1.070785 1.066101 Squares 1.095785 1.118510 1.129314 1.000000 1.000000 1.000000 Right triangles 1.111863 1.126951 1.133634 1.010482 1.005391 1.002733 Equilateral triangles 1.177503 1.201918 1.213527 1.074570 1.074570 1.074570 As one advantage of using an implicit method is that we are not limited by an explicit time step restriction of the form (4.32), we consider three implicit time steps given by k1 = 3kexp , k2 = 2k1 , and k3 = 4k1 . We then maximize over a discrete sample of θ ∈ [0, π/4] and over all wavenumbers (nx , ny ), in order to compute maximum eigenvalue for each of the generating patterns. As the number of iterations required to converge to a given tolerance scales like the reciprocal of the logarithm of the spectral radius, we compare the efficiency of the generating patterns by considering the ratio  log λmax (RJmin ) , log (λmax (RJ∗ )) where λmax (RJ∗ ) is the largest eigenvalue of RJ∗ , for ∗ = H, S, R, E, and λmax (RJmin ) is the smallest among all λmax (RJ∗ ). This ratio corresponds to the ratio of iterations required to converge to a given tolerance when compared with the most efficient among the generating patterns. The results obtained for p = 0, 1, 2, 3, and k = k1 , k2 , k3 for each generating pattern are shown in Table 4.1 and Figure 4.2. 153 1.2 1.2 1 1 0.8 0.8 k1 k2 k3 k1 k2 k3 (a) p = 0 (b) p = 1 1.2 1.2 1 1 0.8 0.8 k1 k2 k3 k1 k2 k3 (c) p = 2 (d) p = 3 Hexagons Squares Right Triangles Equilateral Triangles Figure 4.2: Ratios of the logarithm of the largest eigenvalues for generating patterns We remark that for degrees 0, 1, and 2 polynomials, the hexagonal mesh resulted in the smallest eigenvalues for all choices of time step considered, and the square mesh resulted in the second-smallest eigenvalues. For degree 3 polynomials, the square mesh resulted in the smallest eigenvalues for all cases considered. We notice a significant decrease in the expected performance of the hexagonal elements in the case of p = 3, although we have noticed that the effect observed in practice is not as significant as the theoretical results would suggest. 154 4.3 Numerical results 4.3.1 Advection with variable velocity field To perform numerical experiments extending the analysis of the advection equation (1.48) beyond the case of a constant velocity β, we consider a variable velocity field β(x, y). In this case, we evaluate the upwind numerical flux (1.51) point-wise. As an example, we define the velocity to be given by the vector field β(x, y) = (2y − 1, −2x + 1) on the spatial domain Ω = [0, 1] × [0, 1]. This velocity field is shown in Figure 4.3. We let the initial conditions be given by the Gaussian centered at (x0 , y0 ) = (0.35, 0.5), u0 (x, y) = exp(−150((x − x0 )2 + (y − y0 )2 )). (4.33) The exact solution is periodic with period π, and is given by the rotation about the center of the domain, u(x, y, t) = exp(−150((x−0.5+0.15 cos(2t))2 +(y −0.5−0.3 cos(t) sin(t))2 )). (4.34) Convergence of the block Jacobi method We consider meshes of the domain created by repeating each of the four generating patterns considered in the previous section. As before, for fixed spatial resolution h, we choose hH , hS , hR , and hE such that the number of degrees of freedom is the same for each mesh. We then solve the advection equation using the backward Euler time discretization, where the block Jacobi iterative method is used to solve the resulting 155 1.0 0.8 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Figure 4.3: Velocity field β(x, y) = (2y − 1, −2x + 1) Table 4.2: Iterations required for the block Jacobi iterative method to converge in the case of a non-constant velocity field. The smallest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Hexagons 33 57 104 21 41 77 24 41 77 21 39 75 Squares 35 61 109 21 42 83 22 42 83 22 42 81 Right triangles 39 68 128 26 51 100 25 51 100 25 51 100 Equilateral triangles 37 67 123 25 47 92 25 47 92 24 47 91 linear system. The zero vector is used as the starting vector for the block Jacobi √ solver. We choose h = 0.05, and since max(x,y) |β(x, y)| = 2, we consider time √ steps of k1 = h/ 2, k2 = 2k1 , k3 = 4k1 . The number of iterations required for the block Jacobi method to converge to a tolerance of 10−14 are given in Table 4.2. The results are similar to those from the analysis performed in Section 4.2.3. We note that the hexagonal and square meshes resulted in the lowest number of Jacobi iterations for all of the test cases considered. In contrast to the results of Section 4.2.3, we do not observe a decrease in the performance of the hexagonal elements for the case of p = 3, and instead the performance is similar among all choices of p considered. 156 Randomly perturbed mesh We now consider the effect of polygonal elements on irregular meshes. To this end, we consider a set of generating points distributed evenly on a Cartesian grid with mesh size h. Then, each point is perturbed by a random perturbation sampled uniformly from the interval [−δ, δ]. We obtain two randomized meshes by constructing the Delaunay triangulation and Voronoi diagram resulting from this set of generating points [6]. The Delaunay mesh consists entirely of triangular elements, whereas the Voronoi diagram is constructed out of arbitrary polygonal elements. Examples of the two meshes considered are shown in Figure 4.4. In contrast to the regular meshes considered in the previous examples, these two meshes do not consist of the same number of elements. The Voronoi diagram consists of about half the number of elements as the Delaunay triangulation. In the test case considered, the randomized polygonal mesh consists of 410 polygonal elements, whereas the randomized triangular mesh consists of 759 triangular elements. The governing equations and set-up is the same as in the previous section. We record the number of block Jacobi iterations required to converge to a tolerance of 10−14 in Table 4.3. Because there is a difference in the number of mesh elements, the resulting linear system will have a different total number of degrees of freedom. This difference will then have an additional effect on the speed of convergence of the block Jacobi method. We note that for polynomials of degree p = 0, 1, 2, 3 and for all choices of time step k considered, solving the system resulting from the Voronoi diagram requires fewer block Jacobi iterations than does solving the system resulting from the corresponding Delaunay triangulation. 157 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (a) Delaunay triangulation (b) Voronoi diagram Figure 4.4: Randomized polygonal and triangular meshes corresponding to the same set of generating points Table 4.3: Iterations required for the block Jacobi iterative method to converge in the case of irregular, randomly perturbed meshes. The smallest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Voronoi diagram 27 32 38 24 33 38 24 32 36 22 31 36 Delaunay triangulation 38 48 52 33 45 48 33 46 50 33 44 48 Convergence of the GMRES method The above analysis focused on the block Jacobi method largely because of the sim- plicity of the method. In practice, more sophisticated iterative methods are often used [104]. In this section, we consider the solution of the linear system (4.2) by means of the GMRES method, using both the block Jacobi and the block ILU(0) preconditioners. Since the computational work increases per iteration in GMRES, we choose a restart parameter of 20 iterations [114]. We repeat the above test case of the advection equation with variable velocity field and record the number of GM- RES iterations required to converge to a tolerance of 10−14 using the block Jacobi preconditioner in Table 4.4. 158 Table 4.4: Iterations required for the GMRES iterative method with block Jacobi preconditioner to converge. The smallest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Hexagons 31 53 92 25 42 80 28 47 86 28 49 90 Squares 37 64 116 27 51 101 27 51 98 27 52 100 Right triangles 40 70 134 33 61 123 31 60 117 29 59 115 Equilateral triangles 39 67 124 33 58 113 32 59 113 31 57 111 We now consider the solution of the above problem using the GMRES method with the block ILU(0) preconditioner. Because of the sensitivity of the block ILU(0) factorization to the ordering of the mesh elements, and for the sake of a fair com- parison between the generating patterns, we consider the natural ordering of mesh elements, illustrated in Figure 4.5. As in the case of the block Jacobi preconditioner, we repeat the test case of the advection equation with variable velocity field. We record the number of GMRES iterations required to converge to the above tolerance using the block ILU(0) preconditioner in Table 4.5. In this case, the square mesh resulting in the smallest number of iterations in all of the trials. The mesh consisting of right isosceles triangles resulted in the largest number of iterations in all trials. We further note that the number of GMRES iterations required when using the block Jacobi preconditioner scales similarly to the number of block Jacobi iterations required, as recorded in Table 4.2. We note that the block ILU(0) preconditioner requires fewer GMRES iterations to converge, and the number of iterations scales more favorably in k, when compared with the block Jacobi preconditioner. 159 7 8 9 7 8 9 4 5 6 4 5 6 1 2 3 1 2 3 5 7 6 8 6 8 5 7 1 3 1 3 2 4 2 4 Figure 4.5: Illustration of the natural ordering of mesh elements Table 4.5: Iterations required for the GMRES iterative method with ILU(0) precon- ditioner to converge. The smallest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Hexagons 8 11 16 10 13 20 11 15 23 10 13 22 Squares 8 10 16 8 11 19 7 10 17 8 10 18 Right triangles 13 19 32 10 14 28 10 15 27 11 14 28 Equilateral triangles 11 15 27 10 12 22 9 12 22 9 12 22 4.3.2 Compressible Euler equations In this section, we consider the compressible Euler equations of gas dynamics, given in Section 1.4.3. We consider the model problem of an unsteady compressible vortex in a rectangular domain [132]. The domain is taken to be a 20 × 15 rectangle and the vortex is initially centered at (x0 , y0 ) = (5, 5). The vortex is moving with the 160 free-stream at an angle of θ. The exact solution is given by    ((y − y0 ) − vt) f (x, y, t) u = u∞ cos(θ) − exp , (4.35) 2πrc 2    ((x − x0 ) − ut) f (x, y, t) v = u∞ sin(θ) − exp , (4.36) 2πrc 2   γ−1 1 2 (γ − 1)M∞ 2 ρ = ρ∞ 1 − exp((f (x, y, t)) , (4.37) 8π 2   γ−1 γ 2 (γ − 1)M∞ 2 p = p∞ 1 − exp((f (x, y, t)) , (4.38) 8π 2 where f (x, y, t) = (1−((x−x0 )−ut)2 −((y −y0 )−vt)2 )/rc2 , M∞ is the Mach number, u∞ , ρ∞ , and p∞ are the free-stream velocity, density, and pressure, respectively. The free-stream velocity is given by (u, v) = u∞ (cos(θ), sin(θ)). The strength of the vortex is given by , and its size is rc . We choose the parameters to be γ = 1.4, M∞ = 0.5, u∞ = 1, θ = arctan(1/2),  = 0.3, and rc = 1.5. In the discontinuous Galerkin discretization of the Euler equations we use the Lax-Friedrichs numerical flux defined by (1.60). We use the backward Euler time discretization, but remark that the corresponding fully-discrete system results in a nonlinear set of equations, which are solved using Newton’s method. Each iteration of Newton’s method requires solving a linear equation of the form (4.2). We set h = 1, and consider three time steps, k1 = 0.03h, k2 = 2k1 , k3 = 4k1 . We use piecewise polynomials of degrees p = 0, 1, 2, 3. Each Newton solve requires between 3 to 8 iterations to converge to within a tolerance of 5 × 10−13 . The tolerance used for the linear solvers is the same as in the previous test cases. 161 Table 4.6: Block Jacobi iterations required per Newton solve of the compressible Euler equations. The lowest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Hexagons 32 49 78 31 50 83 50 90 158 53 97 171 Squares 34 51 89 31 54 92 54 99 181 55 105 201 Right triangles 37 56 97 41 64 112 58 101 189 59 113 217 Equilateral triangles 37 57 95 39 62 113 54 99 179 60 114 215 The Block Jacobi method Each iteration of Newton’s method requires the solution of a linear system of equa- tions. We solve these systems using the block Jacobi method. We compute the total the number of Jacobi iterations required to complete one solve of Newton’s method, and report the results in Table 4.6. We note that for each choice of p and time step k, the hexagonal mesh required the fewest number of block Jacobi iterations. As in the previous numerical experiments, we do not see a decrease in performance for the hexagonal elements in the case of p = 3. The square mesh resulted in the second-smallest number of iterations for most of the cases considered, while the two configurations of triangles resulted in generally similar numbers of iterations. The GMRES method We now repeat the above test case, using the GMRES method to solve the resulting linear systems. We consider both the block Jacobi and block ILU(0) preconditioners. We then compute the total number of GMRES iterations required to complete one solve of Newton’s method. As in Section 4.3.1, the ordering of the mesh elements has a significant effect on the effectiveness of the block ILU(0) approximate factorization. For this reason, we use the natural ordering of elements, depicted in Figure 4.5. We 162 Table 4.7: GMRES with block Jacobi preconditioner. Iterations required per Newton solve of the compressible Euler equations. The lowest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Hexagons 55 74 106 50 92 126 61 110 153 76 141 195 Squares 62 84 155 52 93 132 67 126 185 78 149 222 Right triangles 63 87 162 81 106 184 96 132 242 85 159 299 Equilateral triangles 66 90 167 81 108 187 72 133 197 85 161 245 Table 4.8: GMRES with block ILU(0) preconditioner. Iterations required per New- ton solve of the compressible Euler equations. The lowest number of iterations in each column is highlighted. p=0 p=1 p=2 p=3 k1 k2 k3 k1 k2 k3 k1 k2 k3 k1 k2 k3 Hexagons 24 32 42 21 36 48 29 48 57 29 50 64 Squares 24 28 45 21 33 40 24 41 49 27 48 60 Right triangles 31 40 70 35 40 60 36 48 69 31 49 75 Equilateral triangles 28 37 65 37 44 70 33 56 68 38 64 80 present the results for the block Jacobi preconditioner in Table 4.7, and for the block ILU(0) preconditioner in Table 4.8. With the block Jacobi preconditioner, the hexagonal mesh required the smallest number of iterations for all test cases considered, and the square mesh the second-smallest. In the case of the block ILU(0) preconditioner, the square mesh required the fewest number of iterations, with the hexagonal mesh usually requiring the second-smallest number of iterations. As we observed in Section 4.3.1, the number of iterations required for both the block Jacobi method and GMRES with the block Jacobi preconditioner scales quite poorly with increasing time steps. The number of GMRES iterations required when using the block ILU(0) preconditioner is significantly better. 163 4.3.3 Inviscid flow problems The following two numerical experiments extend the above results to larger-scale, more realistic flow problems. These problems, in contrast to the preceding test cases, are characterized by a large number of degrees of freedom, the presence of geometric features and wall boundary conditions, variably-sized mesh elements, and shocks. As in the previous section, the equations considered here are the compressible Euler equations. For the following two problems, we choose the finite element function space to consist of piecewise constant functions (corresponding to p = 0), which results in a finite-volume-type discretization. This choice of discretization allows for the solution of problems with shocks, without the use of slope limiters, artificial viscosity, or other shock-capturing techniques [76]. The Roe numerical flux is used as an approximate Riemann solver for these problems. Subsonic flow over a circular cylinder For a first test case, we consider the inviscid flow over a circular cylinder at Mach 0.2. The computational domain is defined as Ω = R \ C, where R = [−10, 30] × [−10, 20], and C is a disk of radius 1 centered at the point (5, 5). Farfield boundary conditions are enforced on ∂R, and a no normal flow condition is enforced on ∂C. The freestream velocity is taken to be unity in the x-direction, and ρ∞ = 1. For this test case we use four unstructured meshes, two consisting entirely of triangles, and two consisting of mixed polygons, generated using the PolyMesher algorithm [125]. All the meshes are created using a gradient-limited element size function that determines the initial distribution of seed points according to the rejection method [103], such that the element edge length near the surface of the cylinder is about one-fifth the edge length of elements away from the cylinder. For both the triangular and polygonal 164 20 6.5 15 6.0 10 5.5 5 5.0 0 4.5 −5 4.0 −10 3.5 −10 0 10 20 30 3.5 4.0 4.5 5.0 5.5 6.0 6.5 Figure 4.6: Overview of the coarse polygonal mesh with 15,404 elements, with zoom- in showing polygonal elements near the surface of the cylinder. meshes, we consider a coarse mesh, with 15,404 elements, and a fine mesh with 62,270 elements. Thus, the average area of each element is the same for both the polygonal and triangular meshes. Additionally, the number of degrees of freedom in the solution is the same, allowing for a fair comparison. The coarse polygonal mesh, and a zoom-in around the surface of the cylinder are shown in Figure 4.6. Starting from freestream initial conditions, we integrate the equations until t = 5 × 10−3 in order to obtain a representative solution. Using this solution, we then compute 10 time steps using a third-order A-stable DIRK method [1]. Each stage of the DIRK method requires the solution of a nonlinear system of equations, which we solve by means of Newton’s method. In each iteration of Newton’s method, we solve the resulting linear system of the form (4.2) using both the block Jacobi method and the preconditioned GMRES method. The nonlinear system is solved to within a tolerance of 10−8 , and each linear system is solved using a relative tolerance of 10−5 . For the GMRES method, we consider two preconditioners: block Jacobi, and block ILU(0). In order to compare the iterative solver performance differences between meshes, we compute the total number of solver iterations required to complete all 10 time steps. The results for the GMRES method are shown in Table 4.9, and for 165 Table 4.9: Total GMRES iterations per 10 time steps for inviscid flow over a circular cylinder. (a) Coarse grid with 15,404 elements ILU Jacobi Ratios ∆t Polygonal Triangular Polygonal Triangular ILU Jacobi 1.0 × 10−1 793 932 2092 3126 0.85 0.67 2.5 × 10−1 1569 1829 4405 6870 0.86 0.64 5.0 × 10−1 2470 3090 7145 11859 0.80 0.60 1.0 3651 4486 11054 18880 0.81 0.59 (b) Fine mesh with 62,270 elements ILU Jacobi Ratios ∆t Polygonal Triangular Polygonal Triangular ILU Jacobi 1.0 × 10−1 1443 1673 4075 6137 0.86 0.66 2.5 × 10−1 2998 3344 8732 12741 0.90 0.69 5.0 × 10−1 4720 5423 14084 21882 0.87 0.64 1.0 7205 8151 22814 34706 0.88 0.66 Table 4.10: Total block Jacobi iterations per 10 time steps for inviscid flow over a circular cylinder. Coarse grid with 15,404 elements Fine mesh with 62,270 elements ∆t Polygonal Triangular Ratio Polygonal Triangular Ratio 1.0 × 10−1 2474 3159 0.78 4788 6281 0.76 2.5 × 10−1 4895 6697 0.73 9609 12406 0.77 5.0 × 10−1 7882 12158 0.65 15580 20946 0.74 1.0 13181 19072 0.69 26628 33934 0.78 the block Jacobi solver in Table 4.10. These results demonstrate a consistent trend, corroborating both the numerical results and the analysis from the previous sections. When using the block Jacobi solver or GMRES with block Jacobi preconditioner, the polygonal mesh results in convergence in between 60–70% of the iterations required for the triangular mesh. The effect is smaller when using the ILU(0) preconditioner, but we do still observe a modest reduction in the number of iterations required. When using the block Jacobi 166 iterative solver, we observe iteration counts very similar to when using GMRES with block Jacobi as a preconditioner. In these cases, the polygonal mesh requires between 70–80% of the iterations as the all-triangular mesh. Supersonic flow over a circular cylinder The next numerical example is designed to investigate the performance of the it- erative solvers for steady-state problems, in the presence of shocks and h-adapted meshes. For this problem, we let the domain be Ω = R \ C, where R = [0, 5] × [0, 10], and, as before, C is a circle of radius one centered at (5, 5). Freestream conditions are enforced at the left, top, and bottom boundaries, an inviscid wall condition is enforced on the boundary of the cylinder, and an outflow condition is enforced on the right boundary. The Mach number is set to M = 2.0, resulting in the formation of a shock upstream from the cylinder. In order to accurately capture the shock, we refine the mesh in its vicinity. As in the previous case, we consider a set of four meshes, two all-triangular, and two polygonal. For both the triangular and polygo- nal meshes, we consider coarse and fine versions, with 31,162 and 95,932 elements, respectively. The coarse mesh is depicted in Figure 4.7a, with Mach isolines overlaid to indicate the position of the shock. Additionally, Mach contours of the steady-state solution are shown in Figure 4.7b. Beginning with freestream initial conditions, the solution rapidly approaches a steady state. We integrate in time until t = 100 in order to obtain an solution which can be used as an initial guess for the steady-state Newton solve. Then, starting with this solution, we set the time-derivative of the solution to zero and solve the resulting nonlinear equations using Newton’s method to find a steady-state solution. The resulting linear system that is required to be solved at each iteration can be thought 167 10 10 1.99 8 1.77 8 1.55 6 1.33 6 1.11 4 0.90 4 0.68 2 0.46 2 0.24 0 0 0.02 0 2 4 0 2 4 (a) Coarse mesh for supersonic test problem, (b) Contours of Mach number for steady showing Mach isolines for steady-state solu- state solution tion Figure 4.7: Overview of the coarse polygonal mesh with 31,162 elements, showing Mach number contours for steady-state solution. of as corresponding to equation (4.2), where formally we set k = ∞. The nonlinear system is solved to within a tolerance of 10−10 , and each linear system is solved using a relative tolerance of 10−5 . Since the mass matrix in (4.2) acts to regularize the linear system, the conditioning becomes worse for larger values of k, and the number of iterations required per linear solve grows. Hence, effective preconditioners are particularly important for the solution of such steady-state problems. For these problems, the block Jacobi iterative solver did not converge in fewer than 10,000 iterations, and so we consider only the GMRES method, using block ILU(0) and block Jacobi preconditioners. We present the comparison of iteration counts for this problem in Table 4.11. On the coarse meshes, the ILU(0) preconditioner required about 73% as many iterations on the polygonal mesh when compared with the triangular mesh. This difference 168 Table 4.11: Total GMRES iterations per steady-state solve for supersonic flow over a cylinder. Coarse grid with 31,162 elements Fine mesh with 95,932 elements Polygonal Triangular Ratio Polygonal Triangular Ratio ILU 469 640 0.73 953 1947 0.49 Jacobi 2340 6464 0.36 – – – is more significant when using the block Jacobi preconditioner, consistent with the results observed in previous section. In this case, the polygonal mesh requires only slightly more than one third the number of iterations as the all-triangular mesh. On the fine mesh, there are close to half a million degrees of freedom. For a problem of this scale, we did not observe convergence in less than 10,000 iterations per linear solve using the block Jacobi preconditioner, and so we only compare performance using the block ILU(0) preconditioner. In this case, the polygonal mesh required about half as many iterations per steady-state solve when compared with the all- triangular mesh. 4.4 Summary In this chapter we have analyzed the effect of the generating pattern of a regular mesh on the convergence of iterative linear solvers applied to implicit discontinuous Galerkin discretizations. We considered four generating patters: a hexagon, a square, two right triangles, and two equilateral triangles. A classical von Neumann analysis applied to the constant-velocity advection equation allowed us to compute the eigenvalues of the block Jacobi matrix, and therefore estimate the speed of convergence of the block Jacobi method. In more than half of the cases considered, the hexagonal generating pattern resulted in the 169 smallest eigenvalues, and in the remaining cases, the square generating pattern re- sulted in the smallest eigenvalues. In order to extend these results beyond the case of the constant-velocity advection equation, we performed numerical experiments on the variable-velocity advection equation and compressible Euler equations. In the case of the advection equation, in all but one case the hexagonal mesh resulted in the fastest convergence, and in the remaining case the square mesh resulted in the fastest convergence. In the case of the Euler equations, the hexagonal mesh resulted in the fastest convergence in all test cases. We additionally considered two irregular meshes resulting from the random per- turbation of a set of regularly-spaced generating points. We obtain a triangular mesh by performing the Delaunay triangulation on these points, and we obtain a polygonal mesh by constructing the Voronoi diagram dual to the Delaunay triangu- lation. Solving the advection equation on these irregular meshes, we observed that the block Jacobi method converged faster on the polygonal mesh in every test case. Additionally, we performed numerical experiments examining the performance of the GMRES iterative method when used with the ILU(0) preconditioner. We found that in all of the test cases, the square generating pattern resulted in the fewest number of GMRES iterations, and in all but two cases, the hexagonal generating pattern resulted in the second-fewest number of iterations. For a final set of numerical experiments, we performed two inviscid fluid flow simulations on sets of coarse and fine meshes. Each mesh was either all-triangular, or was composed of arbitrary polygons. We measured iteration counts for both time- dependent and steady-state problems, using the block Jacobi method, and GMRES with block ILU(0) and block Jacobi preconditioners. We found that the polygonal 170 meshes resulted in faster convergence of the iterative solvers, with a larger difference being observed for the block Jacobi method and preconditioner. This difference was more pronounced for the steady-state problem, with a quite significant difference observed on the fine mesh using GMRES with ILU(0). These results suggest that certain types of polygonal meshes have the advantage of rapid convergence of iterative solvers. Future research directions involve the study of accuracy of DG methods on polygonal and polyhedral meshes, efficient computa- tion of quadrature rules over arbitrary polygonal domains and the extension of the above results to three spatial dimensions. Chapter Five Case studies 172 In this chapter, we consider two challenging case studies that are intended to illustrate some of the main properties of the methods described in the preceding chapters, and to demonstrate the viability of applying these techniques to large- scale, realistic problems. These cases are designed to illustrate the use of both the fully-implicit Runge-Kutta time integration schemes develops in Chapter 2 and the very high-order approximate tensor-product preconditioners developed in Chapter 3. The first test case, discussed in Section 5.1, is the transitional flow over the SD7003 airfoil, which has been well-studied in the context of implicit large eddy sim- ulation and high-order CFD methods [51, 52, 127]. The availability of large amounts of reference data (both computational and experimental) allow for evaluation of the proposed methods [13, 19, 90, 115]. In this chapter, a large-scale parallel compu- tational simulation of transitional flow over the airfoil is performed, demonstrating the effectivity of high-order fully-implicit, stage-parallel Runge-Kutta methods when applied to realistic CFD problems on fully unstructured meshes. The second problem, discussed in Section 5.2, is the simulation of two-dimensional flow over a rotating vertical-axis wind turbine. This problem requires the treatment of moving geometries, which we discretize using an arbitrary Lagrangian-Eulerian (ALE) formulation. We first formulate the ALE method, and then perform several benchmark test cases in order to verify consistency and convergence of the ALE scheme. Finally, we use approximate Kronecker-product preconditioners in order to solve the linear systems resulting from the implicit time integration of these schemes. 173 4 4 2 2 0 0 −2 −2 −4 −4 −2.5 0.0 2.5 5.0 7.5 10.0 −2.5 0.0 2.5 5.0 7.5 10.0 0.4 0.4 0.2 0.2 0.0 0.0 −0.2 −0.2 −0.4 −0.4 −0.25 0.00 0.25 0.50 0.75 1.00 1.25 −0.25 0.00 0.25 0.50 0.75 1.00 1.25 (a) Coarse mesh with 1646 quadrilaterals (b) Refined mesh with 1940 quadrilaterals Figure 5.1: Two-dimensional meshes used for SD7003 test case, showing anisotropic boundary-layer elements. The mesh is refined around the wake of the airfoil and near the point of separation. 5.1 Benchmark problem: transitional flow over the SD7003 airfoil In this section, we consider the transitional flow over the Selig-Donovan SD7003 airfoil. This benchmark problem was selected as a challenging test case for the Second International Workshop on High-Order CFD Methods, and has been the subject of extensive computational and experimental studies [13, 19, 51, 52, 90, 115]. The governing equations are the three-dimensional compressible Navier-Stokes equations, given in Section 1.4.4. The Mach number is taken to be M∞ = 0.1, the Reynolds number is taken to be Re = 60,000, and the angle of attack is α = 8◦ . 174 The computational domain is given by [−4.5, 10.5] × [0, 0.2] × [−5, 5] with the airfoil removed. The airfoil has unit chord length. A no-slip adiabatic wall condition is enforced at the surface of the airfoil, and farfield conditions are enforced at the domain boundaries. Periodic conditions are enforced in the spanwise direction. First, we generate an unstructured two-dimensional cross-sectional quadrilateral mesh, which is shown in Figure 5.1a. This mesh features highly anisotropic el- ements near the surface of the airfoil in order to resolve the resulting boundary layer. The two-dimensional mesh is then extruded in the y-dimension to obtain a three-dimensional hexahedral mesh. The mesh is decomposed into 8 blocks in the y-dimension. We also create a refined quadrilateral mesh, which features smaller el- ement sizes in the vicinity of features of interest. The refined two-dimensional mesh is shown in Figure 5.1b. This mesh is then extruded in the y-dimension, resulting in a three-dimensional hexahedral mesh with 10 elements in the spanwise direction. For both meshes, we use p = 3 bicubic polynomials, resulting in a total of 3,893,760 degrees of freedom on the coarse mesh, and 6,208,000 degrees of freedom on the fine mesh. In order to generate an initial condition for this problem, we first run a two- dimensional simulation. Starting from freestream conditions, we integrate the prob- lem until t = 10 tc , where tc represents the characteristic time scale of the problem. We obtain the solution that is shown in Figure 5.2. This two-dimensional solution is then extruded to generate a three-dimension solution field, where the y component of the velocity is set to zero. The extruded solution is used as the initial conditions for the full three-dimensional simulation. The three-dimensional compressible Navier- Stokes equations are then integrated for an additional 20 tc . We discard the results from the first 10 tc of the simulation in order to allow for the smoothing of any ini- tial transients. In Figure 5.3, we show a snapshot of the three-dimensional solution 175 at time t = 15 tc . Shown are isosurfaces of the Q-criterion, which, as discussed in Section 2.3.2, is given by the difference of the symmetric and antisymmetric parts of of the velocity gradient tensor, 1 1 Q= (Ωij Ωij − Sij Sij ) = − ui,j uj,i (5.1) 2 2 and can be used for vortex identification [45, 65]. In the context of this problem, it is of interest to study the formation of a lami- nar separation bubble towards the leading edge of the airfoil [51]. A large number of experimental and computational studies have investigated these features. In partic- ular, we compare our results with the computational results of Gabraith and Visbal [51, 52], Boom and Zingg [19], Beck et al. [13], and Vermeire et al. [130]. We also compare certain quantities of interest with the experimental results of Selig et al. [115]. Of interest are the distributions of the pressure coefficient and skin friction coefficient along the surface of the airfoil. These quantities are evaluated pointwise along the surface of the airfoil, and then integrated in the spanwise direction. We compute the temporal mean of these quantities, averaged over t ∈ [10 tc , 20 tc ]. The dimensionless coefficient of pressure is defined by p − p∞ Cp = 1 , (5.2) ρ kv∞ k2 2 ∞ where p is the pointwise static pressure, and p∞ , ρ∞ and v∞ are the freestream pressure, density and velocity, respectively. The pressure coefficient for the coarse and fine meshes is shown in Figure 5.4. Good agreement is observed when comparing with other computational studies, with the exception of that of Boom and Zingg. 176 1.005 0.6 1.000 0.4 0.2 0.995 0.0 0.990 −0.2 −0.4 0.985 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ρ Figure 5.2: Two-dimensional solution (density) for flow over SD7003 airfoil at t = 10 tc Figure 5.3: Snapshot of three-dimensional solution at t = 15 tc , showing isosurfaces of Q-criterion, colored by velocity magnitude The skin friction coefficient is defined by τw Cf = 1 , (5.3) ρ kv∞ k2 2 ∞ where τw is the skin shear stress on the surface of the airfoil. The separation and 177 reattachment locations are given by the zeros of Cf . Plots of the skin friction dis- tribution are shown in Figure 5.5. Fair agreement is observed when comparing with the results of Galbraith and Visbal and Beck et al. All of the computational studies give similar results for the separation location, but significant differences are seen in the reattachment location. Additional comparisons with experimental and computa- tional results are shown in Table 5.1. Here we also compare the mean aerodynamic loads, integrated over the surface of the airfoil. The lift coefficient gives reasonable agreement with the experimental data, but the drag coefficient is overpredicted, as is the case with all of the other computational results considered. The separation and reattachment locations give fair agreement with the other computational results. 5.1.1 Performance comparison We are interested in studying the performance characteristics of time integration schemes for this test problem. We consider both implicit and explicit time integra- tion. For implicit methods, we compare the third-order L-stable Radau IIA fully- implicit method with the third-order L-stable DIRK of Alexander [1]. A detailed comparison of these methods is found in Chapter 2. We also consider the standard four-stage, fourth-order explicit Runge-Kutta method. We choose a representative, time-accurate implicit time step of ∆t = 10−2 tc . For the explicit method, we use the largest stable time step, which we experimentally determine to be 4 × 10−6 tc . We obtain a representative solution by integrating the equations until t = 10 tc , at which point we study the performance characteristics for an additional 5×10−2 tc . Of interest are the run times, parallel scaling, and preconditioner performance for these methods. For the DIRK method, we use a block ILU(0) preconditioner, given by Algorithm 2.1. For the fully-implicit IRK methods, we use the the stage-uncoupled 178 p = 3, coarse p = 3, refined 3 Boom and Zingg 3 Boom and Zingg Galbraith and Visbal Galbraith and Visbal Beck et al. (p = 7) Beck et al. (p = 7) 2 2 −Cp Vermeire et al. Vermeire et al. −Cp 1 1 0 0 −1 −1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 x x Figure 5.4: Distribution of pressure coefficient on suction and pressures sides of SD7003 airfoil for coarse (left) and fine (right) meshes 0.03 0.03 p = 3, coarse p = 3, refined Boom and Zingg Boom and Zingg 0.02 Galbraith and Visbal 0.02 Galbraith and Visbal Beck et al. (p = 7) Beck et al. (p = 7) 0.01 0.01 Cf Cf 0 0 −0.01 −0.01 −0.02 −0.02 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 x x Figure 5.5: Distribution of skin friction coefficient on the suction side of SD7003 airfoil for coarse (left) and fine (right) meshes Table 5.1: Comparison of experimental and computational results for lift CL , drag CD , separation location xs , and reattachment location xr for SD7003 airfoil Study CL CD xs xr DOF/comp. DG (coarse) 0.95 0.045 0.038 0.294 0.78 × 106 DG (fine) 0.95 0.045 0.039 0.295 1.24 × 106 Beck et al. (p = 7) 0.93 0.050 0.030 0.336 4.55 × 106 Boom and Zingg 0.97 0.034 0.037 0.200 4.48 × 106 Galbraith and Visbal 0.91 0.043 0.040 0.280 5.70 × 106 Selig et al. (experimental) 0.92 0.029 — — — 179 ILU preconditioner described in Section 2.2.4 and the stage-parallel variant of this solver, detailed in Section 2.2.6. We run this numerical experiment on the NASA Advanced Supercomputing fa- cility’s Pleiades supercomputer. We perform runs using 60, 120, 240, 480, and 960 cores, allowing us to study the parallel scalability of the algorithms. We present the results for both the fine and coarse meshes in Figure 5.6. We notice that for both meshes, the stage-parallel solver resulted in the fastest runtimes. As observed by Vermeire et al. [130], implicit time stepping proves to be more efficient for this problem when compared with explicit time stepping, due to the restrictive time step stability condition. Good parallel scaling is observed for both cases until the number of processors is increased such that the number of mesh elements per partition is less than 100, at which point we observe slightly degraded parallel scaling performance. Additionally, we present the number of equivalent multiplications for each method. For the DIRK solver, this number corresponds exactly to the number of iterations required in order for GMRES to converge. Since the a matrix-vector product with the Jacobian matrix for the fully-coupled IRK solver requires two residual Jaco- bian matrix-vector products, each GMRES iteration for the fully coupled solver corresponds to two equivalent multiplications. As the number of mesh partitions increases, the the ILU(0) preconditioner becomes less effective because terms that would induce coupling between mesh partitions are dropped from the incomplete factorization, resulting in the observed increase in equivalent multiplications as the number of processors increases. This results in a benefit of using the stage-parallel solver, for which the mesh can be partitioned into a factor of s fewer subdomains, where s is the number of IRK stages We also observe the improved strong parallel scaling of the stage-parallel solver when the number of processors is equal to 960. 180 Equivalent multiplications 10,000 Wall-clock time (s) 1,750 1,500 1,000 1 1 1,250 100 60 120 240 480 960 60 120 240 480 960 Number of processors Number of processors Stage-parallel IRK IRK DIRK RK4 (a) Coarse mesh with 13,168 hexahedra Equivalent multiplications 10,000 2,000 Wall-clock time (s) 1,750 1 1,000 1 1,500 100 60 120 240 480 960 60 120 240 480 960 Number of processors Number of processors Stage-parallel IRK IRK DIRK RK4 (b) Fine mesh with 19,400 hexahedra Figure 5.6: Log-log plots showing the performance and parallel scaling of IRK and DIRK solvers for the SD7003 test case 5.1.2 Summary This large-scale benchmark test problem demonstrates the effectiveness of an un- structured high-order approach using implicit time integration. Good agreement with previous computational and experimental results were observed. The use of a high-order method allows for accurate results despite a relatively small number of degrees of freedom. Efficient parallel scaling was observed, and the fastest runtimes 181 were found using the fully-implicit Radau IIA Runge-Kutta methods. An advantage was found to using the stage-parallel solver in the strong scaling limit. 5.2 High-order simulation of vertical-axis wind tur- bines In this section, we consider the use of a high-order DG discretization in order to simulate a rotating, vertical-axis wind turbine. Recent work on this subject has demonstrated the viability of implicit large-eddy simulation using high-order meth- ods for the study of these vertical-axis wind turbines [68, 69, 70]. In the following section, we are interested in studying the effectiveness of the approximate tensor- product preconditioner developed in Chapter 3 for the implicit time integration of the resulting semi-discrete systems. This section is organized as follows. First we describe the arbitrary Lagrangian- Eulerian method used to discretize the rotating geometry. Then, we verify the cor- rectness of this discretization using two benchmark problems. Finally, we apply this method to the simulation of a three-blade vertical-axis wind turbine, and compare the performance of the approximate tensor-product preconditioner with that of the exact block Jacobi preconditioner. 5.2.1 Arbitrary Lagrangian-Eulerian formulation In order to simulate the flow over these vertical-axis wind turbines, we must incor- porate the domain motion into the discretization. The method used presently is 182 an arbitrary Lagrangian-Eulerian (ALE) formulation [44]. In this case, we consider a time-dependent domain Ω(t) which is obtained by rotating the fixed, reference domain Ω0 = Ω(0). Thus, the domain is given by  Ω(t) = (x, y) = R(θ)(X, Y )T : (X, Y ) ∈ Ω0 , (5.4) where θ(t) = ωt is the angle of rotation (depending on a constant angular velocity ω), and R(θ) is the rotation matrix    cos(θ) − sin(θ)  R(θ) =  . (5.5) sin(θ) cos(θ) We define the transformation mapping G(X, t) = R(t)X. We let x denote physical coordinates on the domain Ω(t), and X denote coordinates on the static reference domain Ω0 , giving x = G(X, t). At this point we introduce some notation. We define the deformation gradient G by    ∂G1 /∂X ∂G1 /∂Y  G= , (5.6) ∂G2 /∂X ∂G2 /∂Y and we define the transformation Jacobian determinant, g = det(G). Since G is a linear orthogonal transformation, we have G = R(θ) and g ≡ 1. We also define the mapping velocity by vG = ∂G/∂t. We consider a conservation law in physical coordinates of the form ∂u + ∇ · F (u, ∇u) in Ω(t). (5.7) ∂t Here F is a flux function, which, as in Section 1.2.2, we decompose into inviscid and 183 viscous parts, F (u, ∇u) = F I (u) + F V (u, ∇u). (5.8) The solution is a function of the form u(x, t). Following the derivation in [106, 136], we obtain a transformed conservation law on the reference domain, ∂uX + ∇X · FX (uX , ∇X uX ) in Ω0 , (5.9) ∂t where ∇X denotes the gradient with respect to the reference coordinates X. The transformed solution uX (X, t) is given in reference coordinates, and is defined by the relation uX (X, t) = gu(x, t), for x = G(X, t). The transformed flux is given by FX = FXI + FXV , where, T FXI (uX ) = gF I (g −1 uX )G−T − uX ⊗ G−1 vG , (5.10) ∂g  −1  −T FXV (uX , ∇X uX ) = gF V g −1 uX , g −1 ∇X uX − g −1 uX ∂X G G , (5.11) where we use the convention that the gradient of uX is a matrix that is given by   ∇X uX = ∂uX /∂X1 , ∂uX /∂X2 . (5.12) In the particular case where G is a rigid motion, these expressions can be simplified using the fact that g = 1 and ∂g/∂t = 0. We obtain the following simplified fluxes, T FXI (uX ) = F I (uX )G−T − uX ⊗ G−1 vG , (5.13)  FXV (uX , ∇X uX ) = F V uX , (∇X uX ) G−1 G−T . (5.14) 184 ALE discontinuous Galerkin discretization We use the standard discontinuous Galerkin method to discretize equation (5.9) over the static reference domain Ω0 , using the fluxes given by (5.13) and (5.14). In order to discretize the inviscid flux, we must define a suitable numerical flux function. We consider an element K in the reference domain, K ⊆ Ω0 , which corresponds to an element k(t) = G(K, t) in physical space. A vector N that is normal to ∂K corresponds with a transformed vector n = gG−T N that is normal to ∂k(t). Thus, the normal flux can be written  T  FXI (uX ) · N = F I (uX )G−T − uX ⊗ G−1 vG ·N (5.15)  −T = F I (uX ) − uX ⊗ vGT ·G N (5.16)  = F I (u) − u ⊗ vG T · n. (5.17) Typically, when defining a numerical flux function corresponding to F I (u)·n, we use the Roe approximate Riemann solver [110], which proceeds by solving the Riemann e Thus, the Roe solver for the trans- problem corresponding to a linearized flux A. e formed Riemann problem with flux (5.17) makes use of the linearization A−(vG ·n)I, the eigenvalues and eigenvectors of which are computed trivially. Now we turn our attention to the discretization of the viscous flux, F V . As in Section 1.2.2, we assume that the viscous flux is linear in the gradient of u, and thus we can write d X ∂u FiV (u, ∇u) = Hij (u) . (5.18) j=1 ∂xj 185 Therefore, the transformed viscous flux FXV has the form V FX,i (uX , ∇X uX ) = FiV (uX , ∇X uX G−1 )G−T (5.19) d X ∂uX −1 −T = H`j (uX ) G G (5.20) j,k,`=1 ∂Xk kj `i d d ! X X ∂uX = H`j (uX )G−1 −T kj G`i . (5.21) k=1 j,`=1 ∂Xk This allows us to write the transformed viscous flux in the form required for the interior penalty method, d X V ∂uX FX,i (uX , ∇X uX ) = HX,ij (uX ) , (5.22) j=1 ∂Xj for matrices d X HX,ij = H`k G−1 −T jk G`i . (5.23) k,`=1 Thus, if we consider the matrices Hij as a 4-tensor H given by    H11 H12  H= , (5.24) H21 H22 then the transformed matrices have the simple form HX = G−1 HG−T . (5.25) We then apply the standard interior penalty method to the transformed viscous fluxes defined by HX . 186 Moving adiabatic wall condition In order to simulate the rotating vertical-axis wind turbines, it is important to cor- rectly implement a moving adiabatic no-slip boundary condition. First, we consider the simple case of a stationary wall. The adiabatic wall condition is then a ho- mogeneous Dirichlet condition on the velocity field, and a homogeneous Neumann condition on the temperature, which is given by cv T = E − 21 kvk2 , where cv is the specific heat capacity at constant volume. We enforce this condition on a subset of the domain boundary, ΓW ⊆ ∂Ω. At this point, we recall the generalized interior penalty formulation given by (1.40), which we repeat here for convenience, Z Z Z I ∂t uh · vh dx − F (uh ) : ∇vh dx + F cI (u− , u+ ) : Jvh K ds h h Ω Z X d Ω Z ( Γ d ) ∂uh X ∂uh − Hij (uh ) : ∇vh dx + Hij (uh ) : Jvh K ds Ω j=1 ∂xj Γ ∂x j (5.26) j=1 Z Z ( d ) Z X ∂v h T + ηe Juh K : Jvh K ds + Juh K : Hij (uh ) ds = f · vh . Γ Γ i=1 ∂xi Ω In order to enforce the Dirichlet condition, we must correctly define the inviscid cI on ∂Ω. We do this by introducing the boundary state, numerical flux function F − − uW h = (ρ , 0, ρE ), and then evaluating the standard numerical flux function. Thus, the third integral in (5.26), when evaluated at a boundary face e ∈ ΓW , becomes Z cI (u− , uW ) : Jvh K ds, F e ∈ ∂ΓW . (5.27) h h e Additionally, this definition gives rise to a corresponding penalty term, Z  ηe u− W h − uh : Jvh K ds, e ∈ ∂ΓW , (5.28) e 187 which penalizes deviation from non-zero velocity at the wall. In order to enforce the Neumann condition on the temperature, we modify the viscous flux. We recall from Section 1.4.4 that the viscous flux is given by    0    F V =  −τ ,  (5.29)   qj − ui τij where τ is the viscous stress tensor, and q is the heat flux. The condition ∇T · n = 0 is equivalent to q · n = 0, and thus at an adiabatic wall, we modify the viscous flux to be    0    F V,W =  −τ .  (5.30)   −ui τij As in the case of the standard viscous flux, we can write this modified flux in the form d X ∂u FiV,W (u, ∇u) = HijW (u) , (5.31) j=1 ∂xj for suitably defined matrices HijW (u). Thus, for the integrals over every edge on the adiabatic wall e ∈ ΓW , we replace Hij with HijW in the formulation (5.26). If we now consider the case of a moving domain with a rigid motion, the procedure is the same, except that we must enforce the inhomogeneous Dirichlet condition v = vG at the wall, where we recall that vG = ∂G/∂t is the mapping velocity. This 188 20 15 10 5 0 −5 −10 −15 −20 −20 −10 0 10 20 30 40 50 60 Figure 5.7: Mesh used for laminar flow over a circular cylinder is achieved by simply modifying the boundary state to be   −  ρ    uW h = −  ρ vG .  (5.32)   ρE − In order to verify the correctness of these boundary conditions, we consider two verification test cases, discussed in the following sections. Verification test case: laminar flow over a circular cylinder We consider the case of laminar, compressible flow over a circular cylinder at Reynolds number Re = 40. We take the domain to be Ω = (C1 ∪ R) \ C2 , where C1 is the disk centered at the origin of radius 20, C2 is the disk centered at the origin of radius 1, and R = [0, 60] × [−20, 20]. A mesh of the domain is shown in Figure 5.7. We consider two equivalent variants of this problem. For the first case, we con- sider the cylinder to be stationary, and enforce a no-slip wall condition at the sur- 189 face of the cylinder. At all other boundaries, we enforce farfield conditions, where v∞ = (1, 0), ρ∞ = 1, and M∞ = 0.2. The right edge of the domain is placed suffi- ciently far from the cylinder to have negligible effects on the flow in the vicinity of the cylinder. For the second variant of this problem, we consider a reference frame moving with the farfield fluid velocity. In this reference frame, the cylinder is now moving with velocity equal to the negative farfield velocity. In the ALE framework, this corresponds to a translational mesh movement, with mesh velocity vG = (−1, 0). At the surface of the cylinder, we enforce the ALE no-slip wall condition, using the boundary state given by (5.32). At all other boundaries, we enforce farfield conditions, where v∞ = (0, 0), and ρ∞ and p∞ are as in the previous case. These two variants are physically equivalent, and thus we expect that the numer- ical solutions should converge to the same results in both cases. We use the mesh pictured in Figure 5.7, and a coarser mesh with about half as many elements. We use a p = 3 piecewise polynomial function space, and solve for the steady solution using Newton’s method. Contours of the Mach number of the solution are shown in Figure 5.8. We can compare the solutions to both of the above variants by translat- ing a solution from the static reference frame to the moving reference frame. This requires appropriately modifying the momentum and kinetic energy of the solution. We refer to the solution computed using the stationary domain as u, and the solution e . After performing this modification, we computing using the moving domain as u compute the L∞ difference ku − u e k∞ between the two solutions, on both the coarse mesh and the fine mesh. We also compute the lift and drag coefficients, integrated fL and C over the surface of the cylinder for both cases, denoted CL and CD , and C fD , for the stationary and moving cases, respectively. Comparisons of these results are shown in Table 5.2. The L∞ difference between these solutions tends to zero as the 190 mesh is refined. We also see that as the mesh is refined, the lift coefficient tends to zero, as is expected due to the symmetric nature of this problem. We also compute the distributions of the coefficient of pressure and skin friction along the surface of the cylinder. These quantities are shown in Figure 5.9. The resulting curves from both the static and moving domain tests display excellent agreement. Verification test case: steady flow over a NACA airfoil For a second verification test case, we consider the steady, laminar flow over a NACA 0012 airfoil. This problem is similar to the benchmark NACA problem from the First International Workshop on High-Order CFD Methods [132]. We consider take the angle of attack to be α = 0◦ , and the Reynolds number to be Re = 1000. We are interested in the steady solution, where freestream conditions are enforced on the domain boundaries, and a no-slip condition is enforced on the surface of the airfoil. The farfield conditions are defined by v∞ = (1, 0), ρ∞ = 1, and M∞ = 0.2. As in the previous test case, we also consider a moving reference frame, where the mesh velocity is given by vG = (−1, 0), and the farfield velocity is given by v∞ = (0, 0). We generate coarse and fine meshes according to the following methodology. We begin by generating a preliminary coarse mesh of the geometry, consisting of 648 quadrilateral elements. We perform anisotropic refinement near the surface of the airfoil in order to properly resolve the boundary layer, resulting in a final coarse mesh with 872 quadrilaterals. We obtain the fine mesh by performing uniform refinement on the preliminary coarse mesh, splitting every quadrilateral into four. We then perform anisotropic refinement to generate the boundary layer, resulting in a final fine mesh with 3024 quadrilateral elements. Both meshes use p = 3 isoparametric curved elements to represent the geometry. The meshes are pictured in Figure 5.10. 191 20 0.231 15 0.206 10 0.180 5 0.154 0.128 0 0.103 −5 0.077 −10 0.051 −15 0.026 -0.000 −20 −20 −10 0 10 20 30 Figure 5.8: Solution (Mach number) for laminar flow over a circular cylinder 1 0.4 0.2 0 −Cp Cf 0 −0.2 Stationary domain −1 Stationary domain −0.4 Moving domain Moving domain −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 x x Figure 5.9: Coefficient of pressure (left) and coefficient of skin friction (right) for laminar flow over circular cylinder Table 5.2: Comparison of static and moving-domain variants of laminar flow over a circular cylinder. Moving domain quantities are indicates by a tilde. Mesh e k∞ ku − u CL fL C CD fD C Coarse 2.53 × 10−5 −3.33 × 10−11 −3.42 × 10−11 1.2441 1.2438 Fine 9.14 × 10−6 −1.80 × 10−12 −4.22 × 10−12 1.2443 1.2440 We numerically find the steady-state solution to this problem using Newton’s method. Pressure contours of the solution are shown in Figure 5.11. As in the case of flow over a cylinder, both the stationary and moving geometry variants of this problem represent the same physical system, and therefore we expect that 192 4 4 2 2 0 0 −2 −2 −4 −4 −2.5 0.0 2.5 5.0 7.5 10.0 −2.5 0.0 2.5 5.0 7.5 10.0 Figure 5.10: Coarse and fine meshes used for laminar NACA test case the numerical solutions should converge to the same answer. We compute the lift fL and the drag coefficients CD and C coefficients CL and C fD on the stationary and moving geometries, respectively. These results are presented in Table 5.3. Since this problem is symmetric about x = 0, the true value for the lift coefficient is zero. fL converge to zero as the mesh is refined. Indeed, we observe that both CL and C Additionally, we observe that both predicted values for the drag coefficient appear to be converging to a value of about 6.055 × 10−2 . We also compute the distributions of the coefficient of pressure and coefficient of skin friction on the surface of the airfoil. These curves, displayed in Figure 5.12, show excellent agreement between the stationary domain and moving domain results. 5.2.2 Problem configuration and results We consider a three-blade vertical-axis wind turbine configuration, with the following parameters. The chord length is taken to be c = 0.265 m, the chord offset is c0 = 0.25c, and the radius of the wind turbine is R = 1 m. The blades are offset at a pitch angle of α0 = 6◦ . We use a moderate Reynolds number of Re = 100,000. The inflow velocity is given by v∞ = (8, 0) m/s. The tip-speed ratio, defined by λ = ωR/u∞ , where ω is the angular velocity of the wind turbine, is given by λ = 1.8. 193 1.00 0.75 18.37 18.30 0.50 18.22 0.25 18.14 0.00 18.06 17.99 −0.25 17.91 −0.50 17.83 −0.75 17.75 17.68 −1.00 −1.0 −0.5 0.0 0.5 1.0 1.5 Figure 5.11: Pressure contours of steady, laminar flow over NACA 0012 airfoil 0.5 0.2 0 −Cp Cf 0 −0.5 −0.2 Stationary domain −1 Stationary domain Moving domain Moving domain 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 x x Figure 5.12: Coefficient of pressure (left) and coefficient of skin friction (right) for laminar flow over NACA 0012 airfoil Table 5.3: Comparison of static and moving-domain variants of laminar flow over NACA 0012 airfoil. Moving domain quantities are indicated by a tilde. Mesh CL fL C CD fD C Coarse −2.04 × 10−6 −1.99 × 10−6 6.0659 × 10−2 6.0653 × 10−2 Fine −6.41 × 10−7 −6.18 × 10−7 6.0555 × 10−2 6.0550 × 10−2 The geometry is discretized using a quadrilateral mesh, shown in Figure 5.13, with 4799 elements. The farfield boundaries are taken to be at a radius of 10 m. We use p = 3 polynomials for the spatial discretization, resulting in a total of 307,136 degrees of freedom. A snapshot of the solution at t = 0.3 is shown in Figure 5.14. 194 1.5 1.0 0.5 0.0 −0.5 −1.0 −1.5 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 Figure 5.13: Mesh of three-blade vertical-axis wind turbine We use the discontinuous Galerkin ALE formulation described above, with mesh mapping G(X, t) = R(ωt)X. We integrate in time using a three-stage, L-stable, third-order DIRK method. We are interested in comparing the performance of the approximate Kronecker-product preconditioner developed in Chapter 3 with the standard block Jacobi preconditioner. For this comparison, we apply both block preconditioners component-wise, rather than on the larger 4 × 4 blocks that coupling the solution components. The largest stable explicit time step for this problem is 5 × 10−8 . We compare iteration counts for a range time steps, up to 16 times the explicit time step. We integrate the equations until t = 0.3 in order to obtain a representative solution about which to linearize. When then compute the average number of GMRES iterations required per linear solve, averaging over the DIRK stages and Newton iterations. These iteration counts are presented in Table 5.4. 195 1.5 1.0 0.5 0.0 −0.5 −1.0 −1.5 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 Figure 5.14: Solution (vorticity), flow over a three-blade vertical-axis wind turbine, at t = 0.3 We observe that for this problem, the Kronecker-product and exact block Ja- cobi preconditioners result in extremely similar iteration counts for all choices of ∆t considered. We also observe an almost linear growth in the number of itera- tions as we increase ∆t. This undesirable property likely results from the use of a component-uncoupled Jacobi preconditioner. We observe that using a block Jacobi preconditioner that inverts the 4 × 4 component blocks give rise to greatly improved convergence, at the cost of a factor of 64 in forming the preconditioner, and a factor of 16 in applying it. However, a comparable improvement in the Kronecker-product preconditioner was not observed, perhaps due to the complicated coupling between components arising from the ALE discretization. Even greater improvements in the iterative convergence for this problem can be achieved by using an ILU(0) incom- plete factorization. Such more sophisticated techniques allow for the use of time 196 Table 5.4: GMRES iteration counts for vertical-axis wind turbine ∆t Jacobi KSVD 5 × 10−8 10 11 1 × 10−7 15 16 2 × 10−7 24 28 4 × 10−7 47 49 8 × 10−7 111 113 steps that are about 500 times larger than the largest stable explicit time step. This example suggests that developing techniques to better couple the solution compo- nents within the Kronecker-product preconditioner could have a large impact on the solver performance. 5.2.3 Summary This test case illustrates the use of tensor-product solvers on problems with moving domains using an arbitrary Lagrangian-Eulerian method. The interior penalty for- mulation is developed for the viscous terms in the transformed equations, and the moving adiabatic wall boundary condition is verified by means of two test problems. The approximate Kronecker-product preconditioner is compared with the standard block Jacobi preconditioner on a three-blade configuration of a vertical axis wind turbine with tip-speed ratio λ = 1.8 at Reynolds number Re = 105 . The Kronecker- product preconditioner compares favorably with the standard block Jacobi precon- ditioner when applied component-wise, however, the convergence of the iterative solver stands to be much improved through better coupling of the components in the preconditioner. This is an important area for future research. Chapter Six Conclusion 198 High-order methods are a very promising area of ongoing research. Methods such as the discontinuous Galerkin method have been shown to result in highly-accurate solutions with relatively few degrees of freedom. It is believed that these methods have the potential for increased efficiency and lower memory and storage costs, which is of increasing importance on upcoming computing architectures. Such methods are considered advantageous for a range of applications, including, for example, the large eddy simulation of turbulent fluid flow. However, high-order discretizations of realistic problems often give rise to very severe stability restrictions, motivating the use of implicit time integration methods. In this thesis, we have studied and developed several preconditioners and solvers designed for the implicit time integration of discontinuous Galerkin discretizations. The DG method results in systems of equations with a large number of a degrees of freedom, making implicit time integration a challenging topic. The development of efficient iterative solvers and preconditioners is essential for good performance. We have extended ILU-based incomplete factorizations to fully-implicit Runge- Kutta time integration methods, demonstrating favorable performance, while allow- ing for the use of highly stable, arbitrary order, stiffly-accurate fully-implicit schemes. These methods also allow for parallelism across the stages of the method, provid- ing a promising approach to achieve parallelism in time, which is expected to be of increasing importance on anticipated computing architectures. We have developed and analyzed a class of approximate Kronecker-product pre- conditioners, which allow for the use of very high-order polynomial spaces. By taking advantage of the tensor-product structure of quadrilateral and hexahedral computa- tional meshes, both the storage costs and computational complexity of implicit time integration can be greatly reduced. These preconditioners can be formed and ap- 199 plied efficiently, using a construction known as the Kronecker-product singular value decomposition. The effectiveness of these preconditioners has been demonstrated on a very wide array of example problems and test cases. The use polygonal meshes instead of traditional quadrilateral or triangular meshes was also studied. A von Neumann analysis demonstrates that quadrilateral and hexagonal meshes give rise to linear systems that allow for faster convergence of iter- ative solvers such as the block Jacobi method. This conclusion is borne out through a series of numerical experiments on a range of unstructured polygonal and triangular meshes, using a GMRES solver with both block Jacobi and ILU(0) preconditioners. Finally, we presented two challenging case studies that demonstrate the effective- ness of these techniques applied to large-scale problems. These problems demonstrate the utility of implicit time integration, and the need for efficient solvers and precon- ditioners. Applying the techniques developed in this thesis to these two problems has shown the benefit of both the use of stage-parallel fully-implicit IRK methods, and tensor-product preconditioners together with high-order DG discretizations. There remains significant room for future work on these topics. The use of stage- parallelism for IRK solvers has proven to be quite effective, but the work in this thesis was limited to methods with at most five stages. The use and development of stage-parallel solvers with higher numbers of stages is of interest. Additionally, the tensor-product preconditioners developed in this work are variants of the tra- ditional block Jacobi method. Preconditioners such as ILU have been shown to be more effective than block Jacobi, and thus the extension of the Kronecker-product preconditioners to ILU-type incomplete factorizations has the potential for a large impact. Similarly, the incorporation of the Kronecker-product preconditioners into a multilevel algorithm such as p-multigrid is also of relevance. Appendix A Butcher tableaux 201 A.1 Radau IIA methods We construct the Butcher tableaux for the Radau IIA methods according to the derivation of Axelsson [4]. The following Mathematica code can be used to gener- ate the abscissa c and matrix of coefficients A given the number of stages s. The coefficients b are identical to the last row of A. Algorithm A.1 Mathematica code to compute the coefficients of Radau IIA schemes 1 q[n_,x_] = LegendreP[n,2 x-1]-LegendreP[n-1,2 x-1]; Define the polynomial Qn (x) according to equation (2.1) from [4] 2 c[n_] := (x /. NSolve[q[n,x] == 0,x]) // Re // (Sort[#,Less] &) The abscissa ck are given by the zeros of Qn (x) 3 `[n_,k_,x_] := q[n,x]/((x-c[n][[k]]) (D[q[n,x],x] /. x -> c[n][[k]])) Define `k (x) as in [4] 4 a[n_,i_,k_] := NIntegrate[`[n,k,x],{x,0,c[n][[i]]}] Rc The quadrature coefficients aik are computed as 0 i `k (x) dx 5 A[n_] := Table[a[n,i,k],{i,1,n},{k,1,n}] Finally, we form the Butcher matrix A For the particular cases of s = 2 and s = 3 the resulting tableaux, which we present below, are quite simple. 1/3 5/12 −1/12 RADAU23: 1 3/4 1/4 (A.1) 3/4 1/4 √ √ √ √ 2 6 11 7 6 37 169 6 2 6 5 − 10 45 − 360 225 − 1800 − 225 + 75 √ √ √ √ 2 6 37 169 6 11 7 6 2 6 5 + 10 225 − 1800 45 − 360 − 225 − 75 RADAU35: √ √ (A.2) 1 4 9 − 366 4 9 + 36 6 1 9 √ √ 4 9 − 366 4 9 + 36 6 1 9 202 A.2 DIRK methods The three-stage, third-order DIRK method derived by Alexander in [1] is given by α α 0 0 τ2 τ2 − α α 0 DIRK33: (A.3) 1 b1 b2 α b1 b2 α where the coefficients are √   √  √   √  6 1 2 2 1 2 α=1+ 2 sin 3 arctan 4 − 2 cos 3 arctan 4 τ2 = (1 + α)/2 (A.4) b1 = −(6α2 − 16α + 1)/4 b2 = (6α2 − 20α + 5)/4 Finally, the six-stage, fifth-order ESDIRK method derived by Boom and Zingg [18] is given by (A.5). K. P. P. p. p. p. p. ESDIRK65: 0 0 0 0 0 0 0 0.556107682272893 0.2780538411364465 0.2780538411364465 0 0 0 0 1.028127096688746 0.3137405401502951 0.4363327154020044 0.2780538411364465 0 0 0 0.540645375074761 0.2741986534107860 −0.0164268277321164 0.0048197082596452 0.2780538411364465 0 0 0.058741042826253 −0.2441776975175844 −3.3203529439447852 0.0477747285706825 3.2974431145814931 0.2780538411364465 0 1 −0.2786732780227907 1.8929947094010862 −0.1280948204262490 −1.3574693381380240 0.5931888860495311 0.2780538411364465 −0.2786732780227907 1.8929947094010862 −0.1280948204262490 −1.3574693381380240 0.5931888860495311 0.2780538411364465 (A.5) 203 Bibliography [1] R. Alexander. “Diagonally implicit Runge-Kutta methods for stiff O.D.E.’s”. In: SIAM Journal on Numerical Analysis 14.6 (Dec. 1977), pp. 1006–1021. doi: 10.1137/0714068. [2] D. N. Arnold. “An interior penalty finite element method with discontinu- ous elements”. In: SIAM Journal on Numerical Analysis 19.4 (Aug. 1982), pp. 742–760. doi: 10.1137/0719052. [3] D. N. Arnold et al. “Unified analysis of discontinuous Galerkin methods for elliptic problems”. In: SIAM Journal on Numerical Analysis 39.5 (2002), pp. 1749–1779. doi: 10.1137/s0036142901384162. [4] O. Axelsson. “A class of A-stable methods”. In: BIT 9.3 (Sept. 1969), pp. 185– 199. doi: 10.1007/bf01946812. [5] A. Baggag, H. Atkins, and D. Keyes. Parallel implementation of the discon- tinuous Galerkin method. Tech. rep. Institute for Computer Applications in Science and Engineering, 1999. [6] T. J. Baker. “Delaunay-Vorono¨ı methods”. In: Handbook of Grid Generation. Ed. by B. K. Soni, N. P. Weatherill, and J. F. Thompson. CRC Press, 1999. [7] G. Balafas. “Polyhedral mesh generation for CFD-analysis of complex struc- tures”. MA thesis. Technische Universit¨at M¨ unchen, 2014. [8] F. Bassi and S. Rebay. “A high order discontinuous Galerkin method for compressible turbulent flows”. In: Discontinuous Galerkin Methods. Ed. by B. Cockburn, G. E. Karniadakis, and C.-W. Shu. Springer Berlin Heidelberg, 2000, pp. 77–88. doi: 10.1007/978-3-642-59721-3_4. [9] F. Bassi and S. Rebay. “A high-order accurate discontinuous finite element method for the numerical solution of the compressible Navier-Stokes equa- tions”. In: Journal of Computational Physics 131.2 (Mar. 1997), pp. 267–279. doi: 10.1006/jcph.1996.5572. [10] F. Bassi and S. Rebay. “GMRES discontinuous Galerkin solution of the com- pressible Navier-Stokes equations”. In: Discontinuous Galerkin Methods. Ed. by B. Cockburn, G. E. Karniadakis, and C.-W. Shu. Springer Berlin Heidel- berg, 2000, pp. 77–88. doi: 10.1007/978-3-642-59721-3_4. 204 BIBLIOGRAPHY 205 [11] F. Bassi et al. “A high-order accurate discontinuous finite element method for inviscid and viscous turbomachinery flow”. In: Proceedings of the Second Eu- ropean Conference on Turbomachinery Fluid Dynamics and Thermodynam- ics. Ed. by R. Decuypere and G. Dibelius. Technologisch Instituut Section on Aeronautics. Antwerp, Belgium, 1997. [12] F. Bassi et al. “Linearly implicit Rosenbrock-type Runge-Kutta schemes ap- plied to the discontinuous Galerkin solution of compressible and incompress- ible unsteady flows”. In: Computers & Fluids 118 (Sept. 2015), pp. 305–320. doi: 10.1016/j.compfluid.2015.06.007. [13] A. D. Beck et al. “High-order discontinuous Galerkin spectral element meth- ods for transitional and turbulent flow simulations”. In: International Jour- nal for Numerical Methods in Fluids 76.8 (Aug. 2014), pp. 522–548. doi: 10.1002/fld.3943. [14] M. Benzi, G. H. Golub, and J. Liesen. “Numerical solution of saddle point problems”. In: Acta Numerica 14 (May 2005), pp. 1–137. doi: 10 . 1017 / s0962492904000212. [15] M. Benzi, W. Joubert, and G. Mateescu. “Numerical experiments with par- allel orderings for ILU preconditioners”. In: Electronic Transactions on Nu- merical Analysis 8 (1999), pp. 88–114. [16] M. Berggren. “A vertex-centered, dual discontinuous Galerkin method”. In: Journal of Computational and Applied Mathematics 192.1 (July 2006), pp. 175– 181. doi: 10.1016/j.cam.2005.04.057. [17] H. Bijl et al. “Implicit time integration schemes for the unsteady compressible Navier-Stokes equations: laminar flow”. In: Journal of Computational Physics 179.1 (June 2002), pp. 313–329. doi: 10.1006/jcph.2002.7059. [18] P. D. Boom and D. W. Zingg. “High-order implicit temporal integration for unsteady compressible fluid flow simulation”. In: 21st AIAA Computational Fluid Dynamics Conference. American Institute of Aeronautics and Astro- nautics, June 2013. doi: 10.2514/6.2013-2831. [19] P. Boom and D. Zingg. “Time-accurate flow simulations using an efficient Newton-Krylov-Schur approach with high-order temporal and spatial dis- cretization”. In: 51st AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition. American Institute of Aeronau- tics and Astronautics, Jan. 2013. doi: 10.2514/6.2013-383. [20] M. E. Brachet et al. “Small-scale structure of the Taylor-Green vortex”. In: Journal of Fluid Mechanics 130.-1 (May 1983), p. 411. doi: 10.1017/ s0022112083001159. [21] F. Brezzi et al. “Discontinuous Galerkin approximations for elliptic prob- lems”. In: Numerical Methods for Partial Differential Equations 16.4 (2000), pp. 365–378. doi: 10.1002/1098-2426(200007)16:4<365::aid-num2>3.0. co;2-y. [22] K. Burrage, C. Eldershaw, and R. Sidje. “A parallel matrix-free implemen- tation of a Runge-Kutta code”. In: Joint Australian-Taiwanese Workshop on Analysis and Applications. 1999. [23] J. C. Butcher. “On the implementation of implicit Runge-Kutta methods”. In: BIT 16.3 (Sept. 1976), pp. 237–240. doi: 10.1007/bf01932265. BIBLIOGRAPHY 206 [24] M. H. Carpenter et al. “Fourth-order Runge-Kutta schemes for fluid mechan- ics applications”. In: Journal of Scientific Computing 25.1-2 (Nov. 2005), pp. 157–194. doi: 10.1007/bf02728987. [25] M. H. Carpenter, S. Viken, and E. Nielsen. “The temporal efficiency of higher order schemes”. In: 41st Aerospace Sciences Meeting and Exhibit. American Institute of Aeronautics and Astronautics, Jan. 2003. doi: 10.2514/6.2003- 86. [26] N. Chalmers, L. Krivodonova, and R. Qin. “Relaxing the CFL number of the discontinuous Galerkin method”. In: SIAM Journal on Scientific Computing 36.4 (Jan. 2014), A2047–A2075. doi: 10.1137/130927504. [27] B. Cockburn and B. Dong. “An analysis of the minimal dissipation local dis- continuous Galerkin method for convection-diffusion problems”. In: Journal of Scientific Computing 32.2 (Mar. 2007), pp. 233–262. doi: 10.1007/s10915- 007-9130-3. [28] B. Cockburn, S. Hou, and C.-W. Shu. “The Runge-Kutta local projection dis- continuous Galerkin finite element method for conservation laws. IV. The mul- tidimensional case”. In: Mathematics of Computation 54.190 (1990), pp. 545– 545. doi: 10.1090/s0025-5718-1990-1010597-0. [29] B. Cockburn, S.-Y. Lin, and C.-W. Shu. “TVB Runge-Kutta local projec- tion discontinuous Galerkin finite element method for conservation laws III: One-dimensional systems”. In: Journal of Computational Physics 84.1 (1989), pp. 90–113. doi: 10.1016/0021-9991(89)90183-6. [30] B. Cockburn and C.-W. Shu. “Runge-Kutta discontinuous Galerkin methods for convection-dominated problems”. In: Journal of Scientific Computing 16.3 (Sept. 2001), pp. 173–261. doi: 10.1023/A:1012873910884. [31] B. Cockburn and C.-W. Shu. “The Local Discontinuous Galerkin Method for Time-Dependent Convection-Diffusion Systems”. In: SIAM Journal on Nu- merical Analysis 35.6 (Dec. 1998), pp. 2440–2463. doi: 10.1137/s0036142997316712. [32] B. Cockburn and C.-W. Shu. “The Runge-Kutta discontinuous Galerkin method for conservation laws V: Multidimensional systems”. In: Journal of Compu- tational Physics 141.2 (Apr. 1998), pp. 199–224. doi: 10.1006/jcph.1998. 5892. [33] B. Cockburn and C.-W. Shu. “The Runge-Kutta local projection P 1 -discontinuous- Galerkin finite element method for scalar conservation laws”. In: ESAIM: Mathematical Modelling and Numerical Analysis 25.3 (1991), pp. 337–361. doi: 10.1051/m2an/1991250303371. [34] B. Cockburn and C.-W. Shu. “TVB Runge-Kutta local projection discon- tinuous Galerkin finite element method for conservation laws. II. General framework”. In: Mathematics of Computation 52.186 (1989), pp. 411–411. doi: 10.1090/s0025-5718-1989-0983311-4. [35] A. Crivellini and F. Bassi. “An implicit matrix-free discontinuous Galerkin solver for viscous and turbulent aerodynamic simulations”. In: Computers & Fluids 50.1 (Nov. 2011), pp. 81–93. doi: 10.1016/j.compfluid.2011.06. 020. BIBLIOGRAPHY 207 [36] E. Cuthill and J. McKee. “Reducing the bandwidth of sparse symmetric ma- trices”. In: ACM Proceedings of the 1969 24th National Conference. ACM ’69. New York, NY, USA: ACM, 1969, pp. 157–172. doi: 10.1145/800195. 805928. [37] G. G. Dahlquist. “A special stability problem for linear multistep methods”. In: BIT 3.1 (Mar. 1963), pp. 27–43. doi: 10.1007/bf01963532. [38] J. DeBonis. “Solutions of the Taylor-Green vortex problem using high-resolution explicit finite difference methods”. In: 51st AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition. American Insti- tute of Aeronautics and Astronautics, Jan. 2013. doi: 10.2514/6.2013-382. [39] J. W. Demmel. Applied Numerical Linear Algebra. Society for Industrial and Applied Mathematics, Jan. 1997. doi: 10.1137/1.9781611971446. [40] L. T. Diosady and D. L. Darmofal. “Preconditioning methods for discon- tinuous Galerkin solutions of the Navier-Stokes equations”. In: Journal of Computational Physics 228.11 (June 2009), pp. 3917–3935. doi: 10.1016/j. jcp.2009.02.035. [41] L. T. Diosady and S. M. Murman. “Tensor-product preconditioners for higher- order space-time discontinuous Galerkin methods”. In: Journal of Computa- tional Physics 330 (Feb. 2017), pp. 296–318. doi: 10.1016/j.jcp.2016.11. 022. [42] B. Diskin and J. L. Thomas. “Comparison of node-centered and cell-centered unstructured finite-volume discretizations: inviscid fluxes”. In: AIAA Journal 49.4 (Apr. 2011), pp. 836–854. doi: 10.2514/1.j050897. [43] B. Diskin et al. “Comparison of node-centered and cell-centered unstructured finite-volume discretizations: viscous fluxes”. In: AIAA Journal 48.7 (July 2010), pp. 1326–1338. doi: 10.2514/1.44940. [44] J. Donea, S. Giuliani, and J. Halleux. “An arbitrary Lagrangian-Eulerian finite element method for transient dynamic fluid-structure interactions”. In: Computer Methods in Applied Mechanics and Engineering 33.1-3 (Sept. 1982), pp. 689–723. doi: 10.1016/0045-7825(82)90128-1. [45] Y. Dubief and F. Delcayre. “On coherent-vortex identification in turbulence”. In: Journal of Turbulence 1 (Jan. 2000), N11. doi: 10.1088/1468-5248/1/ 1/011. [46] I. S. Duff and G. A. Meurant. “The effect of ordering on preconditioned conjugate gradients”. In: BIT Numerical Mathematics 29.4 (1989), pp. 635– 657. doi: 10.1007/BF01932738. [47] J. A. Escobar-Vargas, P. J. Diamessis, and C. F. Van Loan. “The numeri- cal solution of the pressure Poisson equation for the incompressible Navier- Stokes equations using a quadrilateral spectral multidomain penalty method”. Preprint available at https : / / www . cs . cornell . edu / cv / ResearchPDF / Poisson.pdf. 2011. [48] D. J. Estep and R. W. Freund. “Using Krylov-subspace iterations in discon- tinuous Galerkin methods for nonlinear reaction-diffusion systems”. In: Dis- continuous Galerkin Methods. Ed. by B. Cockburn, G. E. Karniadakis, and C.-W. Shu. Springer Berlin Heidelberg, 2000, pp. 77–88. doi: 10.1007/978- 3-642-59721-3_4. BIBLIOGRAPHY 208 [49] K. J. Fidkowski et al. “p-multigrid solution of high-order discontinuous Galerkin discretizations of the compressible Navier-Stokes equations”. In: Journal of Computational Physics 207.1 (July 2005), pp. 92–113. doi: 10.1016/j.jcp. 2005.01.005. [50] R. Frank, J. Schneid, and C. W. Ueberhuber. “Order results for implicit Runge-Kutta methods applied to stiff systems”. In: SIAM Journal on Nu- merical Analysis 22.3 (June 1985), pp. 515–534. doi: 10.1137/0722031. [51] M. Galbraith and M. Visbal. “Implicit large eddy simulation of low Reynolds number flow past the SD7003 airfoil”. In: 46th AIAA Aerospace Sciences Meeting and Exhibit. American Institute of Aeronautics and Astronautics, Jan. 2008. doi: 10.2514/6.2008-225. [52] M. Galbraith and M. Visbal. “Implicit large eddy simulation of low-Reynolds- number transitional flow past the SD7003 airfoil”. In: 40th Fluid Dynamics Conference and Exhibit. American Institute of Aeronautics and Astronautics, June 2010. doi: 10.2514/6.2010-4737. [53] R. V. Garimella, J. Kim, and M. Berndt. “Polyhedral mesh generation and optimization for non-manifold domains”. In: Proceedings of the 22nd Interna- tional Meshing Roundtable. Springer International Publishing, 2014, pp. 313– 330. doi: 10.1007/978-3-319-02335-9_18. [54] A. George. “Nested dissection of a regular finite element mesh”. In: SIAM Journal on Numerical Analysis 10.2 (Apr. 1973), pp. 345–363. doi: 10.1137/ 0710032. [55] G. H. Golub, F. T. Luk, and M. L. Overton. “A block Lanczos method for com- puting the singular values and corresponding singular vectors of a matrix”. In: ACM Transactions on Mathematical Software 7.2 (June 1981), pp. 149–169. doi: 10.1145/355945.355946. [56] J. Gopalakrishnan and G. Kanschat. “A multilevel discontinuous Galerkin method”. In: Numerische Mathematik 95.3 (Sept. 2003), pp. 527–550. doi: 10.1007/s002110200392. [57] D. Gottlieb and E. Tadmor. “The CFL condition for spectral approximations to hyperbolic initial-boundary value problems”. In: Mathematics of Compu- tation 56.194 (May 1991), pp. 565–565. doi: 10.1090/s0025- 5718- 1991- 1066833-9. [58] S. Gottlieb and C.-W. Shu. “Total variation diminishing Runge-Kutta schemes”. In: Mathematics of Computation of the American Mathematical Society 67.221 (Jan. 1998), pp. 73–85. doi: 10.1090/s0025-5718-98-00913-2. [59] S. Gottlieb, C.-W. Shu, and E. Tadmor. “Strong stability-preserving high- order time discretization methods”. In: SIAM Review 43.1 (Jan. 2001), pp. 89– 112. doi: 10.1137/s003614450036757x. [60] E. Hairer, S. P. Nørsett, and G. Wanner. Solving Ordinary Differential Equa- tions I. Springer-Verlag Berlin Heidelberg, 1993. doi: 10.1007/978-3-540- 78862-1. [61] E. Hairer and G. Wanner. Solving Ordinary Differential Equations II. Springer- Verlag Berlin Heidelberg, 1996. doi: 10.1007/978-3-642-05221-7. BIBLIOGRAPHY 209 [62] R. Hartmann and P. Houston. “An optimal order interior penalty discontinu- ous Galerkin discretization of the compressible Navier–Stokes equations”. In: Journal of Computational Physics 227.22 (Nov. 2008), pp. 9670–9685. doi: 10.1016/j.jcp.2008.07.015. [63] R. Hartmann and P. Houston. “Symmetric interior penalty DG methods for the compressible Navier-Stokes equations. I. Method formulation.” In: Inter- national Journal of Numerical Analysis and Modeling 3.1 (2006), p. 1. [64] M. Hestenes and E. Stiefel. “Methods of conjugate gradients for solving linear systems”. In: Journal of Research of the National Bureau of Standards 49.6 (Dec. 1952), p. 409. doi: 10.6028/jres.049.044. [65] J. C. R. Hunt, A. A. Wray, and P. Moin. Eddies, streams, and convergence zones in turbulent flows. Tech. rep. CTR-S88. Center for Turbulence Research, 1988. [66] L. O. Jay and T. Braconnier. “A parallelizable preconditioner for the iter- ative solution of implicit Runge-Kutta-type methods”. In: Journal of Com- putational and Applied Mathematics 111.1-2 (Nov. 1999), pp. 63–76. doi: 10.1016/s0377-0427(99)00132-6. [67] G.-S. Jiang and C.-W. Shu. “Efficient implementation of weighted ENO schemes”. In: Journal of Computational Physics 126.1 (June 1996), pp. 202–228. doi: 10.1006/jcph.1996.0130. [68] S. Kanner and P.-O. Persson. “Validation of high-order wall-resolved large- eddy simulation of vertical-axis wind turbines”. In: 2018 Wind Energy Sym- posium. American Institute of Aeronautics and Astronautics, Jan. 2018. doi: 10.2514/6.2018-0990. [69] S. Kanner, L. Wang, and P.-O. Persson. “Implicit large-eddy simulation of 2D counter-rotating vertical-axis wind turbines”. In: 34th Wind Energy Sym- posium. American Institute of Aeronautics and Astronautics, Jan. 2016. doi: 10.2514/6.2016-1731. [70] S. Kanner and P.-O. Persson. “Validation of a High-Order Large-Eddy Sim- ulation Solver Using a Vertical-Axis Wind Turbine”. In: AIAA Journal 54.1 (Jan. 2016), pp. 101–112. doi: 10.2514/1.j054138. [71] R. Kl¨ofkorn. “Efficient matrix-free implementation of discontinuous Galerkin methods for compressible flow problems”. In: Proceedings of ALGORITMY. ˇ coviˇc. 2012, pp. 11–21. Ed. by A. Handloviˇcov´a, Z. Minarechov´a, and D. Sevˇ [72] L. Krivodonova and R. Qin. “An analysis of the spectrum of the discontinuous Galerkin method”. In: Applied Numerical Mathematics 64 (Feb. 2013), pp. 1– 18. doi: 10.1016/j.apnum.2012.07.008. [73] M. Kronbichler et al. “Comparison of implicit and explicit hybridizable dis- continuous Galerkin methods for the acoustic wave equation”. In: Interna- tional Journal for Numerical Methods in Engineering 106.9 (Oct. 2015), pp. 712– 739. doi: 10.1002/nme.5137. [74] E. J. Kubatko, C. Dawson, and J. J. Westerink. “Time step restrictions for Runge-Kutta discontinuous Galerkin methods on triangular grids”. In: Journal of Computational Physics 227.23 (Dec. 2008), pp. 9697–9710. doi: 10.1016/j.jcp.2008.07.026. BIBLIOGRAPHY 210 [75] A. Kværnø. “Singly diagonally implicit Runge-Kutta methods with an explicit first stage”. In: BIT Numerical Mathematics 44.3 (Aug. 2004), pp. 489–502. doi: 10.1023/b:bitn.0000046811.70614.38. [76] R. J. LeVeque. Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, 2002. doi: 10.1017/cbo9780511791253. [77] H. Luo, J. D. Baum, and R. L¨ohner. “A discontinuous Galerkin method based on a Taylor basis for the compressible flows on arbitrary grids”. In: Journal of Computational Physics 227.20 (Oct. 2008), pp. 8875–8893. doi: 10.1016/ j.jcp.2008.06.035. [78] H. Luo, J. D. Baum, and R. L¨ohner. “On the computation of steady-state compressible flows using a DG method”. In: Proceedings of the Fourth Inter- national Conference on Computational Fluid Dynamics. Ed. by H. Deconinck and E. Dick. Springer Berlin Heidelberg, 2006, pp. 47–52. doi: 10.1007/978- 3-540-92779-2_4. [79] R. E. Lynch, J. R. Rice, and D. H. Thomas. “Direct solution of partial differ- ence equations by tensor product methods”. In: Numerische Mathematik 6.1 (Dec. 1964), pp. 185–199. doi: 10.1007/bf01386067. [80] J. A. Mackenzie and K. W. Morton. “Finite volume solutions of convection- diffusion test problems”. In: Mathematics of Computation 60.201 (Jan. 1993), p. 189. doi: 10.2307/2153161. [81] T. A. Manteuffel. “An incomplete factorization technique for positive definite linear systems”. In: Mathematics of Computation 34.150 (May 1980), pp. 473– 473. doi: 10.1090/s0025-5718-1980-0559197-0. [82] G. Manzini, A. Russo, and N. Sukumar. “New perspectives on polygonal and polyhedral finite element methods”. In: Mathematical Models and Meth- ods in Applied Sciences 24.08 (July 2014), pp. 1665–1699. doi: 10 . 1142 / s0218202514400065. [83] H. M. Markowitz. “The elimination form of the inverse and its application to linear programming”. In: Management Science 3.3 (Apr. 1957), pp. 255–269. doi: 10.1287/mnsc.3.3.255. [84] P. Mocz et al. “A discontinuous Galerkin method for solving the fluid and magnetohydrodynamic equations in astrophysical simulations”. In: Monthly Notices of the Royal Astronomical Society 437.1 (Oct. 2013), pp. 397–414. doi: 10.1093/mnras/stt1890. [85] R. D. Nair, M. N. Levy, and P. H. Lauritzen. “Emerging numerical methods for atmospheric modeling”. In: Numerical Techniques for Global Atmospheric Models. Springer Berlin Heidelberg, 2011, pp. 251–311. doi: 10.1007/978- 3-642-11640-7_9. [86] C. R. Nastase and D. J. Mavriplis. “High-order discontinuous Galerkin meth- ods using an hp-multigrid approach”. In: Journal of Computational Physics 213.1 (Mar. 2006), pp. 330–357. doi: 10.1016/j.jcp.2005.08.022. [87] A. Nigro et al. “Up to sixth-order accurate A-stable implicit schemes applied to the discontinuous Galerkin discretized Navier–Stokes equations”. In: Jour- nal of Computational Physics 276 (2014), pp. 136–162. doi: http://dx.doi. org/10.1016/j.jcp.2014.07.028. BIBLIOGRAPHY 211 [88] A. Nigro et al. “Modified extended BDF scheme for the discontinuous Galerkin solution of unsteady compressible flows”. In: International Journal for Nu- merical Methods in Fluids 76.9 (2014), pp. 549–574. doi: 10.1002/fld.3944. [89] W. Oaks and S. Paoletti. “Polyhedral mesh generation”. In: Proceedings of the 9th International Meshing Roundtable. 2000, pp. 57–67. [90] M. Ol et al. “Comparison of laminar separation bubble measurements on a low Reynolds number airfoil in three facilities”. In: 35th AIAA Fluid Dynamics Conference and Exhibit. American Institute of Aeronautics and Astronautics, June 2005. doi: 10.2514/6.2005-5149. [91] S. A. Orszag. “Spectral methods for problems in complex geometries”. In: Journal of Computational Physics 37.1 (1980), pp. 70–92. doi: http://dx. doi.org/10.1016/0021-9991(80)90005-4. [92] W. Pazner and P.-O. Persson. “Approximate tensor-product preconditioners for very high order discontinuous Galerkin methods”. In: Journal of Compu- tational Physics 354 (Feb. 2018), pp. 344–369. doi: 10.1016/j.jcp.2017. 10.030. [93] W. Pazner and P.-O. Persson. “High-order DNS and LES simulations using an implicit tensor-product discontinuous Galerkin method”. In: 23rd AIAA Computational Fluid Dynamics Conference. American Institute of Aeronau- tics and Astronautics, June 2017. doi: 10.2514/6.2017-3948. [94] W. Pazner and P.-O. Persson. “Interior penalty tensor-product precondition- ers for high-order discontinuous Galerkin discretizations”. In: 2018 AIAA Aerospace Sciences Meeting. American Institute of Aeronautics and Astro- nautics, Jan. 2018. doi: 10.2514/6.2018-1093. [95] W. Pazner and P.-O. Persson. “On the convergence of iterative solvers for polygonal discontinuous Galerkin discretizations”. In: Communications in Applied Mathematics and Computational Science 13.1 (Feb. 2018), pp. 27– 51. doi: 10.2140/camcos.2018.13.27. [96] W. Pazner and P.-O. Persson. “Stage-parallel fully implicit Runge-Kutta solvers for discontinuous Galerkin fluid simulations”. In: Journal of Compu- tational Physics 335 (Apr. 2017), pp. 700–717. doi: 10.1016/j.jcp.2017. 01.050. [97] W. Pazner et al. “A high-order spectral deferred correction strategy for low Mach number flow with complex chemistry”. In: Combustion Theory and Modelling 20.3 (Mar. 2016), pp. 521–547. doi: 10.1080/13647830.2016. 1150519. [98] J. Peraire and P.-O. Persson. “The compact discontinuous Galerkin (CDG) method for elliptic problems”. In: SIAM Journal on Scientific Computing 30.4 (Jan. 2008), pp. 1806–1824. doi: 10.1137/070685518. [99] J. Peraire and P.-O. Persson. “High order discontinuous Galerkin methods for CFD”. In: Advances in Computational Fluid Dynamics. Ed. by Z. J. Wang. World Scientific, Mar. 2011, pp. 119–152. doi: 10 . 1142 / 9789814313193 _ 0005. [100] J. Peraire et al. “Adaptive remeshing for compressible flow computations”. In: Journal of Computational Physics 72.2 (Oct. 1987), pp. 449–466. doi: 10.1016/0021-9991(87)90093-3. BIBLIOGRAPHY 212 [101] M. Peric. “Flow simulation using control volumes of arbitrary polyhedral shape”. In: ERCOFTAC bulletin 62 (2004), pp. 25–29. [102] P.-O. Persson. “High-order LES simulations using implicit-explicit Runge- Kutta schemes”. In: 49th AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition. American Institute of Aeronautics and Astronautics, Jan. 2011. doi: 10.2514/6.2011-684. [103] P.-O. Persson. “Mesh generation for implicit geometries”. PhD thesis. Mas- sachusetts Institute of Technology, 2005. [104] P.-O. Persson. “Scalable parallel Newton-Krylov solvers for discontinuous Galerkin discretizations”. In: 47th AIAA Aerospace Sciences Meeting includ- ing The New Horizons Forum and Aerospace Exposition. American Institute of Aeronautics and Astronautics, Jan. 2009. doi: 10.2514/6.2009-606. [105] P.-O. Persson and J. Peraire. “An efficient low memory implicit DG algorithm for time dependent problems”. In: 44th AIAA Aerospace Sciences Meeting and Exhibit. American Institute of Aeronautics and Astronautics, Jan. 2006. doi: 10.2514/6.2006-113. [106] P.-O. Persson, J. Bonet, and J. Peraire. “Discontinuous Galerkin solution of the Navier-Stokes equations on deformable domains”. In: Computer Methods in Applied Mechanics and Engineering 198.17-20 (Apr. 2009), pp. 1585–1595. doi: 10.1016/j.cma.2009.01.012. [107] P.-O. Persson and J. Peraire. “Newton-GMRES preconditioning for discon- tinuous Galerkin discretizations of the Navier-Stokes equations”. In: SIAM Journal on Scientific Computing 30.6 (Jan. 2008), pp. 2709–2733. doi: 10. 1137/070692108. [108] W. H. Reed and T. R. Hill. “Triangular mesh methods for the neutron trans- port equation”. In: Los Alamos Report LA-UR-73-479 (1973). [109] W. M. van Rees et al. “A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers”. In: Journal of Computational Physics 230.8 (Apr. 2011), pp. 2794–2805. doi: 10.1016/j.jcp.2010.11.031. [110] P. L. Roe. “Approximate Riemann solvers, parameter vectors, and difference schemes”. In: Journal of Computational Physics 43.2 (Oct. 1981), pp. 357– 372. doi: 10.1016/0021-9991(81)90128-5. [111] P. L. Roe. “The use of the Riemann problem in finite difference schemes”. In: Proceedings of the Seventh International Conference on Numerical Methods in Fluid Dynamics. Springer Berlin Heidelberg, 1989, pp. 354–359. doi: 10. 1007/3-540-10694-4_54. [112] J. Ruppert. “A Delaunay refinement algorithm for quality 2-dimensional mesh generation”. In: Journal of Algorithms 18.3 (May 1995), pp. 548–585. doi: 10.1006/jagm.1995.1021. [113] Y. Saad and M. H. Schultz. “GMRES: A generalized minimal residual algo- rithm for solving nonsymmetric linear systems”. In: SIAM Journal on Scien- tific and Statistical Computing 7.3 (July 1986), pp. 856–869. doi: 10.1137/ 0907058. BIBLIOGRAPHY 213 [114] Y. Saad. Iterative Methods for Sparse Linear Systems. Society for Industrial and Applied Mathematics, Jan. 2003. doi: 10.1137/1.9780898718003. [115] M. S. Selig et al. Summary of Low-Speed Airfoil Data. Vol. 1. SoarTech Pub- lications, 1995. [116] K. Shahbazi. “An explicit expression for the penalty parameter of the interior penalty method”. In: Journal of Computational Physics 205.2 (May 2005), pp. 401–407. doi: 10.1016/j.jcp.2004.11.017. [117] J. Shen, T. Tang, and L.-L. Wang. “Separable Multi-Dimensional Domains”. In: Spectral Methods. Springer Berlin Heidelberg, 2011, pp. 299–366. doi: 10.1007/978-3-540-71041-7_8. [118] J. R. Shewchuk. “Delaunay refinement algorithms for triangular mesh gen- eration”. In: Computational Geometry 22.1-3 (May 2002), pp. 21–74. doi: 10.1016/s0925-7721(01)00047-5. [119] C.-W. Shu. “Total-variation-diminishing time discretizations”. In: SIAM Jour- nal on Scientific and Statistical Computing 9.6 (Nov. 1988), pp. 1073–1084. doi: 10.1137/0909073. [120] C.-W. Shu and S. Osher. “Efficient implementation of essentially non-oscillatory shock-capturing schemes”. In: Journal of Computational Physics 77.2 (Aug. 1988), pp. 439–471. doi: 10.1016/0021-9991(88)90177-5. [121] C.-W. Shu and S. Osher. “Efficient implementation of essentially non-oscillatory shock-capturing schemes, II”. In: Upwind and High-Resolution Schemes. Springer Berlin Heidelberg, 1989, pp. 328–374. doi: 10.1007/978-3-642-60543-7_14. [122] R. M. Smith and A. G. Hutton. “The numerical treatment of advection: a performance comparison of current methods”. In: Numerical Heat Transfer 5.4 (Oct. 1982), pp. 439–461. doi: 10.1080/10407788208913458. [123] V. Springel. “E pur si muove: Galilean-invariant cosmological hydrodynamical simulations on a moving mesh”. In: Monthly Notices of the Royal Astronomical Society 401.2 (Jan. 2010), pp. 791–851. doi: 10.1111/j.1365-2966.2009. 15715.x. [124] J. J. B. de Swart, W. M. Lioen, and W. A. van der Veen. Specification of PSIDE. Tech. rep. Centrum Wiskunde & Informatica, 1998. [125] C. Talischi et al. “PolyMesher: a general-purpose mesh generator for polygonal elements written in Matlab”. In: Structural and Multidisciplinary Optimiza- tion 45.3 (Jan. 2012), pp. 309–328. doi: 10.1007/s00158-011-0706-z. [126] E. F. Toro. Riemann Solvers and Numerical Methods for Fluid Dynamics. Springer Berlin Heidelberg, 2009. doi: 10.1007/b79761. [127] A. Uranga et al. “Implicit large eddy simulation of transitional flows over airfoils and wings”. In: 19th AIAA Computational Fluid Dynamics. American Institute of Aeronautics and Astronautics, June 2009. doi: 10.2514/6.2009- 4131. [128] C. F. Van Loan and N. Pitsianis. “Approximation with Kronecker products”. In: Linear Algebra for Large Scale and Real-Time Applications. Springer Nether- lands, 1993, pp. 293–314. doi: 10.1007/978-94-015-8196-7_17. BIBLIOGRAPHY 214 [129] C. F. Van Loan. “The ubiquitous Kronecker product”. In: Journal of Com- putational and Applied Mathematics 123.1-2 (Nov. 2000), pp. 85–100. doi: 10.1016/s0377-0427(00)00393-9. [130] B. C. Vermeire, F. D. Witherden, and P. E. Vincent. “On the utility of GPU accelerated high-order methods for unsteady flow simulations: A compari- son with industry-standard tools”. In: Journal of Computational Physics 334 (Apr. 2017), pp. 497–521. doi: 10.1016/j.jcp.2016.12.049. [131] P. E. Vos, S. J. Sherwin, and R. M. Kirby. “From h to p efficiently: implement- ing finite and spectral/hp element methods to achieve optimal performance for low- and high-order discretisations”. In: Journal of Computational Physics 229.13 (July 2010), pp. 5161–5181. doi: 10.1016/j.jcp.2010.03.031. [132] Z. Wang et al. “High-order CFD methods: current status and perspective”. In: International Journal for Numerical Methods in Fluids 72.8 (Jan. 2013), pp. 811–845. doi: 10.1002/fld.3767. [133] T. Warburton and T. Hagstrom. “Taming the CFL number for discontinuous Galerkin methods on structured meshes”. In: SIAM Journal on Numerical Analysis 46.6 (Jan. 2008), pp. 3151–3180. doi: 10.1137/060672601. [134] C. C. de Wiart et al. “Assessment of a discontinuous Galerkin method for the simulation of vortical flows at high Reynolds number”. In: International Journal for Numerical Methods in Fluids 74.7 (Nov. 2013), pp. 469–493. doi: 10.1002/fld.3859. [135] M. Yano and D. L. Darmofal. “An optimization-based framework for anisotropic simplex mesh adaptation”. In: Journal of Computational Physics 231.22 (Sept. 2012), pp. 7626–7649. doi: 10.1016/j.jcp.2012.06.040. [136] M. J. Zahr and P.-O. Persson. “An adjoint method for a high-order discretiza- tion of deforming domain conservation laws for optimization of flow prob- lems”. In: Journal of Computational Physics 326 (Dec. 2016), pp. 516–543. doi: 10.1016/j.jcp.2016.09.012. [137] M. J. Zahr and P.-O. Persson. “Performance tuning of Newton-GMRES meth- ods for discontinuous Galerkin discretizations of the Navier-Stokes equations”. In: 21st AIAA Computational Fluid Dynamics Conference. American Insti- tute of Aeronautics and Astronautics, June 2013. doi: 10.2514/6.2013- 2685.