Data-Driven Parallel Scientific Computing: Multi-Fidelity Information Fusion Algorithms and Applications to Physical and Biological Systems by Paris Perdikaris M.Sc., Brown University; Providence, R.I., U.S.A., 2010 M.Eng., National Technical University of Athens; Athens, Greece, 2009 A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in The Division of Applied Mathematics at Brown University PROVIDENCE, RHODE ISLAND May 2015 c Copyright 2015 by Paris Perdikaris This dissertation by Paris Perdikaris is accepted in its present form by The Division of Applied Mathematics as satisfying the dissertation requirement for the degree of Doctor of Philosophy. Date George Em. Karniadakis, Ph.D., Advisor Recommended to the Graduate Council Date Johannes O. Royset, Ph.D., Reader Date Daniele Venturi, Ph.D., Reader Approved by the Graduate Council Date Peter M. Weber, Dean of the Graduate School iii Vita Paris Perdikaris was born on March 27, 1984, in Athens, Greece, and grew up in the coastal town of Paleo Faliro. He completed studies leading to a Masters of Engineering in Naval Architecture and Marine Engineering at the National Technical University of Athens, receiving the Thomaidio award for scientific publications in 2009. As a PhD candidate in the Division of Applied Mathematics at Brown University, Paris has been advised and mentored by Prof. George Em. Karniadakis. In 2009 he received a University Fellowship for graduate studies, and completed a comprehensive two semester, 8-course training in Applied Mathematics. In addition to teaching assistantships, Paris has worked on several projects funded by major U.S agencies, all under the supervision of Prof. George Em. Karniadakis. His current work aims at setting the foundations of a new type of data-driven scientific computing, bridging traditional techniques, e.g., spectral element methods, with contemporary machine learning algorithms. iv Peer-reviewed publications 1. Perdikaris, P., D. Venturi, and G. E. Karniadakis. Multi-fidelity information fusion algorithms for high dimensional systems and massive data-sets. SIAM Journal of Scientific Computing, (to be submitted), 2015. 2. Perdikaris, P., D. Venturi, J. O. Royset, and G. E. Karniadakis. Multi-fidelity modeling via recursive co-kriging and Gaussian Markov random fields. Pro- ceedings of the Royal Society A, (under review), 2015. 3. Perdikaris, P., J. A. Insley, L. Grinberg, Y. Yu, M. E. Papka, and G. E. Kar- niadakis. Investigating flow-structure interactions in cerebral aneurysms. Su- percomputing, 2014. 4. Perdikaris, P., L. Grinberg, and G. E. Karniadakis. An effective fractal-tree closure model for simulating blood flow in large arterial networks. Annals of biomedical engineering, 1-11, 2014. 5. Perdikaris, P. and G. E. Karniadakis. Fractional-order viscoelasticity in one- dimensional blood flow models. Annals of biomedical engineering, 42:1012- 1023, 2014. 6. Perdikaris, P., L. Kaiktsis, and G. S. Triantafyllou. Chaos in a cylinder wake due to forcing at the Strouhal frequency. Physics of Fluids, 21:101705, 2009. v Preface and Acknowledgments This thesis has been compiled over a mosaic of research projects I have been involved in over the last 6 years at Brown. It all started in the summer of 2010 when I took a leap from my naval engineering background to study the viscoelastic behavior of soft tissue. This problem introduced me to the fascinating world of modeling mutli- physics phenomena in the human circulation – a biological system that is the envy of every engineer. This resulted to a series of findings that were centered around three open questions in computational hemodynamics, namely creating robust models that exhibit low sensitivity to parametrizations, addressing the closure problem for simu- lating blood flow in truncated arterial domains, and developing efficient techniques for model inversion and parameter calibration. In the summer of 2014 my research activities took a very interesting and exciting twist, as I was captured by the potential application of machine learning techniques in multi-fidelity modeling, design optimization under uncertainty, data assimilation, and model inversion. I soon realized that the practical applicability of such tech- niques is hindered by the prohibitive computational cost of training surrogate models in the presence of large data-sets, and scaling the predictive inference schemes to high dimensional problems. This led to a dedicated two-semester effort to address these challenges and design a tractable and scalable framework that opens the path to a new direction in data-driven scientific computing. There are many people, without whom, this work would have been infeasible. vi First, I would like to sincerely thank my advisor, Prof. George Em. Karniadakis. This thesis would be impossible to be conceived, implemented, and published without his strong support. George gave me the opportunity to join his vibrant research group 6 years ago, and has been a limitless resource of inspiration, dedication, and mentorship ever since. Throughout this time, he provided an open highway for me to express myself, feed my curiosity, and develop both as a person and as a scientist. His intuition, ideas, and constructive criticism have been invaluable to the development of this thesis. Next, I would like to genuinely thank Prof. Daniele Venturi. Daniele has been a good friend and an ideal colleague throughout my years at Brown. This thesis would have never reached its current form without his help and contributions. As we are taking different paths away from Brown, I take his work ethics and originality of thinking as examples of true scholarship, and I look forward to a productive collaboration in the years to come. I would like to thank Prof. Johannes O. Royset for our ongoing and productive collaboration, and for taking the time to participate in my thesis committee. I also thank Dr. Leopold Grinberg for our fruitful collaboration, and for sharing his extensive experience on high performance scientific computing. I am also very grateful to Prof. George Triantafyllou, and Prof. Lambros Kaiktsis at the National Technical University of Athens that fully supported and encouraged my endeavor for advanced studies. I owe a wholehearted thanks to all faculty members and staff at the Division of Applied Mathematics at Brown University. The achievements and deep scientific knowledge of the first have served as an ocean of inspiration, while the administrative support and efficiency of the later have simplified many every-day tasks. I would like vii to especially thank Prof. Basilis Gidas for being a good friend, mentor, and resource- ful explorer of Providence’s gastronomic hideouts. I also thank Prof. Constantine Dafermos for supporting my pursuit for graduate studies at Brown, and for being an example of scientific rigor and integrity. Many thanks also go to Madeline Brewster and Jean Radican for their administrative support. My gratitude also goes to the CCV, ORNL, and ALCF staff for their support and training on large-scale paral- lel computing, and especially to Dr. Joseph Isley for his artistic touch in scientific visualization. I thank all my classmates and lab-mates for providing a healthy and joyful work environment. Especially, I thank the “old guard” of the 37 Manning basement: Heyrim, Mingge, Summer, Changho, Joon, Mohsen, Handy, Xiu, and Yue, for filling the space with their light and making those long nights of work a bit less lonely. A very special thanks goes to her highness, Sophia Sa, for being the y´ang to my y´ın, constantly challenging her limits of patience, and bringing color to the tedious days of this dissertation. I feel very grateful that I got the chance to meet and interact with many wonderful people here at Brown. I’ll probably forget some, but among all, I thank Babis for his camaraderie and thirst for adventures, Socrates for his wisdom and voice of reason, Foteini for giving me my first tour of Thayer St., Ares for his genuine friendship and unique perspectives, Vaios for letting me win at pool, Jesus for his unbearable lightness of being, Maria for her limitless amount of positive energy, Kostas and Anastasia, Alexandra, Evgenios, Dimitra, the Sa family, Mary and Christopher, and last, but not least, Tasos for being a patient roommate for over than 3 years. A very well deserved expression of gratitude also goes to my good friends in Greece, Christos, Giannis, Mimis, and the Jinetera crew, for helping me recharge my viii batteries every summer, and for constantly radiating positive vibes from 4,767 miles away. Last but not least, I am beyond thankful to my family, Vaso, Myrto, and Angelos. If words were to exist for expressing my appreciation for their unconditional love and never-ending support, you would surely find them in this sentence. This work is dedicated to them with immense gratitude. ix Contents Vita iv Preface and Acknowledgments vi 1 Introduction 1 1.1 Data-driven Scientific Computing . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Part 1: Scalable multi-fidelity information fusion algorithms . 3 1.1.2 Part 2: Open questions in cardiovascular mathematics . . . . 4 2 A General Framework for Multi-fidelity Information Fusion 6 2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Multi-fidelity modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3.1 Kriging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3.2 Recursive co-kriging . . . . . . . . . . . . . . . . . . . . . . . 14 2.3.3 Maximum likelihood estimation . . . . . . . . . . . . . . . . . 16 2.4 Multi-fidelity in models and in probability space . . . . . . . . . . . . 17 2.5 Challenges to be addressed . . . . . . . . . . . . . . . . . . . . . . . . 20 2.5.1 Bottlenecks of learning from large data-sets . . . . . . . . . . 20 2.5.2 Bottlenecks of scaling to high dimensions . . . . . . . . . . . . 21 2.6 Demonstration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.6.1 Multi-fidelity modeling of ocean waves . . . . . . . . . . . . . 22 2.6.2 Data-assimilation of coastal bathymetry data . . . . . . . . . 24 2.6.3 Effect of low correlation . . . . . . . . . . . . . . . . . . . . . 28 3 Spectral/hp Element Methods for Inference with Gaussian Markov Random Fields 32 3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2 Gaussian Markov random fields and the SPDE approach . . . . . . . 33 3.2.1 The Mat´ern covariance model . . . . . . . . . . . . . . . . . . 34 3.2.2 Connection to stochastic partial differential equations (SPDE) 35 3.3 Hilbert space approximation . . . . . . . . . . . . . . . . . . . . . . . 37 3.3.1 Weak form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.3.2 Choosing an expansion basis . . . . . . . . . . . . . . . . . . . 39 x 3.4 The spectral/hp element method . . . . . . . . . . . . . . . . . . . . 41 3.4.1 Overview of the method . . . . . . . . . . . . . . . . . . . . . 41 3.4.2 Properties of the nodal basis . . . . . . . . . . . . . . . . . . . 43 3.4.3 Treatment of boundary conditions . . . . . . . . . . . . . . . . 44 3.4.4 Computation of the covariance and precision matrices . . . . . 47 3.5 Model fitting and the kriging predictor . . . . . . . . . . . . . . . . . 50 3.5.1 Maximum Likelihood Estimation with GMRFs . . . . . . . . . 54 3.6 Multi-fidelity modeling via recursive co-kriging and Gaussian Markov random fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.6.1 Ordinary kriging with GMRFs . . . . . . . . . . . . . . . . . . 55 3.6.2 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.7.1 Helmholtz equation in 2D . . . . . . . . . . . . . . . . . . . . 62 3.7.2 A prototype problem in risk-averse design . . . . . . . . . . . 67 3.7.3 Multi-fidelity regression of a random function . . . . . . . . . 73 3.7.4 Stochastic incompressible flow . . . . . . . . . . . . . . . . . . 74 3.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4 Multi-fidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data-sets 82 4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.2 Machine learning in the frequency domain . . . . . . . . . . . . . . . 87 4.2.1 Bottlenecks of Maximum Likelihood Estimation . . . . . . . . 87 4.2.2 Frequency-domain sample variogram fitting . . . . . . . . . . 88 4.3 Kernel design in high-dimensions . . . . . . . . . . . . . . . . . . . . 90 4.4 Implementation aspects . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.5.1 Borehole function . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.5.2 Trid function in 100 dimensions . . . . . . . . . . . . . . . . . 102 4.5.3 Sobol function in 1,000 dimensions . . . . . . . . . . . . . . . 105 4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5 Modeling Continuum Blood Flow and Biomechanics 112 5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.2 Nonlinear 1D blood flow models . . . . . . . . . . . . . . . . . . . . . 114 5.2.1 Accounting for arterial viscoelasticity . . . . . . . . . . . . . . 115 5.2.2 The Discontinuous Galerkin scheme . . . . . . . . . . . . . . . 120 5.2.3 Boundary Conditions . . . . . . . . . . . . . . . . . . . . . . . 124 5.3 Linear 1D blood flow models . . . . . . . . . . . . . . . . . . . . . . . 125 5.4 3D blood flow models . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.4.1 Flow in rigid vessels . . . . . . . . . . . . . . . . . . . . . . . 127 5.4.2 Flow-structure interactions . . . . . . . . . . . . . . . . . . . . 131 5.5 Coupling of the 3D and 1D solvers . . . . . . . . . . . . . . . . . . . 133 5.5.1 The coupling algorithm . . . . . . . . . . . . . . . . . . . . . . 136 5.5.2 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 xi 5.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 5.6.1 3D/1D comparison in a straight pipe . . . . . . . . . . . . . . 143 5.6.2 3D-1D coupling for a straight pipe . . . . . . . . . . . . . . . 146 5.6.3 3D-1D coupling for a Y-shaped bifurcation . . . . . . . . . . . 149 6 Fractional-order Viscoelasticity in Arterial Biomechanics 155 6.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 6.2 Viscoelastic Models of the Arterial Wall . . . . . . . . . . . . . . . . 159 6.2.1 The Fractional SLS model (Fractional Kelvin-Zener model) . . 160 6.2.2 Evaluation of Hereditary Integrals . . . . . . . . . . . . . . . . 164 6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 6.3.1 Simulation setup . . . . . . . . . . . . . . . . . . . . . . . . . 168 6.3.2 Comparison of integer- and fractional-order viscoelastic models 169 6.3.3 Global sensitivity analysis of the stochastic fractional SLS model174 6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 7 A Fractal-tree Closure Model for Simulating Blood Flow in Large Arterial Networks 180 7.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 7.2 Fractal-tree closure model . . . . . . . . . . . . . . . . . . . . . . . . 184 7.3 Implementation aspects . . . . . . . . . . . . . . . . . . . . . . . . . . 186 7.4 Case studies and simulation setup . . . . . . . . . . . . . . . . . . . . 187 7.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 7.5.1 Model sensitivity on the cut-off radius . . . . . . . . . . . . . 190 7.5.2 Overcoming the cut-off radius sensitivity . . . . . . . . . . . . 192 7.5.3 The Fahraeus-Lindqvist effect . . . . . . . . . . . . . . . . . . 195 7.5.4 Effect of distal outflow boundary conditions . . . . . . . . . . 196 7.5.5 3D-1D coupling in a cerebral network with 10 outlets . . . . . 198 7.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 8 Calibration of Blood Flow Simulations via Multi-fidelity Surrogates205 8.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 8.2 Efficient global optimization . . . . . . . . . . . . . . . . . . . . . . . 209 8.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 8.3.1 Y-shaped bifurcation . . . . . . . . . . . . . . . . . . . . . . . 212 8.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 9 Conclusions and Future Work 222 9.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 9.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 xii List of Tables 2.1 Design optimization: Examples of possible design criteria. . . . . . . . 11 3.1 Helmholtz equation in 2D: Leading size dimension of the precision Qu and homogeneous-homogeneous Schur complement SHH matrices for different number of expansion modes n. . . . . . . . . . . . . . . . . . 65 3.2 Stochastic incompressible flow: Size of the uniform discretization grid in the (σ1 , σ2 )-plane, and number of samples per grid-point for each of the probabilistic methods employed. . . . . . . . . . . . . . . . . . 78 4.1 Sobol function: Breakdown of the total computational cost with re- spect to the implementation steps outlined in Sec. 4.4. . . . . . . . . 108 5.1 3D/1D comparison in a straight pipe: Non-dimensional time-step ∆t and computation time per cycle for each simulation considered. . . . 143 5.2 3D/1D comparison in a straight pipe: Relative L2 error at the outlet. 144 5.3 3D-1D coupling: 3D-1D coupling for a straight pipe: Relative L2 error at the outlet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 5.4 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Relative L2 error at outlet #1. . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.5 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Relative L2 error at outlet #2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.1 Comparison of integer- and fractional-order viscoelastic models: Vis- coelastic model parameters. . . . . . . . . . . . . . . . . . . . . . . . 171 7.1 Fractal tree closure models: Total number of vessels and average num- ber of generations attached to each outlet of the parent CoW and arm networks as a function of the fractal-tree cut-off radius rcut . . . . . . . 190 7.2 3D-1D coupling in a cerebral network with 10 outlets: Simulation prop- erties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 7.3 3D-1D coupling in a cerebral network with 10 outlets: Relative errors of the pressure gradient ∆p between the inlet and each of the 10 outlets of the 3D domain. We consider rcut = 100µm to be our reference case. 200 7.4 3D-1D coupling in a cerebral network with 10 outlets: Relative errors of the flow-rate measured at each of the 10 outlets of the 3D domain. We consider rcut = 100µm to be our reference case. . . . . . . . . . . 200 xiii 8.1 Blood flow in a symmetric Y-shaped bifurcation: Optimal terminal resistances and corresponding relative L2 -error in the inlet pressure wave over one cardiac cycle, for each iteration of the EGO algorithm. The comparison is done with respect to the inlet pressure wave popt corresponding to the optimal configuration of resistances predicted by the reference solution obtained using 10,000 high-fidelity samples. . . 219 xiv List of Figures 2.1 Schematic representation of the multi-fidelity design under uncertainty framework: the quantity of interest Y(x; ξ) is a random field in the design space, realizations of which could originate from information sources of variable-fidelity (such as computer codes, experiments, ex- pert opinion, etc.). A response surface S(x) encodes the dependence of a derived quantity of interest f (Y(x; ξ)) on the input design variables x and uncertain parameters ξ. The ability to efficiently construct re- sponse surfaces (e.g. S(x) = E[f (Y(x; ξ))]) allows one to asses the system’s performance as a function of the random inputs and iden- tify optimal configurations based on the desired design criteria (see Table 2.1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Gaussian process regression: Left: Samples of a zero-mean Gaussian prior over functions. Middle: Samples of the calibrated Gaussian pos- terior in view of noise-less data (dashed lines), and the conditional mean of Z|Y (solid line).Right: Samples of the calibrated Gaussian posterior in view of noisy data (dashed lines), and the conditional mean of Z|Y (solid line). The grey shadowed regions depict the enve- lope defined by two standard deviations (figure taken from Rasmussen (2006)). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Multi-fidelity in models and in probability space: m models of vari- able fidelity in physical space are driven by random input, producing a random response surface Ym (x; ξ). For example, the expectation of a derived quantify of interest E[f (Ym (x; ξ))] can be estimated by employing p methods of variable fidelity in probability space. . . . . . 18 2.4 Multi-fidelity in models and in probability space: Family of response surfaces resulting from simultaneously addressing multi-fidelity in mod- els and in probability space. Physical model fidelity is increased along the columns (red arrow) and probability space model fidelity increases along the rows (blue arrow). The yellow arrow represents a possible optimal information fusion path in the combined fidelity space. . . . . 20 2.5 Multi-fidelity modeling of ocean waves: Exact response surface and co-kriging predictor constructed using 310 low-fidelity and 80 high- fidelity observations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.6 Multi-fidelity modeling of ocean waves: Left: Poin-twise L∞ error of the co-kriging predictor (310 low-fidelity and 80 high-fidelity obser- vations). Right: Point-wise variance of the co-kriging predictor. The reference solution was obtained using 2,500 high-fidelity samples. . . . 25 xv 2.7 Multi-fidelity modeling of ocean waves: Visual comparison of the ex- act solution (2,500 high-fidelity samples) (right), and the co-kriging predictor (310 low-fidelity and 80 high-fidelity observations) (left). . . 25 2.8 Data-assimilation of coastal data: Left: Kriging predictor (100 noisy observations). Right: Kriging variance. . . . . . . . . . . . . . . . . . 26 2.9 Data-assimilation of coastal data: Left: Kriging predictor (100 noisy observations). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.10 Data-assimilation of coastal data: Left: Robustness of the riging pre- dictor with respect to corrupted measurements. (Left to right) Pre- dictions yˆ versus corrupted data y for increasing noise amplitudes. . . 27 2.11 Effect of low correlation: Ordinary kriging predictor (red dashed line) through 6 equispaced observations (red circles) of fh (x) (black solid line). The gray shaded area corresponds to two standard deviations of the Gaussian predictor. . . . . . . . . . . . . . . . . . . . . . . . . 30 2.12 Effect of low correlation: Left: Two-level recursive co-kriging predictor for a low-fidelity model fl (x) with c = 0.5.Right: Two-level recursive co-kriging predictor for a low-fidelity model fl (x) with c = 0.4. In both cases, the training data-set consists of 11 equispaced observations of fl (x), and 6 equispaced observations of fh (x), while the gray shaded area depicts two standard deviations of the Gaussian predictor. . . . 31 3.1 The spectral/ hp element method: Sketch of the unstructured tessel- lation of the SPDE domain Ω the polynomial basis employed. The solution domain is decomposed into non-overlaping elements; within each element the solution is approximated by vertex, edge, face and (in 3D) interior modes. The shape functions associated with the vertex, edge and face modes for a fourth-order polynomial expansion (P=4) defined on triangular and quadrilateral faces are shown in color. . . . 42 3.2 Helmholtz equation in 2D: Exponential convergence rate of the nodal spectral element discretization for a Helmholtz equation in 2D. The error between the numerical and the exact solution has been computed for different numbers of expansion modes n, using the L∞ , L2 , and H 1 norms, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.3 Helmholtz equation in 2D: Condition number of the precision Qu and homogeneous-homogeneous Schur complement SHH for different num- ber of expansion modes n. . . . . . . . . . . . . . . . . . . . . . . . . 65 3.4 Helmholtz equation in 2D: Sparsity patterns of the Helmholtz, preci- sion and homogeneous-homogeneous Schur complement matrices for a second-order polynomial expansion basis (n = 3). . . . . . . . . . . 65 3.5 Helmholtz equation in 2D: Sparsity patterns of the Helmholtz, preci- sion and homogeneous-homogeneous Schur complement matrices for a third-order polynomial expansion basis (n = 4). . . . . . . . . . . . 66 3.6 Helmholtz equation in 2D: Sparsity patterns of the Helmholtz, preci- sion and homogeneous-homogeneous Schur complement matrices for a sixth-order polynomial expansion basis (n = 7). . . . . . . . . . . . 66 3.7 Helmholtz equation in 2D: Observation points, kriging prediction and pointwise L∞ error for a single observation point located at (1,0). Top to bottom: Nodal spectral element discretization with 4 quadrilateral elements and 3,5,7, and 9 expansion modes, respectively. . . . . . . . 68 xvi 3.8 Helmholtz equation in 2D: Observation points, kriging prediction and kriging variance for a nodal spectral element discretization with 4 quadrilateral elements and n = 7 expansion modes. Top to bottom: Increasing the number of randomly selected observations from 1,5,10, to 15, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.9 A prototype problem in risk-averse design: Exact response surface R0.8 (Y ) and co-kriging predictor constructed using 120 low-fidelity (SG-L2) and 10 high-fidelity (PCM) observations. (x?1 , x?2 ) denotes the optimal solution to the optimization problem of Eq. 3.63, while the inset plot shows the point-wise absolute error of the co-kriging predictor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.10 Multi-fidelity regression of a random function: Exact response surface S = E[Y (x; ξ)] and co-kriging predictor constructed using four levels of fidelity: 80 S11 points (fc (x; ξ), SG-L2), 40 S12 points (fc (x; ξ), SG-L5), 10 S21 points (fe (x; ξ), SG-L2), and 5 S22 points (fe (x; ξ), SG-L5). The inset plot shows the point-wise variance of the co-kriging predictor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.11 Incompressible stochastic flow past a cylinder: (Left) The computa- tional mesh consists of 412 triangular spectral elements of polynomial order 11. (Right) Standard deviation of the stream-wise velocity com- ponent at time t = 4. The quantity of interest here is the base pressure coefficient CBP , where pB denotes the fluid pressure on the cylinder surface at 180o from the stagnation point (base pressure), while U∞ and p∞ are the free-stream velocity and pressure, respectively. . . . . 77 3.12 Incompressible stochastic flow past a cylinder: Probabilistic colloca- tion (PCM) samples of the velocity inflow profile on a uniform 6 × 5 (σ1 , σ2 )-grid. At each grid point, a total of 64 samples is used to in- tegrate out the randomness introduced by (ξ1 , ξ2 ) in Eq. 3.68. Note that σ1 controls the skewness of the inflow profile, while σ2 acts as a noise amplifier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 3.13 Incompressible stochastic flow past a cylinder: Response surface of the 0.6-superquantile risk of the base pressure coefficient R0.6 (CBP ), computed using a 3-level recursive co-kriging scheme, on 100 low- fidelity (MC), 15 intermediate-fidelity (SG-L2), and 5 high-fidelity (PCM) observation points in the (σ1 , σ2 )-plane. The exact surface corresponds to 357 uniformly distributed observations of the highest fidelity model (PCM). The inset plot shows the point-wise variance of the co-kriging predictor. . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.1 Kernel design in high-dimensions: Sketch of the undirected graph de- fined by the Sobol sensitivity indices of 5-dimensional function y(x1 , x2 , x3 , x4 , x5 ). Here we have one third-order interaction, four second-order interac- tions and five first-order terms. The size of the lines and the circles indicate the magnitude of the sensitivities. The corresponding maxi- mal cliques are C1 = {x1 , x2 , x3 }, C2 = {x1 , x4 }, C3 = {x5 }. . . . . . . 93 4.2 Borehole function: Sketch of the undirected graph of the RS-HDMR sensitivity indices generated using 1,024 observations of S11 (x). The first-order sensitivity indices are Si = [0.837, 0.0, 0.0, 0.042, 0.0, 0.041, 0.041, 0.0], while the non-zero pair-wise sensitivities are S14 = 0.011, S16 = 0.011, and S14 = 0.010, capturing 99.7% of the total variance in the obser- vations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 xvii 4.3 Borehole function: Left: Density plot of the of the frequency distribu- tion of the exact solution E[y] (blue solid line), versus the estimated E[ˆy ] (dashed lines) resulting from co-kriging and the HDMR represen- tation. The red dashed line corresponds to the final co-kriging pre- dictor accounting for information fusion along the path S11 → S12 → S21 → S22 . Right: Scatter plot of the exact solution E[y] (black dashed line), versus the co-kriging predictor E[ˆy ] at each level (circles) at 2,000 randomly sampled test locations. The black circles correspond to the final co-kriging predictor accounting for information fusion along the path S11 → S12 → S21 → S22 . . . . . . . . . . . . . . . . . . . . . . . 102 4.4 Trid function: Sketch of the undirected graph of the RS-HDMR sensi- tivity indices generated using 8,196 observations of y1 (x). The struc- ture is in agreement with the definition of the Trid function in Eq. 4.28, suggesting the presence of only pairwise interactions that lead to 99 maximal cliques, each containing two active dimensions. . . . . . . . . 103 4.5 Trid function: Left: Density plot of the of the frequency distribution of the exact solution y = y3 (x? ) (blue solid line), versus the estimated yˆ(x? ) (dashed lines) resulting from co-kriging and the HDMR repre- sentation. The red dashed line corresponds to the final co-kriging pre- dictor accounting for information fusion along the path y1 → y2 → y3 . Right: Scatter plot of the of the of the exact solution y = y3 (x? ) (black dashed line), versus the co-kriging predictor yˆ(x? ) at each level (circles) at 2,000 test locations, randomly sampled within unit hyper- spheres centered at observations. The black circles correspond to the final co-kriging predictor accounting for information fusion along the path y1 → y2 → y3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.6 Sobol function: Left: Density plot of the of the frequency distribution of the exact solution y(x? ) (blue solid line), versus the estimated yˆ(x? ) (dashed lines) resulting from kriging and the HDMR representation. Right: Scatter plot of the of the of the exact solution y(x? ) (black dashed line), versus the kriging predictor yˆ(x? ) (black circles) at 2,000 test locations, randomly sampled within unit hyper-spheres centered at observations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.1 Nonlinear 1D blood flow: Flow in a 1D compliant artery (from Sherwin et al. (2003)). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.2 Nonlinear 1D blood flow: viscoelastic models considered. Left: Voigt. Right: Kelvin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.3 Nonlinear 1D blood flow: Left: Sketch of the RCR outflow model. Right: Sketch of the forward and backward characteristics at the 1D- 0D coupling interface (figure taken from Alastruey et al. (2008)). . . . 125 5.4 3D-1D coupling: Computational domain for 3D-1D overlapping cou- pling for a straight pipe. . . . . . . . . . . . . . . . . . . . . . . . . . 133 5.5 3D-1D coupling: Sketch of the 3D-1D weak coupling algorithm . . . . 136 5.6 3D-1D coupling: Flow-rate interpolation for intermediate asynchronous 1D solver steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 5.7 3D-1D coupling: Normal flux close to the outflow boundary of a straight pipe (left) and a converging pipe with 5 degrees tapering (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.8 3D/1D comparison in a straight pipe: Comparison of computed flow- rates at the inlet and outlet. . . . . . . . . . . . . . . . . . . . . . . . 145 xviii 5.9 3D/1D comparison in a straight pipe: Comparison of computed pres- sures at the inlet and outlet. . . . . . . . . . . . . . . . . . . . . . . . 145 5.10 3D-1D coupling: 3D-1D coupling for a straight pipe: Comparison of computed flow-rates at the inlet and outlet (RC-coupling, no overlap). 148 5.11 3D-1D coupling: 3D-1D coupling for a straight pipe: Comparison of computed pressures at the inlet and outlet (RC-coupling, no overlap). 148 5.12 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Consid- ered model for the Y-shaped bifurcation: full 3D domain (left), 3D-1D non-overlapping domain (center), 3D-1D overlapping domain (right). The orange region depicts the overlap of the 3D and 1D domains. . . 150 5.13 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Flow- rate oscillations at the 3D-1D non-overlapping interface #1 (RC- coupling). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 5.14 3D-1D coupling: Comparison of computed flow-rates at the inlet and outlets (RC-coupling, 3D-1D overlap). . . . . . . . . . . . . . . . . . 153 5.15 3D-1D coupling: Comparison of computed pressures at the inlet and outlets (RC-coupling, 3D-1D overlap). . . . . . . . . . . . . . . . . . 154 6.1 Simulation setup: Patient-specific cranial network from Grinberg et al. (2011a) and PC-MRI flowrate measurements at the four inlets (L. ICA, R. ICA, L. VA, R. VA) . . . . . . . . . . . . . . . . . . . . . . . 169 6.2 Comparison of integer- and fractional-order viscoelastic models: Com- parison of four viscoelastic model parameter sets: Variability in the computed flowrate at outlets # 23, 30, 44 and 50. . . . . . . . . . . . 172 6.3 Comparison of integer- and fractional-order viscoelastic models: Com- parison of four viscoelastic model parameter sets: Variabilty in the computed pressure wave and corresponding pressure-area hysteresis loop at the inlet (L. ICA) and distal outlet # 29. . . . . . . . . . . . 173 6.4 Comparison of integer- and fractional-order viscoelastic models: Com- puted pressure and pressure-area hysteresis loop at the inlet (L. ICA) for different values of the fractional order 0 ≤ α ≤ 1. The dashed line corresponds to the integer-order model SLS1 (recovered for α = 1.0) with a wider relaxation spectrum: τσ = 0.025s, ττσ = 4. . . . . . . . . 174 6.5 Global sensitivity analysis of the stochastic fractional SLS model: Com- puted mean flowrate (in black) and a variation of ±1 standard devia- tion (in red) at distal outlets # 44 and 50. . . . . . . . . . . . . . . . 175 6.6 Global sensitivity analysis of the stochastic fractional SLS model: Com- puted mean pressure (in black) and a variation of ±1 standard devia- tion (in red) at the inlet (L. ICA) and distal outlet # 50. . . . . . . . 176 6.7 Global sensitivity analysis of the stochastic fractional SLS model: Vari- ance decomposition of the computed pressure waveform at the inlet (L. ICA). Si and Sij are the global sensitivity indices of each random parameter and their pairwise interactions, respectively. . . . . . . . . 177 7.1 Fractal-tree closure model: Left: Fractal-trees are attached at each outlet of a patient-specific arterial network, resulting to an arterial domain with millions of compliant vessels in which blood flow is mod- eled using Eq. 5.1. Right: Flow is driven by high resolution, patient- specific PC-MRI measurements at the inlets (right vertebral artery R. VA, left vertebral artery L. VA, right internal carotid artery R. ICA and left internal carotid artery L. ICA). . . . . . . . . . . . . . . . . . 186 xix 7.2 Case studies and simulation setup: A detailed arterial network of the arm, courtesy of Watanabe et al. (2013) (276 arteries with 105 terminal outlets, 124 bifurcation points, 18 merging junctions and 85 perforator vessels). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 7.3 Model sensitivity on the cut-off radius: Sensitivity on the cut-off radius rcut : Computed pressure (blue) and flowrate (red) waves or different values of the cut-off radius rcut . Right: Basilar artery (CoW case). Left: Radial artery (arm case). . . . . . . . . . . . . . . . . . . . . . . 191 7.4 Overcoming the cut-off radius sensitivity: Left: Decreasing L/r ansatz for different thresholds r? . Right: Volume of the generated arterial tree as a function of the cut-off radius for different L/r relations (CoW case). The dashed line corresponds to the estimated target volume of 35mL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 7.5 Overcoming the cut-off radius sensitivity: Waveform convergence with decreasing cut-off radius using the L/r ansatz : Computed pressure (blue) and flowrate (red) waves for different values of the cut-off radius rcut . Left: Basilar artery (CoW case). Right: Radial artery (arm case). Inscribed tables contain the relative error (L2 norm) of the pressure wave as we decrease the cut-off radius (reference cases: rcut = 20µm for CoW and rcut = 10µm for the arm). . . . . . . . . . . . . . . . . . 195 7.6 The Fahraeus-Lindqvist effect: Computed pressure waves for different blood rheology models: radius-depended viscosity suggested by Pries et al. (1992) (non-Newtonian, solid line) versus constant blood vis- cosity (Newotnian, dashed line) Left: Left internal carotid artery (L. ICA, CoW case). Right: Axillary artery (arm case). . . . . . . . . . . 196 7.7 Effect of distal outflow boundary conditions: Effect of distal outflow boundary conditions: Computed pressure (solid lines) and flowrate (dashed lines) waves for different types of distal outlfow boundary conditions: Absorbing (red), constant pressure (green), and 3-element RCR windksessels (blue). Left: Anterior communicating artery (AcoA, CoW case). Right: Brachial artery (arm case). . . . . . . . . . . . . . 198 7.8 3D-1D coupling in a cerebral network with 10 outlets: Network of 20 cranial arteries with 1 inlet (red) and 10 outlets. . . . . . . . . . . . . 198 7.9 3D-1D coupling in a cerebral network with 10 outlets: Comparison of computed pressure at the outlets (left: 3D with 0D-RC outflow, right: 3D with 1D elastic fractal networks). . . . . . . . . . . . . . . . . . . 201 7.10 3D-1D coupling in a cerebral network with 10 outlets: Comparison of computed flow-rates at the outlets (left: 3D with 0D-RC outflow, right: 3D with 1D fractal networks). . . . . . . . . . . . . . . . . . . 201 8.1 Blood flow in a symmetric Y-shaped bifurcation: Flow is driven by a physiological flow-rate wave at the inlet, while outflow boundary conditions are imposed through 3-element windkessel models. . . . . . 212 8.2 Blood flow in a symmetric Y-shaped bifurcation: Detailed response surface and identification of the minimum error in the inlet systolic pressure as a function of the total resistance parameters imposed at the outlets. The surface is constructed by probing a nonlinear 1D-FSI model on 10,000 uniformly spaced samples in the space of inputs X . . 215 xx 8.3 Blood flow in a symmetric Y-shaped bifurcation: Zeroth iteration of the EGO algorithm. Left: Exact solution versus the co-kriging pre- dictor for the inlet systolic pressure error, trained on 100 low-fidelity and 25 high-fidelity observations. Right: Map of the corresponding expected improvement. . . . . . . . . . . . . . . . . . . . . . . . . . . 216 8.4 Blood flow in a symmetric Y-shaped bifurcation: Fourth iteration of the EGO algorithm. Left: Exact solution versus the co-kriging pre- dictor for the inlet systolic pressure error, trained on 100 low-fidelity and 25 high-fidelity observations. Right: Map of the corresponding expected improvement. . . . . . . . . . . . . . . . . . . . . . . . . . . 218 8.5 Blood flow in a symmetric Y-shaped bifurcation: Convergence of the inlet pressure waveform for the first 3 EGO iterations. The exact so- lution corresponds to the reference results obtained from 10,000 high- fidelity samples, defining the target inlet systolic pressure p?s . . . . . . 219 9.1 Deep networks: (a) Sketch of a deep network of vector-valued random fields f (L) ∈ Rm . In this particular example we have an observation model y (L) = f (L) for each layer, but we can also have situations in which we have hidden (unobserved) layers. The state space variables (e.g., the design variables) are denoted by x ∈ Rn . The multi-objective response - random field approximating the QoI by Y (x). The deep network allows us to perform information fusion of different infor- mation sources by using scalable machine learning algorithms. (b) Recursive co-kriging method as a particular instance of a deep linear Gaussian network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 xxi Chapter One Introduction 2 1.1 Data-driven Scientific Computing This thesis takes aim at two directions. The first direction involves setting the foundations for a “new” type of data-driven scientific computing, essentially creat- ing a platform for blending multiple information sources of variable fidelity, e.g., experimental data, high-fidelity numerical simulations, expert opinion, etc., towards creating tractable paths to analyzing the response of complex physical and biological systems. Standing on the verge of traditional scientific computing and contempo- rary statistical learning techniques, we elaborate on a novel multi-fidelity information fusion framework that allows for the seamless integration of surrogate-based opti- mization and uncertainty quantification, and enables the development of efficient algorithms for data assimilation, design optimization, model inversion, and beyond. The second direction is focused on addressing open questions in the modeling of the human circulatory system, especially the interplay between blood flow and biomechanics in the brain. Despite great growth in computing power and algorith- mic sophistication, in-silico modeling of blood flow and arterial mechanics is still limited to truncated arterial domains, a fact that introduces the need for reduced- order models and parametric representations to account for the neglected mesoscale dynamics. The implicit need of such simplified representations gives rise to a series of open questions in cardiovascular mathematics, three of which will be at the focal point of our attention. In what follows, we provide a summary of how the content of this thesis has been organized. 3 1.1.1 Part 1: Scalable multi-fidelity information fusion al- gorithms In Chapter 2 we put forth a general framework for multi-fidelity information fusion that simultaneously takes into account multi-fidelity in physical models (e.g., direct numerical simulations versus empirical formulas), as well as multi-fidelity approx- imations in probability space (e.g., sparse grids vs. tensor product multi-element probabilistic collocation). We will describe the building blocks of our construction, namely Gaussian process regression and recursive inference schemes based on auto- regressive stochastic modeling. This exposition aims to lay the methodological foun- dations for the first part of this thesis, provide motivation, and state the challenges that will be addressed in the subsequent chapters. These challenges are primarily centered around two long-standing pathologies in statistical learning, the “big-N ” problem for training surrogates on large data-sets, and the curse of dimensionality. In Chapter 3 we equip the proposed multi-fidelity information fusion framework with efficient predictive inference algorithms based on multi-level recursive co-kriging and Gaussian Markov random fields (GMRFs). We consider families of GMRFs that arise as solutions to certain stochastic partial differential equations (SPDEs), and we introduce a nodal spectral/hp element discretization that leads to high- order accurate approximations with sparse inverse covariance matrices. Leveraging this sparsity, we develop efficient parallel solvers for constructing response surfaces of complex stochastic dynamical systems that significantly outperform the O(N 3 ) barrier of maximum likelihood estimation techniques. The effectiveness of the new algorithms is demonstrated in numerical examples involving the reconstruction of gappy fields, a prototype problem in risk-averse design, the regression of random functions, as well as uncertainty quantification in fluid mechanics involving random 4 laminar wakes behind circular cylinders. In Chapter 4 we introduce a methodology for multi-fidelity information fusion and predictive inference in high dimensional input spaces and in the presence of mas- sive data-sets. The proposed methodology further extends the paradigm introduced in Chapter 2, in the context of seamlessly integrating surrogate-based modeling and uncertainty quantification. Scaling to high dimensions is achieved by data-driven dimensionality reduction techniques based on hierarchical functional decompositions and a graph-theoretic approach for constructing custom auto-correlation models for Gaussian process regression. Multi-fidelity information fusion is facilitated through stochastic auto-regressive schemes and frequency-domain machine learning algo- rithms that scale linearly with the data. The effectiveness of the new algorithms is demonstrated in several benchmark problems involving the regression of multi- fidelity deterministic and stochastic functions in up to 1,000 input dimensions and 105 data-points on a standard desktop computer. 1.1.2 Part 2: Open questions in cardiovascular mathematics In Chapter 5 we present outline the basic tools employed in this this thesis for study- ing blood flow and arterial biomechanics in-silico. In particular, we provide a detailed presentation of one-dimensional (1D) models of blood flow in compliant arteries that will be extensively used in Chapters 6-8. Combining accuracy in predicting key flow features and low computational cost, such models enable a broad range of system- atic studies that would otherwise be intractable if one solely relied on expensive three-dimensional formulations. Moreover, we consider possible interface conditions for the coupling of the geometrically heterogeneous 3D and 1D solvers, and test a novel approach based on overlapping domains. These methods set the ground for the 5 subsequent findings presented Chapters 6-8. There, we provide systematic studies that aim to elucidate three open questions in cardiovascular mathematics. The first question involves devising reduced-order models that are robust with respect to model misspecification and sensitivity in the input parameters. This is a key question, as model reduction typically introduces a number of parameters are very hard to estimate in the clinical setting, and increased sensitivity to them jeopardizes the applicability of such simplified models. In Chapter 6 we address this issue in the context of designing accurate and robust constitutive models for the viscoelastic response of soft tissue under stress. The second open question is related to outflow boundary conditions imposed at the outlets of truncated arterial domains. Typically, this problem is treated using lumped parameter models, although the flow solution is known to exhibit high sensitivity to such parametrizations. In Chapter 7 we present an effective solution to this closure problem by modeling the neglected downstream vasculature as self- similar fractal trees, in which blood flow dynamics are governed by nonlinear 1D conservation laws. The third question is related to the inevitable step of model parameter cali- bration that often results to a prohibitive computational cost. Our treatment here leverages directly the developments presented in the first part of this thesis, as we construct a multi-fidelity, surrogate-based framework for model inversion. The pro- posed methodology targets the accurate construction of high-dimensional response surfaces, and the efficient pursuit to identify global optima while keeping the number of expensive model evaluations at a minimum. To this end, we utilize the machine learning techniques developed throughout Chapters 2- 5, towards exploiting the re- sulting predictive inference schemes within a Bayesian optimization setting. Chapter Two A General Framework for Multi-fidelity Information Fusion 7 2.1 Motivation Progress in perceptibly diverse areas of science such as numerical analysis and sci- entific computing, design optimization, uncertainty quantification, and statistical learning, have started to carve an emerging trend in engineering design, in which decision making becomes increasingly more data-driven rather than merely relying on empirical formulae and expert opinion. A set of versatile tools, ranging from ex- periments to stochastic simulations can offer invaluable input towards not only per- formance optimization, but also risk assessment, cost effectiveness and operational aspects of a design. A smart management of these powerful resources is expected to play a vital role, especially in the case of novel, high-performance unconventional designs where simple empirical extrapolation can provide no guarantee of optimality. However, employing ensembles of high-fidelity experiments or computer simulations for performing data-driven design optimization for realistic cases may easily become a task of prohibitive cost. A viable solution to this problem is offered by considering surrogate modeling. A surrogate model can be simply thought of as an intermediate agent that ab- sorbs information coming from a realization of what we consider to be as the true response of a system (up to measurement error), and is able to perform inexpensive predictions as to what the true response might look like in regions where no a-priori information is available. Modern statistical learning techniques have opened the path for constructing sophisticated surrogates that are able to accurately capture intrinsic features of the response of complex engineering systems (see Forrester et al. (2008), Rasmussen (2006), Hastie et al. (2009)). At the heart of surrogate-based modeling we find classical statistical methods, such as Gaussian process regression, members of which are the so-called kriging and co-kriging predictors (see Cressie and Cassie 8 (1993), Rasmussen (2006)). Initially introduced in geostatistics by the pioneering work of Matheron (1963) and Krige (1951), kriging has been a flexible and reliable tool in spatial statistics and has extensively served a widespread spectrum of applications, ranging from classical data assimilation in environmental sciences to modern machine learning techniques in robotics. However, it was the seminal work of Sacks et al. (1989) that introduced kriging in the context of design and analysis of computer experiments, and consequently led to a rapidly growing field of application in which kriging has been used as a predictive tool for constructing response surfaces of engineering systems by exploring the spatial correlation between the output of deterministic computer codes. Simpson et al. (2008) provides a comprehensive summary of the evolution and recent advancements of this metamodeling approach in the context of multidisciplinary design optimization. Of particular importance to our study is the work of Kennedy and O’Hagan (2000) that introduced the use of first-order auto-regressive stochastic models for predicting the output of a complex computer code when fast approximations are available. We recognize that this approach establishes a coherent mathematical framework for blending heterogeneous variable-fidelity information sources, creating a natural setting for multi-fidelity modeling. Forrester et al. (2007) adopted this methodology to build a two-level co-kriging scheme and successfully applied it for optimizing the drag/dynamic pressure ratio of a transonic aircraft wing. Although the auto-regressive scheme of Kennedy and O’Hagan (2000) has served well many disciplines and applications, it suffers from the same limitations as any kriging/co-kriging method. The biggest practical limitation stems from the need to repeatedly invert large, dense, ill-conditioned covariance matrices at the machine 9 learning stage, where the metamodel is trained on a set of known observations. To address this issue, we have aligned our work with some new interesting findings that can significantly enhance the computational efficiency of kriging/co-kriging. First, the treatise of Le Gratiet and Garnier (2014) suggests that co-kriging schemes aris- ing from the model of Kennedy and O’Hagan (2000) with s-levels of variable-fidelity information sources can be effectively decoupled, and equivalently formulated in a recursive fashion as s independent kriging problems. This allows for the construction of predictive co-kriging schemes by solving a sequence of simpler kriging problems that involve covariance matrices of smaller dimensions (and potentially lower condi- tion numbers), which can be learned from the data by performing optimization in lower dimensional spaces (compared to the coupled setting of Kennedy and O’Hagan (2000)). The aim of this chapter is to establish a new mathematical framework for surrogate- based building of response surfaces that simultaneously takes into account multi- fidelity in models (e.g., high-fidelity direct numerical simulations vs. low-fidelity empirical formulas) as well as multi-fidelity in probability space (e.g., high-fidelity tensor product multi-element probabilistic collocation vs low-fidelity sparse grid quadratures). We incorporate elements of statistical learning in an auto-regressive co-kriging methodology to cross-correlate ensembles of multi-fidelity surrogates from which we can accurately and efficiently extract the response of complex non-linear stochastic dynamical systems. This framework targets a seamless integration of surrogate-based optimization and uncertainty quantification, providing a launchpad for contemporary engineering design under uncertainty, inverse problems, and be- yond. This chapter is structured as follows. In Sec. 2.3 we outline the basic mathe- matical concepts upon which the proposed framework is built. In particular, Sec. 10 2.2, 2.3 describe the setting of multi-fidelity modeling and introduces the core con- cept of Gaussian process regression in the form of kriging, co-kriging and multi-level recursive co-kriging. We then define a general auto-regressive setting under which multi-fidelity in models and multi-fidelity in probability space can be addressed in unison. These methods are the cornerstone of this thesis and provide the basis for the algorithms and results presented in Chapters 3, 4. 2.2 Multi-fidelity modeling Consider a general system whose response can be evaluated using different models. The response typically depends on a set of design parameters x ∈ Rn and is subject to uncertainties encoded by a set of random parameters, ξ ∈ Rm . Given a set of design criteria, our goal is to construct the corresponding response surface that quantifies the dependence of a derived quantity of interest, f (Y(x; ξ)), on the input design variables x and uncertainties ξ (see Fig. 2.1). Design criteria come in different flavors and generally reflect our priorities and objectives in identifying configurations of the design variables x that qualify as optimal. For example, one may pursue to maximize the performance of a system for a wide range of operating conditions (see Phadke (1995)), identify a design with a probability density function (PDF) that matches a target performance (see Seshadri et al. (2014)), or minimize the risk associated with undesirable realizations (see Rockafellar and Royset (2014), and Table 2.1). Although many realistic cases seek designs that satisfy multiple objectives, in what follows we will focus on single-objective optimization problems just to enhance the clarity of our presentation. The process usually referred to as multi-fidelity modeling, elaborates on efficiently 11 Figure 2.1: Schematic representation of the multi-fidelity design under uncertainty framework: the quantity of interest Y(x; ξ) is a random field in the design space, re- alizations of which could originate from information sources of variable-fidelity (such as computer codes, experiments, expert opinion, etc.). A response surface S(x) en- codes the dependence of a derived quantity of interest f (Y(x; ξ)) on the input design variables x and uncertain parameters ξ. The ability to efficiently construct response surfaces (e.g. S(x) = E[f (Y(x; ξ))]) allows one to asses the system’s performance as a function of the random inputs and identify optimal configurations based on the desired design criteria (see Table 2.1). Design criteria max E[f (Y(x; ξ))] and min Var[f (Y(x; ξ))] (Phadke (1995)) x x min ||p(Y; x) − q(Y)||, where q(Y) is the PDF of a target performance (Seshadri et al. (2014)) x min R[f (Y(x; ξ))], where R is a risk measure (Rockafellar and Royset (2014)) x Table 2.1: Design optimization: Examples of possible design criteria. constructing response surfaces by correlating surrogate models of different fidelity. On one side of the fidelity spectrum one may have cheaper surrogate models, which are fast to compute but less trustworthy (e.g. potential flow solvers, empirical formu- las, etc), while on the other side we have high-fidelity models that enjoy our outmost trust but can be very expensive to compute (e.g. direct numerical simulations, exper- iments, etc). In many cases, exploring the correlation between the two allows us to efficiently construct an accurate representation of the response surface by performing relatively few evaluations of an expensive high-fidelity model and more evaluations of a cheaper surrogate. A linking mechanism that undertakes the task of information fusion naturally arises in the context of multivariate Gaussian regression. Next, we introduce this concept in the context of ordinary kriging and co-kriging. 12 3 2 2 2 1 1 1 output, y Z|Y Z|Y Z|Y 0 0 0 −1 −1 −1 −2 −2 −2 −3 0 0.5 1 0 0.5 1 input, x −5 0 5 x input, x x x input, x (a) (b) (c) Figure 2.2: Gaussian process regression: Left: Samples of a zero-mean Gaussian prior over functions. Middle: Samples of the calibrated Gaussian posterior in view of noise-less data (dashed lines), and the conditional mean of Z|Y (solid line).Right: Samples of the calibrated Gaussian posterior in view of noisy data (dashed lines), and the conditional mean of Z|Y (solid line). The grey shadowed regions depict the envelope defined by two standard deviations (figure taken from Rasmussen (2006)). 2.3 Methods The basic building block of the proposed multi-fidelity information fusion framework is Gaussian process regression (GPR). One way of viewing GPR is as defining a prior distribution over functions, which is then calibrated in view of data using an appropriate likelihood function, resulting to a posterior distribution with predictive capabilities (see Fig. 2.2). In what follows we provide an overview of the key steps in this construction, in the context of the ordinary kriging and recursive co-kriging predictors. The reader is referred to Rasmussen (2006) for a detailed exposition to the subject. 2.3.1 Kriging The main idea here is to model N scattered observations y of a quantity of interest (QoI) Y (x) as a realization of a Gaussian random field Z(x), x ∈ Rd . The observa- tions could be deterministic or stochastic in nature and may well be corrupted by 13 modeling errors or measurement noise E(x), which is thereby assumed to be a zero- mean Gaussian random field, i.e E(x) ∼ N (0, σ2 I). Therefore, we have the following observation model Y (x) = Z(x) + E(x). (2.1) The prior distribution on Z(x) is completely characterized by a mean µ(x) = E[Z(x)] and covariance κ(x, x0 ; θ) function, where θ is a vector of hyper-parameters. Typi- cally, the choice of the prior reflects our belief on the structure, regularity, and other intrinsic properties of the quantity of interest Y (x). However, our primary goal here is not just drawing random fields from the prior but to incorporate the knowledge contained in the observations y in order to reconstruct the field Y (x). This can be y |y, θ), where yˆ(x? ) contains achieved by computing the conditional distribution π(ˆ the predicted values for Y (x) at a new set of locations x? . If a Gaussian prior is as- y |y, θ) is obviously Gaussian and provides sumed on the hyper-parameters θ then π(ˆ a predictive scheme for the estimated values yˆ. Once Z(x) has been trained on the observed data (see Sec. ??), its calibrated mean µ ˆ 2 , and noise variance ˆ, variance σ ˆ2 are known and can be used to evaluate the predictions yˆ, as well as to quantify σ the prediction variance s2 as (see Jones (2001) for a derivation), yˆ(x? ) = µ ˆ + rT (R + σ ˆ2 I)−1 (y − 1ˆ µ), (2.2) [1 − rT (R + σ ˆ2 I)−1 r]2   2 ? 2 T 2 −1 s (x ) = σ ˆ 1 − r (R + σ ˆ I) r + , (2.3) 1T (R + σ ˆ2 I)−1 1 where R = κ(x, x0 ; θ) is the N × N correlation matrix of Z(x), r = κ(x, x? ; θ) is a 1 × N vector containing the correlation between the prediction and the N training 14 points, and 1 is a 1 × N vector of ones. This is a linear regression scheme known as the best linear unbiased predictor (BLUP) in the statistics literature (see Cressie and Cassie (1993)). Note, that for σ2 = 0 the predictor exactly interpolates the training data y, returning zero variance at these locations. 2.3.2 Recursive co-kriging We provide a brief overview of the recursive co-kriging scheme recently put forth by Le Gratiet and Garnier (2014) - a more efficient version of the well-known auto- regressive inference scheme proposed by Kennedy and O’Hagan (2000) in the context of predicting the output from a complex computer code when fast approximations are available. To this end, suppose that we have s levels of information sources producing outputs yt (xt ), at locations xt ∈ Dt ⊆ Rd , sorted by increasing order of fidelity and modeled by Gaussian processes Zt (x), t = 1, . . . , s. Then, the auto-regressive scheme of Kennedy and O’Hagan (2000) reads as Zt (x) = ρt−1 (x)Zt−1 (x) + δt (x), t = 2, ..., s, (2.4) where δt (x) is a Gaussian field independent of {Zt−1 , . . . , Z1 } and distributed as δt ∼ N (µδt , σt2 Rt ). Also, ρ(x) is a scaling factor that quantifies the correlation between {Zt (x), Zt−1 (x)}. In the Bayesian setting, ρ(x) is treated as a random field with an assigned prior distribution that is later calibrated to the data. Here, to simplify the presentation we assume that ρ is a deterministic scalar, independent of x, and learned from the data through maximum likelihood estimation (see Sec. ??). The key idea put forth by Le Gratiet and Garnier (2014) is to replace the Gaus- 15 sian field Zt−1 (x) in Eq. 3.55 with a Gaussian field Z˜t−1 (x) that is conditioned on all known observations {yt−1 , yt−2 , . . . , y1 } up to level (t − 1), while assuming that the corresponding experimental design sets Di , i = 1, . . . , t − 1 have a nested struc- ture, i.e. D1 ⊆ D2 ⊆ · · · ⊆ Dt−1 . This essentially allows to decouple the s-level auto-regressive co-kriging problem to s independent kriging problems that can be efficiently computed and are guaranteed to return a predictive mean and variance that is identical to the coupled scheme of Kennedy and O’Hagan (2000). To under- line the advantages of this approach, note that the scheme of Kennedy and O’Hagan requires inversion of covariance matrices of size st=1 Nt × st=1 Nt , where Nt is the P P number of observed training points at level t. In contrast, the recursive co-kriging approach involves the inversion of s covariance matrices of size Nt × Nt , t = 1, . . . , s. Once Zt (x) has been trained on the observed data {yt , yt−1 , . . . , y1 } (see Sec. ??), the optimal set of hyper-parameters {ˆ µt , σ ˆ2t , ρˆt−1 , θˆt } is known and can be used ˆt2 , σ to evaluate the predictions yˆt , as well as to quantify the prediction variance s2t at all points in x?t (see Le Gratiet and Garnier (2014) for a derivation), yˆt (x?t ) = µ ˆt + ρˆt−1 yˆt−1 (x?t ) + rtT (Rt + σ ˆ2t I)−1 [yt (xt ) − 1ˆµt − ρˆt−1 yˆt−1 (xt )], (2.5) [1 − rtT (Rt + σ ˆ2t I)−1 rt ]2   2 ? 2 2 ? 2 T 2 −1 st (xt ) = ρˆt−1 st−1 (xt ) + σ ˆt 1 − rt (Rt + σ ˆt I) rt + , 1Tt (Rt + σ ˆ2t I)−1 1t (2.6) where Rt = κt (xt , x0 t ; θˆt ) is the Nt × Nt correlation matrix of Zt (x), rt = κt (xt , x?t ; θˆt ) is a 1 × Nt vector containing the correlation between the prediction and the Nt training points, and 1t is a 1 × Nt vector of ones. Note that for t = 1 the above scheme reduces to the ordinary kriging predictor of Eq. 3.53, 2.3. Also, κt (xt , x0 t ; θt ) is the auto-correlation kernel that quantifies spatial correlations at level t. 16 We recognize that such recursive auto-regressive schemes can provide a rigor- ous and tractable workflow for multi-fidelity information fusion. This suggests a general framework that targets the seamless integration of surrogate-based predic- tion/optimization and uncertainty quantification, allowing one to simultaneously address multi-fidelity in physical models (e.g., direct numerical simulations vs. ex- periments) as well as multi-fidelity in probability space (e.g., the sparse grid quadra- tures of Novak and Ritter (1996) vs. the multi-element probabilistic collocation of Foo et al. (2008)). 2.3.3 Maximum likelihood estimation Estimating the hyper-parameters requires learning the optimal set of {µt , σt2 , σ2t , ρt−1 , θt } from all known observations {yt , yt−1 , . . . , y1 } at each inference level t. In what follows we will confine the presentation to maximum likelihood estimation (MLE) procedures for the sake of clarity. However, in the general Bayesian setting all hyper-parameters are assigned with prior distributions, and inference is performed via more costly marginalization techniques, typically using Markov Chain Monte Carlo integration (see Rasmussen (2006), Hastie et al. (2009)). Parameter estimation via MLE at each inference level t is achieved by minimizing the negative log-likelihood of the observed data yt , n 1 min log(σt2 ) + log |Rt (θt ) + σ2t I|+ (2.7) 2 {µt ,σt ,σ2t ,ρt−1 ,θt } 2 2 1 + [y (xt ) − 1t µt − ρt−1 yˆt−1 (xt )]T [Rt (θt ) + σ2t I]−1 [yt (xt ) − 1t µt − ρt−1 yˆt−1 (xt )], 2 t 2σt 17 where we have highlighted the dependence of the correlation matrix Rt on the hyper- parameters θt . Setting the derivatives of this expression to zero with respect to µt , ρt−1 and σt2 , we can express the optimal values of µˆt , ρˆt−1 and σ ˆt2 as functions of the correlation matrix (Rt + σ2t I), (µˆt , ρˆt−1 ) = [hTt (Rt + σ2t I)−1 ht ]−1 hTt (Rt + σ2t I)−1 yt (xt ) (2.8) 1 ˆt2 = ˆt − ρˆt−1 yˆt−1 (xt )]T [Rt + σ2t I]−1 [yt (xt ) − 1t µ σ [yt (xt ) − 1t µ ˆt − ρˆt−1 yˆt−1 (xt )] , c (2.9)   Nt − 1,  t=1 where ht = [1t yˆt−1 (xt )], and c = σ2t , θˆt } . Finally, the optimal {ˆ  N − 2,  t>1 t can be estimated by minimizing the concentrated restricted log-likelihood min log |Rt (θt ) + σ2t I| + c log(ˆ σt2 ). (2.10) {σ2t ,θt } 2.4 Multi-fidelity in models and in probability space We can build further upon the presented co-kriging framework to formulate a gen- eral methodology that can simultaneously address multi-fidelity in physical models as well as multi-fidelity in probability space. As it is often the case in realistic design scenarios, the output of a system may well be sensitive to a set of inputs ξ that exhibit random variability. Consequently, decision making towards identifying an optimal design is typically informed by exploring the measures of uncertainty that describe the response of the underlying stochastic dynamical system. This response 18 Multi-fidelity in models Multi-fidelity in probability space Figure 2.3: Multi-fidelity in models and in probability space: m models of variable fidelity in physical space are driven by random input, producing a random response surface Ym (x; ξ). For example, the expectation of a derived quantify of interest E[f (Ym (x; ξ))] can be estimated by employing p methods of variable fidelity in prob- ability space. is often characterized by non-Gaussian statistics that can be estimated numerically by utilizing appropriate sampling and integration techniques. The potential non- Gaussianity in the system response should not be confused with the Gaussian na- ture of the kriging/co-kriging predictors. The former is an inherent property of the dynamical system that generates the observed data, while the later introduces a modeling framework for information fusion. Similarly to having multi-fidelity in models, methods of different fidelity can also be incorporated in probability space to provide an accurate quantification of uncertainty introduced by random input. This structure is schematically illustrated in Fig. 2.3, where m models of variable fidelity in physical space are driven by random input, hence producing a random response surface Ym (x; ξ). In return, any uncertainty quantification measure of Ym (x; ξ), such as for e.g. the expectation E[Ym (x; ξ)] or the risk R[Ym (x; ξ)], can be estimated using a set of p variable-fidelity methods, such as Monte Carlo integration (see Binder and Heermann (2010)) or probabilistic multi-element collocation on tensor product grids (see Foo et al. (2008), and Fig. 2.3). 19 This construction results in a family of response surfaces that can be organized hierarchically in a p × m matrix, where physical model fidelity is increased along the columns and probability space model fidelity increases along the rows (see Fig. 2.4). Then, it is meaningful to allow information fusion along the {→}, {↓}, {↑→}, {↓→} directions by employing the autoregressive co-kriging framework presented in Sec. 2.2, 2.3.2. For example, moving along the purely vertical direction {↓} results to the following autoregressive expression for the expectation E[f (Ym (x; ξ))]: Ek+1 [f (Yl (x; ξ))] = ρk+1 Ek [f (Yl (x; ξ))] + δk+1 (x), k ≤ p, l ≤ m, (2.11) where the k-index increases with the fidelity of the estimator of E[f (Y(x; ξ))] in probability space, while the l-index increases with model fidelity in physical space. This structure gives rise to the very interesting task of identifying an optimal path traversal between different models for building an accurate representation of the target response surface. This is an open question that we plan to address in a future study. A possible way of attacking this problem is through stochastic dynamic programming techniques for guiding an optimal allocation of available computational resources (see Royset (2013)). Alternatively, one could employ random graph theory to identify optimal information/entropy diffusion paths, where each graph node is weighted by the fidelity and corresponding cost of each model, while edge weights represent the degree of correlation between different models. 20 Figure 2.4: Multi-fidelity in models and in probability space: Family of response surfaces resulting from simultaneously addressing multi-fidelity in models and in probability space. Physical model fidelity is increased along the columns (red arrow) and probability space model fidelity increases along the rows (blue arrow). The yellow arrow represents a possible optimal information fusion path in the combined fidelity space. 2.5 Challenges to be addressed 2.5.1 Bottlenecks of learning from large data-sets The computational cost of calibrating model hyper-parameters through MLE is dom- inated by the inversion of correlation matrices (Rt + σ2t I)−1 at each iteration of the minimization procedure in Eq. 2.10. The inversion is typically performed using the Cholesky decomposition that scales as O(Nt3 ), leading to a severe bottleneck in the presence of moderately big data-sets. This is typically the case for high-dimensional problems where abundance of data is often required for performing meaningful in- ference. This pathology is further amplified in cases where the noise variance σ2t is negligible and/or the observed data points are tightly clustered in space. Such cases introduce ill-conditioning that may well jeopardize the feasibility of the inversion as well as pollute the numerical solution with errors. Moreover, if an anisotropic correlation kernel κt (xt , x0 t ; θt ) is assumed, then the vector of correlation lengths θt is d-dimensional, leading to an increasingly complex optimization problem (see Eq. 2.10) as the dimensionality of the input variables xt increases. These shortcom- ings render the learning process intractable for large data-sets, and suggest seeking alternative routes to parameter estimation. 21 In Chapter 3 we introduce a new inference methodology that exploits the solu- tion of certain stochastic partial differential equations, introducing Gaussian Markov random fields with sparse inverse covariances. Essentially, this enables the design of algorithms that leverage this sparsity, leading to a significant speed up of the inference process. 2.5.2 Bottlenecks of scaling to high dimensions In general, the design of predictive inference schemes in high-dimensions suffers from the well known curse of dimensionality, as the number of points needed to explore the input space in its entirety increases exponentially with the dimension. This implicit need for big data introduces a severe deadlock for scalability in machine learning algorithms as they often involve the repeated inversion of covariance matri- ces that quantify the spatial cross-correlations in the observations. Again, this leads to the so called “big N ” problem - an expression used to characterize the demanding operational count associated with handling data-sets comprised of N observations (N > 1000). The implications of such large data-sets on learning algorithms are well known, leading to a O(N 3 ) scaling for implementations based on maximum likelihood estimation. Another problem arising in high-dimensions stems from the the common strategy for constructing auto-correlation models by taking the product of one-dimensional autocorrelation kernels. This typically results to an anisotropic covariance model, which assumes that all dimensions actively interact with each other. However, as the dimensionality is increased, one would hope to find sparsity in the input space, i.e. dimensions with negligible or very weak pairwise interactions. 22 In Chapter 4 we will present a set of new algorithms that introduce a tractable solution path in high dimensional input spaces and in the presence of massive data- sets. 2.6 Demonstration Here we present two demonstrations to highlight the applicability of the proposed framework to problems of different nature. The first benchmark involves the accurate reconstruction of the response predicted by variable fidelity partial differential equa- tion models that describe the dynamics of deep water waves. The analysis is based on the recent work of Cousins and Sapsis (2015a,b), Mohamad et al. (2015).The sec- ond example demonstrates the use of the presented methods for data-assimilation of noisy field measurements for the reconstruction of the bathymetry map in a coastal region. Finally, we present a representative case in one input dimension, where the proposed multi-fidelity formulation fails to result to an improved predictor due to lack of adequate correlations between different fidelity levels. 2.6.1 Multi-fidelity modeling of ocean waves Understanding and predicting large ocean waves is critical due to the catastrophic damage they inflict on ships and other coastal structures. However, understanding these waves is challenging: due to the large amplitudes of the waves, their dynamics are nonlinear. To better understand these extreme waves, we model the behavior of wave groups. Some wave groups will focus, increasing in amplitude, while others disperse an decrease in amplitude. We wish to quantify how this behavior depends 23 on the amplitude A and length scale L of the group. Specifically, we determine whether or not a particular group will focus and, if so, the corresponding increase in amplitude. High fidelity, nonlinear partial differential equation (PDE) models reproduce these extreme wave group dynamics well, but are expensive to solve. Thus it is computationally advantageous to use reduced order models. These reduced order models qualitatively reproduce the true dynamics but have error of up to 30% for some groups. Using the proposed multi-fidelity information fusion framework allows us to approximate the true dynamics by combining a small number of simulations of the high fidelity model with many simulations of the reduced order model. The quantity of interest here is the spatio-temporal maximum of the wave en- velope amplitude u normalized by the initial wave amplitude, i.e., Y = max |u|/A0 . This quantity is 1 when defocusing occurs and the amplitude decreases. Values of this ratio larger than 1 indicate that the associated group focuses, increasing in am- plitude. For more details on this modeling approach the reader is referred to the recent work of Cousins and Sapsis (2015a,b), Mohamad et al. (2015). In Fig. 2.5 we present the response surface of the quantity of interest Y as a function of the input variables x = (A0 , L0 ), where L0 is the initial wavelength. The response has been reconstructed using 80 samples of an expensive to compute high- fidelity PDE model supplemented with 310 observations coming from a reduced order low-fidelity model. We observe that the resulting multi-fidelity co-kriging predictor is in good agreement with the “exact” solution obtained using 2,500 samples of the high-fidelity code. The accuracy of the prediction is demonstrated in Fig. 2.6(a), where we plot the point-wise error in the L∞ norm, while Fig. 2.6(b) shows the spatial distribution of the predictor’s uncertainty. Finally, in Fig. 2.7 we present a 24 Exact (2500 high-fidelity samples) Co-kriging Low-fidelity samples (310 pts) High-fidelity samples (80 pts) Y L0 A0 Figure 2.5: Multi-fidelity modeling of ocean waves: Exact response surface and co- kriging predictor constructed using 310 low-fidelity and 80 high-fidelity observations. visual comparison of the exact solution versus the co-kriging surrogate. It is evident that the proposed multi-fidelity analysis can accurately capture the transition in the dynamics that results to a jump in the response surface, but also result to orders of magnitude in speed-up for obtaining an accurate representation. 2.6.2 Data-assimilation of coastal bathymetry data Our objective here it to use reconstruct the bathymetry field of a coastal region from noisy measurements. To this end, we have available 1,321 depth measurements (cour- tesy of Mike Sacarny, MIT) and we will test the robustness of the kriging predictive distribution with respect to corrupting the data with additional Gaussian noise. In Fig. 2.8(a),(b) we present the kriging prediction and variance, respectively, using a 25 L∞ error Cokriging Variance 0.5 0.3 0.5 0.45 0.45 0.02 0.25 0.018 0.4 0.4 0.016 0.35 0.35 0.2 0.014 0.3 0.3 0.012 A0 0.25 0.15 A0 0.25 0.01 0.2 0.2 0.008 0.1 0.15 0.15 0.006 0.1 0.1 0.004 0.05 0.05 0.002 0.05 0 0 5 10 15 20 25 30 35 40 45 5 10 15 20 25 30 35 40 45 L0 L0 (a) (b) Figure 2.6: Multi-fidelity modeling of ocean waves: Left: Poin-twise L∞ error of the co-kriging predictor (310 low-fidelity and 80 high-fidelity observations). Right: Point-wise variance of the co-kriging predictor. The reference solution was obtained using 2,500 high-fidelity samples. # !"* !") !"( !"' A0 A0 !"& !"% !"$ !"# !"! ! L0 L0 (a) (b) Figure 2.7: Multi-fidelity modeling of ocean waves: Visual comparison of the exact solution (2,500 high-fidelity samples) (right), and the co-kriging predictor (310 low- fidelity and 80 high-fidelity observations) (left). 26 Depth(m) Depth(m) ) !)(#'$* * !)(#'$* "#& !)(#'$) & !)(#'$) " Longitude Longitude ! !)(#'$% !)(#'$% $#& + !)(#'$' !)(#'$' $ " !)(#'" !)(#'" $ (#& !)(#'"$ !)(#'"$ ( !"#!$% !"#!$%& !"#!$' !"#!$'& !"#!" !"#!"(& !"#!$% !"#!$%& !"#!$' !"#!$'& !"#!" !"#!"(& Latitude Latitude (a) (b) Figure 2.8: Data-assimilation of coastal data: Left: Kriging predictor (100 noisy observations). Right: Kriging variance. 5/2 Mat´ern covariance model, and 100 observations that may contain measurement noise. We observe that this results to a smooth bathymetry map that is in agreement with the exact topology of the region (see Fig. 2.10), while the kriging uncertainty is negligible, except for a region far away from the observations (see bottom right corner of Fig. 2.8(b). Finally, we asses the robustness of the kriging scheme with respect to adding a Gaussian perturbation in the original measurements. In Fig. 2.10 we present an illustration of the predictive capacity of the kriging predictor yˆ as a function of the noise amplitude. In all cases, the kriging predictor was able to filter out the noise components and return a smooth response surface. These demonstrations serve as motivation of some of the capabilities that the proposed framework has to offer. In Chapter 3 we provide a detailed overview of the methods employed to produce the aforementioned results, as well as a number of benchmark problems that further illustrate the potential applications of this work. 27 Figure 2.9: Data-assimilation of coastal data: Left: Kriging predictor (100 noisy observations). 0% noise 10% noise 50% noise 7 7 7 6 6 6 5 5 5 4 4 4 yˆ yˆ yˆ 3 3 3 2 2 2 1 1 1 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 y y y (a) (b) (c) Figure 2.10: Data-assimilation of coastal data: Left: Robustness of the riging pre- dictor with respect to corrupted measurements. (Left to right) Predictions yˆ versus corrupted data y for increasing noise amplitudes. 28 2.6.3 Effect of low correlation Our objective here is to highlight the importance of cross-correlations between out- puts of different fidelity in obtaining an accurate recursive co-kriging predictor. In absence of such correlations, the proposed methodology is not expected to behave much better than fitting a kriging predictor through the observed points at the high- est fidelity output. Here, we demonstrate this remark through a fabricated example in one input dimension, and in absence of measurement noise. To this end, consider the high-fidelity output to return realizations of a sine function in 1D fh (x) = sin(6πx) (2.12) Then, let fl (x) be a low-fidelity model described as fl (x) = c sin(6πx) + cos(2πx) + 3e−x/2 , (2.13) where c is a parameter that controls the degree of correlation between fl (x) and fh (x). Obviously, values of c around 1 yield the strongest correlation, while values away from 1 lead to very weak correlations between fh (x) and fl (x). Assume now that we have 6 equispaced observations of fh (x) in [0, 1]. In Fig. 2.11 we present the result of fitting an ordinary kriging predictor with a squared expo- nential kernel on this dataset. Evidently, the resolution of the observations is not adequate to resolve the variability in the exact solution, returning an inaccurate predictor with high variance at locations away from the observed points. Next, we try to construct a more accurate predictor by employing additional in- 29 formation from a lower fidelity model fl (x) through recursive co-kriging. To this end, we have considered two cases that reflect the dependence of the proposed methodol- ogy on the intensity of spatial cross-correlations between different fidelity levels. In the first case, we take c = 0.5 in Eq. 2.13, and train a recursive co-kriging predictor on a set of 11 equispaced low-fidelity observations supplemented by 6 equispaced data-points of the high-fidelity model. The result is shown in Fig. 2.12(a), yielding a reasonably good fit to the exact solution, with a low associated uncertainty. The beneficial effect of injecting low-fidelity data is also indicated by the scaling factor ρ in the auto-regressive scheme of Eq. 3.55. Here, the MLE-based training scheme returned ρ = 1.9247, suggesting a strong correlation between fl (x) and fh (x). This is in contrast with the behavior demonstrated in Fig. 2.12(b), where the scaling pa- rameter in the low-fidelity model of Eq. 2.13 is adjusted to c = 0.4. This choice leads to a weak spatial cross-correlation between fl (x) and fh (x), and as a result the information carried by the 11 equispaced observations of fl (x) has a negligible con- tribution in improving the accuracy of the co-kriging predictor. In fact, the resulting predictor nearly collapses to the estimate obtained using only the 6 high-fidelity observations in the kriging model of Fig. 2.11. The inadequacy of the low-fidelity model to provide useful information in this case is further confirmed by the scaling factor ρ, with MLE returning ρ = 0.1683 – a value more than 10 times less than the one predicted for c = 0.5. 30 Exact Kriging y x Figure 2.11: Effect of low correlation: Ordinary kriging predictor (red dashed line) through 6 equispaced observations (red circles) of fh (x) (black solid line). The gray shaded area corresponds to two standard deviations of the Gaussian predictor. In summary, we have presented a case where injecting low-fidelity information leads to no improvement in the co-kriging predictor due to very weak spatial correla- tions between the low- and high-fidelity models. In such cases, one may expect that recursive co-kriging will result to nothing better than a kriging predictor through the highest fidelity points. However, the framework provides a mechanism for detecting this pathology, and that is done by monitoring the scaling factor ρ appearing in the auto-regressive scheme of Eq. 3.55, and is learned from the data via MLE. 31 Exact Exact Co-kriging Co-kriging Low-fidelity model Low-fidelity model c = 0.5 c = 0.4 y y x x (a) (b) Figure 2.12: Effect of low correlation: Left: Two-level recursive co-kriging predictor for a low-fidelity model fl (x) with c = 0.5.Right: Two-level recursive co-kriging predictor for a low-fidelity model fl (x) with c = 0.4. In both cases, the training data- set consists of 11 equispaced observations of fl (x), and 6 equispaced observations of fh (x), while the gray shaded area depicts two standard deviations of the Gaussian predictor. Chapter Three Spectral/hp Element Methods for Inference with Gaussian Markov Random Fields 33 3.1 Motivation In this chapter we study the applicability and performance of spectral/hp element methods for spatial statistics using Gauss Markov models. This work is motivated by the recent findings of Lindgren et al. (2011) that highlight the link between Gaus- sian fields and Gaussian Markov random fields through finite element discretization of stochastic partial differential equations. This approach has very promising impli- cations as it circumvents the need to invert the full and ill-conditioned covariance matrices resulting from standard kriging techniques, while it enables the use of fast and scalable algorithms, thus opening the path to efficiently performing statistical inference on very big data sets. Here, we introduce the use of high-order nodal spec- tral/hp element methods and highlight all related implementation aspects. Moreover, we describe how this approach can be efficiently integrated into standard model fit- ting and statistical inference techniques, and utilized towards performing kriging predictions that satisfy Dirichlet and/or Neumann boundary conditions. The pro- posed algorithms are integrated in the general multi-fidelity framework proposed in Chapter 2, and results are presented for several benchmark problems. 3.2 Gaussian Markov random fields and the SPDE approach Let Y (x) be an observation of a latent Gaussian field, u(x), under a zero mean Gaussian measurement noise, (x), with variance σ 2 , 34 Y (x) = u(x) + (x), (3.1) and let µ(x) and κ(x, x0 ) be the mean value function and the auto-correlation func- tions of u(x), respectively. Additional assumptions on µ(x) may distinguish the above linear spatial prediction model into simple Kriging (if µ is assumed to be known), ordinary Kriging (if µ is unknown but independent of x), and universal Kriging (if µ is unknown and is represented as a linear combination of deterministic basis functions, see Cressie and Cassie (1993)). 3.2.1 The Mat´ ern covariance model A popular choice for the covariance functions in spatial statistics is the Mat´ern covariance family (see Murphy (2012), Cressie and Cassie (1993)), 21−ν σ 2 C(h) = d d (κ||h||)ν Kν (κ||h||), h ∈ Rd , (3.2) (4π) Γ(ν + 2 2 )κ2ν where h = ||x−x0 ||2 , ν determines the mean-square differentiability of the underlying process, κ is a scaling parameter related to the correlation length of the process, ρ, as √ ρ = 8ν/κ, σ 2 is the marginal variance of the process, while Γ(·) and Kν (·) are the Euler gamma and modified Bessel function of the second kind, respectively. Using the asymptotic property ν−1 Kν (t) − Γ(ν)2 → 0, as t → 0, (3.3) tν we can obtain a closed form expression for the variance of a process with Mat´ern covariance, as 35 d π 2 σ 2 Γ(ν) C(0) = (3.4) Γ(ν + d2 )κ2ν Moreover, we can express the associated spectral density of the Mat´ern covariance function as σ2 S(k) = d (3.5) (κ2 + ||k||2 )ν+ 2 Note that for the spacial case of ν = 0.5, the Mat´ern covariance simply reduces the exponential covariance model, while when ν → ∞ we recover the Gaussian covariance function (see Murphy (2012)). 3.2.2 Connection to stochastic partial differential equations (SPDE) A powerful result by Whittle (1963) shows that a random process u(x) with spectral density given by Eq. 3.5 is a solution to the SPDE α (κ2 − ∇2 ) 2 u(x) = W(x), x ∈ Rd , α = ν + d/2, κ > 0, ν > 0, (3.6) where W(x) is Gaussian white noise. In literature, solutions to Eq. 3.6 are referred to as Mat´ern fields and are proved to be the only stationary solutions to this SPDE (see Whittle (1963)). The key idea put forth by Lindgren et al. (2011) was to employ Hilbert space approximation theory to approximate the solution to Eq. 3.6 using a projection onto 36 a finite dimensional space spanned by basis functions φ1 (x), ..., φn (x) as n X u(x) = φk (x)wk (3.7) k=0 where {wk } are a set of unknown expansion coefficients. In what follows, we will refer to the basis functions {φk (x)} using the terms “modes” and “basis functions” interchangeably, while the term “degrees of freedom” will refer to the expansion coefficients {wk }. A critical observation of Lindgren et al. (2011), citing the work of Rozanov (1982), states that a random field has a Markov property if and only if the reciprocal of its spectrum is a polynomial. Thereby, the Mat´ern fields generated by Eq. 3.6, having a spectrum given by Eq. 3.5, inherit the Markov property when α assumes integer values. Exploiting this Markov property, Lindgren et al. (2011) were able to utilize the finite dimensional representation of u(x) given in Eq. 3.7 to construct a Gauss Markov random field (GMRF) with local neighborhood and precision matrix Q that represents the Gaussian field u(x) in the sense that Q−1 is close to the covariance of u(x), denoted by Σ, in some norm. The main advantage here is that the Markov property, along with the choice of a suitable basis, results to a sparse precision matrix Q, and, thus, enables the use of fast numerical methods for sparse matrices, significantly enhancing the computational efficiency of Bayesian inference. 37 3.3 Hilbert space approximation 3.3.1 Weak form Consider the SPDE of Eq. 3.6 with α = 2 in a two- or three-dimensional region Ω with boundary ∂Ω: (κ2 − ∇2 )u(x) = W(x), (3.8) subject to u|∂ΩD = qD (∂ΩD ), ∇u · n|∂ΩN = qN (∂ΩN ), (3.9) where qD and qN are Dirichlet and Neumann boundary conditions, n is the out- ward normal, and ∂ΩD ∪ ∂ΩN = ∂Ω. Introduction of the inner product (f, g) = R Ω f (x)g(x)dx allows us to construct the weak form of this problem by multiplying Eq. 3.8 by a test function v(x) that is homogeneous on all Dirichlet boundaries, that is, v(x) belongs to the test space V = {v | v ∈ H 1 , v(0) = 0}. Integrating over the domain Ω we obtain (κ2 u, v) − (∇2 u, v) = (W, v) (3.10) Applying the divergence theorem to the diffusion term we have Z 2 (κ u, v) + (∇u, ∇v) = (W, v) + v∇u · ndS, (3.11) ∂Ω Once a finite dimensional expansion basis for representing the solution u(x) over a tessellation of the domain Ω is chosen, the discrete form of Eq. 3.11 reads as 38 Hw = f (3.12) where w is the vector of the unknown Galerkin coefficients of the solution expansion modes, and H is the discrete Helmholtz operator defined as H = κ2 M + S, (3.13) with M and S being the mass and stiffness matrices, respectively, resulting from projecting the solution u(x) onto a finite dimensional subspace spanned by the ex- pansion modes {φk (x)}, with entries computed as Mij = (φi , φj ) (3.14) Sij = (∇φi , ∇φj ) (3.15) Finally, the right hand side term f = fW + fBC appearing in Eq. 3.12 results from the projection of the white noise forcing onto the basis, plus the forcing due to boundary fluxes, respectively, fW = (W, v) (3.16) Z fBC = v∇u · ndS (3.17) ∂Ω 39 3.3.2 Choosing an expansion basis The choice of the expansion basis in Eq. 3.7 is of critical importance both in terms of obtaining an accurate solution to the SPDE, as well as for developing robust and computationally efficient Bayesian inference algorithms. To illustrate the important role played by the choice of the expansion basis, let us start from computing the expectation of the Galerkin expansion coefficients w using Eq. 3.12 E[w] = H−1 fBC (3.18) Similarly, we can compute the covariance matrix of the Galerkin expansion coeffi- cients, Σw , as Σw = E[wwT ] − E[w]E[wT ] = H−1 E[fW fW T ]H−1 = H−1 MH−1 (3.19) where we have used that W(x) is a zero mean Gaussian process, the forcing term f is a deterministic quantity, and the matrix H is symmetric. Therefore, the Galerkin ex- pansion coefficients Σw form a Gaussian random field distributed as w ∼ N (H−1 fBC , Σw ). Consequently, the precision matrix of the Galerkin expansion coefficients, Qw , is computed as Qw = Σ−1 −1 w = HM H (3.20) Then, the precision matrix of the solution field u(x), denoted by Qu , can be recovered 40 as Qu = ΦQw ΦT (3.21) where Φ is a matrix containing the basis functions {φk (x)} evaluated at the grid point locations x that define the tessellation of Ω. We can now advocate that the effect of choosing a suitable set of basis func- tions is threefold. First, the choice of basis determines the stability, converge and accuracy properties of the discretization scheme to Eq. 3.8. Second, it determines the structure, degree of sparsity and condition number of the mass, stiffness and Helmholtz operators, which, in turn, are responsible for passing these properties to the precision and covariance matrices of the Galerkin coefficients w, according to Eqs. 3.19,3.20. Finally, the chosen basis affects the structure, degree of sparsity and condition number of the precision matrix, Qu , of the solution field, u(x), as it determines the sparsity of the basis matrix Φ in Eq. 3.21. Lindgren et al. (2011) have addressed these issues by choosing to discretize Eq. 3.8 using a linear finite element basis. This choice is appealing as it immediately implies that the basis matrix Φ in Eq. 3.21 is the identity matrix. Moreover, as a result of mass matrix lumping, the mass matrix is diagonal, suggesting that the finite di- mensional approximation to the Gaussian random field u(x) is a Gaussian Markov random field (GMRF) with sparse precision matrix. This consequence is highly desirable as it enables the use of fast algorithms for sparse matrices at the model fitting stage. However, we should note here that the mass matrix lumping technique employed by Lindgren et al. (2011) is an approximation that may degrade the ac- curacy and converge properties of the linear finite element basis (see Karniadakis and Sherwin (2013)), although results of Bolin and Lindgren (2009) have shown negligible differences in the resulting kriging errors between the exact finite element 41 representation (no mass lumping) and the Markov approximation (mass lumping). Bolin and Lindgren (2009) introduced B-spline and Daubechies wavelets bases to demonstrate the advantages of the SPDE approach over two of the most popular co- variance approximation techniques in spatial statistics, namely, covariance tapering and the convolution-fields method. 3.4 The spectral/hp element method 3.4.1 Overview of the method Here, we advocate the use of high-order nodal spectral/hp element bases that attain all the aforementioned desired properties while providing a flexible framework for the development of fast and scalable algorithms. We begin with providing a brief overview of the discretization method and the numerical scheme employed, while for a complete treatment we refer to Karniadakis and Sherwin (2013). First, the computational domain is formed by constructing a tessellation of Ω that may consist of triangular and/or quadrilateral elements in 2D, or a combina- tion of tetrahedra, hexahedra, prisms, and pyramids in 3D. Within each of the Nel elements, the solution to the SPDE (Eq. 3.8) is approximated in terms of a polyno- mial expansion of the form Ndof Nel X P X X u(x) = wi Φi (x) = wpe φep (xe (ξ)), (3.22) i=1 e=1 p=0 where Ndof is the total number of degrees of freedom, ξ defines a mapping from 42 Figure 3.1: The spectral/ hp element method: Sketch of the unstructured tessella- tion of the SPDE domain Ω the polynomial basis employed. The solution domain is decomposed into non-overlaping elements; within each element the solution is approximated by vertex, edge, face and (in 3D) interior modes. The shape func- tions associated with the vertex, edge and face modes for a fourth-order polynomial expansion (P=4) defined on triangular and quadrilateral faces are shown in color. the physical space to the standard element, and φep (xe (ξ)) are local to each element polynomials of order P , which when assembled together under the mapping xe (ξ) result to a a C 0 continuous global expansion Φp (x). The modes can be distinguished between vertex (linear) modes, edge, face, and bubble (interior) modes as depicted in Fig. 3.1. Note that the interior modes have zero support on the elemental boundaries, thus the boundary and interior degrees of freedom can be numerically decoupled through a technique known as substructuring, where the Schur complement of the boundary system is constructed. The boundary degrees of freedom, corresponding to adjacent elements, are coupled due to the requirement of C0-continuity. The local p-type basis functions employed in this study are constructed using the nodal Legendre cardinal functions, yielding the following expression in the standard element 43 (ξ − 1)(ξ + 1)∂ξ Lp (ξ) φep (ξ) = , 0 ≤ p ≤ P, (3.23) P (P + 1)Lp (ξP )(ξ − ξP ) where LP is the Legendre polynomial of order P . This nodal p-type basis has become known in the literature as “spectral elements”, due to Patera (1984), although in the original work of Patera, Chebyshev polynomials where used instead of Legendre. Two- and three-dimensional basis expansions can be constructed in terms of a tensor product of the above one-dimensional expansion as: φepq (ξ1 , ξ2 ) = φep (ξ1 )φeq (ξ2 ), 0 ≤ p, q; p ≤ P1 , q ≤ P2 in 2D (3.24) φepqr (ξ1 , ξ2 , ξ2 ) = φep (ξ1 )φeq (ξ2 )φer (ξ3 ), 0 ≤ p, q, r; p ≤ P1 , q ≤ P2 , r ≤ P3 in 3D, (3.25) where the polynomial order of the multi-dimensional expansions may differ in each coordinate direction as denoted by the bounds P1 , P2 , P3 . Figure 3.1 illustrates the resulting basis defined on triangular and quadrilateral faces for P = 4. 3.4.2 Properties of the nodal basis The choice of this nodal spectral element basis has several advantageous properties. First, it results to a sparse, symmetric positive-definite stiffness matrix, S. Moreover, under-integration of the inner products defining the entries of the mass matrix, M, using the Gauss-Legendre-Lobatto quadrature at the (P + 1) nodal locations, causes M to become diagonal due to the discrete orthogonality property of the basis (see Karniadakis and Sherwin (2013)). The error introduced from this reduced order 44 quadrature rule is, however, consistent with the approximation error of the expansion (see Karniadakis and Sherwin (2013)). In contrast with the mass matrix lumping technique employed in the finite element formulation of Lindgren et al. (2011), the inexact integration employed in the spectral element setting is not an approximation, and has a direct equivalence to the error of approximating a smooth function (see Karniadakis and Sherwin (2013)). Another favorable property of this basis arises when the numerical quadrature order is equal to the polynomial order plus one (same as above). Then, the modal expansion coefficients are simply the physical values of the solution at the quadrature points, and the basis matrix Φ appearing in Eq. 3.21 simply reduces to the identity matrix. This, along with the diagonal property of the mass matrix, naturally implies that the approximation to the solution of the SPDE in Eq. 3.8 will be a Gaussian Markov random field (GMRF) with a sparse precision matrix, given by Eq. 3.21. 3.4.3 Treatment of boundary conditions In order to enforce the set of boundary conditions described in Eq. 3.9 we recall that the space of the test functions v(x) in a Galerkin expansion is homogeneous and is therefore zero on all Dirichlet boundaries. As we can only solve for the components of u(x) that live in the same space, we employ the technique of lifting a known solution (see Karniadakis and Sherwin (2013)) and decompose the solution u(x) into its homogeneous uH (x) and non-homogeneous uD (x) components, where u(x) = uH (x)+uD (x), and uD (x) satisfies all Dirichlet boundary conditions qD (∂ΩD ). We note that the important component of this definition is that the homogeneous solution has zero contribution from modes which are nonzero on Dirichlet boundaries. The homogeneous solution also contains the interior degrees of freedom which are 45 defined to be zero on the boundaries. Moreover, recalling that v(x) is defined to vanish on all Dirichlet boundaries we can state the Galerkin problem of Eq. 3.11 as: Find uH in the trial space X = {u | u ∈ H 1 , u(0) = qD } such that Z 2 H H (κ u , v) + (∇u , ∇v) = (W, v) + vqN dS − (∇uD , ∇v) (3.26) ∂ΩH 3.4.3.1 Solving the global matrix system Recalling that the boundary and interior degrees of freedom are decoupled, we can order the global boundary degrees of freedom (indexed by the superscript b) in a continuous fashion, such that degrees of freedom associated with the unknown homogeneous part of the solution uH (x) are followed by the known degrees of freedom associated with the non-homogenous part uD (x) that is defined through Dirichlet boundary conditions. Finally, the decoupled interior degrees of freedom (indexed by the superscript i) are placed at the end of the pile. Following this numbering scheme, the global matrix system introduced in Eq. 3.12 takes the form      b  HHH HHD HHI  wH   fHb        H   wb  =  fb (3.27)  DH HDD HDI   D   D       i HIH HID HII wH fHi | {z } | {z } | {z } H w f where the vertical and horizontal lines are used to distinguish between the boundary and interior contributions. Using the technique known as substructuring, we can statically condense the global matrix system by removing the known, non-homogeneous boundary degrees 46 b of freedom that correspond to Dirichlet boundary conditions, wD ,      b  HHH HHI   wH − fHb b HHD wD = , (3.28)      i HIH HII wH fHi − HID wD b | {z } | {z } | {z } HH wH fH Then, we can separately solve for the unknown boundary and interior degrees of freedom in two steps as, b −1 i SHH wH = fHb − HHI HII b f − SHD wD (3.29) i HII wH = f i − HIH wH b b − HID wD , (3.30) where the matrix S is known as the Schur complement and is computed as,    SHH SHD  −1 S=  = HHD − HHI HII HID (3.31) SDH SDD This solution framework gives rise to a number of observations. First, we note that the size of the Schur complement matrix SHH in Eq. 3.31 can be significantly smaller that the global Helmholtz matrix H, while its condition number can be no larger than the condition number of the complete system in Eqs. 3.12,3.27 (see Karniadakis and Sherwin (2013)). Moreover, the Schur complement matrix SHH is symmetric and positive definite, allowing for the use of efficient and scalable precon- ditioners to accelerate the iterative solution of Eq. 3.29 (see Sherwin and Casarin (2001), Grinberg et al. (2009)). Second, since the interior modes are decoupled be- tween elements, Eq. 3.30 can be solved in parallel, locally to each element. If an iter- 47 ative solver is used for Eq. 3.29, the method requires the storage only of the following matrices, and only at an elemental level (no global matrix assembly is required (see −1 −1 Karniadakis and Sherwin (2013))): S, HHI HII , HIH , HID , HII . In fact, the square −1 matrices S, HII are symmetric, hence their storage cost is further reduced. These features are of significant importance, as they provide a cornerstone for achieving scalability and computational efficiency, especially for very large systems. 3.4.4 Computation of the covariance and precision matrices Recall the Galerkin formulation in Eq. 3.26, and let us compute the expected value of the expansion coefficients corresponding to the homogeneous part of the solution uH (x) E[wH ] = H−1 H H fBC , (3.32) where HH is the homogeneous Helmholtz operator, constructed from the blocks of the full Helmholtz system that correspond to the homogeneous modes, as shown H in Eq. 3.52. Moreover, the deterministic forcing term fBC now accounts for both Dirichlet and/or Neumann boundary conditions, taking the form Z H fBC = vqN dS − (∇uD , ∇v) (3.33) ∂ΩH Similarly, we can compute the covariance matrix of the homogeneous Galerkin expansion coefficients, ΣH w , as 48 −1 −1 −1 −1 ΣH T T T w = E[wH wH ] − E[wH ]E[wH ] = HH E[fW fW ]HH = HH MH HH (3.34) where we have used that W(x) is a zero mean Gaussian process, the forcing term H fBC is a deterministic quantity, and the matrix HH is symmetric. Therefore, the Galerkin expansion coefficients, wH , corresponding to the homogeneous part of the solution, uH (x), form a Gaussian random field distributed as wH ∼ N (H−1 H H H fBC , Σw ). Consequently, the corresponding precision matrices QH H w , and Qu are computed as QH H −1 w = (Σw ) = HH M−1 H H H H = Qu (3.35) since the nodal basis we have employed results to an identity matrix Φ in Eq. 3.21. Recall that the nodal spectral element basis with inexact integration results to a fully diagonal mass matrix MH , therefore the homogeneous part of the solution uH (x) to the SPDE (Eq. 3.8) has a sparse precision matrix, QH u , hence is a Gaussian Markov random field. In fact, any other choice of basis that does not produce a fully diagonal mass matrix would not attain this property, jeopardizing computational efficiency in the model fitting process. To clarify this point, let n be the total number of homogeneous degrees of freedom of the nodal spectral element expansion, that is the matrices HH , MH , ΣH H w , Qu appearing in Eqs. 3.34,3.35 are all square, symmetric positive definite, of size (n × n). It is clear that when MH is fully diagonal, the precision matrix QH u can be computed from Eq. 3.35 just at the cost of matrix multiplication, as the inversion of MH is straightforward. On the contrary, the computation of the covariance ΣH w in Eq. 3.34 requires an additional cost of up to O(n3 ) operations due to the inversion of the Helmholtz operator HH . Moreover, the resulting covariance ΣH H w is a full matrix, as opposed to the precision Qu that 49 is sparse. Therefore, gains in computational efficiency are made if the model fitting process can be formulated in terms of operations involving the sparse precision matrix of the GMRF uH (x) ∼ N (H−1 H H −1 H H fBC , (Qu ) ). Finally, we note that Qu can be used to consistently sample the a GMRF that approximates the homogeneous part of the solution to Eq. 3.8. This solution vector can then be augmented with the known, non-homogeneous contribution arising from Dirichlet data to obtain a Mat´ern field that satisfies the SPDE (Eq. 3.8), subject to Dirichlet and/or Neumann boundary conditions (Eq. 3.9). The most widely used way of efficiently accessing the covariance in practice is through the Cholesky decomposition of the precision, leveraging on its sparsity for the GMRF case (see Rue and Held (2005)). Here, we present an alternative way to construct the covariance matrix by explicitly computing each of its constituting blocks    ΣHH ΣHI  ΣH w =  H  = Σu , (3.36) ΣIH ΣII where each block reduces to the following expressions using Eqs. 3.29,3.30 and thanks to the properties of the nodal spectral element basis −1 −1 T −1 −1 −1 −1 T ΣHH = SHH MHH (SHH ) + (SHH HHI HII )MII (SHH HHI HII ) (3.37) −1 −1 −1 T ΣHI = (SHH HHI HII )MII (HII ) (3.38) −1 −1 −1 T ΣIH = ΣTHI = HII MII (SHH HHI HII ) (3.39) −1 T ΣII = HII MII (HII ) (3.40) Note that this approach requires only the inversion of the homogeneous-homogeneous block of the Schur complement matrix SHH and the interior-interior block of the 50 Helmholtz operator HII . The first, can be significantly smaller in size than the global Helmoltz system in Eq. 3.34 and can be effectively preconditioned when solved iteratively, while the latter consists of fully decoupled blocks that can be inverted element-wise in parallel. Moreover, it requires no additional storage since the re- −1 −1 quired matrices SHH , HHI HII , HII are already available from Eqs. 3.29,3.30,3.31. Whether this approach results to computational gains versus the popular Cholesky decomposition will be the subject of a future study. Here, we note that this compar- ison is expected to be heavily influenced by the correlation length of the underlying process u(x) and the resulting condition number of the global matrix system. With that in mind, the proposed approach may outperform the Cholesky decomposition for large systems and short correlation lenghts. 3.5 Model fitting and the kriging predictor In this section we provide a brief overview of the model fitting procedure needed to train the observation model of Eq. 3.1 on a set of observed data. For more details on parameter estimation for Bayesian inference we refer the reader to (see Murphy (2012), Rasmussen (2006), Rue and Held (2005)). Generally speaking, a Bayesian hierarchical model typical consists of (at least) a data model, characterized by the conditional probability distribution π(y|u) that describes how observations arise assuming a known process u(x), a latent model, characterized by a conditional probability π(u|θ) describing how the latent process u(x) behaves assuming a set of known parameters θ, and a set of parameters charac- terized by their probability distribution π(θ) which describes our, sometimes vague, 51 prior knowledge of the parameters. Using Bayes’ theorem, these probability distributions can be utilized to provide an expression for the posterior distribution π(u|y) that characterizes the statistics of the latent process u(x), given a set of observations y π(y|u)π(u) π(u|y) = ∝ π(y|u)π(y) (3.41) π(y) Given arbitrary data we typically don’t know the expectation or precision, but let these follow some known parametric structure that depends on a few parameters θ. Then, π(y|u) ∼ N (Au, Q−1  (θ)), π(u) ∼ N (µ(θ), Q−1 u (θ)), π(u|y) ∼ N (µu|y (θ), Q−1 u|y (θ)) (3.42) where A is an operator that restricts the latent process u(x) to the locations of the observed data, π(u|y) is the distribution describing how the latent field u(x) behaves given the some observations y, Q is the precision of the measurement noise appearing in the right hand side of our observation model, while µ(θ) is the mean of the latent process u(x) (see Eq. 3.1). At locations where observations exist, or where u(x) satisfies a known set of Dirichlet boundary conditions, µ(θ) attains these values up to measurement error, controlled by the zero-mean process (x) (see Eq. 3.1). The model fitting process consists of estimating the parameters θ, and possibly evaluating the parameter uncertainty based on the observed data. To this end, we need to compute the following posterior 52 Z Z π(θ|y) ∝ π(y|θ)π(θ) = π(y, u|θ)π(θ)du = π(y|u, θ)π(u|θ)du π(θ) (3.43) Instead of explicitly integrating over u, a simple trick yields a different derivation of π(θ|y): π(z|y, θ) π(θ|y) ∝ π(θ)π(y|θ) = π(θ)π(y|θ) , (3.44) π(z|y, θ) for any arbitrary z. Hence, we may pick z = µu|y , and using the fact that u(x) and (x) are Gaussian processes, we can express the posterior π(y|θ) as |Qu |1/2 |Q |1/2   1 T T  π(y|θ) ∝ exp − (µu|y − µ) Qu (µu|y − µ) + (y − Aµu|y ) Q (y − Aµu|y ) , |Qu|y |1/2 2 (3.45) where we recall that µ is the mean of the latent process u(x). Then, the set of variables that parametrize the observation model of Eq. 3.1 can be estimated be maximizing the posterior of Eq. 3.45, as θˆ = arg max π(y|θ) (3.46) θ Moreover, the precision Qu|y , mean µu|y , and variance Var(u|y) can be explicitly expressed as 53 Qu|y = Qu + AT Q A (3.47) µu|y = µ + Q−1 T u|y A Q (y − Aµ) (3.48) Var(u|y) = Q−1 u|y (3.49) Once the optimization problem for identifying the model parameters is solved (Eq.3.46), the kriging predictor and kriging variance of the observation model (Eq. 3.1) are readily available through µu|y , and Var(u|y) (Eqs. 3.48,3.49), respectively. The step that determines the efficiency of the parameter estimation workflow is the inversion of the precision Q−1 u|y in computing µu|y (Eq. 3.48). The most widely used approach is based on the Cholesky decomposition of the precision Qu and can be outlined in the following steps (see Rue and Held (2005)): 1. Rewrite Eq. 3.48 as: µu|y = µ − Q−1 T −1 T −1 −1 u A (AQu A + Q ) (Aµ − y) 2. Compute the Cholesky decomposition of Qu such that: Qu = RT R 3. Use back-substitution to calculate B = Q−1 T −1 T −1 T u A = R (R ) A 4. Compute R2 using the Cholesky decomposition of AB + Q−1 T  = R2 R2 5. Compute C = B(AB + Q−1  ) −1 = BR2−1 (R2T )−1 6. Compute µu|y = µ − C(Aµ − y) Alternatively, if we employ the block covariance method (Eq. 3.36) proposed in the last part of Sec. 3.4.4, we can compute µu|y as: 1. Rewrite Eq. 3.48 as: µu|y = µ − Σu AT (AΣu AT + Q−1 −1  ) (Aµ − y) 54 2. Compute B = Σu AT 3. Compute the Cholesky decomposition of AB + Q−1 T  = R2 R2 4. Compute C = B(AB + Q−1  ) −1 = BR2−1 (R2T )−1 5. Compute µu|y = µ − C(Aµ − y) where in both cases, the size of the matrix (AB + Q−1  ) is determined by the number of observations, hence it is typically expected to be much smaller than the size Qu or Σu themselves. 3.5.1 Maximum Likelihood Estimation with GMRFs In practice, the optimization problem for estimating the model parameters θ (Eq. 3.46) is formulated as a global minimization problem of the negative log-likelihood L: 1 log Qu + log Q − log Qu|y − (µu|y − µ)T Qu (µu|y − µ) − (y − Aµu|y )T Q (y − Aµu|y ) ,  L=− 2 (3.50) where recall that all Qu|y , Qu , Q , µu|y , µ are parametrized by the corresponding entry in the θ vector (see Eq. 3.42). Minimization of L is typically achieved through gradient descent algorithms. Once the solver has converged to a minimum and a set of optimal parameters θˆ has been obtained, it is straightforward to perform kriging predictions and estimate the kriging variance through µu|y , and Var(u|y) (Eqs. 3.48,3.49), respectively. 55 3.6 Multi-fidelity modeling via recursive co-kriging and Gaussian Markov random fields The proposed algorithms can be integrated in the general multi-fidelity informa- tion fusion framework introduced in Chapter 2. Here, we outline the procedure of recasting the problem in terms of GMFRs, and provide an overview of the key implementation aspects. 3.6.1 Ordinary kriging with GMRFs Here we present a brief overview of the kriging predictive scheme leveraging the sparsity of GMRF precision matrices. The reader is referred to Lindgren et al. (2011), Rasmussen (2006) for a detailed exposition of the subject. As usual, we model scattered observations of a field Y (x) as a realization of a Gaussian random field Z(x). These observations could be corrupted by model error or measurement noise that is described by a Gaussian process E(x), leading to an observation model of the form Y (x) = Z(x) + E(x). (3.51) Consequently, the random field (Z(x), Y (x))T has a multivariate normal distribution       T  Z(x)   µ   Σ ΣA  ∼ N  ,  , (3.52)   Y (x) Aµ AT Σ AΣAT + ΣE 56 where, µ = E[Z] is the mean value of Z, Σ is the covariance of Z, ΣE is the covariance of the noise E(x), and A is a simple matrix that restricts the random field Z(x) to the locations of observed data y. Starting from the conditional distribution of Z|Y we can derive equations for the expectation µZ|Y (x), and covariance ΣZ|Y (x) of the ordinary kriging predictor as µZ|Y (x) = µ + (ΣAT )(AΣAT + ΣE )−1 (y − Aµ), (3.53) ΣZ|Y (x) = Σ − (ΣAT )(AΣAT + ΣE )−1 (AT Σ). (3.54) Typically, the covariances Σ and ΣE are parametrized by a set of hyper-parameters, {θ, θE }, that can be learned from the data y by maximizing the posterior π(y|θ, θE ) using Maximum Likelihood Estimation (MLE) or in a fully Bayesian setting (e.g. Markov Chain Monte Carlo integration). This essentially introduces a machine learn- ing problem for computing the optimal values of the unknown parameters {µ, θ, θE } from the data. The noise process E(x) accounts for epistemic uncertainties that lead to differ- ences between our observables and the real response of the system due to inaccurate or uncertain modeling assumptions. In the experimental setting this may correspond to noise or bias in the measurements, while in the numerical setting it may account for round-off or truncation errors of the numerical scheme. In general, modeling the nature of E(x) results in an intrinsically hard problem that is beyond the scope of the present study. Without loss of generality, in what follows we will assume that E(x) is a zero-mean Gaussian white noise process with variance σE , i.e E(x) ∼ N (0, σE I). This should not be confused with aleatoric uncertainties that arise from randomness in the input parameters of the system that will be taken into account by the random vector ξ (see Fig. 2.1, in Chapter 2, Sec. 2.2). 57 3.6.1.1 Ordinary co-kriging with GMRFs The idea of kriging is naturally extended to the multivariate setting, in which the observation model for Y (x) depends on more than one covariates. Here, we present a brief overview of the approach of Kennedy and O’Hagan (2000), which introduced a co-kriging model based on a first-order autoregressive relation between model output of different levels of fidelity. Suppose we have s levels of variable-fidelity model output (yt (x))st=1 at locations s x ∈ Dt=1 sorted by increasing fidelity and modeled as observations of a Gaussian field (Zt (x))st=1 . Then, ys (x) denotes the output of the most accurate and expensive model, while y1 (x) is the output of the cheapest least accurate surrogate at our disposal. The autoregressive co-kriging scheme of Kennedy and O’Hagan (2000) reads as Zt (x) = ρt−1 (x)Zt−1 (x) + δt (x), t = 2, ..., s, (3.55) where δt (x) is a Gaussian field independent of {Zt−1 (x), ..., Z1 (x)}, and distributed with expectation µδt and covariance Σt , i.e. δt ∼ N (µδt , Σt ). Also, ρ(x) is a scaling factor that quantifies the correlation between the model outputs (yt (x), yt−1 (x)). In a Baysian setting, ρ(x) is treated as a random field with an assigned prior distribution that is later fitted to the data through inference. Here, to simplify the presentation we have assume that ρ is a deterministic scalar, independent of x, and learned from the data through maximum likelihood estimation (see Sec. 3.6.2). 58 The derivation of this model is based on the Markov property cov {Zt (x), Zt−1 (x0 )|Zt−1 (x)} = 0, ∀x 6= x0 , (3.56) which, according to Kennedy and O’Hagan (2000), translates into assuming that given Zt−1 (x), we can learn nothing more about Zt (x) from any other model output Zt−1 (x0 ), for x 6= x0 . The resulting posterior distribution at the tth co-kriging level has a mean and covariance given by µZt |yt ,yt−1 ,...,y1 (x) = µt + (Σt At T )(At Σt At T + σt I)−1 (yt − At µt ), (3.57) ΣZt |yt ,yt−1 ,...,y1 (x) = ΣZt + ρ2t−1 ΣZt−1 + ρ2t−1 ρ2t−2 ΣZt−2 + ... + (Πs−1 2 t=1 ρt ) ΣZ1 , where µt is the mean value of Zt (x), Σt is a covariance matrix comprised by t blocks, representing all cross-correlations between levels {t, t − 1, ..., 1}, and ΣZt = Σt − (Σt At T )(At Σt At T + σt I)−1 (ATt Σt ) is the covariance of the co-kriging predictor at level t. Also, σt is the variance of the noise at level t, and At is a simple matrix that restricts a realization of the Gaussian field Zt (x) to the locations of observed data yt (x) at level t. Similarly to ordinary kriging, the set of unknown parame- ters {µt , ρt−1 , ..., ρ1 , θt , ..., θ1 , σt , ..., σ1 } are determined from the data using machine learning techniques. To simplify the presentation, we have assumed that the scaling factor ρ attains scalar values, hence it does not depend on x. Although this co-kriging framework provides an elegant way of blending variable fidelity model output, it may easily become computationally infeasible if many levels of fidelity and/or a large number of data observations are available. The computa- tional barrier is imposed by the need to repeatedly invert the full covariance matrix Σt at the model fitting stage, with the number of different fidelity levels resulting to 59 an increasing matrix size, while large observation data-sets introduce ill-conditioning. These are well known limiting factors for kriging but they become even more evi- dent for co-kriging models where the total number of observations is the sum of the observations at all fidelity levels. However, recent findings of Le Gratiet and Garnier (2014) provide an equivalent formulation that overcomes the computational com- plexity issues of the Kennedy and O’Hagan model by decoupling the aforementioned s-level co-kriging scheme into s independent kriging problems. The resulting recur- sive scheme is proved to provide a predictive mean and variance that is identical to the coupled Kennedy and O’Hagan model, although it could potentially lead to a drastic reduction of the size and condition number of the covariance matrices that need to be inverted. The key idea behind the derivation of Le Gratiet and Garnier (2014) is to replace the Gaussian process Zt−1 (x) in Eq. 3.55, with a process Z˜t−1 (x) that is conditioned by all the known observations {yt−1 , yt−2 , ..., y1 } up to level (t − 1), while assuming that the corresponding experimental design sets (Di )t−1 i=1 have a nested structure, i.e D1 ⊆ D2 ⊆ ... ⊆ Dt−1 . Then, the problem at the tth -level reduces to an ordinary kriging predictor with mean and covariance given by µZt |yt ,yt−1 ,...,y1 (x) = ρt−1 µZt−1 |yt−1 ,...,y1 (x) + µt + + (Σt At T )(At Σt At T + σt I)−1 [yt − At (ρt−1 µZt−1 |yt−1 ,...,y1 (x) + µt )], ΣZt |yt ,yt−1 ,...,y1 (x) = ρ2t−1 ΣZt−1 + Σt − (Σt At T )(At Σt At T + σt I)−1 (ATt Σt ). (3.58) To underline the potential benefits of this approach, we note that the matrix Σt in the Kennedy and O’Hagan model (see Eq. 3.57) has size st=1 nt × st=1 nt , P P where nt is the number of observations at the tth fidelity level. On the other hand, the recursive co-kriging approach involves the inversion of s covariance ma- 60 trices (Σt )st=1 (see Eq. 3.58) of size nt × nt , where nt is the number of observa- tions yt (x) at level t (see Le Gratiet and Garnier (2014)). Moreover, we note that at each recursive level, the number of unknown parameters to be learned from the data reduces to {µt , ρt−1 , θt , σt }, compared to the large parametric set of {µt , ρt−1 , ..., ρ1 , θt , ..., θ1 , σt , ..., σ1 } of the coupled Kennedy and O’Hagan scheme. 3.6.2 Workflow Algorithms 1-4 summarize the main implementation aspects of the proposed multi- level co-kriging framework, formulated using sparse GMRF precision matrices. Namely, we provide pseudocode for the main co-kriging recursion loop (see Algorithm 1), the hyperparameter learning through maximum likelihood estimation (see Algorithm 2), the computation of the negative log-likelihood function of the co-kriging posterior (see Algorithm 3), and the computation of the co-kriging variance (see Algorithm 4). Algorithm 1 Multi-level recursive co-kriging with GMRFs % Loop through all recursion levels (lowest to highest fidelity) for t = 1 : s do % Load input design variables and observations at level t (x ∈ Dt ) x ← input design data and prediction points At ← simple restriction matrix (see Sec:3.6.1.1) yt (At x) ← observed data % Compute the mass and stiffness matrices of a finite element (FEM) % discretization of the design space [M, S] ← FEM discretization() % Learn the co-kriging hyperparameters {µt , ρt−1 , κt , τt , σt } through MLE {θopt , µopt , Q−1 c } ← MLE optimization(x, yt , At , M, S, t) % Evaluate the co-kriging predictor and variance at level t from Eq.3.58 % and proceed to the next level t = t + 1 µZt |yt ,yt−1 ,...,y1 ← µopt ΣZt |yt ,yt−1 ,...,y1 ← Variance(θopt , Q−1 c , t) % Finally, return the co-kriging predictor and variance at the highest level of fidelity return µZs |ys ,ys−1 ,...,y1 , ΣZs |ys ,ys−1 ,...,y1 61 Algorithm 2 Maximum likelihood estimation for hyperparameter learning procedure MLE optimization(x, yt , At , M, S, t) if t > 1 then y ? ← yt − ρt−1 At µZt−1 |yt−1 ,yt−2 ,...,y1 µt ← µt + ρt−1 µZt−1 |yt−1 ,yt−2 ,...,y1 θ? ← {µt , ρt−1 , κt , τt , σt } else y ? ← yt θ? ← {µt , κt , τt , σt } % Learn the optimal hyperparameters that minimize the negative % log-likelihood of the co-kriging posterior {θopt , µopt , Q−1 c } ← min ? NegLogLikelihood(θ? , y ? , At , M, S) θ return {θopt , µopt , Qc−1 } Algorithm 3 Computation of the negative log-likelihood of the co-kriging posterior procedure NegLogLikelihood(θ? , y ? , At , M, S) % Construct the discrete Helmholtz operator of the SPDE (see Eq. 3.12) H ← τ12 (κ2t M + S) % Construct the sparse precision matrix Qt (see Eq. 3.21), % and the diagonal noise precision QE Qt ← HM −1 H QE ← σ1t I % Factorize Qt = U U T using the Cholesky decomposition U ← Cholesky(Qt ) % Compute the conditional precision Qc Qc ← Qt + ATt QE At % Compute the conditional predictor µ? , using the Woodbury matrix identity % and Cholesky back-substitution to efficiently compute Q−1 c µ? ← µt + Q−1 c A Q t E (y ? − A µ t t ) 1 NegLogLikelihood ← − {log |Qt | + log |QE | − log |Qc |− 2 − (µ? − µt )T Qt (µ? − µt ) + (y ? − At µ? )T QE (y ? − At µ? ) }  Return {NegLogLikelihood, µ? , Q−1 c } Algorithm 4 Computation of the co-kriging variance procedure Variance(θopt , Q−1 c , t) if t > 1 then return diag(ρ2t−1 ΣZt−1 |yt−1 ,yt−2 ,...,y1 + Q−1 c ) else return diag(Q−1c ) 62 3.7 Results In this section we present representative results for using the proposed multi-fidelity information fusion with GMRFs in different contexts. First, we test the accuracy of the nodal spectral element basis, and consider a problem of gappy field reconstruction from a deterministic 2D Helmholtz equation. Next, we present results on a prototype benchmark in risk-averse design, where a stochastic optimization problem is solved to identify the optimal design of a structural element. Then, we consider a test problem involving the regression of the mean field of a random function when a family of low-fidelity representations is available. Finally, we conclude with an uncertainty quantification problem in fluid mechanics that involves stochastic laminar wakes past a circular cylinder. 3.7.1 Helmholtz equation in 2D 3.7.1.1 Convergence rate First, we demonstrate the exponential convergence rate of the nodal spectral element discretization described in Sec. 3.4. To this end, we consider the Helmholtz equation in a bounded two-dimensional domain Ω ∇2 u − λu = f, (3.59) where λ is a prescribed constant, f is a known forcing term, and u is the unknown scalar field we wish to solve for. At the domain boundary, we assume Dirichlet boundary conditions u|∂Ω = u0 (x). Here, we have considered a square domain, Ω = 63 {(x, y) : 0 ≤ x ≤ 2, −1 ≤ y ≤ 1}, discretized using four equi-spaced quadrilateral spectral elements, with 2 elements in each direction. Choosing λ = 100 and f = −(λ + 2π 2 ) cos(πx) cos(πy) we can obtain the analytic solution to the Helmholtz problem as u = cos(πx) cos(πy) (3.60) Using the same polynomial expansion order in each element, we vary the poly- nomial order systematically and compute the corresponding L∞ and L2 errors of the numerical solution against the analytical solution. At the domain boundaries the numerical solution is set to the analytic expression of Eq. 3.60. Figure 3.2 shows the computed numerical solution for polynomial expansion order P = 5, and the corresponding exponential error decay as a function of the element order. Convergence rate 0 10 L∞ L21 H −5 10 Error −10 10 −15 10 2 4 6 8 10 12 n Figure 3.2: Helmholtz equation in 2D: Exponential convergence rate of the nodal spectral element discretization for a Helmholtz equation in 2D. The error between the numerical and the exact solution has been computed for different numbers of expansion modes n, using the L∞ , L2 , and H 1 norms, respectively. 64 3.7.1.2 Gappy field reconstruction using GMRF kriging Having demonstrated the convergence properties of the nodal spectral element ba- sis, we now focus our attention on employing the observation model of Eq. 3.1 for performing reconstruction of gappy fields. In what follows, we consider the case of ordinary kriging, in which the mean of the latent model u(x) is assumed to be unknown but independent of x, although any other form of kriging is also directly applicable. The latent process u(x) is assumed to satisfy the SPDE of Eq 3.6, pos- sibly subject to any a-priory known boundary conditions. As discussed in Sec. 3.4, the SPDE under the nodal spectral element discretization generates a GMRF with sparse precision. Again, we consider the 2D Helmholtz problem of Eq. 3.59, for which an exact solution is readily available. The goal here is to train the prediction model of Eq. 3.1 based on a set of known observations such that it is able to generate a reasonably ac- curate reconstruction of the solution to the Helmohlotz problem. Here, we introduce gappyness in the interior of the domain, while the SPDE satisfies the exact solution through Dirichlet boundary conditions at all four boundaries of the square. Pa- rameter estimation has been performed using the Maximum Likelihood Estimation (MLE) methodology outlined in Sec. 3.5.1. Figures 3.3-3.6 and Table 3.1 show the size, structure, sparsity pattern and condi- tion number of the computed mass, stiffness, precision and homogeneous-homogeneous Schur complement matrices for different number of expansion modes n. The results correspond to 4-element quadrilateral nodal spectral element discretization of the SPDE (Eq. 3.6). We observe that the size and condition number of SHH exhibit good scaling with increasing polynomial order and are considerably smaller than the condition number and size of the precision Qu . This fact may favor the block co- 65 variance approach suggested in Sec. 3.4 for large problems where the large condition number of Qu may prohibit the use of Cholesky decomposition, as the covariance can then be computed by inverting the homogeneous-homogeneous Schur comple- ment SHH . n Qu SHH 4 10 Qu 3 25 5 SHH 3 4 49 9 Condition number 10 2 10 5 81 13 7 169 21 1 10 9 289 29 0 10 0 5 n 10 15 12 529 41 Figure 3.3: Helmholtz equation in 2D: Table 3.1: Helmholtz equation in 2D: Condition number of the precision Qu and Leading size dimension of the precision homogeneous-homogeneous Schur com- Qu and homogeneous-homogeneous Schur plement SHH for different number of ex- complement SHH matrices for different pansion modes n. number of expansion modes n. H Qu SHH 0 0 0 2 10 10 4 20 20 6 0 10 20 0 10 20 0 2 4 6 nz = 145 nz = 369 nz = 21 Figure 3.4: Helmholtz equation in 2D: Sparsity patterns of the Helmholtz, preci- sion and homogeneous-homogeneous Schur complement matrices for a second-order polynomial expansion basis (n = 3). 66 H Qu SHH 0 0 0 10 10 20 20 5 30 30 40 40 50 50 10 0 20 40 0 20 40 0 5 10 nz = 385 nz = 1213 nz = 65 Figure 3.5: Helmholtz equation in 2D: Sparsity patterns of the Helmholtz, preci- sion and homogeneous-homogeneous Schur complement matrices for a third-order polynomial expansion basis (n = 4). H Qu SHH 0 0 0 5 50 50 10 100 100 15 150 150 20 0 50 100 150 0 50 100 150 0 10 20 nz = 2353 nz = 11281 nz = 341 Figure 3.6: Helmholtz equation in 2D: Sparsity patterns of the Helmholtz, preci- sion and homogeneous-homogeneous Schur complement matrices for a sixth-order polynomial expansion basis (n = 7). In Fig. 3.7 we present the effect of increasing the the number of modes in the basis expansion on the accuracy of the resulting kriging predictor. To this end we fix a single observation point located in (0, 1) and increase the polynomial order of the nodal basis expansion that discretizes Eq. 3.6. We observe that as the polyno- mial order is gradually increased from 3 to 9, the kriging predictor becomes up to 3 orders of magnitude more accurate, clearly indicating the benefits of high-order approximation. Moreover, note that as the generated predictor automatically satis- 67 fies the boundary conditions, only one observation point at the center of the domain is sufficient to accurately reconstruct the unknown field. Alternatively, in Fig. 3.7 we fix the number of expansion modes to n = 7, and we observe the increase of the predictor’s accuracy, accompanied by the decrease in the kriging variance, as we add randomly sampled observations. 3.7.2 A prototype problem in risk-averse design Consider a rectangular cross section of a short structural column, with depth x1 and width x2 , under uncertain yield stress and uncertain bending and axial loads. Assuming an elastic-perfectly plastic material, a limit state function that quantifies a relationship between loads and capacity is described by the random field (see Kuschel and Rackwitz (1997)) 4ξ1 ξ22 Y (x1 , x2 ; ξ1 , ξ2 , ξ3 ) = + − 1, (3.61) x21 x2 ξ3 x21 x22 ξ32 where the bending moment load ξ1 , the axial load ξ2 , and the material yield stress ξ3 are distributed as ξ1 ∼ N (2000, 400) ξ2 ∼ N (500, 100) ξ3 ∼ log N (5, 0.5). Note that lower values of Y indicate more favorable capacity-to-load ratios, while high values correspond to critical regimes where failure may occur. Our aim here is to identify an optimal configuration (x?1 , x?2 ) in the two-dimensional design space 68 Exact Kriging predictor L∞ error !"' !"&# !"& !"%# y y y !"% !"$# !"$ !"!# ! x x x !"'& !"'% !"'$ !"'# !"' y y y !"!& !"!% !"!$ !"!# x x x !"!) !"!( !"!' !"!& y y y !"!% !"!$ !"!# x x x !) '(%& %" %! %& $ y y y # " ! x x x Figure 3.7: Helmholtz equation in 2D: Observation points, kriging prediction and pointwise L∞ error for a single observation point located at (1,0). Top to bottom: Nodal spectral element discretization with 4 quadrilateral elements and 3,5,7, and 9 expansion modes, respectively. 69 Exact Kriging predictor Kriging variance '(&! "! &% &$ &# &" y y y &! % $ # " ! x x x !"!# !"!'& !"!'% !"!'$ y y y !"!'# !"!' !"!!& !"!!% !"!!$ !"!!# ! x x x ! '(&! &" &! % y y y $ # " ! x x x !) '(&! &" y y y &! " ! x x x Figure 3.8: Helmholtz equation in 2D: Observation points, kriging prediction and kriging variance for a nodal spectral element discretization with 4 quadrilateral el- ements and n = 7 expansion modes. Top to bottom: Increasing the number of randomly selected observations from 1,5,10, to 15, respectively. 70 D1 × D2 = [2, 20] × [2, 10] that satisfies a set of prescribed criteria. Here, we let this goal translate to identifying a design that minimizes the area of the column with the associated design risk staying below a given threshold c. To quantify the risk associated with a given response Y (x; ξ), we employ a superquantile risk measure, which for a parameter α ∈ [0, 1) is defined as (see Rockafellar and Royset (2014)):   1 Rα (Y (x; ξ)) = min c + E[max{Y (x; ξ) − c, 0}] . (3.62) c∈R 1−α For α = 0, the superquantile risk measure of a continuously distributed random field Y (x; ξ), reduces to the expectation operator, i.e., R0 = E[Y (x; ξ)], thus char- acterizing a risk-neutral situation. However, for non-zero values α ∈ (0, 1), the superquantile risk of Y (x; ξ) returns the average of the (1 − α)% highest realizations of Y (x; ξ), therefore informing a risk-averse decision making mindset (see Rockafellar and Royset (2014)). The design problem can be formulated in a constrained minimization framework which reads as min x1 x2 x1 ∈D1 ,x2 ∈D2 (3.63) subject to Rα (Y ) ≤ c The expectation operator E[max{Y (x; ξ)−c, 0}] appearing in the computation of the superquantile risk Rα (Y (x; ξ)) can be approximated using a variety of different probabilistic methods of different cost and fidelity. Here, we choose to discretize the three dimensional parametric space of (ξ1 , ξ2 , ξ3 ) using two different models. The high-fidelity model is a probabilistic collocation method (PCM) (see Foo et al. (2008)) on a tensor product grid with 10 quadrature points along each random dimension (total of 103 quadrature points), while the low-fidelity model is a Smolyak sparse grid level-2 quadrature (SG-L2) (see Novak and Ritter (1996)) with a total of only 71 58 quadrature points. This setup yields a simple application of the proposed multi-fidelity methodology, where we have a single model in physical space (Eq. 3.61) and two levels of fidelity in probability space (PCM, SG-L2). Employing a two-level recursive co-kriging algo- rithm (see Sec. 3.6.1.1) we can explore the correlations between the two probabilistic models, and build an accurate representation of the target response surface Rα (Y ) by mainly utilizing the realizations of the lower-fidelity model (SG-L2, 120 points in design space), guided by few realizations of the higher fidelity model (PCM, 10 points in design space). In Fig. 3.9, we present the computed response surface R0.8 (Y ) as a function of the design variables (x1 , x2 ) considering the 20% highest realizations of Y . The exact solution corresponds to computing R0.8 (Y ) using 106 Monte Carlo samples (see Binder and Heermann (2010)). The resulting response surface becomes singular as (x1 → 0, x2 → 0), while it attains a very flat profile in the rest of the design space. This flat behavior suggests that the identification of an optimal design as a solution to Eq. 3.63 is highly sensitive to the accuracy of the computed response surface. Here, we have chosen a design criterion of R0.8 (Y ) ≤ −0.99, yielding an optimal solution for (x?1 = 13.94, x?2 = 5.07) that does not depend on further resolution refinement. Although this is a pedagogical prototype case in risk-averse design under uncer- tainty, it does not fully reflect the strengths of the proposed multi-fidelity framework as the two probabilistic methods (PCM, SG-L2) both happen to produce very similar estimations of the mean E[Y ]. The next example illustrates better the capabilities of the proposed methodology in the context of multi-fidelity regression of a random function. 72 Co-kriging absolute error 10 0.8 0.8 9 0.7 8 0.6 0.6 7 0.5 6 Co-kriging Co-kriging x2 5 0.4 0.4 Exact Exact (106(10 MC samples) 4 0.3 3 0.2 0.2 2 0.1 1 5 10 15 20 20 0.0 0 R0.8 (Y ) x1 High-fidelity data Low-fidelity data (PCM, 10 points) (SG-L2, 120 points) (x⋆1 , x⋆2 ) x2 x1 Figure 3.9: A prototype problem in risk-averse design: Exact response surface R0.8 (Y ) and co-kriging predictor constructed using 120 low-fidelity (SG-L2) and 10 high-fidelity (PCM) observations. (x?1 , x?2 ) denotes the optimal solution to the optimization problem of Eq. 3.63, while the inset plot shows the point-wise absolute error of the co-kriging predictor. 73 3.7.3 Multi-fidelity regression of a random function Here, we consider a system with two input design variables x = (x1 , x2 ), subject to external uncertainties, described by four standard normal random variables ξ = (ξ1 , ξ2 , ξ3 , ξ4 ). Let the response of this system, denoted by Y (x; ξ) be described by the random function 2 fe (x; ξ) = ξ1 sin2 (5ξ1 x1 + 2ξ2 ξ3 x2 ) + 2ξ42 e−x1 (ξ2 +ξ3 )(x2 −0.5) cos2 (4x1 + x2 ). (3.64) Now, assume that fe (x; ξ) returns the real high-fidelity response but is expensive to evaluate. On the other hand, let fc (x; ξ) be a low-fidelity cheap to evaluate surrogate model that we can sample extensively as fc (x; ξ) = 1.7fe (x; ξ) + 2ξ1 ξ2 sin(x1 + x2 ) + 5ξ3 ξ42 e−x1 sin(x1 + 7x2 ). (3.65) Our goal here is to employ the proposed multi-fidelity framework to construct the response surface of the mean field S(x) = E[Y (x; ξ)]. In order to approximate the expectation operator, we employ two methods of different fidelity in probability space. To this end, we choose our high-fidelity probabilistic method to be a Smolyak sparse grid level-5 quadrature (SG-L5) (see Novak and Ritter (1996)) that discretizes the four-dimensional parameter space using 4,994 quadrature points. Similarly, the low-fidelity method in probability space is a coarser, level-2 sparse grid quadrature (SG-L2) with just 57 quadrature points (see Novak and Ritter (1996)). Therefore, our multi-fidelity setup consists of two models in physical space (fe (x; ξ), fc (x; ξ)), and two models in probability space (SG-L5, SG-L2). This results in a family of re- 74 sponse surfaces Sij , that can be organized as      S11 S12   ESG-L2 [fc (x; ξ)] ESG-L5 [fc (x; ξ)]   =  (3.66) S21 S22 ESG-L2 [fe (x; ξ)] ESG-L5 [fe (x; ξ)] Fig. 3.10 demonstrates the response surface produced using a 4-level recursive co- kriging scheme, traversing the available models and data in the order S11 → S12 → S21 → S22 , i.e., from lowest to highest fidelity. The resulting response surface is compared with an “exact” solution that is obtained by Monte Carlo integration of the high-fidelity physical model fe (x; ξ) using 106 samples. Evidently, the highly non-linear response of the mean field is captured remarkably well by just using five observations of the expensive highest-fidelity model S22 , supplemented by a number of inaccurate low-fidelity observations from (S11 , S12 , S21 ). This observation is further confirmed by the uncertainty of the predictor, quantified by the co-kriging variance (see Fig. 3.10, inset), which is bounded by 10−3 . 3.7.4 Stochastic incompressible flow We consider two-dimensional (2D) unsteady incompressible flow past a circular cylin- der of diamterer D, subject to a random inflow boundary condition. The flow is governed by the Navier-Stokes equations subject to the incompressibility constraint   ∂v + v · (∇v) = −∇p + ν∇2 v   ∂t (3.67)  ∇ · v = 0  75 Co-kriging variance −4 ×10 +,!* ' 10 !* !"+ ) !"& 8 ( Exact Exact (106(10 MC samples) !"* ' Co-kriging Co-kriging !"% 6 & SS11 data (80 points) x2!") % SS12 data (40 points) !"$ 4 $ SS21 data (10 points) !"( !"# # 2 " SS22 data (5 points) !"' ! S12 ! !' 0 ! !"# !"$ x1 !"% !"& S22 E[Y ] S11 S21 x1 x2 Figure 3.10: Multi-fidelity regression of a random function: Exact response surface S = E[Y (x; ξ)] and co-kriging predictor constructed using four levels of fidelity: 80 S11 points (fc (x; ξ), SG-L2), 40 S12 points (fc (x; ξ), SG-L5), 10 S21 points (fe (x; ξ), SG-L2), and 5 S22 points (fe (x; ξ), SG-L5). The inset plot shows the point-wise variance of the co-kriging predictor. 76 where v is the fluid velocity vector, p is the fluid pressure, t is time, and ν is the kine- matic viscosity of the fluid. The system is discretized in space using the spectral/hp element method (SEM), according to which the computational domain is decomposed into a set of polymorphic non-overlapping elements (see Karniadakis and Sherwin (2013)). Within each element, the solution is approximated as a linear combination of hierarchical, mixed-order, semi-orthogonal Jacobi polynomial expansions Karni- adakis and Sherwin (2013). Temporal integration is based on decoupling the velocity and pressure by applying a high-order time-splitting scheme (see Karniadakis and Sherwin (2013)). The random inflow has a parametric expression of the from  πy  h  πy   πy i U∞ (σ1 , σ2 ; ξ1 , ξ2 ) = 1 + σ1 sin + σ2 ξ1 sin + ξ2 cos . (3.68) 9 9 9 where (σ1 , σ2 ) are parameters controlling the amplitude of the skewness of the inflow noise (design variables), y is the coordinate transverse to the flow, and (ξ1 , ξ2 ) are random variables with standard normal distributions. Our goal here is to construct the response surface for the 0.6-superquantile risk of the base pressure coefficient R0.6 (CBP ) (see Fig. 3.11) at Reynolds number Re = U∞ D ν = 100. The proposed multi-fidelity modeling framework is illustrated here considering a single model in physical space and three models in probability space. The physical model returns realizations of the flow field solution produced by direct numerical simulations of Eq. 3.67 using the spectral/hp element method. All simulations are started from a zero-velocity initial condition, and are integrated until a stable limit- cycle state is reached. This state is characterized by the well known von-Karman 77 5 pB − p∞ y0 CBP = 1 2 2 ρU∞ (a) –5 –10 0 x 10 20 Figure 3.11: Incompressible stochastic flow past a cylinder: (Left) The computational mesh consists of 412 triangular spectral elements of polynomial order 11. (Right) Standard deviation of the stream-wise velocity component at time t = 4. The quantity of interest here is the base pressure coefficient CBP , where pB denotes the fluid pressure on the cylinder surface at 180o from the stagnation point (base pressure), while U∞ and p∞ are the free-stream velocity and pressure, respectively. vortex shedding pattern in which the unsteady separation of the flow around the cylinder gives rise to a time-periodic base pressure signal. We choose three models in probability space to compute the expectation operator in Eq 3.62. The highest fidelity model is probabilistic collocation on a tensor product grid (PCM) (see Foo et al. (2008)), the intermediate fidelity is a Smolyak sparse grid level-2 quadrature (SG-L2) Novak and Ritter (1996), and lowest fidelity model is Monte Carlo inte- gration (MC) (see Binder and Heermann (2010)). For each model, we construct a uniform grid in the (σ1 , σ2 )-plane, and perform an ensemble of simulations at each grid-point according to the corresponding sampling strategy for integrating out (ξ1 , ξ2 ) (see Table 3.2). Fig. 3.12 illustrates the PCM samples of the inflow boundary condition for different values of (σ1 , σ2 ). This uncertainty quantification task results to a total of 5,526 Navier-Stokes simulations that were performed in parallel on one rack of IBM BG/Q (16,384 cores) with a CPU time of about 40 minutes. 78 σ1 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.025 σ2 0.050 0.075 0.1 Figure 3.12: Incompressible stochastic flow past a cylinder: Probabilistic collocation (PCM) samples of the velocity inflow profile on a uniform 6×5 (σ1 , σ2 )-grid. At each grid point, a total of 64 samples is used to integrate out the randomness introduced by (ξ1 , ξ2 ) in Eq. 3.68. Note that σ1 controls the skewness of the inflow profile, while σ2 acts as a noise amplifier. Method (σ1 , σ2 )-grid Samples/grid-point Total samples PCM 6×5 64 1920 SG-L2 11 × 9 22 2178 MC 21 × 17 4 1428 Table 3.2: Stochastic incompressible flow: Size of the uniform discretization grid in the (σ1 , σ2 )-plane, and number of samples per grid-point for each of the probabilistic methods employed. Fig. 3.13 shows the response surface for the 0.6-superquantile risk of the base pressure coefficient R0.6 (CBP ), computed using a 3-level recursive co-kriging. The predictive scheme is trained using 100 low-fidelity (MC), 15 intermediate fidelity (SG- L2), and only 5 high-fidelity (PCM) observation points in the (σ1 , σ2 )-plane. The 79 accuracy of the co-kriging predictor is assessed against an “exact” response surface constructed by using 357 uniformly distributed observations of the highest fidelity model (PCM), returning an error of 0.051 in the L2 norm. More importantly, the uncertainty of the co-kriging predictor is negligible, as its variance is firmly bounded below 3 × 10−5 . Note that if we only used a 1-level kriging scheme trained on the 5 high-fidelity observations, the resulting L2 is 3.536. It is evident that incorporating information from lower fidelity sources can drastically improve the accuracy of the metamodel. 3.8 Discussion The scope of this work is to define a comprehensive mathematical framework that allows for the seamless integration of surrogate-based optimization and uncertainty quantification in engineering design problems. The proposed methodology targets the construction of response surfaces of complex stochastic dynamical systems by blending multiple information sources via auto-regressive stochastic modeling. We develop a general setting, under which multi-fidelity in models and multi-fidelity in probability space can coexist in tandem. A computationally efficient framework is developed using multi-level recursive co-kriging and Gaussian Markov random fields, resulting in very robust schemes in two- and three-dimensions. The capabilities of our implementation were tested for four different cases. In all cases, we were able to successfully exploit cross-correlations within ensembles of surrogate models and efficiently construct an accurate estimate for the response of the given deterministic or stochastic system. 80 Co-kriging variance −5 ×10 +,!* 3 !* ) ( ' 2 Exact (357 PCM samples) & σ2 % Co-kriging $ MC data (100 points) 1 # SG-L2 data (15 points) " ! PCM data (8 points) 0 σ1 ! PCM R0.6 (CBP ) MC SG-L2 σ2 σ1 Figure 3.13: Incompressible stochastic flow past a cylinder: Response surface of the 0.6-superquantile risk of the base pressure coefficient R0.6 (CBP ), computed using a 3- level recursive co-kriging scheme, on 100 low-fidelity (MC), 15 intermediate-fidelity (SG-L2), and 5 high-fidelity (PCM) observation points in the (σ1 , σ2 )-plane. The exact surface corresponds to 357 uniformly distributed observations of the highest fidelity model (PCM). The inset plot shows the point-wise variance of the co-kriging predictor. 81 A limiting factor of our presentation stems from the fact that all demonstrations were confined to two-dimensional problems. Although this significantly contributes towards enhancing the clarity of the presented concepts, we acknowledge that it does not establish a direct link to realistic high-dimensional design problems. At this point we should highlight that one may construct latent variable models through Principal Component Analysis (PCA), non-linear PCA, and projection-pursuit methods (see Hastie et al. (2009)), thus performing a data-driven dimensionality reduction that yields a new set of input coordinates that spans a space of lower dimension. Chapter Four Multi-fidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data-sets 83 4.1 Motivation In pursuit of studying the response of complex engineering and physical systems one has to inevitably face the challenge of interpreting vast amounts of information and quantify the effect of a large number of decision/input variables on the system’s output. Decision making in such data-rich, yet budget-constrained environments, suggests the adoption of an information-driven, machine learning mindset that en- ables a resourceful way of combining versatile tools, ranging from experiments to stochastic simulations, ultimately shaping new frontiers in data analytics, surrogate- based modeling, design optimization, and beyond. Ever since the pioneering work of Sacks et al. (1989), the use of surrogate models for the design and analysis of computer experiments has undergone great growth, establishing Gaussian process regression (GPR) as a general and flexible tool for building inexpensive predictive schemes that are capable of emulating the response of complex systems. Furthermore, the use of GPR within auto-regressive stochastic models, such as the widely used scheme put forth by Kennedy and O’Hagan (2000) and the efficient recursive implementation of Le Gratiet and Garnier (2014), allows for exploring spatial cross-correlations between heterogeneous information sources. In Chapter 2 we argued that this offers a general platform for developing multi- fidelity information fusion algorithms that simultaneously account for variable fi- delity in models (e.g., high-fidelity direct numerical simulations versus low-fidelity empirical formulae) as well as variable fidelity in probability space (e.g., the high- fidelity tensor product multi-element probabilistic collocation of Foo et al. (2008) versus the low-fidelity sparse grid quadratures of Novak and Ritter (1996)). Al- though this construction is appealing to a wide range of applications, it is mainly limited to low-dimensional input spaces and moderately sized data-sets. 84 A common strategy for constructing auto-correlation models for GPR in high di- mensions is by taking the product of one-dimensional autocorrelation kernels. This typically results to an anisotropic covariance model, which assumes that all dimen- sions actively interact with each other. However, as the dimensionality is increased, one would hope to find sparsity in the input space, i.e. dimensions with negligible or very weak pairwise interactions. This observation has been widely studied in the literature and has motivated the use of additive models by Hastie and Tibshirani (1990). Durrande et al. (2011) have recently adopted this approach in the context of GPR, advocating versatility in constructing custom auto-correlation kernels that respect the structure in the observed data. This suggests that, having a way to quantify the active interactions in the data, one can tailor an auto-correlation model that closely adheres to those trends. To this end, Muehlenstaedt et al. (2012) have employed functional ANOVA decompositions to compute the degree to which each input dimension, and their pairwise interactions, contribute to the total variability in the observations, and used the corresponding sensitivity indices to construct an undirected graph that provides insight into the structure of possible additive auto- correlation kernels that best suit the available data. Although, this approach is evidently advantageous for scaling GPR to high dimensional problems, it may still suffer from computational tractability issues in presence of big data-sets. In general, the design of predictive inference schemes in high-dimensions suffers from the well known curse of dimensionality, as the number of points needed to explore the input space in its entirety increases exponentially with the dimension. This implicit need for big data introduces a severe deadlock for scalability in ma- chine learning algorithms as they often involve the repeated inversion of covariance matrices that quantify the spatial cross-correlations in the observations. This defines the so called “big N ” problem - an expression used to characterize the demanding 85 operational count associated with handling data-sets comprised of N observations (N > 1000). The implications of such large data-sets on learning algorithms are well known, leading to a O(N 3 ) scaling for implementations based on maximum likeli- hood estimation. Addressing this challenge has received great attention over the last decades and several methods have been proposed to alleviate the computational cost (see Dietrich and Newsam (1997), Lindgren et al. (2011), Stein et al. (2013)). In Chapter 3 we presented a possible solution path via leveraging the sparsity of GMFR representations. However, in spaces with more than 3 input dimensions this sparsity is lost as the construction of finite element discretizations becomes intractable. Here, we will focus our attention on the frequency-domain learning approach recently put forth by De Baar et al. (2013) that entirely avoids the inversion of covariance matri- ces at the learning stage and is applicable to a large class of wide-sense stationary auto-correlation models. This essentially enables the development O(N ) algorithms, hence opening the path to predictive inference on massive data-sets. The aim of this work is to build upon, and further extend, the mathematical framework put forth in Chapter 2, enabling multi-fidelity information fusion for high dimensional systems and in view of massive data-sets. The building block of our con- struction is GPR and multi-fidelity regression via recursive co-kriging (see Le Gratiet and Garnier (2014)). Motivated by the work of Muehlenstaedt et al. (2012) we pursue scaling to high dimensions by employing hierarchical functional decompositions that reveal structure in the data and inspire a graph-theoretic approach for constructing customized auto-correlation kernels that exploit sparsity in the space of inputs. To this end, we propose a novel, data-driven dimensionality reduction technique based on local projections, and we utilize the Fourier projection-slice theorem (see Levoy (1992)) to decompose the high-dimensional machine learning problem into a series of tractable, low-dimensional problems that can be solved in parallel using O(N ) fast 86 learning algorithms in the frequency domain. This chapter employes and further extend the methods presented in Chapter 2, and is structured as follows. In Sec. 4.2.2 we provide an overview of the frequency- domain approach of De Baar et al. (2013) that bypasses the shortcomings of maxi- mum likelihood estimation and enables fast learning from massive data-sets. Subse- quently, in Sec. 4.3 we elaborate on kernel design in high dimensions. In particular, we outline the a data-driven hierarchical functional decomposition based on RS- HDMR expansions (see Rabitz et al. (1999)), and describe a graph-theoretic approach inspired by Muehlenstaedt et al. (2012) for tailoring GPR auto-correlation kernels to the data. Moreover, we discuss how to decompose the global high-dimensional learning problem to a series of local solves using a projection-based dimensionality reduction technique in conjunction with the Fourier projection-slice theorem (see Levoy (1992)). In Sec. 4.4 we conclude with a summary of the proposed workflow, underlining key implementation aspects. The capabilities of the proposed method- ology are demonstrated through three benchmark problems. First, in Sec. 4.5.1 we employ a multi-fidelity modeling approach for constructing the mean field response of a stochastic flow through a borehole. Second, in Sec. 4.5.2 we present results on the regression of a deterministic function in 100 input dimensions by blending information from three variable fidelity observation models. Lastly, in Sec. 4.5.3 we present an extreme case of performing kriging in 1,000 input dimensions and 105 data-points. 87 4.2 Machine learning in the frequency domain 4.2.1 Bottlenecks of Maximum Likelihood Estimation Recall the expression for the recursive co-kriging predictor, variance, and likelihood, respectively, presented in Chapter 2 (see Eq. 2.5, 2.6, 2.7) yˆt (x?t ) = µ ˆ2t I)−1 [yt (xt ) − 1ˆ ˆt + ρˆt−1 yˆt−1 (x?t ) + rtT (Rt + σ µt − ρˆt−1 yˆt−1 (xt )], (4.1) [1 − rtT (Rt + σ ˆ2t I)−1 rt ]2   2 ? 2 2 ? 2 T 2 −1 st (xt ) = ρˆt−1 st−1 (xt ) + σ ˆt 1 − rt (Rt + σ ˆt I) rt + , 1Tt (Rt + σ ˆ2t I)−1 1t (4.2) n 1 L({µt , σt2 , σ2t , ρt−1 , θt }) = log(σt2 ) + log |Rt (θt ) + σ2t I|+ (4.3) 2 2 1 + [yt (xt ) − 1t µt − ρt−1 yˆt−1 (xt )]T [Rt (θt ) + σ2t I]−1 [yt (xt ) − 1t µt − ρt−1 yˆt−1 (xt )], 2σt2 The computational cost of calibrating model hyper-parameters through maxi- mum likelihood estimation (MLE) is dominated by the inversion of correlation ma- trices (Rt + σ2t I)−1 at each iteration needed for the maximization of the likelihood function in Eq. 4.3. The inversion is typically performed using the Cholesky de- composition that scales as O(Nt3 ), leading to a severe bottleneck in the presence of moderately big data-sets. This is typically the case for high-dimensional prob- lems where abundance of data is often required for performing meaningful inference. This pathology is further amplified in cases where the noise variance σ2t is negligible and/or the observed data points are tightly clustered in space. Such cases introduce 88 ill-conditioning that may well jeopardize the feasibility of the inversion as well as pol- lute the numerical solution with errors. Moreover, if an anisotropic correlation kernel κt (xt , x0 t ; θt ) is assumed, then the vector of correlation lengths θt is d-dimensional, leading to an increasingly complex optimization problem as the dimensionality of the input variables xt increases. These shortcomings render the learning process intractable for large data-sets, and suggest seeking alternative routes to parameter estimation. Next, we describe a method that bypasses the deficiencies of MLE and enables the development of fast learning algorithms that scale linearly with the data. 4.2.2 Frequency-domain sample variogram fitting Following the approach of De Baar et al. (2013) we employ the Wiener-Khinchin theorem to fit the auto-correlation function of a wide-sense stationary random field to the power spectrum of the data. The later contains sufficient information for ex- tracting the second-order statistics that fully describe the Gaussian predictor Zt (x). Therefore, the model hyper-parameters at each inference level t can be learned in the frequency domain by fitting the Fourier transform of the sample variogram as n X min | log[wˆt (ξi )]2 − log [ˆ at (ξi ; σ2t , θt )]|2 , (4.4) {σ2t ,θt } i=1 2 where ξ is a d-dimensional array of frequencies, [w(ξ)] ˆ is the amplitude of the Fourier ˆ(ξ; σ2t , θt ) is the Fourier coefficients in the modal representation of the data yt (x), a transform of the auto-correlation function {κt (xt , x0 t ; θt ) + σ2t δ(||xt − x0 t ||)}, with δ(·) denoting the Dirac delta function, and || · || measures distance in an appropriate norm. The Fourier coefficients w(ξ) ˆ can be efficiently computed with O(Nt log Nt ) cost using the fast Fourier transform (FFT) for regularly spaced samples, or the 89 non-uniform fast Fourier transform (NUFFT) (see Greengard and Lee (2004)) for irregularly spaced samples. Moreover, for a wide class of auto-correlation functions, ˆ(ξ; σ2t , θt ) is analytically available, thereby each evaluation the Fourier transform of a of the objective function in the minimization of Eq. 4.4 can be carried out with a linear cost, i.e. O(Nt ). This directly circumvents the limitations of hyper-parameter learning using MLE approaches, namely the cubic scaling associated with invert- ing dense ill-conditioned correlation matrices, and therefore it enables parameter estimation from massive data-sets. Although the Wiener-Khinchin theorem relies on the assumption of stationarity, modeling of non-stationary response can be also accommodated by learning a bi- jective warping of the inputs that removes major non-stationary effects (see Snoek et al. (2014)). This mapping essentially warps the inputs into a jointly stationary space, thus allowing the use of standard wide-sense stationary kernels that enable fast learning in the frequency domain. A limitation of frequency-domain sample variogram (FSV) fitting is that the summation in Eq. 4.4 is implicitly assumed to take place over all dimensions, i.e. over all Ntd frequencies in ξ. Although this is tractable for low-dimensional problems, it may easily lead to prohibitive requirements both in terms of memory storage and operation count as the dimensionality increases. In the next section we present a novel methodology for scalable hyper-parameter learning from massive data-sets in high-dimensions. 90 4.3 Kernel design in high-dimensions In many physical systems only relatively low-order correlations of the input variables will have an impact on the output (see Rabitz et al. (1999)). In such cases, high- dimensional model representations (HDMR) are proven to dramatically reduce the computational effort in representing the input-output relationships. The general form of the HDMR expansion for a high-dimensional real-valued function y(x) = y(x1 , x2 , . . . , xd ) X X X y(x) = y0 + yi (xi ) + yij (xi , xj ) + yijk (xi , xj , xk ) + · · · , (4.5) 1≤i≤d 1≤i 100) when only a limited num- ber of observations is available. Performing meaningful inference in these data-poor environments becomes a daunting task, and is currently a subject of active research. 111 Although our current presentation mainly employed simplistic sampling strategies, such as uniform and Latin Hypercube sampling, we believe that the design of more sophisticated adaptive sampling techniques will be key to overcoming the aforemen- tioned limitation. Chapter Five Modeling Continuum Blood Flow and Biomechanics 113 5.1 Motivation The emerging multidisciplinary approach to modeling complex biological systems has had a big impact over the recent years in the development of numerical methods and algorithms for studying such systemic processes that span a large spectrum of spatial and temporal scales. Our particular area of interest here is arterial blood flow modeling, where fluid solvers have become a standard research tool for analyzing and predicting the the underlying dynamics in scales ranging from the continuum (larger systemic arteries) to the atomistic level (small arterioles and veins). In this work, the goal is to study and analyze the potentiality and implementation aspects of using one-dimensional models as representations of the downstream vasculature, coupled as outflow boundary conditions to three-dimensional domains. The need of considering such reduced order models arises from the problem size in conjunction with the barrier imposed by the available computational resources. To this end, resolving the full problem in three dimensions involves O(109 ) degrees of freedom and more than 10T b of computer memory. This conflict can be greatly alleviated if one-dimensional or lumped parameter models are employed for closing the problem. This chapter takes aim at three directions. First, we briefly present the for- mulation of one-dimensional (1D) models. Second, we present an overview of a spectral/hp element scheme for resolving three-dimensional (3D) continuum blood flow in fixed, as well as flexible arteries. Lastly, we propose and test an algorithm for the weak coupling of 3D-1D blood flow models. This exposition sets the ground for the contributions presented in Chapters 6, 7, and 8. 114 5.2 Nonlinear 1D blood flow models We consider viscous incompressible 1D flow in a compliant tube. The flow dynamics is governed by a nonlinear hyperbolic system of partial differential equations that can be directly derived from the Navier-Stokes equations under the assumptions of axial symmetry, dominance of the axial velocity component, radial displacements of the arterial wall, and constant internal pressure on each cross section (see Formaggia et al. (2010)). Figure 5.1: Nonlinear 1D blood flow: Flow in a 1D compliant artery (from Sherwin et al. (2003)). The conservation of mass and momentum can be formulated in space-time (A, U ) variables as (see Sherwin et al. (2003) for a detailed derivation) :   ∂A ∂(AU )   + =0     ∂t ∂x (5.1)   ∂U ∂U 1 ∂p U   =−    +U + Kr , ∂t ∂x ρ ∂x ρA where x is the axial coordinate across the vessel’s length, t is time, A(x, t) is the cross- sectional area of the lumen, U (x, t) is average axial fluid velocity, Q(x, t) = AU is the mass flux, p(x, t) is the internal pressure averaged over the tube’s cross-section, and Kr is a friction parameter that depends on the velocity profile chosen (see Sherwin 115 et al. (2003)). Here we use an axisymmetric fluid velocity u(x, r, t) profile that satisfies the no-slip condition.   r ζ  ζ +2 u(x, r, t) = U 1− , (5.2) ζ R with R(x, t) being the lumen radius, ζ a constant, and r the radial coordinate. Following Formaggia et al. (2010), ζ = 9 gives a good fit to experimental blood flow RR data and ζ = 2 returns the parabolic flow profile. Moreover, U = R12 0 2rudr, and 2µ A ∂u the friction parameter can be expressed as Kr = [ ] U R ∂r R = −22µπ (see Sherwin et al. (2003)), with µ being the blood viscosity that is a function of the lumen radius and the blood hematocrit, based on the findings of Pries et al. (1992). In order to close the system of Eq. 5.1, we need to provide a pressure area-relation. The simplest choice is to model the arterial wall as a purely elastic material. Then, the Laplace tube law yields (see Sherwin et al. (2003)) √ πEh √ p p = pext + √ √ ( A − A0 ) (5.3) (1 − ν 2 ) A0 A Here, pext is the external pressure on the arterial wall, E is the Young modulus of the wall, h is the wall thickness, and ν is the Poisson ratio. 5.2.1 Accounting for arterial viscoelasticity We can further extend the capabilities of this formulation to capture more realistic viscoelastic arterial wall behaviors. In particular, while most 1-D models treat such interactions as purely elastic, the arterial walls are anisotropic and heterogeneous with complex nonlinear stress-strain relationships that give rise to phenomena such 116 as creep, stress relaxation and hysteresis (see Fung (1990)). Even though the elastic response is dominant for most vessels, the viscoelastic properties of the wall can have a significant impact on the flow, especially in distal locations (see Reymond et al. (2011)). The Voigt model is the simplest viscoelastic model that features creep and hysteresis. It depends only on the viscoelastic modulus of the wall, γ; which makes it appealing for calibration to patient-specific data. However, if a more accurate representation of viscoelasticity is desired, a more involved modeling approach needs to be considered. The Kelvin model is a three-parameter linear viscoelastic model that serves exactly this purpose. It provides a discrete relaxation spectrum and enables us to resolve all the aforementioned viscoelastic effects. The only drawback is the inherent difficulty in estimating the required parameters from in-vivo data. However, this can be partially circumvented by employing a continuous relaxation spectrum. (a) (b) Figure 5.2: Nonlinear 1D blood flow: viscoelastic models considered. Left: Voigt. Right: Kelvin. 5.2.1.1 The Voigt arterial wall model The stress-strain relation for a Voigt-type material is given by: d σ = E + η (5.4) dt 117 where r(p − pext ) 1 r − r0 σ= and  = (5.5) h 1 − ν 2 r0 are the stress and the strain respectively, while η is the viscoelastic modulus of the wall. Substituting Eq. 5.5 into 5.4 and using the conservation of mass, one arrives at the Voigt pressure-area relation: √ p Γ ∂(AU ) p = pext + β( A − A0 ) − √ (5.6) A ∂x with √ √ πEh 2 πηh β= , Γ= (1 − ν 2 )A0 3 A0 Note that the inclusion of viscoelasticity adds a dissipative term to our system. In this case the system of Eqs. 5.1, 5.6 is no longer hyperbolic. However, the elastic contribution is still dominant thus allowing us to consider exactly the same numerical scheme used for the purely elastic case. Details of this implementation are provided in Sec. 5.2.2. 5.2.1.2 Fung’s QLV and the Kelvin model Here we present an overview of the theory of Quasi Linear viscoelasticity (QLV). The reader is referred to Fung (1990) for a detailed exposition on biomechanics. Modeling the response of soft tissue is often characterized by a constant hysteresis over a wider frequency range. Based on this approach we have a stress-strain relation of the form: 118 ∞ ∂e (x, γ) Z σ(x, t) = G(t − γ) dγ (5.7) 0 ∂γ where G(t) is the stress relaxation function and e (x, t) is the static elastic response of the tissue. The presence of the convolution integral in Eq. 5.7 makes the stress depended upon the strain time history. It is the choice of the stress relaxation function that ultimately determines the model. In particular, we have: • Choosing: G(t − γ) = ηδ(t − γ), we get the Voigt model described in the previous section (δ is the Dirac delta function). • Choosing: τ − τσ − (t−γ) G(t − γ) = 1 + e τσ , (5.8) τσ where η E1 + E2 τσ = and τ = η (5.9) E2 E1 E2 yields the Kelvin model. Note that τ is the relaxation time of constant strain and τσ is the relaxation time of constant stress. The Kelvin model is the simplest viscoelastic model that can capture effects of creep, stress relaxation and storage of strain energy at equilibrium. It is a linear model that employs only two discrete relaxation times. • Choosing: R τ2 1 − (t−γ) 1+c τ1 e τ dτ G(t − γ) = τ Rτ 1 + c τ12 τ1 dτ yields a quasi-linear viscoelastic model with a continuous relaxation spectrum within the finite time interval [τ1 , τ2 ]. 119 Using Eq. 5.5 and the Kelvin stress relaxation function of Eq. 5.8, we can integrate Eq. 5.7 by parts to arrive at the Kelvin pressure-area relation: t τσ − τ Z τ (t−γ) p(x, t) = pext + pe (x, t) + e− τσ pe (γ)dγ, (5.10) τσ τσ2 0 where p p pe (x, t) = β( A(x, t) − A0 ) √ πEh is the static elastic response, and β = . (1 − ν 2 )A0 As in the Voigt model, the pressure-area relation of Eq. 5.10 introduces dissipa- tion to our system, thus violating hyperbolicity. Again, the elastic response is an order of magnitude greater than the viscoelastic, hence the numerical method will be a modification of the one used for the elastic case. The following table summarizes the features of each arterial wall model presented above: Model Creep Stress Relaxation Hysteresis Rel. Spectrum Elastic - - - - Voigt X - X - Kelvin X X X {τ }, {τσ } General QLV X X X [τ1 , τ2 ] 120 5.2.2 The Discontinuous Galerkin scheme The system of equations in Eq. 5.1 can be recast in a conservative form as (see Sherwin et al. (2003)):         ∂U ∂F(U)  A  AU   0  0 + = S(U), U =  , F =  + v  , S(U) =  ,    ∂t ∂x U U2 + pe p U −22µπ ρA 2 ρ ρ (5.11) where pv (x, t) is the the viscoelastic component of the total pressure. Here, in order to analyze the system, we consider the case of zero forcing a purely elastic pressure- area relation (Eq. 5.11 with pv (x, t) = 0). Then, the system of Eq. 5.11 forms a non-linear hyperbolic system, which in conservative form reads as ∂U ∂U + H(U) = 0, (5.12) ∂t ∂x where      A   u A  U=  , H= , (5.13) 1 u ρDA u and the arterial distensibility D is defined as 1 dA 2 D= = √ . (5.14) A dp β A Under the assumption that A > 0, the matrix H has two real eigenvalues: 1 λ1,2 (H) = u ± c, c= √ . (5.15) ρD 121 Then the matrix of left eigenvectors of H is:   c A 1  L=  , (5.16)  − Ac 1 where    λ1 o  −1 LH = ΛL, with, Λ =   , and H = L ΛL (5.17) 0 λ2 Hence we obtain: ∂U ∂U L + ΛL = 0, (5.18) ∂t ∂x ∂W which can be decoupled by changing variables ∂U = L, W = [W1 , W2 ]T :   ∂W1 ∂W1   + λ1 =0  ∂t ∂x    (5.19)     ∂W2 + λ2 ∂W2 = 0    ∂t ∂x leading to an analytical expression for the forward and backward Riemann invariants, respectively s β  1/4 1/4  W1,2 = U ± 4 A − A0 (5.20) 2ρ The computational domain Ω consists of arterial segments, which can be divided in N el elemental non-overlapping regions Ωe = (xLe , xR R L e ), such that xe = xe+1 for e = 1, .., N el. The discontinuous Galerkin formulation requires, for each element e = 1, . . ., Nel , the resolution of the system: 122 p ∂ Uˆi,e Z Z ∂Fi xR Je = −Je Lp dξ−Lp [Fiu − Fi ]xeL +Je Lp Si dξ, p = 0, . . . , P, i = 1, 2, ∂t Ωe ∂x e Ωe (5.21) where U has been discretized by Uδ written in terms of orthonormal Legendre polynomials Lp (x): P X δ U |Ωe = ˆp Lp Ue p=0 Each element is mapped onto a reference element Ωst = {−1 ≤ ξ ≤ 1} called the ”standard element” and Je is the Jacobian of the corresponding affine mapping: 1 (1 − ξ) (1 + ξ) Je = (xR L e − xe ), xe (ξ) = xLe + xR e 2 2 2 Fu is the upwinded flux that propagates information between the elemental re- gions and the bifurcations of the system. At the inlet and outlet boundary elements, the fluxes are upwinded by means of the boundary conditions. The hyperbolicity of the system requires one boundary condition at each terminal end. An Adams-Bashforth scheme is used for the time integration: p ∂ Uˆi,e   p = f Uˆi,e ∂t with, 123 Z Z   ˆ p ∂Fi 1 xR Ui,e = − Lp dξ − Lp [Fiu − Fi ]xeL + Lp Si dξ, Ωe ∂x Je e Ωe then, n+1 n  n−1   p  p 3∆t  ˆ p n  ∆t p Uˆi,e = Uˆi,e + f Ui,e − f Uˆi,e 2 2 In the case of a viscoelastic wall model, the flux F has to be separated in an elastic part Fe and a viscoelastic part so that: F = Fe + Fv , where:     uA  0  Fe =  , Fv =  v    u2 pe p 2 + ρ ρ √ √ with pe = pext + β( A − A0 ) for the elastic and Voigt cases, while pe = pext + τ √ √ τσ β( A − A0 ) for the Kelvin model. In particular for the Voigt model we have   0 FVoigt =   v Γ ∂(AU )  −√ A ∂x Similarly, for the Kelvin model we have:   0 FKelvin =  τ − τ Z t (t−γ)   v σ   − τ e e σ p (γ)dγ τσ2 0 Normally, the evaluation of the convolution integral in the Kelvin pressure-area 124 relation requires a number of operations that scale quadratically with the number of considered history points. Following the approach of Steele et al. (2011), we can explore the properties of the exponential term, we can derive a simple update rule that is in-depended of the number of history points considered: simple update rule: Z t+∆t (t+∆t−γ) Z t+∆t (t+∆t−γ) − − ∆t It+∆t = e τσ e p (γ)dγ = e τ σ It + e− τσ pe (γ)dγ 0 t where the last term contributes for the time interval [t, t + ∆t] and is computed using the trapezoidal rule. This effectively allows us to considered the full history of the arterial wall response without introducing any computational bottleneck. 5.2.3 Boundary Conditions At bifurcation and junction points of the network we require continuity of the Rie- mann invariants, conservation of mass and continuity of total pressure (see Sherwin et al. (2003) for a detailed analysis). Terminal vessels are typically coupled to 0-D lumped parameter models. The most suitable choice for one dimensional blood flow simulations is the RCR model, where we specify the total peripheral resistance downstream of the outlet Rt , and a corresponding compliance C (see Alastruey et al. (2008)): s dpc pc − pout . c0 ρ 1 β 14 C = A∗ U ∗ − , R2 = Rt − R1 , R1 = , c0 = A (5.22) dt R2 A0 2ρ 0 125 A& U & R!" R!# q!"#$ P(A& ) C! p!% p!"#$ (a) (b) Figure 5.3: Nonlinear 1D blood flow: Left: Sketch of the RCR outflow model. Right: Sketch of the forward and backward characteristics at the 1D-0D coupling interface (figure taken from Alastruey et al. (2008)). Following Alastruey et al. (2008), we solve the coupling as ρc0 ρc0 ∗ ∗ √ p F(A∗ ) = [UL + 4c(AL )]A∗ − 4 (A )A − β( A∗ − A0 ) + pc = 0 (5.23) A0 c Finally, the outflow boundary condition is obtained by imposing AL = AR , and UR = 2U ∗ − UL . 5.3 Linear 1D blood flow models The analysis can be further simplified if one adopts a linear formulation (see Sherwin et al. (2003), Alastruey et al. (2008)). This can be achieved by expressing System 5.11 in terms of the (A, p, Q) variables, with Q = AU , and linearizing them about the reference state (A0 , 0, 0), with β and A0 held constant along x. This yields 126 ∂ p˜ ∂ q˜ C1D + = 0, (5.24) ∂t ∂x ∂ q˜ ∂ p˜ L1D + = −R1D q˜, (5.25) ∂t ∂x a ˜ p˜ = , (5.26) C1D where a ˜, p˜ and q˜ are the perturbation variables for area, pressure, and volume flux, respectively, i.e., (A, p, Q) = (A + a ˜, p˜, q˜), and 2(ζ + 2)πµ ρ A0 R1D = , L1D = , C1D = , (5.27) A20 A0 ρc20 are the viscous resistance to flow, blood inertia and wall compliance, respectively per unit length of vessel (see Alastruey et al. (2008)). For a purely elastic arterial wall behavior it follows that ∂ p˜ ∂p 1 = = (5.28) ∂˜a ∂A A=A0 C1D Then, the corresponding linear forward and backward Riemann invariant can ana- lytically be expressed p˜ ρc0 W1,2 = q˜ ± , Z0 = , (5.29) Z0 A0 where Z0 is the characteristic impedance of the vessel. 127 5.4 3D blood flow models 5.4.1 Flow in rigid vessels Here, we provide an overview of the sub-iterative scheme for solving the incompress- ible Navier-Stokes equations put forth by Baek and Karniadakis (2011). To this end, we consider 3D unsteady incompressible flow in a rigid domain Ω, described by the Navier-Stokes equations subject to the incompressibility constraint:   ∂v + v · (∇v) = −∇p + ν∇2 v + f   ∂t (5.30)  ∇ · v = 0  where v is the velocity vector, p is the pressure, t is time, ν is the kinematic vis- cosity of the fluid, and f are the external body forces. The system is discretized in space using the spectral/hp element method (SEM) (see Karniadakis and Sher- win (2013)), according to which, the computational domain Ω is decomposed into a set of polymorphic non-overlapping elements Ωei ⊂ Ω, i = 1, ..., N el. Within each element, the solution is approximated as a linear combination of hierarchical, mixed- order, semi-orthogonal Jacobi polynomial expansions (see Karniadakis and Sherwin (2013)). This hierarchical structure consists of separated vertex (linear term) Φk (x), edge Ψk (x), face Θk (x) and interior (or bubble) modes Λk (x). According to this de- composition, the polynomial representation of a field v(t, x) at any point xj is given by the linear combination of the basis functions multiplied by the corresponding modal amplitudes: 128 Nv Ne Nf Ni X X X X v(t, xj ) = ˆ kV (t)Φk (xj ) v + ˆ kE (t)Ψk (xj ) v + ˆ kF (t)Θk (xj ) v + ˆ kI (t)Λk (xj ) v k=1 k=1 k=1 k=1 (5.31) The numerical solution of the above system is based on decoupling the veloc- ity and pressure by applying a high-order time-splitting scheme (see Karniadakis and Sherwin (2013)). We solve for an intermediate velocity field v? in physical space (Equation 5.32) before a Galerkin projection is applied to obtain the weak formulation for the pressure and velocity variables (Equations 5.33, 5.34), which are computed by solving a Poisson and a Helmholtz problem, respectively. At time step n and sub-iteration step k we solve Equations 5.33, 5.34 for pnk and vkn , respectively, using previous time-step solutions vn−q , q = 1, .., Je , where Je denotes the order of the stiffly stable time integration scheme used. Je−1 X ? αk vn−q − ∆t Nnk−1 + f  v = (5.32) q=0 ∂pnk   1 pnk Lˆ =− (∇ · v? , φ) + , φ − LD pˆD (5.33) ∆t ∂n ∂vkn   1 ∆tν vkn Hˆ = (v? − ∆t∇pnk , φ) + , φ − HD v ˆ n+1,D (5.34) γ0 γ0 ∂n where we define Nnk−1 = vk−1 n n · ∇vk−1 (5.35) n n Vk−1 = ν∇ × (∇ × vk−1 ) (5.36) 129 Also, γ0 , αk are the coefficients of a backward differentiation formula, βk are the coefficients of the stiffly stable time integration scheme, H = M − ∆tν γ0 , and M and L ˆ and pˆ are the unknown are the mass and stiffness matrices, respectively. Moreover, v modal amplitudes of the velocity and pressure variables, while variables with a D superscript are considered to be known. In particular, the last term in Equations 5.33, 5.34 is due to lifting a known solution for imposing Dirichlet boundary condi- tions, and the corresponding operators HD and LD need to be constructed only for these boundaries (see Karniadakis and Sherwin (2013)). The system is closed with a Neumann ∂v/∂n = 0 boundary condition for the velocity at the outlets, and a con- sistent Neumann pressure boundary condition at the boundaries with a prescribed Dirichlet condition for the velocity field (see Karniadakis and Sherwin (2013)): γ0 vkn − Je−1 " Je−1 # n−q P ∂pnk q=0 αk v X =− + [βk (Nnk−1 + Vk−1 n )]n−k · n (5.37) ∂n ∆t k=0 At the first sub-iteration step k = 1, the solution from previous sub-iterations is not available and Nn0 , V0n are extrapolated in time as follows: Je X Nn0 = βq [v · ∇v]n−q (5.38) q=1 Je X V0n = ν βq [∇ × (∇ × v)]n−q (5.39) q=1 ˆ kn and pˆnk we can accelerate the convergence of the iterative pro- After computing v cedure by applying the Aitken under-relaxation scheme to the computed velocity field: ˜kn = λk v v n ˜k−1 + (1 − λk )vkn (5.40) 130 where λk ∈ [0, 1] is computed using the classical Aitken rule: (Qk−1 − Qk ) · Qk n λk = λk−1 + (λk−1 − 1) , Qk = v ˜k−1 − vkn (5.41) ||Qk−1 − Qk ||2 Finally, convergence is checked by requiring ||vkn − vk−1 n || and ||pnk − pnk−1 || to be less than a pre-specified tolerance. For a complete analysis of this scheme the reader is referred to Karniadakis and Sherwin (2013), Baek and Karniadakis (2011). Results in the context of arterial blood flow simulations can be found in Grinberg et al. (2011a), while parallel per- formance has been documented in Grinberg et al. (2011b). 5.4.1.1 Treatment of boundary conditions Given the Fourier representation of the volumetric flow-rate (VFR) measurements at the ascending aorta, the inflow is driven by a time-depended velocity boundary condition based on the Womersley profile. This model takes into account the pulsatile nature of the flow and it corresponds to the exact solution of pulsatile flow in a straight rigid pipe. At the outlets we employ a lumped parameter RC model that results to a Dirich- let pressure boundary condition which is determined by the VFR Q(t), the resistance R and the capacitance C (see Grinberg and Karniadakis (2008)): dp(t) p(t) + RC = Q(t)R (5.42) dt As reported by Grinberg and Karniadakis (2008), the advantage of this boundary 131 condition is the attenuation of high frequency oscillations. Our approach in choosing the R and C parameters follows the analysis presented by Grinberg and Karniadakis (2008), according to which we set an arbitrary value for the resistance Rj at the j th outlet and we calculate the resistances at all other outlets as: Q¯j Ri = Rj ¯ (5.43) Qi by utilizing the given patient-specific mean VFR measurements at these locations. 5.4.2 Flow-structure interactions We consider the solution of the 3D Navier-Stokes equations taking into account arte- rial compliance. Given proper initial conditions, the system of equations describing the fluid flow is derived in the arbitrary Lagrangian-Eulerian (ALE) reference frame as (see Baek and Karniadakis (2012), Yu et al. (2013))  ∂ ? v + (v − w) · (∇v) = −∇p + ν∇2 v + f    ∂t M (5.44)  ∇ · v = 0, in Ω(t),  u = v = w, x ∈ ∂Ωs , where u, v and w are the structure, fluid and mesh velocities, respectively, and ∂Ωs denoted the fluid-solid interface. The (?) symbol implies that the derivative is taken on the mesh frame M. At the interface between fluid subdomain and solid subdomain, Dirichlet and Neumann type boundary conditions are imposed for the velocity and pressure, respectively. To model the response of the arterial wall, consider a structure initially occupying domain Ω0 being deformed to a new configuration Ω at time t. Let X and x(X, t) 132 denote the coordinate in the undeformed configuration and the displacement at the location X and at time t. Then such deformation can be described as  2 ρ ∂ x = ∇ · τ + ρb,   x∈Ω ∂t2 (5.45)  t = τ · n,  x ∈ ∂Ω, where τ, b, t and ρ are the stress tensor, external body force, external traction force, and structural mass density, respectively, and x(X, t) is the unknown displacement. Under the assumption that deformation is small and material is linear elastic, the stress tensor τ = C, where C is the material matrix with two material constants, Young’s modulus (E) and Poisson constant (ν) and  = 1/2(∇x + ∇xT ), where T indicates the transpose of a tensor or matrix. For extensions to hyper-elastic and viscoelastic constitutive laws the reader is referred to Yu et al. (2012). The fluid and solid domains are coupled through kinematic and dynamics bound- ary condition at the interface   vf · nf = −us · ns ,  (5.46)  σf · nf = −σs · ns  The above equations are discretized using the spectral element method put forth by Baek and Karniadakis (2012), Yu et al. (2013). As described, the fluid solver em- ploys the ALE formulation and updates the mesh at every time step. The solid solver employs an elastodynamic equation with traction boundary condition at the fluid interface. Both solvers are based on third-order accurate time- stepping schemes. Stability and convergence of the solution are enhanced by performing sub-iterations in conjunction with Aitken relaxation. The relaxation procedure is crucial as it leads to stable coupling even for small mass ratios – a common problem in FSI simulations, 133 recently addressed by Yu et al. (2013). 5.5 Coupling of the 3D and 1D solvers In this section we aim to explore the coupling of nonlinear 1D model to 3D flow sim- ulations in rigid arteries. While reduced order models, such as 0D lumped parameter models and 1D linearized models have been used extensively in the past as outflow boundary conditions, here our ultimate goal is to model the downstream vasculature as a fractal network (see Chapter 7, and Zamir (1999)), in which the fully non-linear 1D problem is solved. In what follows, we present the main algorithm along with the sets of possible coupling conditions, and several test cases. Figure 5.4: 3D-1D coupling: Computational domain for 3D-1D overlapping coupling for a straight pipe. We consider a rigid tube 3D domain Ω3D with its outlet being coupled to a compliant 1D vessel Ω1D (with or without an overlapping region). At each time- step, the hyperbolic 1D model requires one boundary condition at its inlet, while the parabolic 3D model requires two boundary conditions at its outlet. A detailed discussion on such coupling conditions, well-possedness and stability of the resulting system has been given by Formaggia et al. (2010), Formaggia et al. (2001), Passerini et al. (2009). Among the admissible choices presented therein, we consider the following cases: 134 • Case 1 (defective):  Qn1D = Qn3D ,       pn+1 n+1 (5.47)  3D = p1D ,  n+1 du1D n+1   ∂v    = ∂n dx • Case 2 (defective):  Qn1D = Qn3D ,        pn+1 n+1 n+1 (5.48)  3D = f (R1D , C1D ),  n+1 du1D n+1   ∂v    = ∂n dx • Case 3:  Qn1D = Qn3D ,        ρ n 2 ρ n+1 2  {−(pn+1 n n T n+1 3D + 2 |u3D | )I − µ(∇u3D + ∇u3D )} · n = {p1D + 2 |u1D | } · n,   ∂v n+1 du1D n+1     = ∂n dx (5.49) • Case 4:  Qn1D = Qn3D ,       √ p √  pρ p W 2 = U 3D n − 2 ( p 3D + β A 0 − β A0 ), (5.50)  2  n+1 du1D n+1   ∂v    = ∂n dx where Q = Au is the flow rate, A the cross sectional area, u the velocity component in the axial direction, p the pressure, W2 the incoming characteristic from the 1D p domain, R = Q the resistance and C the corresponding compliance. Note that suggested sets of coupling conditions only differ in the treatment of the pressure Dirichlet boundary condition applied at the outlet of the 3D domain. In case 1, the 1D model is driven by the flow-rate Q3D (imposed as an inflow 135 du Dirichlet boundary condition), while it returns the pressure p1D and the flux dx to be imposed as Dirichlet an Neumann outflow boundary conditions at the 3D outlet, respectively. In case 2, the idea is exactly the same as in case 1 but here we impose the Dirichlet boundary condition for the pressure at the 3D outlet using a low pass RC- filter to attenuate any spurious oscillations coming from the downstream 1D model. In comparison to case 1, this leads to faster convergence of the CG solver for the pressure int the 3D domain when time-extrapolation is used to accelerate the 3D solver (see Grinberg and Karniadakis (2011)), thus reducing the total computation time. Note that since the 1D domain can only provide a single averaged value for the pressure and the outflow flux (instead of point-wise data naturally required by the 3D domain), cases 1,2 give rise to a defective boundary condition problem. In case 3, the traction at the 3D outlet is set to be equal to the total pressure at the 1D inlet. In the cases we study here, this formulation is identical to the conditions listed in case 1. However, when the non-linear terms in the Navier-Stokes are written in rotational form, it provides a non-defective outflow boundary condition as it can return point-wise data for the pressure (see Formaggia et al. (2001)). In case 4, point-wise pressure data is recovered by exploiting the continuity of the forward and backward Riemann invariants at the coupling interface. However, it is only applicable for the case of coupling compliant 3D-1D coupling (see Papadakis (2009)). In what follows we will confine our study to the rigid 3D - flexible 1D coupling, thus we will only employ, test and compare cases 1,2 of the coupling conditions presented above. 136 5.5.1 The coupling algorithm The goal is to utilize the 1D model for providing the 3D domain with an accurate pressure boundary condition at time level (n + 1). To this end, we need to integrate the 1D domain from tn to tn+1 driven by the inflow imposed by Qn3D . Figure 5.5: 3D-1D coupling: Sketch of the 3D-1D weak coupling algorithm Typically the explicit integration scheme of the hyperbolic 1D system requires a smaller time-step than the one used in the 3D counterpart. For more accurate ˜ n+1 to be results we can provide the 1D domain with a time-extrapolated flow-rate Q 3D used for the intermediate 1D time-steps: ˜ n+1 = f (Qn3D , Qn−1 , Qn−2 ) Q 3D 3D 3D Hence, between tn and tn+1 the flow-rate at the 1D inlet can be computed using ˜ n+1 as shown in Fig.6: interpolation between Qn3D and Q 3D 137 Figure 5.6: 3D-1D coupling: Flow-rate interpolation for intermediate asynchronous 1D solver steps. Finally, at tn+1 we can provide the 3D domain with the corresponding boundary conditions for the outflow pressure and flux (either directly or through a low-pass RC filter) and solve for the field variables v and p as described in Section 2. We have implemented and tested this coupling using the spectral element solver NektarG for the 3D domain, and the nonlinear 1D Discontinuous Galerking solver described in Sec. 5.2 for the 1D domain. The implementation of the required commu- nication between the 3D and 1D solvers is based on non-blocking calls of the Message Passing Interface library (MPI) which involve only the CPU ranks handling elements on each coupling interface. The overlap between communication and computation makes the additional computational cost of this implementation negligible as long as the size of the 1D network is relatively small (up to a few thousands of vessels). For 1D networks consisting of tenths of thousands vessel or even millions, the serial 1D 138 solver causes a computation bottleneck. In Chapter 7 of this thesis we will address this issue by introducing a parallel implementation of the 1D solver that allows near real-time solutions in fractal arterial networks with millions of arteries. Finally, we note that a strong coupling between the geometrically heterogeneous solvers can be achieved by the means of sub-iterations of the aforementioned partitioned algorithm. 5.5.2 Remarks 5.5.2.1 Overlapping vs non-overlapping coupling All results on 3D-1D coupling that can be found in the literature have employed a non-overlapping domain decomposition strategy. Here, motivated by the work of Grinberg et al. (2011b), we introduce and analyze the potentiality of using an overlapping domain region between the 3D and 1D models. In the context of spec- tral methods, Karniadakis and Sherwin (2013)) have shown that the time-splitting scheme used for the 3D model introduced a numerical boundary layer of thickness δt ≈ (ν∆t)Je that decays exponentially over the distance s away from the domain boundary (error ∝ e−s/δt ). As demonstrated by the results presented by Grinberg et al. (2011b), extracting the coupling quantities from the inner elements of the two communicating domains reduces the numerical boundary layer errors and enhances the stability of the coupling. As we will see in the Results section, this approach indeed returns more accurate results, but more importantly, overcomes the stability issues that the non-overlapping setup may suffer from. 139 ∂v 5.5.2.2 Imposing the outflow flux ∂n A standard way for imposing the Neumann outflow boundary condition in stand- ∂v alone 3D flow solvers is to assume = 0, where n is the normal direction to the ∂n outlet’s cross section. This may not be the case for unsteady flow in geometries different than the straight tube. To demonstrate this we set up a simple simulation of Womerseley flow in a pipe with 5o tapering along the axial direction and compared the results against the straight pipe case. (a) (b) Figure 5.7: 3D-1D coupling: Normal flux close to the outflow boundary of a straight pipe (left) and a converging pipe with 5 degrees tapering (right). As we can see in 5.7 the normal flux close to the outflow boundary takes values ∂v away from zero for the diverging pipe case, indicating that imposing = 0 as a ∂n Neumann outflow boundary condition is not consistent. When coupling a 3D domain to a 1D network, we can utilize the information com- ing from the downstream 1D domain in order to impose a more accurate Neumann ∂v ∂u1D boundary condition on . To this end, we can either directly impose or com- ∂n ∂x ∂v ∂u1D pute a linear combination (flux upwinding) f ( , ) where time-extrapolated ∂n ∂x values can also be considered: 140 Je−1 ∂vn+1 X  ∂vn−k dun+1  ∂v ∂u1D 1D = f( , )= ξk λ − (1 − λ) (5.51) ∂n ∂n ∂x k=0 ∂n dx where 0 ≤ λ ≤ 1. Typically, we choose λ = 0.5 (central averaging). 5.5.2.3 Arterial wall stiffness of the 1D model Since we consider the coupling between rigid 3D and compliant 1D domains, an inconsistency arises at the coupling interface, where the continuity of cross sectional area is violated. Comparing the solutions computed from the two different models (3D vs 1D) given an identical simulation setup, we notice that while we can get a good match for the flow-rate, the predicted solutions for the the propagating pressure wave are completely different. This is of course a well expected behavior and it is attributed to the different arterial wall response (rigid vs compliant). As first put forth in Grinberg et al. (2011a), a straightforward way for the compliant 1D model to reproduce the results of the rigid 3D solver is to make the arterial wall model stiffer by scaling appropriately the spring constant β that appears in the 1D pressure-area relation (see Eq. 5.3): √ πEh β = βscal , βscal > 1 (5.52) (1 − ν 2 )A0 ? Practically, we can find an appropriate scaling factor βscal such that the 1D solver accurately reproduces the results predicted by the full 3D model. However, if the ? chosen value for βscal is not close to the “best fit” value βscal , the 1D solver under ? ? predicts the pressure wave for βscal << βscal and over-predicts it for βscal >> βscal . 141 The identification of the appropriate scaling factor is case depended. However, we ? can explicitly derive a formula for βscal based on the physiological properties of the network. To this end, consider a single vessel in an arterial network and let Rt be the total peripheral resistance downstream, and R1 the vessel’s characteristic impedance: s . c0 ρ 1 β 14 R1 = , c0 = A (5.53) A0 2ρ 0 Then, we must have that Rt − R1 ≥ 0, and therefore, we can scale β and stiffen the 1D model respecting the upper bound: 3 2Rt2 A02 R1 ≤ Rt ⇒ βscal ≤ (5.54) βρ In simulations, the total resistance Rt is typically prescribed for each outlet that is attached to a 0-D model and its values can be obtained either by tuning the model to match patient specific data, or, more generally, by interpolating available data sets in the literature (see for e.g., Reymond et al. (2011)). Here, we also propose an alternative way for stiffening the wall response of the 1D model. To this end, we introduce a scaling factor τ˜ to the 1D pressure-area relation as: √ p p = pext + β( A − A0 ) · τ˜ (5.55) where the values of τ˜ are computed using the following low pass RC-filter analogue: ∂ τ˜ n √ p o τ˜ + RC = 1 + R · β( A − A0 ) (5.56) ∂t 142 Obviously, for R = 0 we recover the fully elastic 1D pressure-area relation, while for R > 0, we get τ˜ > 1, resulting to a reduced distensibility of the arterial wall cross section. The main advantage of this approach is that it achieves an accurate fit to √ the solution predicted by the full 3D model when we a-priori choose R = F (t)β A, where F (t) = (1 − e−20(t−t0 ) )2 is a filter function used to filter out any erroneous oscillations that occur during the first time-steps. This choice is based on the fact √ that the factor β A eliminates the vessel’s distensibility, which for a vessel governed . by the 1D elastic pressure-area relation is defined by: D = A1 dA dp = β √2 A . Finally. the 0.18 capacitance in the RC-filter equation for τ˜ is typically set as C = √ . β A0 5.6 Results In this section we append some representative results for our 3D-1D weak coupling algorithm. We start the section with a basic test case of Womersley flow in a straight pipe in order to compare the results obtained by the full 3D/1D models and highlight the different results obtained between the rigid 3D and the compliant 1D case. Next, using the same straight pipe setup, we truncate the 3D domain and couple its outlet to a 1D segment. The results presented aim to answer the question of how stable and accurate are the predictions given by our coupled 3D-1D solver when compared against the full 3D case. The same comparison is also carried out for a slightly more challenging configuration: a Y-shaped bifurcation geometry with two outlets. 143 5.6.1 3D/1D comparison in a straight pipe Here we consider a straight pipe of length L = 2cm, and radius r = 3mm. For the 3D model impose the exact Womersley solution as an inflow boundary condition on the velocity, while a lumped parameter RC network (R = 100, C = 0.002) provides the Dirichlet outflow boundary condition for the pressure. Similarly, the 1D model is driven by the same Womersley solution, while we use a three-element windkessel model (RCR) that exactly matches the outflow boundary condition of the 3D setup. p The Womersley number of the flow is W s = 0.5D ω/ν = 5.585, the Reynolds Um D number is computed as Re = ν = 135 and the period of each cycle is T = 0.423s. The 3D computational domain is discretized by N el = 14245 spectral elements, while the 1D domain consists of only 1 DG element. In both cases the solution is represented within each element using polynomial expansions of 4th order while time-integration is 2nd order accurate. Finally, the 3D simulation was performed using 1024 processors on a Cray XK6 machine (ORNL Jaguar). Regarding the arterial wall stiffness of the 1D model, three cases are considered: 1.) fully elastic model, where β is set according to the data published by Olufsen (1999), 2.) stiff elastic model (denoted as βstif f ), where the initial β has been scaled to 1.4 · 104 β (setting βscal = 1.4 · 104 results to a good fit of the 3D predictions), and 3.) stiff elastic model, where the scaling factor τ˜ is computed using the RC-filter equation introduced in Sec. 5.5.2.3 (denoted as β − RC). 3D 1D elastic 1D βstif f 1D β − RC ∆t 0.002 0.06 0.005 0.005 Computation time/cycle 1 hr 1s 20 s 20 s Table 5.1: 3D/1D comparison in a straight pipe: Non-dimensional time-step ∆t and computation time per cycle for each simulation considered. 144 As we can see in the following figures, the 1D model captures the flow-rate wave for any configuration of the arterial wall model used. However, although the average pressure over the cycle is accurate, the actual pressure waveform predicted by the fully elastic 1D model is inconsistent with the solution in the 3D domain. For a better fit we need to stiffen the 1D wall model. To this end, choosing βscal = 1.4 · 104 results to a good approximation, respecting the upper bound presented in ? ? Sec. 5.5.2.3, Eq. 5.54. Moreover, when βscal << βscal or βscal >> βscal we get under/over-shoots in the predicted waveform as demonstrated in Fig. 5.9 for βscal = 103 and βscal = 5 · 104 . Finally, choosing to stiffen the 1D model by using the β − RC approach seems more appealing since accurate results are obtained without any parameter calibration needed. Table 5.2, summarizes the relative L2 errors computed by comparing each of the considered 1D simulations to the full 3D case. Relative L2 error 1D elastic 1D βstif f 1D β − RC ||Q3D − Q1D ||2 5.001895e-02 1.105535e-02 1.080173e-02 ||Q3D ||2 ||p3D − p1D ||2 6.203276e-01 1.836266e-02 4.339720e-02 ||p3D ||2 Table 5.2: 3D/1D comparison in a straight pipe: Relative L2 error at the outlet. 145 Flowrate at the inlet Flowrate at the outlet 0.12 0.12 Full 3D Full 1D elastic 0.1 0.1 Full 1D βscal = 1.4 · 104 β Full 1D β − RC Full 1D βscal = 103 β Full 1D βscal = 5 · 104 β 0.08 0.08 ) ) 3 3 Q ( cm Q ( cm s s 0.06 0.06 0.04 0.04 0.02 0.02 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) t (sec) (a) (b) Figure 5.8: 3D/1D comparison in a straight pipe: Comparison of computed flow-rates at the inlet and outlet. Pressure at the inlet Pressure at the outlet 35 35 Full 3D Full 3D Full 1D elastic Full 1D elastic 30 30 Full 1D βscal = 1.4 · 104 β Full 1D βscal = 1.4 · 104 β Full 1D β − RC Full 1D β − RC Full 1D βscal = 103 β Full 1D βscal = 103 β 25 25 Full 1D βscal = 5 · 104 β Full 1D βscal = 5 · 104 β p (mmHg) p (mmHg) 20 20 15 15 10 10 5 5 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) t (sec) (a) (b) Figure 5.9: 3D/1D comparison in a straight pipe: Comparison of computed pressures at the inlet and outlet. As we can see, the elastic 1D model returns inconsistent predictions for the pressure. The best match is achieved when we choose to stiffen the arterial wall response of the 1D model using a scaling factor close to the one predicted by Eq. 5.54 in Sec. 5.5.2.3. This results to a relative error less than %2, while using the β − RC stiffening approach results to a slightly higher error for the pressure (%4.3). 146 5.6.2 3D-1D coupling for a straight pipe This is a basic test case for analyzing the accuracy of our coupling algorithm. We consider exactly the same setup presented in Section. 4.1, but now the 3D domain is truncated at z = 1.4cm and the remaining portion of the pipe is replaced by an overlapping or non-overlapping 1D segment of length l = 12mm or l = 6mm, respectively. In the overlapping case we choose the overlap length to be equal to twice the radius loverlap = 2r = 6mm. The set of 3D-1D coupling conditions considered corresponds to cases 1,2 presented in Section 3 and will be denoted as p-coupling and RC-coupling, respectively. Regarding the arterial wall stiffness of the 1D model, we consider the same three cases presented above: 1.) fully elastic model, where β is set according to the data published by Olufsen (1999), 2.) stiff elastic model (denoted as βstif f ), where the initial β has been scaled to 1.533 · 104 β (using Eq. 5.54, Eq. 5.54), and 3.) stiff elastic model, where the scaling factor τ˜ is computed using the RC- filter equation introduced in Sec. 5.5.2.3 (denoted as β − RC). The truncated 3D computational domain consists of N el = 9908 spectral elements and the computation time for 1 cycle on 1024 processors (Cray XT5) was 37mins; a %40 speedup compared to the full 3D case. The time step of the 3D solver was ∆t3D = 0.002 non-dimensional time units, while the 1D solver was advanced in time using ∆t1D = ∆t3D /5 for stability reasons. Results are presented for the following cases: 1.) Full 3D domain, 2.) 3D-1D coupling with a fully elastic pressure-area relation (β is set according to the data published by Olufsen (1999)), 3.) 3D-1D coupling with a stiff 1D model (denoted ? ? as βscal , with the fixed scaling factor βscal = 1.533 · 104 for a good fit of the 3D predictions), and, 4.) 3D-1D coupling with a stiff 1D model (denoted as β − RC, where the scaling factor τ˜ is computed using the RC-filter equation introduced in 147 ? ? Eq. 5.54). Using βscal << βscal or βscal >> βscal we observe again under/over predictions in the pressure wave exactly similar to the ones presented in Sec. 5.6.1 so we will omit presenting them again here. The coupling interface conditions used correspond to cases 1,2 (see Sec. 5.5), i.e the pressure at the 3D outflow is either imposed directly from the 1D domain or through an RC filter (denoted as p-coupling and RC-coupling, respectively). Table 5.3 summarizes the computed errors for each case considered. RC-coupling p-coupling Relative L2 error elastic ? βscal β − RC ? βscal ||Q3D − Q1D ||2 5.001895e-02 1.518175e-02 1.525898e-02 1.518176e-02 No overlap ||Q3D ||2 ||p3D − p1D ||2 6.203276e-01 1.802136e-02 4.413450e-02 1.802138e-02 ||p3D ||2 ||Q3D − Q1D ||2 5.001213e-02 1.413325e-02 1.512868e-02 1.413327e-02 Overlap ||Q3D ||2 ||p3D − p1D ||2 6.073351e-01 1.496531e-02 4.302153e-02 1.496537e-02 ||p3D ||2 Table 5.3: 3D-1D coupling: 3D-1D coupling for a straight pipe: Relative L2 error at the outlet. As expected, the rigid 3D - elastic 1D coupling, although stable, yields inaccurate predictions for the pressure wave propagation. Only by ’stiffening’ the wall response of the 1D model we can achieve a good fit to the full 3D case. In particular, both ? ? methods (βscal , β − RC) where stable, with βscal resulting to better accuracy. The stiff 1D cases require a smaller time-step than the elastic 1D model but this does not have any significant impact on the overall computation cost of this simulation. Moreover, we observe agreement in the results for either set of interface coupling conditions. However, the RC-coupling seems more preferable since it was less com- 148 putationally expensive than p-coupling (by a factor of about 10% per cycle). This difference may be attributed in the ’filtered’ Dirichlet pressure boundary condition imposed at the 3D outlet, which potentially helps reduce the number of iterations of the CG pressure solver. Flowrate at the inlet Flowrate at the outlet 0.12 0.12 Full 3D 3D-1D elastic 0.1 0.1 3D-1D β − RC ⋆ 3D-1D βscal 0.08 0.08 ) ) 3 3 Q ( cm Q ( cm s s 0.06 0.06 0.04 0.04 0.02 0.02 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) t (sec) (a) (b) Figure 5.10: 3D-1D coupling: 3D-1D coupling for a straight pipe: Comparison of computed flow-rates at the inlet and outlet (RC-coupling, no overlap). Pressure at the inlet Pressure at the outlet 30 30 Full 3D Full 3D 3D-1D elastic 3D-1D elastic 25 3D-1D β − RC 25 3D-1D β − RC ⋆ ⋆ 3D-1D βscal 3D-1D βscal 20 20 p (mmHg) p (mmHg) 15 15 10 10 5 5 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) t (sec) Figure 5.11: 3D-1D coupling: 3D-1D coupling for a straight pipe: Comparison of computed pressures at the inlet and outlet (RC-coupling, no overlap). The results are in agreement to what we saw in the full 3D vs full 1D comparison. Again, the elastic 1D model returns inconsistent predictions for the pressure. The 149 best match is achieved when we choose to stiffen the arterial wall response of the 1D model using the scaling factor predicted by Eq. 5.54 in Sec. 5.5.2.3. Again, this results to a relative error less than %2, while using the β − RC stiffening approach results to a slightly higher error for the pressure (%4.3). Both the non-overlapping and overlapping configurations return consistent re- sults, with the later demonstrating a marginally better accuracy. Although, all coupling configurations where stable for this case, this may not be the case in gen- eral. As we will see in the next section, stability issues arise when a slightly more challenging test case is addressed. 5.6.3 3D-1D coupling for a Y-shaped bifurcation We consider a Y-shaped bifurcation domain with one inlet and two outlets. The dimensions of each segment of the full 3D domain are: {l1 = 10mm, r1 = 3mm}, {l2 = 25mm, r2 = 3mm}, {l3 = 20mm, r3 = 1.5mm}. The simulation is driven p by prescribing a Womerseley inflow at the inlet with W s = 0.5D ω/ν = 5.585, Um D Re = ν = 135, and T = 0.423s. The constant mode of the Womerseley solution is scaled so that the magnitude of the imposed flow-rate lies in the physiological range for vessels of this size. For the full 3D reference case, the RC outflow boundary condition is used, and the RC parameters are chosen such that the flow-rate ratio at the two outlets is Q1 /Q2 = 13. The computational domain for the full 3D case consists of N el = 45441 spectral elements with the polynomial expansion within each element being of 4th order. Each cycle took approximately 73mins on 2048 cores of IBM BlueGene/P (Intrepid ANL) using a non-dimensional time-step ∆t3D = 0.002 and 2nd order time integration. 150 For the 3D-1D coupling, the original domain is truncated such that: {l1 = 10mm, r1 = 3mm}, {l2 = 15mm, r2 = 3mm}, {l3 = 15mm, r3 = 1.5mm}, with the re- maining topology being represented by 1D domains. This truncated 3D domain is then discretized in space using N el = 29106 spectral elements with 4th poly- nomial expansions within each element. Similarly, the 1D domains consist of 1 (non-overlapping case) or 2 (overlapping case) DG elements of the same order. For the overlapping case, the 3D-1D overlap length was set equal to twice the radius: 1 2 loverlap = 2r1 = 6mm and loverlap = 2r2 = 3mm. To ensure the stability of the 1D solver we use 5 inner time-steps with ∆t1D = ∆t3D /5. Figure 5.12: 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Con- sidered model for the Y-shaped bifurcation: full 3D domain (left), 3D-1D non- overlapping domain (center), 3D-1D overlapping domain (right). The orange region depicts the overlap of the 3D and 1D domains. The configurations considered are: 1.) Full 3D domain, 2.) 3D-1D coupling with a fully elastic pressure-area relation (β is set according to the data published by 151 ? Olufsen (1999)), 3.) 3D-1D coupling with a stiff 1D model (denoted as βscal , using the scaling factor presented in Eq. 5.54 in Sec. 5.5.2.3), and, 4.) 3D-1D coupling with a stiff 1D model (denoted as β − RC, where the scaling factor τ˜ is computed using the RC-filter equation introduced in Sec. 5.5.2.3). 1.5 2.5 Full 3D Full 3D ⋆ 3D-1D elastic 3D-1D βscal -RC 2 1 ) ·10−7 ) ·10−8 1.5 3 3 Q ( cm Q ( cm s s 1 0.5 0.5 0 0 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02 t (sec) t (sec) (a) (b) Figure 5.13: 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Flow-rate oscillations at the 3D-1D non-overlapping interface #1 (RC-coupling). Unlike the straight pipe case where all configurations and models were stable, here we encounter issues that render all the non-overlapping coupling cases to be unstable. Wave reflections at the 3D-1D interfaces may contaminate the solution and amplify high modes, inducing oscillations in the flow-rate and pressure waves. These instabilities are manifested in an explosive manner in the case of stiff 1D models, leading to an immediate blow-up during the first hundred time-steps (see Fig. 5.13(a)). In the case of elastic 1D domains, the instability appears milder but still the solution diverges at some point during the first quarter of the cycle (a few thousand time-steps, see Fig. 5.13(b)). Moreover, the instabilities are even stronger when p-coupling is used instead of RC-coupling, even if we try to filter the initial oscillations due to non divergence free initial conditions. 152 RC-coupling p-coupling Relative L2 error elastic ? βscal β − RC ? βscal ||Q3D − Q1D ||2 Unstable Unstable Unstable Unstable No overlap ||Q3D ||2 ||p3D − p1D ||2 Unstable Unstable Unstable Unstable ||p3D ||2 ||Q3D − Q1D ||2 1.834868e-01 1.444825e-02 Unstable Unstable Overlap ||Q3D ||2 ||p3D − p1D ||2 7.080739e-01 3.316665e-02 Unstable Unstable ||p3D ||2 Table 5.4: 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Relative L2 error at outlet #1. RC-coupling p-coupling Relative L2 error elastic ? βscal β − RC ? βscal ||Q3D − Q1D ||2 Unstable Unstable Unstable Unstable No overlap ||Q3D ||2 ||p3D − p1D ||2 Unstable Unstable Unstable Unstable ||p3D ||2 ||Q3D − Q1D ||2 3.745957e-01 2.403443e-02 Unstable Unstable Overlap ||Q3D ||2 ||p3D − p1D ||2 7.025703e-01 4.286251e-02 Unstable Unstable ||p3D ||2 Table 5.5: 3D-1D coupling: 3D-1D coupling for a Y-shaped bifurcation: Relative L2 error at outlet #2. The only way we could obtain both stable and accurate results was by using the 3D-1D RC-coupling with an overlapping region between the coupled domains. As we can see in Tables 5.4, 5.5 and Figs. 5.14, 5.15, the relative error in the flow-rate and the pressure between the full 3D and 3D-1D coupling cases was less that 5% ? when the 1D vessels where stiffened according to Eq. 5.54 in Sec. 5.5.2.3 (case βscal ). Interestingly, the overlapping coupling between the rigid 3D domain and the purely elastic 1D domain was initially stable but then starts diverging after the second cycle (Fig. 5.14(c)). 153 Flowrate at the inlet Flowrate at outlet #1 0.12 0.1 Full 3D 0.09 3D-1D elastic ⋆ 0.1 3D-1D βscal 0.08 0.07 0.08 0.06 ) ) 3 3 Q ( cm Q ( cm s s 0.06 0.05 0.04 0.04 0.03 0.02 0.02 0.01 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) t (sec) (a) Flowrate at outlet #2 (b) 7 Full 3D 3D-1D elastic ⋆ 3D-1D βscal 6 5 ) ·10−3 4 3 Q ( cm s 3 2 1 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) (c) Figure 5.14: 3D-1D coupling: Comparison of computed flow-rates at the inlet and outlets (RC-coupling, 3D-1D overlap). 154 Pressure at the inlet Pressure at outlet #1 50 50 Full 3D Full 3D 45 3D-1D elastic 45 3D-1D elastic ⋆ ⋆ 3D-1D βscal 3D-1D βscal 40 40 35 35 30 30 p (mmHg) p (mmHg) 25 25 20 20 15 15 10 10 5 5 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) t (sec) Pressure at outlet #2 50 Full 3D 45 3D-1D elastic ⋆ 3D-1D βscal 40 35 30 p (mmHg) 25 20 15 10 5 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 t (sec) Figure 5.15: 3D-1D coupling: Comparison of computed pressures at the inlet and outlets (RC-coupling, 3D-1D overlap). Chapter Six Fractional-order Viscoelasticity in Arterial Biomechanics 156 6.1 Motivation The vascular wall is a heterogeneous soft tissue with complex bio-mechanical prop- erties that vary over different locations in the arterial tree and the temporal state of the response (deformation/relaxation). It is comprised of three different layers: the intima (mainly endothelial cells), the media (mainly smooth muscle cells), and the adventitia (mainly collagenous fibers) (see Fung (1990)). This structure high- lights two key functions, namely, reversible energy storage (elasticity) and energy dissipation (viscosity). Constitutive laws for biological tissue can be derived using integer-order differen- tial equations that model stress-strain relations using additive combinations of purely elastic and viscous elements (see Fung (1990)). The simplest integer-order models of linear viscoelasticity are the Voigt and Standard Linear Solid (SLS or Kelvin-Zener) models (see Eringen (1980)). The Voigt model can be constructed by the parallel combination of a spring and a dashpot and it is the simplest model that accounts for creep and hysteresis phenomena. Similarly, the SLS model is constructed using the parallel combination of a spring with a spring and a dashpot in series, and it accounts for creep, hysteresis and stress relaxation phenomena. Due to their simplicity, these models have been used in several 1D blood flow studies to describe arterial wall viscoelasticity (see Matthys et al. (2007), Reymond ˇ c et al. (2006), Raghu et al. (2009), Steele et al. (2011), DeVault et al. (2008), Cani´ et al. (2011)). Matthys et al. (2007) employed a Voigt viscoelastic model within a 1D flow solver and validated results against in vitro measurements of flow in 37 silicone branches representing the largest central systemic arteries in the human. In Rey- mond et al. (2009), a more generalized integer-order viscoelastic model is considered, 157 and 1D simulations are compared against in vivo measurements in a patient-specific network of 94 large systemic arteries. Both studies show good agreement between simulations and measured data, indicating that 1D models may capture the most sig- nificant features of the pulsatile flow and pressure waveforms. However, they mainly focus on larger systemic arteries, where the arterial wall response is known to be predominantly elastic (see Craiem and Armentano (2007), Craiem et al. (2008)), whereas the effect of wall viscoelasticity on hemodynamics is minor and only mani- fested in distal locations (see Reymond et al. (2009)), where the arterial wall appears to be more muscular. The theory of quasilinear viscoelasticity (QLV) theory (see Fung (1990)) pro- vides a path to constructing integer-order models that accurately capture the time- depended response of the arterial wall; however, estimating the elastic and viscoelas- tic parameters that define these models in different anatomic locations still remains a very challenging task. In particular, one needs to address the lack of clinical data, the local and patient-specific nature of parameter values, the dynamic variation that parameters such as the Young’s modulus E may exhibit during the cardiac cycle, etc. Also, as reported by Doehring et al. (2005), the sensitivity on these parameters (especially the long viscoelastic relaxation time) is high. These are key limiting fac- tors for considering integer-order QLV models in 1D blood flow simulations of large arterial networks with high spatial variability in mechanical properties and presence of regions where the viscoelastic response of the wall may be significant. An alternative approach to modeling the viscoelastic behavior of arteries based on fractional calculus (see Mainardi (2010), Podlubny (1998), Magin (2006)) has been introduced by Craiem and Armentano (2007), Craiem et al. (2008), Doehring et al. (2005), where the cell and tissue biomechanics are modeled by fractional-order dif- ferential equations. Under this framework, a spring represents a zero-order element 158 (purely elastic response) and a dashpot corresponds to a first-order element (purely viscous response). A fractional order 0 < α < 1 results in a new flexible element type, the so-called spring-pot (see Magin (2006)), which essentially interpolates be- tween the purely elastic and viscous behaviors and can be thought of as a large set of weighted integer-order spring/dashpot pairs arranged in parallel (see N¨asholm and Holm (2013)). This gives rise to a very interesting physical interpretation: the frac- tional order introduces a fractal tree-like hierarchical structure, naturally allowing for separation of material and global time scales (see Doehring et al. (2005), Figure 1 therein). In contrast to the discrete relaxation spectrum of linear Voigt and SLS integer-order models, fractional-order models exhibit a continuous relaxation distri- bution, spanning all frequencies above zero, up to infinity. Therefore, fractional-order models can be naturally put under the context of quasi-linear viscoelasticity and be considered as good candidates for modeling biological tissue, which in reality shows continuous relaxation (see Fung (1990)). Recent experimental studies have confirmed the ability of fractional-order models to capture the dynamic response of the vascular wall. Specifically, Craiem and Ar- mentano (2007) proposed a fractional-order Voigt model (FO-Voigt) and accurately described the viscoelastic mechanical response of the aorta of a sheep in-vivo with a minimal set of estimated parameters. Also, Craiem et al. (2008) performed uni- axial loading experiments on strips cut from healthy human aortas, and a FO-Voigt model was successfully fitted to measured stress-relaxation data. Similarly, Craiem et al. (2008) calibrated FO-Voigt and fractional-order SLS (FO-SLS) models to accu- rately reproduce in-vivo data from ascending aorta segments of four human donors. Doehring et al. (2005) integrated a fractional stress relaxation function correspond- ing to a FO-SLS model in the QLV framework to successfully model aortic valve cusp biomechanics, while demonstrating significantly lower sensitivity on the input 159 parameters as compared to a standard QLV model with a continuous relaxation spec- trum. These recent findings indicate that fractional order models may offer a new powerful alternative for describing arterial wall mechanics, reducing the parameter estimation count and overcoming the sensitivity limitations of integer-calculus-based QLV models. In this chapter, we present a nonlinear, time-domain 1D blood flow formulation integrated with integer- and fractional-order viscoelastic wall models. We perform blood flow simulations in a large patient-specific network and compare integer- and fractional-order SLS models, calibrated with experimental data reported in the lit- erature. Our main scientific goal is to quantify the effect of the fractional-order and the viscoelastic relaxation response on the computed flow and pressure wave prop- agation. Finally, we present a global sensitivity analysis study for which we have considered a stochastic fractional-order SLS model by introducing uncertainty in the parameters that define its viscoelastic behavior. 6.2 Viscoelastic Models of the Arterial Wall The theory of quasi-linear viscoelasticity (see Fung (1990)) provides a unifying frame- work under which general soft tissue constitutive laws can be formulated. Such mod- els are often characterized by a constant hysteresis over a wider frequency range. Based on this approach we have a stress-strain relation of the form ∞ ∂e (x, γ) Z σ(x, t) = G(t − γ) dγ, (6.1) 0 ∂γ 160 where G(t) is the stress relaxation function and e (x, t) is the static elastic response of the tissue. The presence of the convolution integral in Eq. 6.1 makes the stress depend upon the strain time history, but it is the choice of the stress relaxation function that ultimately determines the tissue model. Typically, one introduces a parametric representation of G(t) that fits given experimental measurements (see Steele et al. (2011), Valdez-Jasso et al. (2011)). In the following sections we present the fractional-order Standard Linear Solid (SLS) model and its implementation in a one-dimensional blood flow solver. 6.2.1 The Fractional SLS model (Fractional Kelvin-Zener model) Before we present the fractional version of the SLS model, we start with the classical definition of the Caputo fractional derivative of order α (see Mainardi (2010))  1 R t f (n) (τ ) dτ, n − 1 < α < n   Γ(n − α) 0 (t − τ )α+1−n  C α 0 Dt f (t) := n , (6.2)  d f (t) , α=n   dtn where α > 0 is a real number, n is an integer, and Γ(·) is the Euler gamma function. We note that for α 6= n, the Caputo derivative is a non-local operator that depends on the history of f in the interval [0, t]. The fractional order generalization of the SLS model is constructed using the parallel combination of a spring with a spring in series with a spring-pot. The stress is related to strain as α αC α σ(t) + τσα C   0 D t σ(t) = E (t) + τ 0 D t (t) (6.3) 161 We observe that simpler fractional- and integer-order viscoelastic models as well as the purely elastic tube law appear as limiting cases in the above equation. Namely, for α = 1, we recover the classic integer-order SLS model, while taking τσ = 0 results to the fractional-order Voigt model. Thermodynamic consistency of the FO-SLS model requires a monotonically de- creasing stress relaxation function, which further implies that the fractional time derivatives of both the stress and the strain must be of the same order α (see Atanack- ovi´c et al. (2011), N¨asholm and Holm (2013)). By employing the Laplace transform, we can express the FO-SLS stress relaxation function as  α  α    α  τ τ t G(t) = E +E − 1 Eα,1 − , (6.4) τσ τσ τσ where Eα,β (t) is the two parameter Mittag-Leffler function, known as the fractional generalization of the exponential ∞ X tn Eα,β (t) := (6.5) n=0 Γ(α − nβ) Compared to the standard exponential function, the Mittag-Leffler function exhibits very different behavior Mainardi (2010): for small times it exhibits a much faster decay than the exponential, while at the limit t → ∞ the decay is only algebraic. This heavy tail decay is what characterizes the Mittag-Leffler function as a super-slow process (see Mainardi (2010)). Another important observation here is that for the FO-SLS model the idea of relaxation times must be revised. Under the fractional setting, they no longer repre- sent single discrete relaxation frequencies but they should be rather be interpreted as 162 break-frequencies in a Cole-Cole model sense, around which the model characteristics change (see N¨asholm and Holm (2013)). Finally, by substituting the FO-SLS stress relaxation function of Eq. 6.4 in Eq. 6.1, integrating by parts and employing the definitions in Eq. 5.5 we arrive at the FO-SLS pressure-area relation p(x, t) = pext + pE (x, t) + pV (x, t), (6.6) where pE , pV correspond to the elastic and viscoelastic pressure contributions, re- spectively. α √  E τ p p (x, t) = β( A − A0 ) τσ  α  Z t α  (6.7) t−γ    V 1 τ p (x, t) = 1− Eα,0 − pE (γ)dγ τσ τσ 0 τ σ We observe that the FO-SLS model introduces stress-strain memory effects that in the long time limit decay algebraically due to the Mittag-Leffler relaxation kernel (see Mainardi (2010)). It is important to note here that thanks to the linearity of the time-fractional ODE governing the fractional-order SLS model (Eq. 6.3), the Laplace transform allows us to arrive to Eq. 6.6, which corresponds to the exact solution of Eq. 6.3. An alternative way of formulating the required pressure-area relation arises from directly discretizing the Caputo time derivatives in the fractional constitute law that defines each model using an appropriate discretization technique. The easiest and most popular way of doing this is by employing the Gr¨ unwald-Letnikov formula (see 163 Podlubny (1998)): ∞ C α −α X k−α−1 0 Dt f (t) = lim ∆t GLαk f (t − k∆t), GLak := GLαk−1 , (6.8) ∆t→0 k=0 k with GLα0 = 1. By substituting the Gr¨ unwald-Letnikov formula in Eq. 6.3, we can formulate the pressure-area relation for the FO-SLS model as ∞ 1 + τα ∆t−α E ∆t−α X p(x, t) = pext + p (x, t)+ GLαk {τα pE (t−k∆t)−τσα p(t−k∆t)}, 1 + τσα ∆t−α 1 + τσα ∆t−α k=0 (6.9) where the last term in the summation is the total elastic and viscoelastic pressure from previous time steps. An attractive feature of this approach is that for a small time step ∆t, which is typically the case for the high-order polynomial approximations employed here (due to the Courant-Friedrichs-Lewy (CFL) condition), the Gr¨ unwald-Letnikov co- efficients exhibit fast decay properties. This enables us to reduce the computation of the convolution sum in Eq. 6.9, by using the “short memory” principle of Pod- lubny (1998), and approximate the viscoelastic memory effects using only a portion of the response history, disregarding any terms in the Gr¨ unwald-Letnikov expansion that are below a cutoff threshold. However, the elastic behavior of larger systemic arteries typically corresponds to low values of the fractional order (see Craiem and Armentano (2007), Craiem et al. (2008)), and the accurate evaluation of these con- volutions using the “short memory” principle requires one to consider history effects from the last four cardiac cycles. Our numerical experiments indicate that this is the minimum amount of time-history required by the Grunwald-Letnikov formula to give numerically stable and convergent results for the problem considered. With our goal being the long-time integration of Eq. 5.1, using a time step as low as ∆t = 10−6 , and with each cardiac cycle being about 1sec long, this results to storing at least 164 4 · 106 values per history point. This value, in combination with the requirement to compute a convolution sum at each quadrature point and for each time step, renders the simulation completely unrealistic even on modern supercomputers. In Section 6.2.2, we will present a formulation that alleviates the computational com- plexity introduced by fractionality, and finally arrive at a computable workflow that employs the exact solution of Eq. 6.3 and accounts for the full time history in the evaluation of hereditary integrals. It is important to note that viscoelasticity adds dissipation, leading to a system that is no longer strictly hyperbolic. However, the conservative elastic contribution is still dominant, thus allowing us to employ the Discontinuous Galerkin methodology presented in Sec. 5.2.2 without any numerical issues. We have observed that in cases in which the viscoelastic response is significant (integer-order models or fractional τ models with α close to 1.0) and the relaxation times ratio τσ becomes too large, the scheme becomes unstable. In such cases, due to the explicit nature of our method, stability is only recovered by decreasing the time-step sufficiently so that it resolves the viscous time scale. Alternatively, this issue could be alleviated by using a total variation diminishing (TVD) implicit time integration scheme (see Shu and Osher (1988)), however at the expense of increased computational cost and implementation complexity. 6.2.2 Evaluation of Hereditary Integrals A major computational challenge in considering viscoelastic models is posed by the need to evaluate the convolution integral in the expression for the viscoelastic pres- sure component, pv (t). This evaluation quantifies the stress-strain memory depen- dencies and needs to be repeated at every time step, and for every quadrature point 165 of the domain. A naive implementation of this operation scales quadratically with the number of time steps, introducing a major computational bottleneck. Moreover, a typical 1D simulation of the hyperbolic system of Eq. 5.1, involving 50 arteries with a polynomial order of P = 6, may impose a very restrictive CFL condition, requiring a time step as low as ∆t = 10−6 . This results to a very expensive computation both in terms of floating point operations and memory requirements. Here, we briefly present the methods and techniques we employed in this work to alleviate this issue and obtain a computable workflow. Integer-order SLS model: For the SLS model we explore the properties of the exponential relaxation kernel and derive a simple updating rule that is independent of the time history imposed by the convolution operator: Z t+∆t (t+∆t−γ) Z t+∆t (t+∆t−γ) − − ∆t It+∆t = e τσ e p (γ)dγ = e τ σ It + e− τσ pe (γ)dγ, (6.10) 0 t where the last term corresponds to the time interval [t, t + ∆t] and can be computed using standard quadrature rules. This effectively allows us to consider the full history of the arterial wall response, without introducing any computational or memory bottleneck, as only the solution from the last time step needs to be stored. Fractional order models: Due to the presence of power law (FOV-Kelvin-Voigt model) or Mittag-Leffler (FOV-SLS) relaxation kernels, fractional order models do not admit the separability concept introduced above to obtain a simple updating rule for the hereditary integral. However, one can still avoid the “brute force” com- putation by appropriately choosing a smart algorithm. Here, we have employed the method developed by Lubich and Sch¨adle (2002). Following their approach, we em- ploy the definition of the inverse Laplace transform, to generally write a hereditary 166 integral in the time interval (a, b) as: Z b Z b Z  1 (t−γ)s f (t − γ)g(γ)dγ = F (s)e ds g(γ)dγ a a 2πi Γ Z Z b (6.11) 1 (t−b)s = F (s)e e(t−γ)s g(γ)dγds, 2πi Γ a where the inner integral, denoted as y(b, a, s), is recognized as the solution at time b of the homogeneous scalar initial value problem: y 0 = sy + g, y(a) = 0 (6.12) Then, we can choose an integer base parameter B and discretize the global time interval in a local sequence of fast growing intervals, Il = [B l−1 ∆t, 2(B l − 1)∆t]. Moreover, we can apply an N point quadrature rule for the numerical evaluation of contour integral in Eq. 6.11, and discretize the convolution operation as: Z b N L X (l) (l) (l) (l) X f (t − γ)g(γ)dγ ≈ wj F (sj )e(t−γl−1 )sj y(γl−1 , γl , sj ), (6.13) a l=1 j=−N (l) (l) where the nodes sj and the weights wj are determined by the choice of the quadra- ture rule employed to evaluate the contour integral in Eq. 6.11 for every approxi- mation interval [t − γl−1 , t − γl ] ⊂ Il . Here, we have chosen a trapezoidal rule with equidistant steps to a parametrized Talbot contour (see Lubich and Sch¨adle (2002)), yet we note that other choices of contours, such as hyperbolas and parabolas, have been successfully employed by L´opez-Fern´andez et al. (2008). Now, the hereditary integral can be evaluated by solving the 2N + 1 ordinary differential equations in Eq. 6.12 at each node sj . Assuming that the convolution kernel f (t) is analytic for t > 0 and locally integrable, L´opez-Fern´andez et al. (2008) have proved that the method computes the convolution integral with spectral accuracy. 167 The key advantages of this algorithm is that it only requires O(nlogn) operations and O(n) memory, compared with the O(n2 ) operations and memory needed by a “brute force” approach for computing convolutions with n steps of history. Another advantage is that, for the FOV-SLS model, we avoid the direct numerical approx- imation of the Mittag-Leffler function, and we only need to evaluate the Laplace transform of its derivative, which is given by the following simple expression:    α  t 1 L Eα,0 − = F (s) = α 1 (6.14) τσ s + τσ We also report that in simulations where we adopted a “brute force” approach for computing Eq. 6.7, and directly evaluated the Mittag-Leffler function using the algorithm by Podlubny, the solver suffered from instabilities triggered by errors in the approximation of the Mittag-Leffler function, with stability being recovered only when the function was approximated to very high accuracy. Since here we are inter- ested in long-time integration of Eq. 5.1, direct approximation of the Mittag-Leffler function for long times and at the required level of accuracy is further problematic as it introduces a significant computational cost. In the next section we present patient-specific simulation results of the 1D model in Eq. 5.1, coupled to purely elastic (se Eq. 5.3), and more general SLS (see Eq. 5.10) and FOV-SLS (Eq. 6.6) viscoelastic models. Integer- and fractional-order Kelvin- Voigt models are just limiting cases of SLS models and will be omitted from our attention in what follows. 168 6.3 Results 6.3.1 Simulation setup All simulations were performed for a large patient-specific network that provides a detailed geometrical representation of the 50 largest arteries in the human brain, see Fig. 6.1. The data was obtained at the Department of Neurosurgery at Children’s Hospital, Boston MA, USA. First, the 3D geometry was reconstructed from high- resolution magnetic resonance images (MRI) using an “in house” developed software package, and then, the 1D domain was extracted from the centerlines of the 3D vas- culature. Consequently, the 1D representation considers arteries as straight tapering tubes, preserving the mean diameter and length of each of the 3D segments, while omitting curvature as well as bifurcation and junction branching angle information. The exact dimensions, model parameters and details of the reconstruction process for this network are given by Grinberg et al. (2011a). Flow is driven from PC-MRI flowrate measurements at the 4 inlets (see Fig. 6.1): internal carotid (ICA) and vertebral (VA) arteries (see Grinberg et al. (2011a)). The 21 distal outlets are coupled to 3-element windkessel models with parameters obtained from (see Grinberg et al. (2011a)). Given the unavailability of pressure data measurements, these parameters are generally hard to estimate and may have a significant effect on the flow characteristics, leading to even incorrect results (see Reymond et al. (2009)). Here, we have observed that the chosen data set leads to low sensitivities and a physiologically correct solution for our case of interest (see Grinberg et al. (2011a)). The purely elastic response of the arterial wall is quantified by the β parameter, the values of which have also been taken from Grinberg et al. (2011a). 169 Regarding spatial discretization, the number of DG elements is primarily chosen with respect to the vessel’s length. Mesh refinement is performed in regions of high flow gradients, with the final choice of the local polynomial order used (typically 3-8) resulting from mesh independence studies that ensure accurate resolution of the flow dynamics. Following Grinberg et al. (2011a), we have a system with 200 degrees of freedom, and the computed solution does not depend on further resolution refinement. The initial conditions for all simulations are (A, U )t=0 = (A0 , 0), and convergence to a periodic solution is obtained after two cardiac cycles. To ensure stability we have chosen a very small time-step ∆t = 6 · 10−6 s, a consequence of the high polynomial order that leads to a strict CFL condition and the viscoelastic dissipation introduced by the wall response. The computation of one cardiac cycle takes 30 seconds for elastic or integer-order viscoelastic models, and 15 minutes for fractional-order models (on a desktop with 8 cores of Intel Xeon E5607@2.27GHz). R. VA 8 R. VA L. ICA R. ICA 6 Q ( cms ) 3 4 2 0 0 0.2 0.4 0.6 0.8 1 t (sec) Figure 6.1: Simulation setup: Patient-specific cranial network from Grinberg et al. (2011a) and PC-MRI flowrate measurements at the four inlets (L. ICA, R. ICA, L. VA, R. VA) 6.3.2 Comparison of integer- and fractional-order viscoelas- tic models 170 We have performed 1D blood flow simulations in the aforementioned cranial net- work using the integer order SLS, and the fractional order SLS pressure-area rela- tion defined in Eqs. 6.6, 6.7. The integer-order SLS pressure-area relation is deter- mined by the triplet {β, τ , τσ }, while, its fractional-order counterpart, is described by {β, α, τ , τσ }. The nature of these parameters is patient-specific and their val- ues vary among anatomic locations, pathologies, age, etc. (see Bia et al. (2005)). Moreover, their accurate estimation in the clinical setting for various locations in an arterial network is a very challenging task. This leads to a very scarce selection of measured data in the literature, mostly corresponding to the aorta, the carotids and femoral arteries under ex-vivo or in-vivo conditions. In this study, we have chosen four different relaxation time parameter sets from the literature, two for the integer order SLS and two for the fractional-order SLS model. The first set (model SLS1) corresponds to the estimations of Valdez-Jasso et al. (2011), where an integer-order SLS model was calibrated to fit measured data for the thoracic descending aorta and the carotid artery under ex vivo and in vivo conditions in ovine and human arteries. The second set (model SLS2) employs the parameters reported by Lundkvist et al. (1996) for a healthy human femoral artery under ex vivo conditions. The third set (model FO-SLS1) is based on the measurements of Doehring et al. (2005), where a fractional-order SLS model was calibrated to ex vivo data for porcine aortic valve cusps. Finally, the fourth set (model FO-SLS2) corresponds to the mean values predicted by Craiem et al. (2008) for a fractional-order SLS model fitted to in-vivo data from human ascending aorta segments. For comparison purposes we have also included the results obtained by Grinberg et al. (2011a) using the 1D model with a purely elastic pressure-area relation (model Elastic). Since our goal here is the comparison of different viscoelastic models, we have kept the same elasticity parameters reported by Grinberg et al. (2011a) for 171 all cases considered. Table 6.1 summarizes the viscoelastic models we have employed in this section and their corresponding parameters. τ Model τ (sec) τσ (sec) τσ α SLS1 (see Valdez-Jasso et al. (2011)) 0.050 0.025 2.00 - SLS2 (see Lundkvist et al. (1996)) 29.3 16.9 1.49 - FO-SLS1 (see Doehring et al. (2005)) 1.84 0.076 24.12 0.29 FO-SLS2 (see Craiem et al. (2008)) 11.74 7.61 1.54 0.20 Table 6.1: Comparison of integer- and fractional-order viscoelastic models: Viscoelas- tic model parameters. In Figs. 6.2,6.3 we present the computed waveforms for flowrate and pressure as well as the pressure-area hysteresis loops due to viscoelastic dissipation at repre- sentative locations of the arterial network. We observe that all wall models except SLS2 lead to very similar flowrate wave propagation. However, with each model introducing a different amount of viscoelastic dissipation (see Fig. 6.3), the resulting local pressure waves present a variability up to 15%, although all models predict a very similar pressure drop between the inlets and the outlets. This is somewhat expected since the wall response is predominately elastic for all cases. The integer- order viscoelastic models correspond to a fractional order of α = 1.0, yet a strong viscoelastic behavior is not observed due to the chosen relaxation times τ ≤ 2τσ in both SLS1 and SLS2. Similarly, the relatively low values of the fractional order α in the FO-SLS1 and FO-SLS2 models leads to a dominant elastic response. Note, however, that the change in relaxation times from model SLS1 to SLS2 results to a significant change in the local pressure wave in each vessel (see Fig. 6.3), with the SLS2 model producing a stiffer response and predicting much smaller cross sectional wall displacements. This finding highlights the parametric sensitivity of the integer order SLS model: the discrete relaxation times {τ , τσ } dictate the viscoelastic wall 172 Domain # 23 Domain # 29 1.6 Elastic 4 SLS1 1.4 SLS2 FOV-SLS1 FOV-SLS2 Q ( cms ) Q ( cms ) 1.2 3 3 3 1 2 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 t (sec) t (sec) Domain # 44 Domain # 50 0.6 0.3 0.5 Q ( cms ) Q ( cms ) 3 3 0.2 0.4 0.1 0.3 0.2 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 t (sec) t (sec) Figure 6.2: Comparison of integer- and fractional-order viscoelastic models: Compar- ison of four viscoelastic model parameter sets: Variability in the computed flowrate at outlets # 23, 30, 44 and 50. response and strongly depend on the anatomic location. This dependence on the relaxation times introduces a caveat for using the integer order SLS model in simu- lations of large patient-specific arterial networks for which we only hope to have an estimate of τ and τσ at few limited anatomic locations. On the other hand, the behavior of the fractional models is dictated by the fractional order α which controls the interplay between elastic energy storage and viscoelastic dissipation. To better understand the role of α we consider a FO-SLS model and vary the fractional order between 0 ≤ α ≤ 1 while keeping the same relaxation timescales with the SLS1 model: τ = 0.050s and τσ = 0.025s. In Fig. 6.4 we present the computed pressure wave at the left internal carotid artery (L. ICA) and the resulting pressure-area hysteresis loops for different values of α along with the purely elastic and integer order viscoelastic response. As expected, for α = 0 we recover the purely elastic stress-strain response, while α = 1 returns the solution of the integer order SLS1 model. For α ≤ 0.5 the response is predominately elastic 173 L. ICA Domain # 29 100 100 Elastic SLS1 SLS2 80 80 FOV-SLS1 FOV-SLS2 p (mmHg) p (mmHg) 60 60 40 40 20 20 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 t (sec) t (sec) L. ICA Vessel # 29 13 2.34 A (cm2 ) A (cm2 ) 12.9 2.32 12.8 Elastic SLS1 2.3 SLS2 FOV-SLS1 12.7 FOV-SLS2 20 40 60 80 100 20 40 60 80 100 p (mmHg) p (mmHg) Figure 6.3: Comparison of integer- and fractional-order viscoelastic models: Compar- ison of four viscoelastic model parameter sets: Variabilty in the computed pressure wave and corresponding pressure-area hysteresis loop at the inlet (L. ICA) and distal outlet # 29. and the solution is insensitive to the choice of relaxation times, with viscoelastic dissipation being only observed for large relaxation times ratio (see Fig. 6.3 for model FO-SLS1). Increasing the fractional order, the viscoelastic response becomes more important and the choice of relaxation times has a significant impact on the computed solution, both in terms of the pressure waveform and the amount of dissipation introduced in the system. This is illustrated in Fig. 6.4, where we have included τ results for the FO-SLS model with a larger relaxation time ratio τσ = 4, while keeping the same short relaxation time τσ = 0.025s. This results to a pronounced viscoelastic response that noticeably affects the pressure wave propagation and the pressure-area hysteresis loops. For α = 1.0 the fractional-order SLS model reproduces the integer-order SLS model behavior and suffers from the same sensitivities. However, experimental stud- ies of Craiem and Armentano (2007), Doehring et al. (2005), Craiem et al. (2008) suggest that α ≈ 0.1 ÷ 0.3 for larger systemic arteries, while they relate higher 174 α = 0.0 13.1 70 α = 0.2 α = 0.4 α = 0.8 13 60 α = 1.0 p (mmHg) α = 1.0, τǫ =4 A (cm2 ) τσ 50 12.9 40 12.8 30 12.7 0 0.2 0.4 0.6 0.8 1 30 40 50 60 70 t (sec) p (mmHg) Figure 6.4: Comparison of integer- and fractional-order viscoelastic models: Com- puted pressure and pressure-area hysteresis loop at the inlet (L. ICA) for different values of the fractional order 0 ≤ α ≤ 1. The dashed line corresponds to the integer-order model SLS1 (recovered for α = 1.0) with a wider relaxation spectrum: τσ = 0.025s, ττσ = 4. fractional orders to smooth muscle activation, which is a key flow auto-regulation agent in arteriolar networks. This may well indicate that muscular small arteries and arterioles exhibit higher fractional orders. Our simulations indicate that for α ∈ [0.0, 0.8], the effect of the FO-SLS model on the computed blood flow is primar- ily determined by the fractional order α with low sensitivity on the relaxation times ratio used. 6.3.3 Global sensitivity analysis of the stochastic fractional SLS model τ We consider a stochastic fractional order SLS model in which α, τσ , and τσ are uniformly distributed random variables with value range taken from the literature: τ α ∈ [0, 1], τσ ∈ [1, 2], and τσ ∈ [0.02, 20.0]. The corresponding three-dimensional parametric space is discretized with the probabilistic collocation method (PCM) on a tensor product grid generated by the nodes and weights of Legendre polynomials (see Xiu and Karniadakis (2002)). The number of collocation points per random dimension is 8, leading to a total of 512 sampling points. The simulation ran in 175 parallel on 2048 cores of BG/Q, resulting to a total runtime of 1 hour for 3 cardiac cycles. In Figs. 6.5,6.6 we present the computed mean flowrate and pressure wave ±1 standard deviation for two representative locations in the network. The flowrate wave shows a deviation of less than 5% from the mean, indicating that wall vis- coelasticity has a minor effect on its propagation. This fact is also in agreement with the results obtained when comparing the integer and fractional-order SLS models (Fig. 6.2). On the contrary, the pressure waveform displays significant variations primarily with respect to the fractional order α, but also with respect to the relax- ation times ratio when α → 1 and the viscoelastic behavior is pronounced. This identifies the role of viscoelastic response as an agent that regulates pressure wave propagation by introducing viscoelastic energy dissipation. This effect is expected to be more pronounced in distal locations where arteries are more muscular and the associated fractional orders are high. Domain # 44 Domain # 50 0.6 0.3 0.5 Q ( cms ) Q ( cms ) 3 3 0.2 0.4 0.1 0.3 0.2 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 t (sec) t (sec) Figure 6.5: Global sensitivity analysis of the stochastic fractional SLS model: Com- puted mean flowrate (in black) and a variation of ±1 standard deviation (in red) at distal outlets # 44 and 50. 176 L. ICA Domain # 29 80 80 p (mmHg) p (mmHg) 60 60 40 40 20 20 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 t (sec) t (sec) Figure 6.6: Global sensitivity analysis of the stochastic fractional SLS model: Com- puted mean pressure (in black) and a variation of ±1 standard deviation (in red) at the inlet (L. ICA) and distal outlet # 50. To further investigate the parametric sensitivity of pressure wave propagation on viscoelasticity we have performed a global stochastic sensitivity analysis study by computing the standard ANOVA decomposition (see Yang et al. (2012)) of the time- depended solution. Figure 6.7 shows the contribution of each random parameter and their second-order interactions to the total variance of the solution corresponding to the pressure at one of the inlets (left internal carotid artery, L. ICA). Clearly, the fractional order α is the most important parameter, with maximal influence dur- ing the systolic phase. This confirms the results presented in the previous sections, illustrating the effect of viscoelastic dissipation (introduced by increasing the frac- tional order) on the local pressure wave. The second most important parameter τ is the relaxation times ratio τσ . In a similar fashion, a larger ratio introduces vis- coelastic dissipation which, in turn, affects the local pressure, especially during the systolic phase. The opposite situation is observed for the short relaxation time τσ , as it only appears to have a noticeable variance contribution during the diastole. Finally, Fig. 6.7 shows the variance contributions of pairwise interactions of the ran- dom parameters, indicating that their effect on the flow is less significant, especially during the systolic phase. We note that although these results are expected for the larger cranial arteries considered here, they provide a clear map of how arterial wall viscoelasticity affects hemodynamics. 177 0.4 {α} 0.3 { ττσǫ } Si , Sij {τσ } 0.2 {α, ττσǫ } {α, τσ } 0.1 { ττσǫ , τσ } 0 0 0.2 0.4 0.6 0.8 1 t (sec) Figure 6.7: Global sensitivity analysis of the stochastic fractional SLS model: Vari- ance decomposition of the computed pressure waveform at the inlet (L. ICA). Si and Sij are the global sensitivity indices of each random parameter and their pairwise interactions, respectively. 6.4 Discussion We have performed deterministic and stochastic 1D blood flow simulations in a large patient-specific cranial network of compliant arteries corresponding to four different viscoelastic parameter sets. To this end, we have developed a fast parallel solver that supports integer and, for the first time, fractional wall models, enabling us to analyze in detail the effect of viscoelasticity on pulse wave propagation. The solver can be made available to the community upon request and it is written in C++, employing the MPI and OpenMP libraries to enable parallel, multithreaded processing. First, we compared several models with the same elasticity parameters while the viscoelastic parameters are calibrated with data found in the literature. This com- parison study indicated that all models lead to a similar flowrate wave propagation but the resulting local pressure waves are highly affected by the amount of viscoelas- tic dissipation introduced by each model. We observed that the viscoelastic behavior 178 of integer-order models is strongly dependent on the choice of the relaxation times, while for fractional models, the response is dictated by the fractional-order. This fact makes fractional-order models very flexible as their conservative or dissipative nature is tuned by a single parameter, the fractional order α, while sensitivity of the solution to the relaxation parameters is only observed as α → 1. Consequently, fractional-order wall models provide a flexible tool, which may not only accurately capture the static response of the arterial wall but also, by dynamically tuning the fractional order α (which can be a function of time-space in fractional PDEs). One very interesting thrust for future work is neurovascular coupling, in which neural activity could be used to dynamically tune a variable-order fractional viscoelastic wall model model, accounting for smooth muscle activation effects that lead to va- sodilation or vasoconstriction and provide a pressure auto-regulation mechanism (see Witthoft and Karniadakis (2012)). To further investigate the behavior of fractional viscoelastic models, a detailed uncertainty quantification study was performed. Using a stochastic wall model, we have computed a broad range of parametric combinations and reported the mean so- lution as well as the variabilities observed. We confirmed that wall viscoelasticity has only minor effects on flowrate wave propagation, while pressure waves and resulting wall displacements present variability in their phase and magnitude depending on the amount of viscoelastic dissipation introduced by the fractional order. Finally, by performing a global sensitivity analysis study, we have quantified the relative impor- tance of the viscoelastic parameters and their second-order interactions throughout the span of a full cardiac cycle. One of the limitations of the present work is that the viscoelastic behavior of each arterial wall model is defined by a single global set of viscoelastic parameters for all vessels. This is due the very limited availability of experimental results for most 179 of the cranial arteries considered here. We believe that tuning the fractional-order locally for every segment will lead to a more concise modeling approach and we hope that future experimental studies will shed more light on how to appropriately choose it for different anatomic locations, pathologies, etc. Finally, we must underline that validation is definitely the only path for accept- ing new tissue models, yet it still remains a very challenging task. In cases where local relaxation measurements are available, simple integer-order viscoelastic mod- els can be calibrated to yield as good results as more general models with more relaxation timescales (see Valdez-Jasso et al. (2011)). However, noninvasive param- eter estimation becomes very hard for most distal anatomic locations, making these models suffer from high sensitivity to the relaxation timescales. These issues are key to the practical applicability of any model, and the main reason that may lead one to consider fractional-order models, leveraging on their better sensitivity proper- ties and ability to model continuous relaxation at the expense of a single additional parameter, the fractional order α. Chapter Seven A Fractal-tree Closure Model for Simulating Blood Flow in Large Arterial Networks 181 7.1 Motivation The field of computational hemodynamics has undergone great growth in the last 20 years, producing more reliable mathematical models, utilizing modern computer architectures for parallel simulations, and allowing for high fidelity simulations in computational domains reconstructed from patient-specific medical imaging data (see Xiao et al. (2013), Grinberg et al. (2013), Reymond et al. (2011)). While simu- lations have significantly pushed the boundaries of problem size, performing full scale blood flow simulations on the human arterial network remains intractable hence the need for formulating a proper closure problem for simulations in truncated domains. It is a common practice to reconstruct blood vessel network from medical images covering a particular region, where the resulting models capture only relatively large vessels and additional modeling is required to account for hemodynamics in the trun- cated vasculature. Specifically, in this article we focus on modeling outflow boundary conditions for simulations in arterial networks with multiple outlets. Generally speaking, there are two options to properly truncate the computational domain a) use flowrate and/or pressure measurements to directly impose patient- specific data at the outlets, i.e., avoid modeling the truncated arterial tree; and b) use low-dimensional models to simulate the flow in the truncated domain. Clearly, in the absence of clinically measured data one has to resort to modeling. Ideally, closure models should be based on sound assumptions with respect to the physics of the problem, be as free as possible of parametric sensitivities, and be fast to solve. In the following we review the current closure models and point to their advantages and limitations. The widely used simplest representation of the sub-pixel vasculature comes from 182 zero-dimensional (0D) lumped parameter models. This approach consists of additive combinations of electrical elements, such as resistors R, capacitors C and inductors L, that aim to capture the effective resistance and compliance effects of the ne- glected arterial networks (see Alastruey et al. (2008), Reymond et al. (2011)). The main advantage of this approach is its implementation simplicity and low computa- tional cost, as it only requires the solution of a linear ordinary differential equation at each terminal outlet. The key disadvantage of this method is the inherent diffi- culty in estimating the 0D model parameters in absence of local flow measurements at the outlets, in combination with the high sensitivity of the flow model output on these parameters. Recent studies have proposed iterative calibration procedures, in which the problem is solved multiple times until a set of parameters that yields a physiological solution is identified (see Blanco et al. (2012), Melani and Quar- teroni (2013), Lombardi (2014a)). Despite their evident limitations, 0D models have been extensively used in the literature (see for e.g., Reymond et al. (2011), Alastruey et al. (2008), Grinberg and Karniadakis (2008), Blanco et al. (2012), Watanabe et al. (2013)), and results for accurately calibrated models have been successfully validated against in-vivo measurements (see Reymond et al. (2011)). However, it seems that the obtained windkessel model parameters are not uniform for each arterial net- work and a calibration procedure has to be repeated to accommodate any network modifications. An alternative closure model, first put forth by Olufsen (1999), considers mod- eling the downstream vasculature as a self-similar network of compliant arteries in which a linearized one-dimensional (1D) model is used to estimate the pressure out- flow boundary condition as a function of the flowrate time history. The structure of the fractal arterial trees is motivated by the findings of Zamir (1999), while the elas- ticity parameters vary with the vessel’s radius by fitting experimental measurements. 183 The resulting outflow boundary condition is obtained by computing a convolution operator that relates pressure to flow time history at each of the outlets. Although this approach offers a more detailed representation of the downstream dynamics, it is known to be highly sensitive to the cut-off radius of the structured tree (see Cousins and Gremaud (2012)), suggesting the need for a calibration procedure. Some other limitations of this method include the high number of simulation cycles required to reach a periodic flow state, the costly computation of the convolutions in every time-step, as well as the neglect of the Fahraeus-Lindqvist effect in small arteries and arterioles (see Pries et al. (1992)) , and the viscoelastic response of the arterial wall. Recent advances have extended the use of the structured tree model to unsteady problems and explored the use of the Laplace transform to speed up the convergence to a periodic flow state (see Cousins et al. (2013)). In the case where flow measurements at the outlets are present, Grinberg and Karniadakis (2008) have proposed a simple way of constructing a two-element 0D model that is able to reproduce the measured data in-silico. According to Grinberg and Karniadakis (2008), the time varying resistance parameters are derived from the measured flowrates, hence no calibration of the 0D model is required, and simulating 5-10% of a cardiac cycle is sufficient for the mass flow distribution to converge to the reference data. In this work our main goal is to construct a robust model that eliminates para- metric sensitivities and is able to provide a physiological closure to patient-specific hemodynamics simulations with a large number of outlets. We simulate the trun- cated network by constructing fractal networks of small arteries and arterioles with radii in the range (10 − 500µm) that bridge the gap between the outlets of a patient- specific domain and the capillary bed. We simulate hemodynamics using a fully nonlinear 1D solver that accounts for non-Newtonian flow effects and wall viscoelas- 184 ticity. As this task requires us to resolve flow in networks of thousands to millions of arteries, we have scaled up a parallel 1D Discontinuous Galerkin solver to mod- ern multi-core architectures, moving closer towards the capability of producing near real-time numerical solutions. This chapter we extend the nonlinear 1D blood flow formulation presented in Chapter 5 to performing simulations in massive fractal trees. In Sec. 7.2 we outline the nonlinear 1D closure model and highlight key implementation aspects. In Sec. 7.2 we present the simulation setup for all test cases considered. Then, in Sec. 8.3 we present the main findings of this study. In particular, we address the flow sensitivity issue on the terminal cut-off radius and propose a way to eliminate it by introducing an ansatz on the vessel’s length-to-radius ratio for small arterioles. Moreover, we study how the predicted hemodynamics is affected by the Fahraeus-Lindqvist effect (see Pries et al. (1992)) in small arteries and arterioles, and by different choices of boundary conditions at the very distal outlets of the fractal domains. Finally, section 7.6 summarizes our key findings and provides motivation on possible future thrusts of research. 7.2 Fractal-tree closure model Here we build upon the structured tree framework first put forth by Olufsen (1999) and model downstream flow dynamics by structured trees that aim to bridge the geometric scale gap between outlets with diameters of O(cm) and the small arterioles that enter the capillary network O(µm). This approach is motivated by the findings of Zamir (1999) and Cassot et al. (2010), which indicate that branching patterns of arteries in the meso-vascular regime resemble fractal laws as follows. A parent vessel 185 of radius r0 bifurcates into two daughter vessels with r1 = δr0 , r2 = βr0 , where  2 z/2 −1/z √ r2 δ = (1 + γ ) , β = δ γ, γ= , r2 ≤ r1 (7.1) r1 Starting from an outlet with radius r0 , the resulting fractal-tree is uniquely deter- mined by the parametric set of {rcut , L/r, δ, β, γ, z}, where L/r is the length over radius ratio of each vessel and rcut is the terminal cut-off radius of the tree (see Fig- ure 7.1), β, δ are the parent-daughter radius ratios, γ quantifies the asymmetry of the network, and z is a power law exponent that describes the fluid flow distribution from parent to daughter vessels at each bifurcation (see Zamir (1999), Cassot et al. (2010)). Of these parameters, only rcut may be considered as user input, while the rest may be estimated from medical imaging either in a patient-specific setting or based on population samples. Although δ, β, γ and z seem to be independent of loca- tion in the arterial tree (see VanBavel and Spaan (1992), Zamir (1999), Cassot et al. (2010)), the length to radius ratio L/r exhibits a local character, varying significantly among different organs and typically taking values within the range L/r = 10 − 60 (see Zamir (1999)). The choice of these parameters determines the size of the gen- erated fractal-tree, and, therefore, the total downstream resistance experienced by each outlet. The elastic property of the arterial wall is estimated using the experimental fit put forth by Olufsen (1999) that relates the Young’s modulus times the thickness of the arterial wall with the vessel radius (see Fig. 3 in Olufsen (1999)). This is the exact relation that has been employed in our work for r > 500µm, where the curve gives a good fit to the observed experimental data. However, for r < 500µm the exponential fit becomes singular as the vessel radius approaches 0. This singularity results to unrealistic predictions, as it implies that small arteries become exponentially stiffer with decreasing radius. To overcome this limitation, we have 186 assumed that vessels with r < 500µm have the same elastic modulus, set to the value predicted by Olufsen’s law for r = 500µm, corresponding to a threshold value of Eh/r0 = 7.35 · 106 g/sec2 /cm. Although this choice is arbitrary, we believe it is realistic as small arteries and arterioles are not known to exhibit mechanical responses that may differ by up to 2 orders of magnitude, as predicted from Olufsen’s fit for r << 500µm. R. VA 8 R. VA L. ICA R. ICA Q ( cms ) 3 6 4 2 0 0 0.2 0.4 0.6 0.8 1 t (sec) Figure 7.1: Fractal-tree closure model: Left: Fractal-trees are attached at each outlet of a patient-specific arterial network, resulting to an arterial domain with millions of compliant vessels in which blood flow is modeled using Eq. 5.1. Right: Flow is driven by high resolution, patient-specific PC-MRI measurements at the inlets (right vertebral artery R. VA, left vertebral artery L. VA, right internal carotid artery R. ICA and left internal carotid artery L. ICA). 7.3 Implementation aspects In this work we model flow dynamics in the fractal network using the non-linear con- servation law of Eq. 5.1. This allows us to simulate unsteady blood flow in arterial networks spanning multiple spatial scales, to naturally account for the vascular resis- tance of the arteriolar regime, as well as to resolve the systemic pressure drop, most 187 of which is known to occur at the level of arterioles (see McDonald (1974)). More- over, we are able to consider important non-Newtonian flow effects in small arteries and arterioles, as well as to include the effects of wall viscoelasticity. From the im- plementation perspective, this approach introduces a computational challenge as the resulting arterial network may consist of millions of degrees of freedom, mandating the need for efficient numerical methods and algorithms. As the cut-off radius is decreased networks consisting of millions of arteries are generated, rendering simulation a non-trivial task. Here, we address this compu- tational challenge by developing a parallel solver based on hybrid programming paradigm that utilizes the MPI and OpenMP libraries, allowing us to decompose the problem among multiple compute nodes as well as explore intra-node parallelism via multithreading. This approach allowed us to scale up our 1D blood flow solver on the IBM Blue Gene/Q architecture and integrate Eq. 5.1 at the rate of 10−8 s per vessel, per time-step. Based on our experience in simulating 1D flow in networks of thousands to millions arteries, about 100-150 of modern CPUs are adequate for simulating one cardiac cycle in about five minutes. We also envision that further reduction in required CPU time is possible by performing code optimization and scaling to larger number of CPUs. 7.4 Case studies and simulation setup Flow in the Circle of Willis (CoW): For our first test case we consider a large patient-specific network consisting of the 50 largest arteries in the human brain, see Fig. 7.1. The data was obtained at the Department of rengachary2005principles at Children’s Hospital, Boston MA, USA. First, the 3D geometry was reconstructed 188 from high-resolution magnetic resonance images (MRI) using an “in house” devel- oped software package, and then, the 1D domain was extracted from the centerlines of the 3D vasculature. Consequently, the 1D representation preserves the mean di- ameter and length of each of the 3D segments, while omitting curvature as well as bifurcation and junction branching angle information. The exact dimensions and elasticity parameters of each arterial segment, as well as details of the reconstruction process are given in Grinberg et al. (2011a). The nonlinear 1D blood flow system (Eq. 5.1) is solved in the patient-specific parent network (50 cranial arteries) and the fractal trees attached to each one of the 21 outlets. The network has 4 inlets (see Fig. 7.1): the two internal carotid (ICA) and two vertebral (VA) arteries, where we apply accurate PC-MRI flowrate measurements (see Grinberg et al. (2011a)). To ensure stability we have chosen a very small time- step ∆t = 10−6 s due to the high polynomial order (ranging between 3-7 depending on the vessel’s length) and the viscoelastic dissipation introduced by the arterial wall response modeled using Eq. 6.6 with α = 1 (integer-order Standard Linear Solid model). The elasticity parameters are taken from Grinberg et al. (2011a), the viscoelastic relaxation times where set to τ = 0.050s and τσ = 0.025s (see Perdikaris and Karniadakis (2014)), and the external pressure pext was taken equal to 15mmHg, corresponding to the mean pressure of the cerebrospinal fluid (see Reymond et al. (2011)). Flow in a detailed network of the arm: For our second test case we con- sider a detailed arterial representation of the arm based on the work of Watanabe et al. (2013). This network consists of 246 arteries with 105 terminal outlets, 124 bifurcation points, 18 merging junctions and 85 perforator vessels (see Watanabe et al. (2013), and Fig. 7.2). Each arterial segment is discretized in space using 1 DG element per centimeter of vessel’s length and a polynomial approximation of 189 second order, while the time-step was taken equal to ∆t = 10−5 s. Following Watan- abe et. al. we have employed a purely elastic constitutive law parametrized by the data reported by Watanabe et al. (2013). Flow is driven by imposing a physiological flowrate waveform at the inlet of the axillary artery based on information available in the medical literature (see Watanabe et al. (2013)). Similarly, the 1D flow equations (Eq. 5.1) are solved treating the parent arm network (276 arteries) and the fractal trees attached to each one of the 103 outlets, as a whole. Figure 7.2: Case studies and simulation setup: A detailed arterial network of the arm, courtesy of Watanabe et al. (2013) (276 arteries with 105 terminal outlets, 124 bifurcation points, 18 merging junctions and 85 perforator vessels). In both the arm and cranial cases we have imposed absorbing wave outflow boundary conditions at all distal outlets of the fractal-trees by setting the corre- sponding Riemann invariant to zero. Blood viscosity is variable in space with the vessel’s radius according to the experimental fits of Pries et al. (1992), while the blood hematocrit is considered constant and equal to 0.45. The initial conditions are (A, U, p)t=0 = (A0 , 0, pext ), and discretization parameters are chosen such that the computed solution does not depend on further resolution refinement. Starting from each outlet of the of the parent CoW or arm network and given a user specified cut-off radius, we construct several generations of asymmetrically bifurcating fractal-trees. The trees are parametrized by {rcut , L/r = 50, δ = 0.9, β = 0.6, γ = 0.41, z = 2.76}, following the values suggested by Zamir (1999). In Table 7.1 we report the size of the resulting arterial network as well as the average number of generations attached to the parent domain for both the CoW and arm cases. 190 CoW Arm rcut (µm) # of vessels Avg. # of gen. # of vessels Avg. # of gen. 200 1,286 14 - - 100 8,456 21 1,400 14 50 56,484 27 9,046 19 30 226,522 32 36,228 24 20 682,450 36 109,567 28 10 4,392,484 43 702,188 34 Table 7.1: Fractal tree closure models: Total number of vessels and average number of generations attached to each outlet of the parent CoW and arm networks as a function of the fractal-tree cut-off radius rcut . 7.5 Results In this section we present the main findings of our study. First, we demonstrate the well known issue of sensitivity on the cut-off radius of the fractal-trees (see Cousins and Gremaud (2012)), and propose an effective way of eliminating it by introducing an ansatz for selecting the length-to-radius ratio of the generated small arteries and arterioles. Second, we study two different blood rheology models in order to address the Fahraeus-Lindqvist effect (see Pries et al. (1992)) in small arteries and arterioles and quantify its effect on hemodynamics. Last, we present results on the effect of imposing different types of outflow boundary conditions at the distal outlets of fractal-trees. 7.5.1 Model sensitivity on the cut-off radius 191 Results of the first set of simulations with fixed L/r ratio are shown in Figure 7.3, The numerical solution is obtained using fractal-trees parametrized by the values suggested by Zamir (1999). Pressure and flow rate waves are probed at the midpoint of the basilar (CoW case) and radial (arm case) arteries, respectively, demonstrating the high sensitivity on the choice of the cut-off radius. In particular, although the model predicts a consistent flow distribution and pressure drop across the arterial network, the computed pressure wave has a non-physiological magnitude, with the systolic pressure exceeding 200mmHg as the cut-off radius is decreased. This ob- servation is in agreement with the results reported by Olufsen (1999), Steele et al. (2007), and Cousins and Gremaud (2012), Cousins et al. (2013), highlighting the main drawback of structured tree-type outflow boundary conditions. A justification of this inconsistency can be given based on the fact that as we decrease the cut-off radius, we greatly increase the size of the generated fractal-trees, leading to a poten- tial overestimation of the total downstream resistance experienced by the outlets of the parent network. 200µm 100µm rcut rcut 20µm 10µm Figure 7.3: Model sensitivity on the cut-off radius: Sensitivity on the cut-off radius rcut : Computed pressure (blue) and flowrate (red) waves or different values of the cut-off radius rcut . Right: Basilar artery (CoW case). Left: Radial artery (arm case). 192 7.5.2 Overcoming the cut-off radius sensitivity One possible solution is to terminate the fractal-tree once the total volume of the generated network reaches a target value that is based on measuring (or estimating) the arterial fraction of the total blood volume considered. This approach can be applied for the CoW case as follows. First, we assume an estimate for the cere- bral blood volume of about 120mL (see Rengachary and Ellenbogen (2005)). Then, following the findings of Ito et al. (2001), we take the true arterial fraction of the cerebral blood volume to be about 30%. Using this estimate we can compute the target arterial volume as 120 · 30% ≈ 35mL. The main drawback of this approach is that it lacks generality as the suggested termination criterion is only applicable for cases where an estimate of the blood volume and its corresponding arterial fraction at the region of interest is known. Another possible solution is to adopt a tiered approach for selecting the parame- ters that govern the structure of fractal-trees, allowing {L/r, δ, β, γ, z} to vary with the vessel’s radius based on experimental data (see VanBavel and Spaan (1992)). This approach was successfully employed by Steele et al. (2007) and Cousins and Gremaud (2012), Cousins et al. (2013), Cousins (2013), although it was reported to be case dependent, potentially leading to inconsistencies if no model calibration is performed (see Cousins (2013)). While our own experience confirms these findings, here we focus our study on testing an alternative way of selecting the length to ra- dius ratio L/r that can effectively eliminate the sensitivity on the cut-off radius and produce physiologically correct results for patient-specific networks without the need of model calibration. To this end, we consider {δ, β, γ, z} to be constant across the tree, while introducing the ansatz that the L/r ratio is a linearly decreasing function once the radius becomes smaller than a given threshold r? (see Figure 7.4). 193 L/r = f (r) ansatz V = f (rcut) 60 45 L/r = 50 constant ⋆ ⋆ ⋆ r150 r250 r500 L/r = f (r), r ⋆ = 150µm 50 40 L/r = f (r), r ⋆ = 250µm L/r = f (r), r ⋆ = 500µm 40 35 V (mL) L/r 30 30 20 25 10 0 20 0 100 200 300 400 500 600 20 40 60 80 100 120 140 160 180 200 rcut (µm) rcut(µm) Figure 7.4: Overcoming the cut-off radius sensitivity: Left: Decreasing L/r ansatz for different thresholds r? . Right: Volume of the generated arterial tree as a function of the cut-off radius for different L/r relations (CoW case). The dashed line corresponds to the estimated target volume of 35mL. In Figure 7.5 we present simulation results where the fractal-trees were generated using the proposed ansatz for the L/r ratio and no prior model calibration was performed. The solution is probed at the same spatial locations, namely the midpoint of the basilar (CoW case) and radial (arm case) arteries, respectively. For the CoW case, we have chosen a threshold of r? = 150µm, as this value generates a network that matches the target volume of 35mL for a cut-off radius rcut = 30µm (see Figure 7.4). We observe that as the cut-off radius is decreased, the propagated flow and pressure waves converge to a physiologically correct plateau. This convergence is best quantified by computing the relative error of the pressure wave between cases with different cut-off radii and a reference case (see table inscribed in Figure 7.5). We also confirm that these results can be also reproduced if we assumed a constant L/r ratio and terminated the fractal-trees once the target volume of 35mL is reached (rcut = 60µm, see Figure 7.4). For the arm case, there does not exist (to our knowledge) an estimate for neither the blood volume nor for the arterial fraction of the blood volume. Hence, for this 194 case, no estimate for the target volume can be computed, and r? needs to be tuned by the user in order for the model to return physiologically correct results. Moreover, pressure wave propagation is expected to be sensitive to the choice of r? , hence a calibration procedures is needed to obtain physiologically sound results. Although this is a limitation of the proposed framework, we believe that it is not a major one as only a single parameter, r? , needs to be calibrated. On the contrary, if one wishes to employ the most widely used lumped closure model, the 3-element windkessel, immediately faces the problem of calibrating 206 resistance and capacitance param- eters (the parent network consists of 276 arteries and 103 outlets) (see Watanabe et al. (2013), Reymond et al. (2011), Xiao et al. (2013)). Consequently, here we have chosen a L/r relation with a threshold radius of r? = 500µm (see Figure 7.4). Similarly, the flowrate and pressure waves converge to a plateau solution that is no longer sensitive on further decreasing the cut-off radius rcut (see Figure 7.5). Here, although the predicted waveforms are able to reproduce a physiological flow distri- bution, pressure drop and key wave propagation features such as the dicrotic and the anacrotic notch (see Figure 7.6), the computed pressure exceeds the physiologically correct range. This is because the arm network (courtesy of Watanabe et al. (2013)) is not patient-specific but has been meticulously constructed from anatomical ref- erences available in the medical community, and our choice of r? = 500µm was an approximation. Specifically, this network is missing several outlets that need to be considered if we hope to compute a physiologically correct pressure wave without the need of several iterations of a model calibration procedure. 195 200µm 100µm rcut rcut 20µm 10µm rcut (µm) 200 100 50 30 rcut (µm) 100 50 30 20 ||p20µm − p(·) ||2 ||p10µm − p(·) ||2 (%) 20.73 12.81 5.72 1.98 (%) 3.3 1.74 1.11 0.74 ||p20µm ||2 ||p10µm ||2 Figure 7.5: Overcoming the cut-off radius sensitivity: Waveform convergence with de- creasing cut-off radius using the L/r ansatz : Computed pressure (blue) and flowrate (red) waves for different values of the cut-off radius rcut . Left: Basilar artery (CoW case). Right: Radial artery (arm case). Inscribed tables contain the relative error (L2 norm) of the pressure wave as we decrease the cut-off radius (reference cases: rcut = 20µm for CoW and rcut = 10µm for the arm). 7.5.3 The Fahraeus-Lindqvist effect In order to accurately model blood flow in small arteries and arterioles (50µm ≤ r ≤ 500µm), one needs to take into account the dependence of the apparent blood viscosity on the hematocrit and the vessel’s radius (the Fahraeus-Lindqvist effect, see Pries et al. (1992)). Here, we have performed two sets of simulations considering the cases of constant blood viscosity (Newtonian flow) versus radius-dependent viscosity (non-Newtonian flow) using the model proposed by Pries et al. (1992). The model is derived from experimental data obtained at high shear rates (≥ 50s−1 ) and provides a clear correlation between blood viscosity in straight tubes, tube diameter and blood hematocrit. Our findings indicate that the effect of non-Newtonian rheology is primarily man- ifested in the propagation of the pressure wave. In Figure 7.6 we present computed pressure waves using the L/r ansatz at the left internal carotid artery (CoW case, 196 r? = 150µm) and the axillary artery (arm case, r? = 500µm). In both cases the fractal-trees are terminated at rcut = 20µm, and the solutions have converged to a plateau where they are no longer sensitive to further decreasing the cut-off radius. We observe that modeling blood as a Newtonian fluid has a significant effect on pressure wave propagation, resulting to an effective stiffening of the arterial wall response that leads to an upward shift of the pressure wave, pushing the predicted values away from the physiological range. On the other hand, the radius-depended viscosity model returned a physiologically correct solution, highlighting the fact that capturing non-Newtonian blood behavior is crucial for correctly modeling hemody- namics in small arteries and arterioles. Our results highlight that different rheology models may result to significantly different predictions far upstream in the network due to the cumulative effect of the large arteriolar networks modeled by the fractal tree. systolic peak anacrotic notch diacrotic notch diastolic runoff systolic upstroke end-diastolic pressure Figure 7.6: The Fahraeus-Lindqvist effect: Computed pressure waves for different blood rheology models: radius-depended viscosity suggested by Pries et al. (1992) (non-Newtonian, solid line) versus constant blood viscosity (Newotnian, dashed line) Left: Left internal carotid artery (L. ICA, CoW case). Right: Axillary artery (arm case). 7.5.4 Effect of distal outflow boundary conditions The generated fractal arterial networks typically contain tens of thousands of distal outlets for which we need to provide a proper outflow boundary condition. Here, we 197 demonstrate that the choice of the prescribed outflow boundary condition has minor effects on the predicted hemodynamics. To this end, we have considered three dif- ferent types: absorbing wave, constant pressure, and 3-element windkessel boundary conditions. Absorbing conditions are applied by zeroing out the incoming Riemann invariant at the outflow boundary interface, ensuring that no wave reflections will occur. The use of constant pressure boundary conditions is primarily motivated by the fact that the pulsatility of the flow fades out as we approach the capillary bed (arterioles with radius r < 30µm), and estimates for the pressure values at these lo- cations do exist in the medical literature (see McDonald (1974)). Another possibility is to use simple 3-element windkessel models, consisting of a resistor in parallel with a resistor and capacitor (RCR), with the RCR parameters being roughly estimated based the total resistance of the constructed network and the target pressure values at the outlet (see Alastruey et al. (2008)). Note that due to very small variation of flowrate at terminals of the fractal network the RCR models becomes effectively the resistance model, i.e., the pressure-flowrate relation can be accurately estimated from P = Rtotal Q. In Figure 7.7 we present the computed flowrate and pressure waveforms at repre- sentative locations for the CoW and arm networks. In both cases we have used the suggested L/r ansatz (CoW: r? = 150µm, arm: r? = 500µm), with the fractal-trees being terminated at rcut = 30µm. The target pressure for the constant pressure and 3-element windkessel boundary conditions was set to 20mmHg, a representative value for small arterioles (see McDonald (1974)). Our results indicate that both the mass flow distribution as well as the local pressure waves converge to a periodic state after two cardiac cycles (starting from zero-flow initial conditions) and show very low sensitivity to the choice of distal outflow boundary conditions. This observation advocates one of the key advantages of the proposed closure model as it seems to 198 effectively eliminate uncertainties in outflow boundary conditions for hemodynamics. Figure 7.7: Effect of distal outflow boundary conditions: Effect of distal outflow boundary conditions: Computed pressure (solid lines) and flowrate (dashed lines) waves for different types of distal outlfow boundary conditions: Absorbing (red), constant pressure (green), and 3-element RCR windksessels (blue). Left: Anterior communicating artery (AcoA, CoW case). Right: Brachial artery (arm case). 7.5.5 3D-1D coupling in a cerebral network with 10 outlets We will use the simulation setup first put forth by Grinberg and Karniadakis (2008) in the context of testing a proposed RC lumped parameter model as an outflow bound- ary condition for arterial networks with multiple outlets. This 3D patient-specific geometry contains 20 cranial arteries reconstructed from high-resolution computed tomography angiography (CTA) images. Figure 7.8: 3D-1D coupling in a cerebral network with 10 outlets: Network of 20 cranial arteries with 1 inlet (red) and 10 outlets. 199 3D domain 1D domain Number of elements 111214 9689 Polynomial order 5 3 Non-dimensional ∆t 0.025 0.005 Number of processors 8176 10 Average cpu-time per time-step 0.52s 0.0008s Wall clock time per one cycle 4.3hrs - Table 7.2: 3D-1D coupling in a cerebral network with 10 outlets: Simulation proper- ties. The 3D computational domain (Fig.16) is driven by imposing a Womersley flow- rate at its inlet with W s = 4.95, Re = 200, and T = 0.575sec. Each of the outlets is coupled to a fractal 1D network generated as described in Sec. 7.2 with a cutoff radius of rcut = 100µm, and a constant L/r = 50 ratio. At the distal end of each fractal network we attach a 0-D RCR windkessel model and set its parameters by crudely extrapolating the values published by Reymond et al. (2011) as a function of the outlet radius. The 1D network accounts for 9689 arterial domains which have being discretized in space using 1DG element per domain. Table 7.2 summarizes the properties of this simulation. For comparison purposes we have also considered a purely 3D simulation of the same topology using a 0-D RC model at the outlets (see Grinberg and Karniadakis (2008)). In Tables 7.3, 7.4 we have summarized the measured errors in the pressure drop ∆p between the inlet and each of the 10 outlets, and the measured flow rate at each of the 10 outlets, respectively. Here, the objective is to compare the errors for different values of the cut-off radius of the fractal trees, rcut = 300, 200, 100µm. In all cases we observe that all modeling approaches return very close predictions for both the measured pressure drops and flowrates. However, we do observe significant variation in the amplitude of the predicted pressure wave. In Figs. 7.9(a),(b) we present the computed pressure wave over one cardiac cycle at each of the outlets for 200 ∆p rcut = 200µm rcut = 300µm # of 1D vessels Outlet L1 L2 L∞ L1 L2 L∞ 100µm → 200µm → 300µm #1 4.47e-02 3.99e-02 3.76e-02 1.06e-01 9.30e-02 8.57e-02 1151 → 209 → 67 #2 1.75e-02 1.38e-02 -9.76e-05 3.19e-02 2.53e-02 -2.82e-04 2611 → 369 → 119 #3 6.54e-02 4.39e-02 -8.61e-03 1.15e-01 7.77e-02 -1.50e-02 649 → 117 → 39 #4 8.92e-03 5.95e-03 5.64e-03 2.13e-02 1.39e-02 1.17e-02 389 → 67 → 21 #5 5.37e-04 1.22e-02 9.58e-03 2.93e-03 2.60e-02 2.20e-02 1511 → 255 → 87 #6 1.14e-03 9.74e-03 8.80e-03 5.69e-03 2.08e-02 1.96e-02 1319 → 211 → 67 #7 1.05e-01 8.04e-02 -2.44e-05 1.91e-01 1.48e-01 1.31e-03 1319 → 211 → 67 #8 1.87e-01 1.28e-01 -1.98e-02 6.67e-01 4.33e-01 -8.13e-02 529 → 87 → 29 #9 9.31e-06 1.14e-02 8.16e-03 2.01e-03 2.418e-02 1.74e-02 211 → 37 → 11 #10 3.05e-03 8.27e-03 4.23e-03 1.42e-02 2.01e-02 6.58e-03 1319 → 211 → 67 Table 7.3: 3D-1D coupling in a cerebral network with 10 outlets: Relative errors of the pressure gradient ∆p between the inlet and each of the 10 outlets of the 3D domain. We consider rcut = 100µm to be our reference case. Q rcut = 200µm rcut = 300µm # of 1D vessels Outlet L1 L2 L∞ L1 L2 L∞ 100µm → 200µm → 300µm #1 4.21e-02 4.20e-02 4.07e-02 4.21e-02 4.20e-02 4.07e-02 1151 → 209 → 67 #2 4.26e-02 4.12e-02 -1.39e-02 4.26e-02 4.12e-02 -1.39e-02 2611 → 369 → 119 #3 1.79e-01 1.74e-01 -5.71e-02 1.79e-01 1.74e-01 -5.71e-02 649 → 117 → 39 #4 1.08e-02 1.05e-02 8.55e-03 1.08e-02 1.05e-02 8.55e-03 389 → 67 → 21 #5 8.33e-03 1.29e-02 1.80e-02 8.33e-03 1.29e-02 1.80e-02 1511 → 255 → 87 #6 8.70e-03 1.16e-02 1.57e-02 8.70e-03 1.16e-02 1.57e-02 1319 → 211 → 67 #7 1.22e-01 1.20e-01 -3.17e-02 1.22e-01 1.20e-01 -3.17e-02 1319 → 211 → 67 #8 2.69e-01 2.65e-01 -7.46e-02 2.69e-01 2.65e-01 -7.46e-02 529 → 87 → 29 #9 5.23e-04 7.68e-03 8.78e-03 5.23e-04 7.68e-03 8.78e-03 211 → 37 → 11 #10 3.79e-03 6.36e-03 2.71e-03 3.79e-03 6.36e-03 2.71e-03 1319 → 211 → 67 Table 7.4: 3D-1D coupling in a cerebral network with 10 outlets: Relative errors of the flow-rate measured at each of the 10 outlets of the 3D domain. We consider rcut = 100µm to be our reference case. a full 3D simulation with 0-D RC boundary conditions versus the coupled 3D-1D case with fractal trees, using rcut = 100µm, and a constant L/r = 50 ratio. We observe that the the 3D-0D model returns a pressure field that is higher than what is physiologically expected. This is primarilly attributed to the miscalibration of the the R, C parameters at each outlet. On the other hand, the 3D-1D coupled approach using the proposed fractal tree closure model is able to return a physiological pressure field at all outlets. This difference is less pronounced for the flow rate wave, as shown in In Figs. 7.10(a),(b), where both 3D and 3D-1D distribute the flow in a similar fashion. 201 80 200 180 70 160 140 60 Pressure (mmHg) Pressure (mmHg) 120 50 100 80 40 60 40 30 20 20 0 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 t (sec) t (sec) (a) (b) Figure 7.9: 3D-1D coupling in a cerebral network with 10 outlets: Comparison of computed pressure at the outlets (left: 3D with 0D-RC outflow, right: 3D with 1D elastic fractal networks). 1.8 2 1.6 1.8 1.6 1.4 1.4 1.2 Flowrate (cm3 /s) Flowrate (cm3 /s) 1.2 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 t (sec) t (sec) (a) (b) Figure 7.10: 3D-1D coupling in a cerebral network with 10 outlets: Comparison of computed flow-rates at the outlets (left: 3D with 0D-RC outflow, right: 3D with 1D fractal networks). 202 7.6 Discussion We have proposed an outflow closure model for blood flow simulations in large arterial networks, with our main goal being to obtain physiological flow predictions while reducing parametric sensitivities and circumventing costly calibration procedures. The model is based on resolving flow in self-similar networks of millions of small arteries and arterioles that are attached on-the-fly to a given parent vascular network of larger systemic arteries. To our knowledge, this is the first time that nonlinear 1D blood flow simulations have been performed in arterial networks of this size. While the proposed model minimizes the overall time to accurate solution, it requires facilitation of parallel computers and a scalable solver. At the same time the number of processors (CPU cores) required for reasonable time-to-solution is still very low (less than 1000).This contribution is vital as, not only it allowed simulating trees with hundreds of thousand of arteries in the present work, but it opens the path to addressing many interesting open questions in computational hemodynamics. For instance, the proposed computational framework may be combined with flow and pressure measurements towards solving inverse problems (such as estimating material properties of the arterial wall, estimating the total volume of an arterial tree, etc). We have suggested a way of addressing cut-off radius sensitivities by introducing a radius-dependent length-to-radius ratio for small arterioles, controlled by a thresh- old parameter, r? . This ansatz led to flow predictions that gradually converge to a plateau solution as the cutoff radius is decreased, overcoming cutoff radius sensi- tivities previously reported for the structured tree outflow boundary condition (see Cousins and Gremaud (2012)). Our proposed fractal closure model was thoroughly tested for two realistic cases 203 of blood flow in the Circle of Willis and a detailed network of the arm vasculature. For the CoW case, where the parent network is patient-specific and the fractal model parameters are set according to a target volume estimate, the proposed model returns physiologically correct results without any user intervention or prior calibration. The key point here is that the volume constraint can be achieved without performing any simulation based optimization (as opposed to tuning to match imposed/measured data), leading to significant savings in computation. Moreover, the volume constraint fits well the spirit of the structured tree as it has a physiological basis and enables the study of cases where having a physiologically correct tree is very important (for e.g. heterogeneous autoregulation). For the arm case, the parent arm network is not-patient specific and a number of outlets may be missing. Moreover, a target volume estimate cannot be readily extracted from the literature, hence the threshold parameter r? needs to be tuned. The main advantage of tuning r? versus merely tuning a constant L/r ratio, is that the resulting model exhibits very low sensitivity on the cut-off radius rcut as well as on the outflow boundary conditions used. The important lesson we learned here is that, in the absence of estimates for arterial blood volume in the truncated network, the convergence of flow and pressure waveforms with respect to the cut-off radius can still be obtained. One limitation of the proposed method is the potential sensitivity on r? for cases where a target volume estimate is not available. Although this introduces the need for calibration of r? , we believe that the proposed framework is still viable as it only requires tuning of a single parameter, in contrast to tuning hundreds of resistance and capacitance parameters needed for the RCR model. Another practical limitation of the suggested methodology stems from the computational cost of solving the nonlinear flow equations in networks with hundreds of thousands of arteries. Such 204 computations can be currently performed in a reasonable time only by utilizing computer clusters with a few hundreds of modern CPUs. An immediate goal arising from this study is to apply the developed closure model for three-dimensional simulations of a blood flow in compliant arteries. That requires developing stable interface condition between the 3D and the 1D solvers and efficient coupling of the two parallel solvers. Another future thrust is to leverage on the robustness and scalability of the developed solver to address open problems in biomedicine, such as constructing global circulation models, performing high dimen- sional uncertainty quantification studies, as well as parameter estimation in inverse problems. Chapter Eight Calibration of Blood Flow Simulations via Multi-fidelity Surrogates 206 8.1 Motivation Inverse problems are ubiquitous in science. Being inherently ill-posed, they require solution paths that often challenge the limits of our understanding, as reflected by our modeling and computing capabilities. Unlike forward problems, in inverse prob- lems we have to numerically solve the principal equations (i.e., the forward problem) multiple times, often hundreds of times. The complexity in repeatedly solving the forward problem is further amplified in the presence of nonlinearity, high-dimensional input spaces, and massive data-sets; all common features in realistic physical and biological systems. The natural setting for model inversion finds itself within the principles of Bayesian statistics, which provides a formal ground for parameter esti- mation, i.e., the process of passing from prior belief to a posterior predictive inference scheme in view of data. Here, we leverage the methodology presented in Chapter 4 to design a framework for parameter estimation in blood flow models of the human circulation. Our approach consists of utilizing all available information sources, e.g., experimental measurements, computer simulations, empirical laws, etc., through a general multi-fidelity information fusion methodology in which surrogate models are trained on available data, therefore enabling one to explore the interplay between all such sources. In general, we model the response of a system as a function y = f (x) of d input parameters x ∈ Rd . The goal of model inversion is to identify the parametric configuration in x that matches a target response y ? . This translates into solving the following optimization problem min ||f (x) − y ? ||, (8.1) x∈Rd 207 in some suitable norm. In practice, x is often a high-dimensional vector and f is a complex, nonlinear, and expensive to compute map that represents the system’s evolving dynamics. These factors render the solution of the optimization problem very challenging and motivate the use of surrogate models as a remedy for obtaining inexpensive samples of f at unobserved locations. To this end, a surrogate model acts as an intermediate agent that is trained on available realizations of f , and then is able to perform accurate predictions for the response at a new set of inputs. Ever since the seminal work of Sacks et al. (1989), the use of surrogates in the design and analysis of computer experiments has undergone great growth, establishing a data- driven mindset for design, optimization, and, as presented in Chapter 3, uncertainty quantification problems. Of particular importance to our work is the approach of Kennedy and O’Hagan (2000) that introduced the use of stochastic auto-regressive maps for building surrogates from a multitude of information sources of variable fidelity. We reckon that this framework enables the meaningful integration of seem- ingly disparate methodologies, and provides a universal platform in which experi- ments, simulations, and expert opinion can coexist in tandem. The main challenges here arise from scaling this surrogate-based approach to constructing response sur- faces in high-dimensional input spaces and performing tractable machine learning on massive data-sets. Here, our aim is to exploit the developments presented in Chapter 4, towards addressing these challenges and enabling the construction of multi-fidelity surrogates for realistic high-dimensional cases. In-silico modeling of blood flow in the human vasculature has received great attention over the last 20 years, resulting in the development of computational tools that helped elucidate key biophysical aspects but also aim to provide a cus- tomized, patient-specific tool for prediction, intervention, and treatment. Despite great growth in computing power and algorithmic sophistication, the applicability of 208 such models is limited to truncated arterial domains, as the complexity introduced from considering circulation in the entire arterial network remains intractable (see Formaggia et al. (2010)). Addressing this complexity often leads to the introduction of a series of assumptions, parameters, and simplified models. Consequently, the physiological relevance of our computations directly relies on the calibration of such parameters and models, which, due to the inherent difficulty of them being deter- mined in the clinical setting, remains very empirical, as the resulting hemodynamic problem could admit an infinite number of solutions (see Quick et al. (2001)). To this end, recent efforts for developing model inversion techniques in hemodynamics have been proposed by Bertoglio et al. (2012), Blanco et al. (2012), Lassila et al. (2013), Lombardi (2014b), Melani (2013). A common theme among these efforts is the utilization of reduced order models that can be sampled extensively and with low computational cost, returning a response f that enables a computationally tractable solution to the optimization problem of Eq. 8.1. Among possible reduced order models candidates, the most widely used are nonlinear 1D fluid-structure interac- tions (FSI) models, linearized 1D-FSI models, and 0-D lumped parameter models (see Formaggia et al. (2010)). The goal of this work is to incorporate elements of statistical learning towards building a surrogate-based framework for solving inverse problems in hemodynam- ics, and beyond. Motivated by methods put forth in Chapters 3, 4, we propose an information fusion framework that can explore cross-correlations between vari- able fidelity blood flow models (e.g., measurements versus 3D-FSI, versus 1D-FSI, versus 0-D models, etc.), allowing for the efficient construction of high-dimensional response surfaces that guide the pursuit for a solution to the optimization problem of Eq. 8.1, while keeping the number of expensive function evaluations at a mini- mum. Moreover, we aim to demonstrate that this framework is robust with respect 209 to model misspecification, resulting for example from inaccurate low fidelity models or noisy measurements. Although our main focus here is on parameter estimation for physiologically correct blood flow simulations, the implications of the proposed methodology are far reaching, and practically applicable to a wide class of inverse problems. This chapter extends the methods presented in Chapters 3, 4 towards designing a tractable methodology from model inversion via multi-fidelity surrogates. In partic- ular, we use Sec. 8.2 to introduce the Efficient Global Optimization algorithm, and highlight how maximizing the expected improvement of the surrogate predictor can lead to efficient sampling strategies for global optimization. The framework is put at test in Sec. 8.3, where we present results for a pedagogical benchmark problem of calibrating the outflow parametrization of a blood flow simulation in a Y-shaped bifurcation. 8.2 Efficient global optimization Our primary goal here is to utilize the surrogate models generated by the recur- sive co-kriging formulation towards identifying the global optimum of the optimiza- tion problem defined in Eq. 8.1. The probabilistic structure of the surrogate pre- dictors enables an efficient solution path to this optimization problem by guiding a sampling strategy that balances the trade-off between exploration and exploita- tion, i.e., the global search to reduce uncertainty versus the local search in regions where the global optimum is likely to reside. One of the most widely used sam- pling strategies in Bayesian optimization that adopts this mindset is the Efficient Global Optimization (EGO) algorithm proposed by Jones et al. (1998). Given a 210 predictive kriging/co-kriging distribution, the EGO algorithm selects points in the input space that maximize the expected improvement of the predictor. To this end, (t) (1) (2) (Nt ) let fmin = min{yt , yt , . . . , yt } be the global minimum of the observed response at the tth inference level. Consequently, the improvement of the Gaussian predictor (t) Zt (x) upon fmin is defined as It (x) = max{fmin − Zt (x), 0}. Then, the infill criterion suggested by the EGO algorithm implies sampling at locations in the input space that maximize the expected improvement ! ! (t) (t) (t) fmin − yˆt (x) fmin − yˆt (x) E[It (x)] = [fmin − yˆt (x)]Φ + sφ , (8.2) st (x) st (x) where Φ(·) and φ(·) are the standard normal cumulative distribution and density function, respectively, and st is the square root of the predictor variance at level t (see Eq. 2.6). Consequently, the value of E[It (x)] is large if either the value predicted (t) by yˆt (x) is smaller than fmin or there is a large amount of uncertainty in the predictor yˆt (x), hence s2t (x) is large. Once a recursive co-kriging cycle has been completed and the final predictor yˆs and variance s2s (x) at level s are known, the expected improvement can be readily computed by Eq. 8.2. Then, then EGO suggests to re-train the surrogates by aug- menting the design sets Dt with a set of points that correspond to locations where the expected improvement is maximized. This procedure iterates until a stopping criterion is met. Due to the potentially high computational cost of evaluating f in Eq. 8.1, it is common to use the maximum number of function evaluations as the stopping criterion or the convergence rate of the objective function. Another ter- mination criterion stems from setting a target value for the expected improvement, allowing the next cycle to be carried out only if the expected improvement is above the imposed threshold. 211 The EGO algorithm can be summarized in the following implementation steps: Step 1: Given samples of the response yt = ft (xt ), t = 1, . . . , s at all fidelity levels we employ the recursive co-kriging scheme outlined in Sec. 2.3.2 to construct a family of predictive surrogates yˆt (x). Step 2: For each surrogate we compute the set of input configurations that maximize the expected improvement, i.e., we solve the optimization problem max E[It (x)]. (8.3) x∈Rd Step 3: We evaluate the response yt = ft (xt ), t = 1, . . . , s at the new points suggested by maximizing the expected improvement at each fidelity level. Step 4: We train again the multi-fidelity surrogates using the augmented design sets and corresponding data to obtain new predictive schemes that resolve in more detail regions of the response surface where the minimum is more likely to occur. Step 5: Perform Step 2 and check if the desired termination criterion is met. This could entail checking whether a maximum number of function evaluations is reached, whether the minimum of the objective function has converged, or whether the max- imum expected improvement is less than a threshold value. If the chosen criterion is not satisfied, we repeat Steps 3-5. 212 8.3 Results 8.3.1 Y-shaped bifurcation We consider a benchmark problem for calibrating the outflow parameters in a Y- shaped bifurcation with one inlet and two outlets. The geometry resembles the characteristic size and properties of a carotid bifurcation, yet is kept symmetric to enhance clarity in our presentation. Figure 8.1 presents a schematic representation of the problem setup. In particular, the effect of the neglected downstream vasculature is taken into account through 3-element windkessel models, while the inflow is driven by a physiologically correct flow-rate wave. 8 (1) (1) R2 7.5 R1 7 6.5 6 C (1) Q(cm3 /s) 5.5 5 (2) R2 4.5 (2) R1 4 3.5 3 0 0.2 0.4 0.6 0.8 1 C (2) sec Figure 8.1: Blood flow in a symmetric Y-shaped bifurcation: Flow is driven by a phys- iological flow-rate wave at the inlet, while outflow boundary conditions are imposed through 3-element windkessel models. Our goal is to calibrate the outflow resistance parameters to obtain a physiolog- ically correct pressure wave at the inlet. To enable a meaningful visualization of the solution process, we confine ourselves to a two-dimensional input space defined by (i) variability in the total resistances imposed at the two outlets, RT , i = 1, 2. The (i) (i) (i) (i) total resistance is defined as RT = R1 + R2 , where R1 corresponds to the charac- teristic impedance of each terminal vessel (see Eq. 5.29). Moreover, the capacitance 213 parameters are empirically set following the findings of Grinberg and Karniadakis (i) (2008) as C (i) = 0.18/RT , i = 1, 2. To this end, our objective is to identify a pair of (i) RT that matches a physiologically correct systolic pressure of p?s = 126mmHg at the inlet. Assuming no prior knowledge on possible sensible combinations in the input (1) (2) x = [RT , RT ]T , we assume a large input space that spans four orders of magnitude: X = [106 , 1010 ]2 measured in (P a · s/m3 ), and consider the following optimization problem x? = argmin |p?s − ps (x)|2 . (8.4) x∈X First, we employ a nonlinear 1D-FSI model in order to construct a detailed rep- resentation of the response surface of the error in systolic pressure, i.e., f (x) = |p?s − ps (x)|2 , that will be later used as a reference solution to asses the accuracy and convergence of the proposed multi-fidelity model inversion techniques. Here, the choice of studying hemodynamics using 1D models is motivated by their ability to accurately reflect the interplay between the parametrization of the outflow and the systolic pressure at the inlet, yet at a very low computational cost (see Sherwin et al. (2003), Perdikaris et al. (2014)). To this end, Figure 8.2 shows the resulting response surface obtained by probing an accurate nonlinear 1D-FSI solver on 10,000 uniformly spaced samples of the input variables. Since we have considered a sym- metric geometry, the resulting response surface exhibits symmetry with respect to the diagonal plane, and can be visibly subdivided using the path of its local extrema into 4 different subregions. The first part is confined in a flat square region near the origin, suggesting that the solution is relatively insensitive to the choice of in- puts, as any combination in the inputs within this region results to a reasonably small error in the systolic pressure. The second and third parts correspond to the 214 symmetric rectangular regions defined by further increasing one of the two resis- tance parameters. There, the response starts to exhibit sensitivity to the increasing input parameter, leading to a noticeable error in the systolic pressure. Lastly, in the fourth region, the solution exhibits strong sensitivity on the inputs, leading to an explosive growth of the systolic pressure deviation as the resistance parameters take larger values. Interestingly, a closer look at the path of local extrema reveals a subtle undershoot in the response surface along the interface of the aforementioned subregions. Although any input combination within the first subregion results in a very small systolic pressure error, the global solution x? of the minimization problem in Eq. 8.4 resides right at the interface of the four subregions (see Fig. 8.2). Note that this topology of the global response is likely to pose serious challenges to any gradient-based optimization strategy, leading to a potentially very large number of function evaluations until convergence, especially in the absence of a very accurate initial guess. 215 x⋆ Figure 8.2: Blood flow in a symmetric Y-shaped bifurcation: Detailed response sur- face and identification of the minimum error in the inlet systolic pressure as a function of the total resistance parameters imposed at the outlets. The surface is constructed by probing a nonlinear 1D-FSI model on 10,000 uniformly spaced samples in the space of inputs X . Now, we turn our attention to solving the optimization problem introduced by Eq. 8.4 using the proposed surrogate-based framework. To this end, in order to build multi-fidelity surrogates for the systolic pressure y = ps (x) we have considered probing two models: high-fidelity solutions are obtained through the aforementioned nonlinear 1D-FSI solver, while the lower fidelity response is measured through a linearized 1D-FSI model. Here, we emphasize that the linearized model has been purposedly derived around a biased reference state, returning erroneous predictions that deviate from the correct solution up to 30%. This is to demonstrate that the proposed methodology is robust with respect to misspecification in the lower fidelity observations. 216 Initially, we perform 20 linearized low-fidelity simulations, supplemented by 5 high-fidelity realizations, all randomly sampled within the bounds that define the space of inputs. Then, based on the computed response yt (xt ), we train a two-level surrogate using the recursive co-kriging scheme (see Sec. 2.3.2) with a stationary Mat´ern 3/2 auto-correlation function (see Rasmussen (2006)). In order to ensure that any non-stationary effects are filtered out of the data, we use a logarithmic warping of the inputs following the ideas put forth bySnoek et al. (2014). The resulting response surface of the error in the systolic pressure at the inlet is presented in comparison with the reference solution in Fig. 8.3(a). Once the surrogate predictor and variance is available, we can compute the spatial distribution of the corresponding expected improvement using Eq. 8.2. This highlights regions of high expected improvement, as illustrated in Fig. 8.3(b), thus suggesting the optimal sampling locations for the next iteration of the EGO algorithm (see Sec. 8.2). Notice that the maximum expected improvement in this zeroth iteration is already attained very close to the global minimum of the reference solution (see Fig. 8.2). Expected Improvement Figure 8.3: Blood flow in a symmetric Y-shaped bifurcation: Zeroth iteration of the EGO algorithm. Left: Exact solution versus the co-kriging predictor for the inlet systolic pressure error, trained on 100 low-fidelity and 25 high-fidelity observations. Right: Map of the corresponding expected improvement. 217 The next step involves performing an additional set of low- and high-fidelity sim- ulations at the suggested locations of high expected improvement (see Sec. 8.2). The resulting observations are augmented to the existing set of data, and the recursive co-kriging predictor is re-trained to absorb this new information. This procedure is repeated until a termination criterion is met. Here, we have chosen to exit the iteration loop once the minimum of the predicted response surface (i.e., the deviation of the inlet systolic pressure from a target value) is less than |p?s − ps (x)|2 < 10−3 . Figure 8.4(a) presents the resulting response surface of the error in the inlet systolic pressure after 4 iterations of the EGO algorithm. The predictor is trained on 100 low-fidelity and 25 high-fidelity realizations, that clearly target resolving the region of the response surface where the global minimum is likely to occur. Similarly, Figure 8.4(b) presents the spatial distribution of the expected improvement after the new data has been absorbed. By comparing Figs. 8.4(b), 8.3(b) we observe that within 4 iterations of the EGO algorithm the expected improvement has been reduced by a factor greater than 5, effectively narrowing down the search for the global minimum in a small region of the response surface. In fact, 4 iterations of the EGO algorithm were sufficient to meet the termination criterion, and, therefore, identify the set of outflow resistance parameters that returns an inlet systolic pressure matching the target value of p?s = 126mmHg. Here, we underline the robustness and efficiency of the proposed scheme, as convergence was achieved using only 25 samples of the high-fidelity nonlinear 1D-FSI model, supplemented by 100 inaccurate low- fidelity samples of the linearized 1D-FSI solver. 218 Expected Improvement Exact response (105 samples) Co-kriging 120 Low-fidelity data (100 points) High-fidelity data (25 points) 100 80 (2) RT 60 |ps − p⋆s |2 40 20 0 10 (2) RT × 109 (1) RT × 109 (1) RT (a) (b) Figure 8.4: Blood flow in a symmetric Y-shaped bifurcation: Fourth iteration of the EGO algorithm. Left: Exact solution versus the co-kriging predictor for the inlet systolic pressure error, trained on 100 low-fidelity and 25 high-fidelity observations. Right: Map of the corresponding expected improvement. In Table 8.1 we summarize the optimal predicted configuration for the total ter- minal resistances, along with the relative L2 -error in the corresponding inlet pressure waveform compared to the optimal reference configuration (see x? in Fig. 8.2), over one cardiac cycle, and for every iteration of the EGO algorithm. The convergence of the inlet pressure waveform to the reference solution is demonstrated in Fig. 8.5. As the EGO iterations pursue a match with the target inlet systolic pressure p?s , the pressure waveform gradually converges to the reference solution after each iteration of the optimizer. 219 (1) (2) ||p−popt ||2 Iteration No. RT × 109 RT × 109 ||popt ||2 1 0.001 5.9612 0.1949 2 4.6479 2.6275 0.0945 3 2.0214 2.9275 0.0012 4 3.7183 3.6275 2.13·10−4 Table 8.1: Blood flow in a symmetric Y-shaped bifurcation: Optimal terminal re- sistances and corresponding relative L2 -error in the inlet pressure wave over one cardiac cycle, for each iteration of the EGO algorithm. The comparison is done with respect to the inlet pressure wave popt corresponding to the optimal configuration of resistances predicted by the reference solution obtained using 10,000 high-fidelity samples. 150 Exact Iteration 1 140 Iteration 2 Iteration 3 130 120 p⋆s p(mmHg) 110 100 90 80 70 60 0 0.2 0.4 0.6 0.8 1 t(sec) Figure 8.5: Blood flow in a symmetric Y-shaped bifurcation: Convergence of the inlet pressure waveform for the first 3 EGO iterations. The exact solution corresponds to the reference results obtained from 10,000 high-fidelity samples, defining the target inlet systolic pressure p?s . 220 8.4 Discussion We have presented a surrogate-based framework for inverse problems targeting pa- rameter calibration in blood flow simulations. The framework is based on multi- fidelity information fusion and Bayesian optimization algorithms, allowing for the accurate construction of response surfaces through the meaningful integration of variable sources of information, e.g., clinical measurements, in vitro experiments, and numerical simulations. Leveraging the properties of the proposed probabilistic inference schemes, we utilize the celebrated Efficient Global Optimization algorithm of Jones et al. (1998) to perform model inversion using only a few samples of ex- pensive high-fidelity model evaluations, supplemented with a number of cheap, but potentially very inaccurate, low-fidelity observations. This approach enables a com- putationally tractable global optimization framework that is able to efficiently zoom in the response surface in search of the global minimum, while exhibiting robustness to misspecification in the lower fidelity models. The capabilities of the proposed methodology are demonstrated through a ped- agogical benchmark problem involving the parametric calibration of a blood flow solver. To this end, we have considered an symmetric Y-shaped network, consisting of one inlet and two terminal outlet vessels that resemble an idealized carotid bifur- cation. The system is closed using a 3-element windkessel model attached to each outlet, and our goal is to identify the configuration of the outflow resistance param- eters such that we obtain a physiologically correct pressure field at the inlet. To this end, we have employed two models of different fidelity. The high-fidelity model is an accurate nonlinear 1D-FSI solver, while the low-fidelity model corresponds to an inaccurate linear 1D-FSI solver that has been purposedly linearized around a biased reference state. By employing a two-level recursive co-kriging scheme, we are able to 221 combine the information contained in a few samples of the high-fidelity model, along with a larger set of low-fidelity observations, towards constructing a multi-fidelity surrogate than can estimate the global representation of the error in the inlet systolic pressure as a function of the total resistance parameters in the outflow. This proba- bilistic representation is then utilized within the EGO loop to iteratively sample the input space towards narrowing down the search for the optimal configuration. To this end, we demonstrated that 4 EGO iterations, requiring 25 high-fidelity and 100 low-fidelity model evaluations, were sufficient to calibrate the outflow parameters, leading to an error in the inlet pressure wave that is less than 10−3 when compared against a reference solution obtained using 10,000 high-fidelity observations. Our exposition here aims to serve as an elucidating example of the key features and capabilities that the proposed framework has to offer. Extending and applying this methodology to more complex and realistic inverse problems requires addressing the challenges of constructing response surfaces in high-dimensions and in the pres- ence of massive data-sets. A possible solution path for addressing these challenges is provided by the methodology presented in Chapter 4. Chapter Nine Conclusions and Future Work 223 9.1 Conclusions In this thesis we have demonstrated how Gaussian process regression and stochas- tic auto-regressive schemes can provide a launchpad for the development of scalable multi-fidelity information fusion algorithms for modeling the response of complex stochastic dynamical systems. In Chapter 2 we put forth a general framework in which multi-fidelity in physical models and multi-fidelity in probability space can coexist in unison. To this end, in Chapter 3 we proposed fast and accurate predictive algorithms in two and three dimensions based on spectral/hp element discretizations, and recursive co-kriging schemes that leverage the sparsity introduced by Gaussian Markov random fields. The tractability of the proposed framework was successfully extended to high-dimensional input spaces in Chapter 4 via a data-driven dimension- ality reduction technique based on hierarchical functional decompositions. Gearing this approach with O(N ) frequency-domain machine learning algorithms, we were able to develop parallel predictive inference algorithms and demonstrate scaling to extreme cases involving 1,000 input dimensions and O(105 ) data-sets on a standard desktop computer! These findings set the foundations of a new type of scientific com- puting which enables a meaningful learning process through exploring and exploiting cross-correlations between a multitude of different methods and information sources, setting a new paradigm in design optimization under uncertainty, data assimilation, model inversion, and beyond. A parallel thrust aimed at addressing a series of open questions in modeling multi- physics interactions in the human vasculature, focusing on blood flow and arterial biomechanics the brain. To this end, in Chapter 6 we demonstrated how fractional- order models offer an accurate and robust alternative to modeling the viscoelastic response of soft tissue, and we performed a series of systematic computational studies 224 to quantify the effect of arterial viscoelasticity on blood flow dynamics. Moreover, in Chapter 7 we proposed an effective closure model for simulating blood flow in truncated arterial domains, and scaled up a nonlinear 1D-FSI solver to perform the biggest simulation of this kind ever, involving more than 4 million blood vessels. This methodology, along with the 3D-1D coupling algorithms described in Chapter 5, can provide a stepping stone towards liberating blood flow simulations from the expen- sive calibration outflow boundary conditions. An alternative approach to parameter estimation was employed in Chapter 7, by addressing the problem of model inversion using multi-fidelity surrogates and Bayesian optimization. This approach leverages directly on the developments presented in Chapters 2-6, and it enables the efficient pursuit to identify global optima while keeping the number of expensive model eval- uations at a minimum. 9.2 Future work We cultivate the vision of building a unified multi-fidelity data-driven framework in high dimensions that can simultaneously pursue the solution of forward and in- verse uncertainty quantification problems as well as design and optimization in high- dimensions and under uncertainty. To this end, plan to employ deep networks of random fields to perform a seamless fusion of different information sources having different levels of fidelity, such as finite elements, particle methods, and noisy exper- imental data. The objective is to represent the QoI Y (x) in terms of a hierarchy of models connected in an appropriate way. Deep networks can be formally defined as a composition of vector-valued random fields f (i) (x) in Rm (i = 1, ..., L), with each field being drawn independently from suitable prior distribution. Most of the work that has been done so far refers to deep networks of Gaussian random fields (see 225 the recent work of Damianou and Lawrence (2012), Duvenaud et al. (2014), Wilson et al. (2011)). However, we can also estimate their distribution by using observation models and epi-splines (see Royset and Wets (2014b,a)) approaches, within different layers of the network. The mapping from the state space input variables x ∈ Rn (e.g., the design variables) to the output of the network can be represented formally as Y (x) = f (L) f (L−1) ...f (2) f (1) (x) · · · .   (9.1) Note that Y (x) is a random field which will allow us to make inferences, e.g., on deterministic quantities such as risk measures. The feature map f (i) within each layer of the network can be generalized and also become itself a function of the state variables x, i.e., f (i) (f (i−1) , x) (see Fig. 9.1). This allows us to avoid certain pathologies exhibited by deep Gaussian networks. Specifically, the representational capacity of networks in the form (9.1) tends to capture fewer degrees of freedom as the number of layers L increases (see, e.g., Duvenaud et al. (2014)). It is often convenient to set up observation models for some or all layers, e.g., in the form y (1) (x) = f (1) (x) , y (2) (x) = f (2) f (1) (x) , x ,  ··· , where y (i) (x) represents data, either deterministic or stochastic. In this way, some or all layers are not hidden, i.e., we can inject additional information at each level i for calibration or validation purposes. For example, such observation models can be used in an epi-spline inference process to estimate the (possibly non-Gaussian) distribution of the random field f (i) , or more simply to learn its covariance structure - if we assume that f (i) is Gaussian. Thus, the deep network sketched in Fig. 9.1 is really a network of information sources at different levels of fidelity, which are combined together to build an approximation of Y (x). A very particular class of 226 (a) (b) Figure 9.1: Deep networks: (a) Sketch of a deep network of vector-valued random fields f (L) ∈ Rm . In this particular example we have an observation model y (L) = f (L) for each layer, but we can also have situations in which we have hidden (unobserved) layers. The state space variables (e.g., the design variables) are denoted by x ∈ Rn . The multi-objective response - random field approximating the QoI by Y (x). The deep network allows us to perform information fusion of different information sources by using scalable machine learning algorithms. (b) Recursive co-kriging method as a particular instance of a deep linear Gaussian network. feature maps f (i) is the linear one, for which we have f (1) (x), f (2) (f (1) , x) = W (1) (x)f (1) (x), f (3) (f (2) , x) = W (2) (x)f (2) (x), ··· (9.2) where W (i) (x) are m × m matrices with i.i.d. Gaussian entries. In this case, the full input-output map (9.1) takes the form Y (x) = W (L−1) (x) · · · W (1) (x)f (1) (x), (9.3) and it obviously yields a Gaussian response Y (x). Within the class of networks (9.3) we have, for example, the recursive co-kriging of Le Gratiet and Garnier (2014) - a more efficient version of the well-known auto-regressive inference scheme proposed by Kennedy and O’Hagan (2000) that was extensively used throughout this thesis. The recursive co-kriging can be seen as a deep linear network of Gaussian processes in which there is an observation model for each layer (see Fig. 9.1(b)). The network 227 basically assimilates information, e.g., coming from stochastic numerical codes at different levels of fidelities, within each layer for the purpose of calibrating each (Gaussian) feature map f (j) . It is important to emphasize that the observation models y (i) = f (i) allows us to train f (i) with both deterministic and stochastic data. In addition, if we have hidden layers then the training procedure is applied to the an entire stack of layers. For example, we can simultaneously train layers 1 and 2 (1 is hidden) by using the observation model y (2) = f (2) (f (1) (x), x). The ultimate goal of this research thrust is to set the stage for the next generation of design and decision making principles in engineering and biomedical applications. An integral part of our future work will be focused on scaling these ideas to realistic scenarios that require the solution of optimization problems involving thousands of decision variables in addition to hundreds of thousands of uncertain parameters. A key aspect of this goal is to also address inference problems in which the observed outputs are no longer scalar quantities, but could possibly be vectors, or even con- tinuous fields. This is motivated by the recent work of Nguyen and Peraire (2015) which provides a framework for seamlessly integrating measurements and stochastic PDE models towards building accurate inference schemes for functional regression. This new paradigm for multi-fidelity stochastic modeling allows to go well be- yond linear maps, recursive co-kriging methods and Gaussian process regression. This yields new classes of inherently non-linear (non-Gaussian) multi-fidelity infor- mation fusion algorithms, which allow us to address surrogate model building in high dimensions. We expect that the implications of such developments will be far reaching and break new ground in uncertainty quantification, design optimization, model inversion, and beyond. Bibliography J. Alastruey, K. Parker, J. Peir´o, and S. Sherwin. Lumped parameter outflow models for 1-D blood flow simulations: effect on pulse waves and parameter estimation. Communications in Computational Physics, 4(2):317–336, 2008. T. Atanackovi´c, S. Konjik, L. Oparnica, and D. Zorica. Thermodynamical restric- tions and wave propagation for a class of fractional order viscoelastic rods. In Abstract and Applied Analysis, volume 2011. Hindawi Publishing Corporation, 2011. H. Baek and G. E. Karniadakis. Sub-iteration leads to accuracy and stability en- hancements of semi-implicit schemes for the Navier–Stokes equations. Journal of Computational Physics, 230(12):4384–4402, 2011. H. Baek and G. E. Karniadakis. A convergence study of a new partitioned fluid– structure interaction algorithm based on fictitious mass and damping. Journal of Computational Physics, 231(2):629–652, 2012. C. Bertoglio, P. Moireau, and J.-F. Gerbeau. Sequential parameter estimation for fluid–structure problems: Application to hemodynamics. International Journal for Numerical Methods in Biomedical Engineering, 28(4):434–455, 2012. D. Bia, I. Aguirre, Y. Z´ocalo, L. Devera, E. C. Fischer, and R. Armentano. Regional differences in viscosity, elasticity, and wall buffering function in systemic arteries: pulse wave analysis of the arterial pressure-diameter relationship. Revista Espa˜ nola de Cardiolog´ıa (English Edition), 58(2):167–174, 2005. K. Binder and D. Heermann. Monte Carlo simulation in statistical physics: an introduction. Springer Science & Business Media, 2010. P. Blanco, S. Watanabe, and R. Feij´oo. Identification of vascular territory resistances in one-dimensional hemodynamics simulations. J. Biomech., 45(12):2066 – 2073, 2012. D. Bolin and F. Lindgren. Wavelet Markov models as efficient alternatives to tapering and convolution fields. In Preprints in Mathematical Sciences. Citeseer, 2009. C. Bron and J. Kerbosch. Algorithm 457: finding all cliques of an undirected graph. Communications of the ACM, 16(9):575–577, 1973. 228 229 ˇ c, C. J. Hartley, D. Rosenstrauch, J. Tambaˇca, G. Guidoboni, and A. Mikeli´c. S. Cani´ Blood flow in compliant arteries: an effective viscoelastic reduced model, numerics, and experimental validation. Annals of Biomedical Engineering, 34(4):575–592, 2006. F. Cassot, F. Lauwers, S. Lorthois, P. Puwanarajah, V. Cances-Lauwers, and H. Du- vernoy. Branching patterns for arterioles and venules of the human cerebral cortex. Brain res., 1313:62–78, 2010. W. Cousins. Boundary Conditions and Uncertainty Quantification for Hemodynam- ics. PhD thesis, North Carolina State University, 2013. W. Cousins and P. Gremaud. Boundary conditions for hemodynamics: The struc- tured tree revisited. J. Comput. Phys., 231(18):6086–6096, 2012. W. Cousins and T. P. Sapsis. Reduced order prediction of rare events in unidirec- tional nonlinear water waves. Journal of Fluid Mechanics, (under review), 2015a. W. Cousins and T. P. Sapsis. The unsteady evolution of localized unidirectional deep water wave groups. Physical review E, (under review), 2015b. W. Cousins, P. Gremaud, and D. Tartakovsky. A new physiological boundary con- dition for hemodynamics. SIAM J. Appl. Math., 73(3):1203–1223, 2013. D. Craiem and R. L. Armentano. A fractional derivative model to describe arterial viscoelasticity. Biorheology, 44(4):251–263, 2007. D. Craiem, F. J. Rojo, J. M. Atienza, R. L. Armentano, and G. V. Guinea. Fractional-order viscoelasticity applied to describe uniaxial stress relaxation of human arteries. Physics in medicine and biology, 53(17):4543, 2008. N. A. Cressie and N. A. Cassie. Statistics for spatial data, volume 900. Wiley New York, 1993. A. Damianou and N. Lawrence. Deep gaussian processes. arXiv preprint arXiv:1211.0358, 2012. J. De Baar, R. P. Dwight, and H. Bijl. Speeding up Kriging through fast estimation of the hyperparameters in the frequency-domain. Computers & Geosciences, 54: 99–106, 2013. K. DeVault, P. A. Gremaud, V. Novak, M. S. Olufsen, G. Vernieres, and P. Zhao. Blood flow in the circle of Willis: modeling and calibration. Multiscale Modeling & Simulation, 7(2):888–909, 2008. C. Dietrich and G. N. Newsam. Fast and exact simulation of stationary Gaussian processes through circulant embedding of the covariance matrix. SIAM Journal on Scientific Computing, 18(4):1088–1107, 1997. T. C. Doehring, A. D. Freed, E. O. Carew, and I. Vesely. Fractional order viscoelas- ticity of the aortic valve cusp: an alternative to quasilinear viscoelasticity. Journal of biomechanical engineering, 127(4):700–708, 2005. 230 N. Durrande, D. Ginsbourger, O. Roustant, and L. Carraro. Additive covari- ance kernels for high-dimensional Gaussian process modeling. arXiv preprint arXiv:1111.6233, 2011. D. Duvenaud, O. Rippel, R. Adams, and Z. Ghahramani. Avoiding pathologies in very deep networks. arXiv preprint arXiv:1402.5836, 2014. A. C. Eringen. Mechanics of continua. Huntington, NY, Robert E. Krieger Publishing Co., 1980. 606 p., 1, 1980. J. Foo and G. E. Karniadakis. Multi-element probabilistic collocation method in high dimensions. Journal of Computational Physics, 229(5):1536–1557, 2010. J. Foo, X. Wan, and G. E. Karniadakis. The multi-element probabilistic collocation method (ME-PCM): Error analysis and applications. Journal of Computational Physics, 227(22):9572–9595, 2008. L. Formaggia, J.-F. Gerbeau, F. Nobile, and A. Quarteroni. On the coupling of 3D and 1D Navier–Stokes equations for flow problems in compliant vessels. Computer Methods in Applied Mechanics and Engineering, 191(6):561–582, 2001. L. Formaggia, A. Quarteroni, and A. Veneziani. Cardiovascular Mathematics: Mod- eling and simulation of the circulatory system, volume 1. Springer Science & Business Media, 2010. A. Forrester, A. Sobester, and A. Keane. Engineering design via surrogate modelling: a practical guide. John Wiley & Sons, 2008. A. I. Forrester, A. S´obester, and A. J. Keane. Multi-fidelity optimization via surro- gate modelling. In Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, volume 463, pages 3251–3269. The Royal So- ciety, 2007. Y.-C. Fung. Biomechanics. Springer, 1990. L. Greengard and J.-Y. Lee. Accelerating the nonuniform fast Fourier transform. SIAM review, 46(3):443–454, 2004. L. Grinberg and G. E. Karniadakis. Outflow boundary conditions for arterial net- works with multiple outlets. Annals of biomedical engineering, 36(9):1496–1514, 2008. L. Grinberg and G. E. Karniadakis. Extrapolation-based acceleration of iterative solvers: Application to simulation of 3D flows. Communication in Computation Physics, 9(3):607–626, 2011. L. Grinberg, D. Pekurovsky, S. Sherwin, and G. E. Karniadakis. Parallel performance of the coarse space linear vertex solver and low energy basis preconditioner for spectral/hp elements. Parallel Computing, 35(5):284–304, 2009. L. Grinberg, E. Cheever, T. Anor, J. R. Madsen, and G. Karniadakis. Modeling blood flow circulation in intracranial arterial networks: a comparative 3D/1D simulation study. Annals of biomedical engineering, 39(1):297–309, 2011a. 231 L. Grinberg, V. Morozov, D. Fedosov, J. A. Insley, M. E. Papka, K. Kumaran, and G. E. Karniadakis. A new computational paradigm in multiscale simulations: Application to brain blood flow. In High Performance Computing, Networking, Storage and Analysis (SC), 2011 International Conference for, pages 1–12. IEEE, 2011b. L. Grinberg, D. A. Fedosov, and G. E. Karniadakis. Parallel multiscale simulations of a brain aneurysm. Journal of computational physics, 244:131–147, 2013. T. Hastie, R. Tibshirani, J. Friedman, T. Hastie, J. Friedman, and R. Tibshirani. The elements of statistical learning, volume 2. Springer, 2009. T. J. Hastie and R. J. Tibshirani. Generalized additive models, volume 43. CRC Press, 1990. H. Ito, I. Kanno, H. Iida, J. Hatazawa, E. Shimosegawa, H. Tamura, and T. Okud- era. Arterial fraction of cerebral blood volume in humans measured by positron emission tomography. Ann. Nucl. Med., 15(2):111–116, 2001. D. R. Jones. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization, 21(4):345–383, 2001. D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455–492, 1998. G. Karniadakis and S. Sherwin. Spectral/hp element methods for computational fluid dynamics. Oxford University Press, 2013. M. C. Kennedy and A. O’Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1–13, 2000. D. Krige. A statistical approach to some mine valuation and allied problems on the Witwatersrand: By DG Krige. PhD thesis, University of the Witwatersrand, 1951. N. Kuschel and R. Rackwitz. Two basic problems in reliability-based structural optimization. Mathematical Methods of Operations Research, 46(3):309–333, 1997. T. Lassila, A. Manzoni, A. Quarteroni, and G. Rozza. A reduced computational and geometrical framework for inverse problems in hemodynamics. International journal for numerical methods in biomedical engineering, 29(7):741–776, 2013. L. Le Gratiet and J. Garnier. Recursive co-kriging model for design of computer experiments with multiple levels of fidelity. International Journal for Uncertainty Quantification, 4(5), 2014. M. Levoy. Volume rendering using the fourier projection-slice theorem. Computer Systems Laboratory, Stanford University, 1992. G. Li, S.-W. Wang, and H. Rabitz. Practical approaches to construct RS-HDMR component functions. The Journal of Physical Chemistry A, 106(37):8721–8733, 2002. 232 F. Lindgren, H. Rue, and J. Lindstr¨om. An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach. Journal of the Royal Statistical Society: Series B (Statistical Method- ology), 73(4):423–498, 2011. D. Lombardi. Inverse problems in 1D hemodynamics on systemic networks: A se- quential approach. Int. J. Numer. Meth. Biomed. Engng., 30(2):160–179, 2014a. D. Lombardi. Inverse problems in 1D hemodynamics on systemic networks: A se- quential approach. International journal for numerical methods in biomedical en- gineering, 30(2):160–179, 2014b. M. L´opez-Fern´andez, C. Lubich, and A. Sch¨adle. Adaptive, fast, and oblivious convolution in evolution equations with memory. SIAM Journal on Scientific Computing, 30(2):1015–1037, 2008. C. Lubich and A. Sch¨adle. Fast convolution for nonreflecting boundary conditions. SIAM Journal on Scientific Computing, 24(1):161–182, 2002. A. Lundkvist, E. Lilleodden, W. Siekhaus, J. Kinney, L. Pruitt, and M. Balooch. Viscoelastic properties of healthy human artery measured in saline solution by AFM-based indentation technique. In MRS Proceedings, volume 436, page 353. Cambridge Univ Press, 1996. R. L. Magin. Fractional calculus in bioengineering, volume 149. Begell House Pub- lishers Redding, 2006. F. Mainardi. Fractional calculus and waves in linear viscoelasticity: an introduction to mathematical models. World Scientific, 2010. G. Matheron. Principles of geostatistics. Economic geology, 58(8):1246–1266, 1963. K. S. Matthys, J. Alastruey, J. Peir´o, A. W. Khir, P. Segers, P. R. Verdonck, K. H. Parker, and S. J. Sherwin. Pulse wave propagation in a model human arterial network: Assessment of 1-D numerical simulations against in vitro measurements. Journal of biomechanics, 40(15):3476–3486, 2007. D. A. McDonald. Blood flow in arteries. 1974. A. Melani. Adjoint-based parameter estimation in human vascular one dimensional models. PhD thesis, Italy, 2013. A. Melani and A. Quarteroni. Adjoint-based parameter estimation in human vascular one dimensional models. PhD thesis, EPFL, 2013. M. Mohamad, W. Cousins, P. Perdikaris, and T. P. Sapsis. Probabilistic framework for extreme event quantification: nonlinear water waves and the response of ocean structures to rogue waves. (in preparation), 2015. T. Muehlenstaedt, O. Roustant, L. Carraro, and S. Kuhnt. Data-driven Kriging models based on FANOVA-decomposition. Statistics and Computing, 22(3):723– 738, 2012. K. P. Murphy. Machine learning: a probabilistic perspective. MIT press, 2012. 233 S. N¨asholm and S. Holm. On a fractional Zener elastic wave equation. Fractional Calculus and Applied Analysis, 16(1):26–50, 2013. N. Nguyen and J. Peraire. Gaussian functional regression for linear partial differential equations. Computer Methods in Applied Mechanics and Engineering, 2015. E. Novak and K. Ritter. High dimensional integration of smooth functions over cubes. Numer. Math., 75(1):79–97, 1996. M. S. Olufsen. Structured tree outflow condition for blood flow in larger systemic arteries. American journal of physiology-Heart and circulatory physiology, 276(1): H257–H268, 1999. G. Papadakis. Coupling 3D and 1D fluid–structure-interaction models for wave propagation in flexible vessels using a finite volume pressure-correction scheme. Communications in numerical methods in engineering, 25(5):533–551, 2009. T. Passerini, M. de Luca, L. Formaggia, A. Quarteroni, and A. Veneziani. A 3D/1D geometrical multiscale model of cerebral vasculature. Journal of Engineering Math- ematics, 64(4):319–330, 2009. A. T. Patera. A spectral element method for fluid dynamics: laminar flow in a channel expansion. Journal of computational Physics, 54(3):468–488, 1984. P. Perdikaris and G. E. Karniadakis. Fractional-order viscoelasticity in one- dimensional blood flow models. Annals of biomedical engineering, 42(5):1012–1023, 2014. P. Perdikaris, L. Grinberg, and G. E. Karniadakis. An effective fractal-tree closure model for simulating blood flow in large arterial networks. Annals of biomedical engineering, pages 1–11, 2014. M. S. Phadke. Quality engineering using robust design. Prentice Hall PTR, 1995. I. Podlubny. Calculation of the Mittag-Leffler function with desired ac- curacy. http://www.mathworks.com/matlabcentral/fileexchange/ 8738-mittag-leffler-function. Accessed: 2012-09-12. I. Podlubny. Fractional differential equations: an introduction to fractional deriva- tives, fractional differential equations, to methods of their solution and some of their applications, volume 198. Academic press, 1998. A. Pries, D. Neuhaus, and P. Gaehtgens. Blood viscosity in tube flow: dependence on diameter and hematocrit. American Journal of Physiology-Heart and Circulatory Physiology, 263(6):H1770–H1778, 1992. C. M. Quick, W. L. Young, and A. Noordergraaf. Infinite number of solutions to the hemodynamic inverse problem. American Journal of Physiology-Heart and Circulatory Physiology, 280(4):H1472–H1479, 2001. ¨ F. Ali¸s, J. Shorter, and K. Shim. Efficient input-output model repre- H. Rabitz, O. sentations. Computer Physics Communications, 117(1):11–20, 1999. 234 R. Raghu, I. E. Vignon-Clementel, C. A. Figueroa, and C. A. Taylor. Compara- tive study of viscoelastic arterial wall models in nonlinear one-dimensional finite element simulations of blood flow. Journal of biomechanical engineering, 133(8): 081003, 2011. C. E. Rasmussen. Gaussian processes for machine learning. 2006. S. S. Rengachary and R. G. Ellenbogen. Principles of neurosurgery. Elsevier Mosby, 2005. P. Reymond, F. Merenda, F. Perren, D. R¨ufenacht, and N. Stergiopulos. Validation of a one-dimensional model of the systemic arterial tree. American Journal of Physiology-Heart and Circulatory Physiology, 297(1):H208, 2009. P. Reymond, Y. Bohraus, F. Perren, F. Lazeyras, and N. Stergiopulos. Validation of a patient-specific one-dimensional model of the systemic arterial tree. Ameri- can Journal of Physiology-Heart and Circulatory Physiology, 301(3):H1173–H1182, 2011. R. T. Rockafellar and J. O. Royset. Engineering decisions under risk-averseness. (under review), 2014. J. Royset and R. Wets. Multi-variate epi-splines and evolving function identification problems. pre-print, University of California, Davis, 2014a. J. Royset and R. J.-B. Wets. From Data to Assessments and Decisions: Epi-Spline Technology. In A. Newman, editor, INFORMS Tutorials. INFORMS, Catonsville, 2014b. J. O. Royset. On sample size control in sample average approximations for solving smooth stochastic programs. Comput. Optim. Appl., 55(2):265–309, 2013. Y. A. Rozanov. Markov random fields. Springer, 1982. H. Rue and L. Held. Gaussian Markov random fields: theory and applications. CRC Press, 2005. J. Sacks, W. J. Welch, T. J. Mitchell, and H. P. Wynn. Design and analysis of computer experiments. Statistical science, pages 409–423, 1989. P. Seshadri, P. Constantine, G. Iaccarino, and G. Parks. Aggressive design: A density-matching approach for optimization under uncertainty. arXiv preprint arXiv:1409.7089, 2014. S. Sherwin, V. Franke, J. Peir´o, and K. Parker. One-dimensional modelling of a vascular network in space-time variables. Journal of Engineering Mathematics, 47 (3-4):217–250, 2003. S. J. Sherwin and M. Casarin. Low-energy basis preconditioning for elliptic substruc- tured solvers based on unstructured spectral/hp element discretization. Journal of Computational Physics, 171(1):394–417, 2001. C.-W. Shu and S. Osher. Efficient implementation of essentially non-oscillatory shock-capturing schemes. Journal of Computational Physics, 77(2):439–471, 1988. 235 T. W. Simpson, V. Toropov, V. Balabanov, and F. A. Viana. Design and analysis of computer experiments in multidisciplinary design optimization: a review of how far we have come or not. In 12th AIAA/ISSMO multidisciplinary analysis and optimization conference, volume 5, pages 10–12, 2008. J. Snoek, K. Swersky, R. S. Zemel, and R. P. Adams. Input warping for Bayesian optimization of non-stationary functions. arXiv preprint arXiv:1402.0929, 2014. B. Steele, M. Olufsen, and C. Taylor. Fractal network model for simulating ab- dominal and lower extremity blood flow during resting and exercise conditions. Comput. Methods Biomech. Biomed. Engin., 10(1):39–51, 2007. B. N. Steele, D. Valdez-Jasso, M. A. Haider, and M. S. Olufsen. Predicting arterial flow and pressure dynamics using a 1D fluid dynamics model with a viscoelastic wall. SIAM Journal on Applied Mathematics, 71(4):1123–1143, 2011. M. L. Stein, J. Chen, M. Anitescu, et al. Stochastic approximation of score functions for Gaussian processes. The Annals of Applied Statistics, 7(2):1162–1191, 2013. S. Surjanovic and D. Bingham. Virtual Library of Simulation Experiments: Test Functions and Datasets. Retrieved March 20, 2015, from http://www.sfu.ca/ ~ssurjano. D. Valdez-Jasso, D. Bia, Y. Z´ocalo, R. L. Armentano, M. A. Haider, and M. S. Olufsen. Linear and nonlinear viscoelastic modeling of aorta and carotid pressure– area dynamics under in vivo and ex vivo conditions. Annals of biomedical engi- neering, 39(5):1438–1456, 2011. E. VanBavel and J. Spaan. Branching patterns in the porcine coronary arterial tree. Estimation of flow heterogeneity. Circ. Res., 71(5):1200–1212, 1992. S. Watanabe, P. Blanco, and R. Feij´oo. Mathematical model of blood flow in an anatomically detailed arterial network of the arm. ESAIM, Math. Model. Numer. Anal., 47(04):961–985, 2013. P. Whittle. STOCHASTIC-PROCESSES IN SEVERAL DIMENSIONS. Bulletin of the International Statistical Institute, 40(2):974–994, 1963. A. Wilson, D. Knowles, and Z. Ghahramani. Gaussian process regression networks. arXiv preprint arXiv:1110.4411, 2011. A. Witthoft and G. E. Karniadakis. A bidirectional model for communication in the neurovascular unit. Journal of theoretical biology, 311:80–93, 2012. N. Xiao, J. D. Humphrey, and C. A. Figueroa. Multi-scale computational model of three-dimensional hemodynamics within a deformable full-body arterial network. Journal of computational physics, 244:22–40, 2013. D. Xiu and G. Karniadakis. The Wiener–Askey polynomial chaos for stochastic dif- ferential equations. SIAM Journal on Scientific Computing, 24(2):619–644, 2002. X. Yang, M. Choi, G. Lin, and G. Karniadakis. Adaptive ANOVA decomposition of stochastic incompressible and compressible flows. Journal of Computational Physics, 231(4):1587–1614, 2012. 236 Y. Yu, H. Baek, M. L. Bittencourt, and G. E. Karniadakis. Mixed spectral/hp ele- ment formulation for nonlinear elasticity. Computer Methods in Applied Mechanics and Engineering, 213:42–57, 2012. Y. Yu, H. Baek, and G. E. Karniadakis. Generalized fictitious methods for fluid–structure interactions: analysis and simulations. Journal of Computational Physics, 245:317–346, 2013. M. Zamir. On fractal properties of arterial trees. Journal of theoretical biology, 197 (4):517–526, 1999. Z. Zhang, M. Choi, and G. E. Karniadakis. Error estimates for the ANOVA method with polynomial chaos interpolation: tensor product functions. SIAM Journal on Scientific Computing, 34(2):A1165–A1186, 2012. T. Ziehn and A. Tomlin. GUI–HDMR–A software tool for global sensitivity analysis of complex models. Environmental Modelling & Software, 24(7):775–785, 2009.