I'm co-author of FEMPAR, a library for the development of Finite Element Multiphysics PARallel solvers for the numerical solution of partial differential equations (PDEs). The key feature of FEMPAR is the full integration of all the different steps of the simulation pipeline, e.g., linearization, discretization and numerical linear algebra, which allows us to explore algorithms that are hard or simply impossible to consider by staggered approaches, e.g., solver-aware and nonlinear preconditioning. FEMPAR is based from inception on a domain decomposition (DD) strategy exploited at all these steps. The ease to combine discretization/solver steps (e.g., merge nonlinear/time loops and Krylov solver iterations) as well as its state-of-the-art implementation of balancing DD algorithms on high core counts (see e.g. Badia et. al.) makes FEMPAR an innovative and distinctive platform which is perfect for developing extreme scale solvers.

FEMPAR is implemented in Fortran 2008 using an object-oriented approach and hybrid OpenMP/MPI through third party libraries. The parallel codes in FEMPAR heavily use standard computational kernels provided by highly-efficient vendor implementations of the BLAS as well as interfaces to sparse linear system solvers, e.g. PARDISO.

We have already analyzed the scalability of FEMPAR in the largest supercomputers in the world (JUQUEEN, HERMIT, CURIE, HELIOS, etc.) up to 458,752 cores with excellent results. In the first figure below weak scalability of FEMPAR in JUQUEEN for the solution of the 3D Poisson problem using a 3-level DD method can be observed. We further stressed the algorithm by considering oversuscription (4 subdomains per core) reaching 1.73M subdomains as shown in the second figure below. In both cases the problem size is about 30,000,000,000 unknowns. These results made FEMPAR belong to the HighQ club of the highest scaling codes on JUQUEEN.

FEMPAR is free software, distributed under the terms of GNU GPL version 3 (a public release will be made soon). The free software (open source) paradigm seems to be the best approach for extreme scale software projects, which require collaborative research between experts in different fields, e.g., applied mathematics, computational engineering/physics and computer science. Nowadays, there are some high-quality libraries for the numerical approximation of problems governed by PDEs using grid-based methods (e.g. deal.II ). These libraries usually provide the discretization/linearization machinery, leaving the numerical linear algebra tasks to external libraries like TRILINOS, PETSC, HYPRE. In the future exascale scenario, this staggered approach might not be the best option.