I'm co-author of FEMPAR, a library for the development of Finite Element Multiphysics PARallel solvers for the
numerical solution of partial differential equations (PDEs).
The key feature of FEMPAR is the full integration of all the different steps of the simulation pipeline, e.g.,
linearization, discretization and numerical linear algebra, which allows us to explore algorithms that are hard
or simply impossible to consider by staggered approaches, e.g., solver-aware and nonlinear preconditioning.
FEMPAR is based from inception on a domain decomposition (DD) strategy exploited at all these steps.
The ease to combine discretization/solver steps (e.g., merge nonlinear/time loops and Krylov solver
iterations) as well as its state-of-the-art implementation of balancing DD algorithms on high core counts
(see e.g. Badia et. al.
) makes FEMPAR an innovative and
distinctive platform which is perfect for developing extreme scale solvers.
FEMPAR is implemented in Fortran 2008 using an object-oriented approach and hybrid OpenMP/MPI through
third party libraries. The parallel codes in FEMPAR heavily use standard computational kernels provided by
highly-efficient vendor implementations of the BLAS as well as interfaces to sparse linear system solvers, e.g.
We have already analyzed the scalability of FEMPAR in the largest supercomputers in the world
(JUQUEEN, HERMIT, CURIE, HELIOS, etc.) up to 458,752 cores with excellent results. In the first figure below
weak scalability of FEMPAR in JUQUEEN for the solution of the 3D Poisson problem using a 3-level DD method
can be observed. We further stressed the algorithm by considering oversuscription (4 subdomains per core)
reaching 1.73M subdomains as shown in the second figure below. In both cases the problem size is about
These results made FEMPAR belong to the HighQ club
of the highest scaling codes on JUQUEEN.
FEMPAR is free software, distributed under the terms of GNU GPL version 3 (a public release will be made soon).
The free software (open source) paradigm seems to be the best approach for extreme scale software projects,
which require collaborative research between experts in different fields, e.g., applied mathematics,
computational engineering/physics and computer science.
Nowadays, there are some high-quality libraries for the numerical approximation of problems governed by PDEs using grid-based methods (e.g. deal.II
These libraries usually provide the discretization/linearization machinery, leaving the numerical linear
algebra tasks to external libraries like
. In the future exascale
scenario, this staggered approach might not be the best option.