
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2026 by the esys.escript Group
% https://github.com/LutzGross/esys-escript.github.io
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%!TEX root = user.tex
\chapter{Using \mpifo with \escript}
\label{CHAP:mpi4py}

\section{Introduction}

\mpifo\footnote{\url{https://mpi4py.readthedocs.io/}} is a Python package that provides bindings to the Message Passing Interface (\MPI) standard.
When \escript is built with \mpifo support enabled, you can use standard \MPI programming techniques directly in your \PYTHON scripts to control how \escript uses parallel resources.

This chapter explains how to use \mpifo to:
\begin{itemize}
\item Pass custom \MPI communicators to \escript domains
\item Run multiple simulations simultaneously on different process groups
\item Implement ensemble simulations and parameter sweeps
\item Couple multiple physics domains on separate communicators
\item Retrieve \MPI communicators from \escript objects for custom communication
\end{itemize}

\subsection{Why use \mpifo with \escript?}

By default, when \escript runs with multiple \MPI processes, it automatically uses all available processes to solve each PDE.
This works well for large, high-resolution problems.
However, there are scenarios where you might want more control:

\begin{enumerate}
\item \textbf{Ensemble simulations}: Run multiple parameter values simultaneously, each on a smaller group of processes
\item \textbf{Parameter sweeps}: Explore parameter space by running many simulations in parallel
\item \textbf{Multi-physics coupling}: Solve different physics on different process groups
\item \textbf{Uncertainty quantification}: Run multiple realizations of stochastic problems
\item \textbf{Optimization}: Evaluate multiple trial solutions in parallel
\end{enumerate}

Using \mpifo gives you direct control over how processes are organized and allows you to implement these patterns using standard \MPI techniques.

\subsection{Requirements}

To use \mpifo with \escript:
\begin{itemize}
\item \escript must be built with \texttt{mpi4py=True} in the build configuration
\item \mpifo must be installed and compiled against the same \MPI implementation as \escript
\item Use \texttt{mpi='auto'} in the build configuration to automatically detect the correct \MPI implementation
\end{itemize}

See the \escript installation guide for detailed instructions.

\subsection{MPI Initialization and the pythonMPI Wrapper}
\label{sec:mpi4py:init}

\textbf{Critical}: \escript requires \MPI to be initialized \textbf{before} the Python interpreter starts to ensure proper initialization of C++ libraries like Trilinos. This section explains why and how to do it correctly.

\subsubsection{Why MPI Initialization Order Matters}

\escript uses the Trilinos library for distributed linear algebra operations. Trilinos contains C++ code with static objects that capture \MPI state when shared libraries are loaded. If \MPI is initialized \textbf{after} Python starts, some C++ static objects may capture an uninitialized \MPI state, leading to inconsistent behavior and errors.

\textbf{Symptoms of incorrect initialization}:
\begin{itemize}
\item Trilinos errors like: \texttt{global index N does not have a corresponding local index in the Directory Map}
\item Hangs during interpolation between function spaces
\item Crashes or incorrect results when using multiple \MPI processes
\end{itemize}

These errors occur because different parts of the C++ code disagree about how many global indices exist, leading to off-by-one errors and invalid memory access.

\subsubsection{The pythonMPI Wrapper}

To solve this problem, \escript provides a custom Python launcher called \texttt{pythonMPI} that ensures \MPI is initialized \textbf{before} Python starts:

\begin{enumerate}
\item The \texttt{pythonMPI} executable (a C++ program) calls \texttt{MPI\_Init\_thread()}
\item \MPI runtime is fully initialized with \texttt{MPI\_COMM\_WORLD} ready
\item \texttt{pythonMPI} then starts the Python interpreter via \texttt{Py\_Main()}
\item Python and all C++ libraries load with \MPI already active
\item All C++ static initialization sees a consistent, initialized \MPI state
\end{enumerate}

The \texttt{run-escript} launcher automatically uses \texttt{pythonMPI} when running with \MPI.

\subsubsection{Correct Usage: Always Use run-escript}

\textbf{CORRECT}:
\begin{verbatim}
# For MPI jobs, always use run-escript
run-escript -n 4 myscript.py

# For serial jobs, run-escript also works
run-escript myscript.py
\end{verbatim}

\textbf{WRONG}:
\begin{verbatim}
# DO NOT use mpirun/mpiexec with python3 directly
mpirun -n 4 python3 myscript.py  # Will cause Trilinos errors!
\end{verbatim}

\subsubsection{What Happens with Direct mpirun python3}

When you run \texttt{mpirun -n 4 python3 myscript.py}, the initialization sequence is incorrect:

\begin{enumerate}
\item Python interpreter starts \textbf{without} \MPI
\item Python imports \mpifo
\item \mpifo calls \texttt{MPI\_Init()} ← \MPI initialized \textbf{after} Python started
\item Python imports \escript modules
\item Some Trilinos C++ static objects were created \textbf{before} \texttt{MPI\_Init()}
\item Later code uses \MPI state from \textbf{after} \texttt{MPI\_Init()}
\item Inconsistent state causes index calculation errors
\end{enumerate}

\textbf{Example error}: For a mesh with 861 nodes (indices 0--860), some code may think there are 862 nodes and try to access index 862, which doesn't exist.

\subsubsection{Import Order When Using run-escript}

When using \texttt{run-escript}, the import order doesn't matter because \MPI is already initialized:

\begin{python}
# Both work with run-escript
from esys.ripley import Rectangle
from mpi4py import MPI
\end{python}

However, for portability and clarity, it's still good practice to import \mpifo first:

\begin{python}
# Best practice - import mpi4py first
from mpi4py import MPI
from esys.escript import *
from esys.ripley import Rectangle
\end{python}

\subsubsection{Why Import Order Matters Without run-escript}

If you run scripts directly with the Python interpreter (e.g., for serial jobs), you \textbf{must} import \mpifo before \escript:

\begin{python}
# CORRECT for direct python3 execution (serial only)
from mpi4py import MPI
from esys.ripley import Rectangle

domain = Rectangle(n0=100, n1=100)
\end{python}

Do \textbf{not} import \escript before \mpifo:

\begin{python}
# WRONG - Will fail
from esys.ripley import Rectangle
from mpi4py import MPI  # Too late!

domain = Rectangle(n0=100, n1=100)  # Error: MPI not initialized
\end{python}

This produces:
\begin{verbatim}
RuntimeError: MPI has not been initialized. MPI must be
initialized before using escript domains. When using Python,
import mpi4py before importing escript, or use the run-escript
launcher for MPI programs.
\end{verbatim}

\subsubsection{Summary: Best Practices}

\begin{itemize}
\item \textbf{Always use} \texttt{run-escript} for \MPI programs (required for correct Trilinos initialization)
\item \textbf{Never use} \texttt{mpirun python3} directly (will cause Trilinos errors)
\item \textbf{Always import} \mpifo before \escript modules (good practice even with \texttt{run-escript})
\item The \texttt{pythonMPI} wrapper ensures \MPI initializes before Python starts
\item This prevents C++ static initialization order issues with Trilinos
\end{itemize}

\section{Basic Concepts}

\subsection{MPI Communicators}

An \MPI communicator defines a group of processes that can communicate with each other.
By default, all processes belong to \texttt{MPI\_COMM\_WORLD}, which includes every process in the job.

In \mpifo, you can access this default communicator:
\begin{python}
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()  # My process rank (0 to size-1)
size = comm.Get_size()  # Total number of processes
\end{python}

\subsection{Splitting Communicators}

You can create smaller communicators by splitting \texttt{MPI\_COMM\_WORLD}:
\begin{python}
# Split into groups based on color
color = rank % 2  # Processes with same color go in same group
sub_comm = comm.Split(color, rank)

# Now sub_comm is a communicator with only part of the processes
sub_rank = sub_comm.Get_rank()
sub_size = sub_comm.Get_size()
\end{python}

Processes with the same \var{color} are grouped together into a new communicator.
The second argument to \texttt{Split()} determines the rank ordering within each new group.

\section{Passing Communicators to \escript Domains}

All \escript domain constructors accept an optional \var{comm} parameter that allows you to specify which \MPI communicator the domain should use:

\begin{python}
from mpi4py import MPI
from esys.ripley import Rectangle

# Create a domain using default MPI_COMM_WORLD
domain = Rectangle(n0=100, n1=100)

# Or explicitly pass MPI_COMM_WORLD
domain = Rectangle(n0=100, n1=100, comm=MPI.COMM_WORLD)

# Or use a custom communicator
sub_comm = MPI.COMM_WORLD.Split(color, rank)
domain = Rectangle(n0=100, n1=100, comm=sub_comm)
\end{python}

The \var{comm} parameter is available in all domain types:
\begin{itemize}
\item \textbf{Finley}: \texttt{ReadMesh()}, \texttt{ReadGmsh()}, \texttt{Rectangle()}, \texttt{Brick()}
\item \textbf{Ripley}: \texttt{Rectangle()}, \texttt{Brick()}, \texttt{MultiRectangle()}, \texttt{MultiBrick()}
\item \textbf{Speckley}: \texttt{Rectangle()}, \texttt{Brick()}
\item \textbf{Oxley}: \texttt{Rectangle()}, \texttt{Brick()} (requires Trilinos)
\end{itemize}

\section{Retrieving Communicators from \escript Objects}

You can retrieve the \MPI communicator from any \escript domain, function space, or data object:

\begin{python}
# From a domain
domain_comm = domain.getMPIComm()

# From a function space
fs = ContinuousFunction(domain)
fs_comm = fs.getMPIComm()

# From a data object
data = Scalar(1.0, fs)
data_comm = data.getMPIComm()

# All return the same communicator
assert MPI.Comm.Compare(domain_comm, fs_comm) == MPI.IDENT
assert MPI.Comm.Compare(fs_comm, data_comm) == MPI.IDENT
\end{python}

This allows you to:
\begin{itemize}
\item Verify which communicator a domain is using
\item Perform custom \MPI operations using the domain's communicator
\item Pass the communicator to other libraries or functions
\end{itemize}

\textbf{Important}: When comparing \MPI communicators, always use \texttt{MPI.Comm.Compare()}, never use \texttt{==}.
The comparison returns \texttt{MPI.IDENT} if the communicators are identical, \texttt{MPI.CONGRUENT} if they have the same processes but different contexts, or \texttt{MPI.UNEQUAL} if they differ.

\section{Simple Example: Ensemble Simulation}

This example demonstrates running multiple parameter values in parallel.
Each process group solves the same problem with a different parameter:

\begin{python}
from mpi4py import MPI
from esys.escript import *
from esys.escript.linearPDEs import LinearPDE
from esys.ripley import Rectangle

# Get MPI information
world_comm = MPI.COMM_WORLD
world_rank = world_comm.Get_rank()
world_size = world_comm.Get_size()

# Define parameter values to explore
parameters = [0.1, 0.5, 1.0, 2.0]
num_params = len(parameters)

# Ensure we have enough processes
if world_size < num_params:
    if world_rank == 0:
        print(f"Need at least {num_params} processes for this example")
    exit(1)

# Assign each process to a parameter
param_id = world_rank % num_params
my_parameter = parameters[param_id]

# Split communicator: processes with same parameter work together
processes_per_param = world_size // num_params
group_comm = world_comm.Split(param_id, world_rank)

# Create domain with group communicator
domain = Rectangle(n0=50, n1=50, comm=group_comm)

# Solve PDE with this parameter value
pde = LinearPDE(domain)
x = domain.getX()
pde.setValue(A=my_parameter*kronecker(2), Y=x[0]*x[1])
solution = pde.getSolution()

# Each group can now process its result independently
if group_comm.Get_rank() == 0:
    result = integrate(solution)
    print(f"Parameter {my_parameter}: Result = {result}")

# Clean up communicator
group_comm.Free()
\end{python}

\textbf{Running this example}:
\begin{verbatim}
mpirun -n 8 run-escript ensemble_example.py
\end{verbatim}

With 8 processes and 4 parameter values, each parameter gets 2 processes.
All 4 simulations run simultaneously.

\section{Advanced Example: Parameter Sweep}

This example shows a more realistic scenario where different grid resolutions are tested simultaneously:

\begin{python}
from mpi4py import MPI
from esys.escript import *
from esys.escript.linearPDEs import LinearPDE
from esys.ripley import Rectangle
import time

world_comm = MPI.COMM_WORLD
world_rank = world_comm.Get_rank()
world_size = world_comm.Get_size()

# Grid resolutions to test
grid_sizes = [
    (10, 10),   # Coarse
    (20, 20),   # Medium
    (40, 40),   # Fine
    (80, 80),   # Very fine
]
num_grids = len(grid_sizes)

# Split into groups
grid_id = world_rank % num_grids
nx, ny = grid_sizes[grid_id]
grid_comm = world_comm.Split(grid_id, world_rank)
grid_rank = grid_comm.Get_rank()

if world_rank == 0:
    print(f"Running {num_grids} grid resolutions simultaneously")
    print(f"Total processes: {world_size}")

# Synchronize before starting
world_comm.Barrier()
start_time = time.time()

# Create domain with this grid size
domain = Rectangle(n0=nx, n1=ny, comm=grid_comm)

# Solve a test problem
pde = LinearPDE(domain)
x = domain.getX()
pde.setValue(A=kronecker(2), Y=exp(-((x[0]-0.5)**2 + (x[1]-0.5)**2)/0.1))
solution = pde.getSolution()

# Compute result
result = integrate(solution)
elapsed = time.time() - start_time

# Leader of each group reports
if grid_rank == 0:
    procs = grid_comm.Get_size()
    print(f"Grid {nx}x{ny} ({procs} proc): "
          f"time={elapsed:.4f}s, result={result:.6f}")

# Gather all results to rank 0
all_results = world_comm.gather((nx, ny, elapsed, result), root=0)

if world_rank == 0:
    print("\nSummary:")
    print("  Grid Size  |  Time (s)  |   Result")
    print("-" * 42)
    for i in range(0, len(all_results), world_size // num_grids):
        nx, ny, t, r = all_results[i]
        print(f"  {nx:3d} x {ny:3d}   |   {t:6.4f}   |  {r:8.6f}")

grid_comm.Free()
\end{python}

\section{Master-Slave Pattern Example}

The master-slave pattern is useful when you have a collection of independent tasks to distribute among worker processes. One process (the master) manages task distribution and result collection, while other processes (slaves) compute results.

This example shows how to use a master-slave pattern to solve magnetotelluric (MT) problems for multiple frequencies:

\begin{python}
from mpi4py import MPI
from esys.escript import *
from esys.finley import Rectangle
from esys.escript.linearPDEs import LinearSinglePDE
import numpy as np

comm_world = MPI.COMM_WORLD
rank = comm_world.Get_rank()
size = comm_world.Get_size()

def solve_MT_for_frequency(freq):
    """Solve MT problem using MPI.COMM_SELF (single process)"""
    # Create independent domain
    domain = Rectangle(n0=200, n1=100, l0=40000, l1=20000,
                      comm=MPI.COMM_SELF)

    # Setup PDE (complex-valued for frequency domain)
    pde = LinearSinglePDE(domain, isComplex=True)
    Mu0 = 4*np.pi*1e-7
    pde.setValue(D=1j*2*np.pi*freq*Mu0)

    # Set resistivity and boundary conditions
    # (simplified - see full example for details)
    pde.setValue(A=100*kronecker(2))  # 100 Ohm-m
    x = domain.getX()
    pde.setValue(q=whereZero(x[1]-20000), r=1)

    # Solve and return impedance
    Hx = pde.getSolution()
    return integrate(abs(Hx))

if rank == 0:
    # MASTER PROCESS
    frequencies = np.logspace(-1, 1, 10)  # 0.1 to 10 Hz
    results = []
    tasks_sent = 0
    tasks_received = 0

    # Send initial tasks to slaves
    for slave in range(1, min(size, len(frequencies)+1)):
        if tasks_sent < len(frequencies):
            comm_world.send(frequencies[tasks_sent], dest=slave, tag=1)
            tasks_sent += 1

    # Receive results and send more tasks
    while tasks_received < len(frequencies):
        # Receive result from any slave
        status = MPI.Status()
        result = comm_world.recv(source=MPI.ANY_SOURCE, tag=2,
                                status=status)
        slave = status.Get_source()
        results.append(result)
        tasks_received += 1

        # Send next task if available
        if tasks_sent < len(frequencies):
            comm_world.send(frequencies[tasks_sent], dest=slave, tag=1)
            tasks_sent += 1

    # Send termination signal
    for slave in range(1, size):
        comm_world.send(None, dest=slave, tag=1)

    print(f"Completed {len(results)} frequencies")

else:
    # SLAVE PROCESS
    while True:
        # Receive task from master
        freq = comm_world.recv(source=0, tag=1)

        # Check for termination
        if freq is None:
            break

        # Solve problem
        result = solve_MT_for_frequency(freq)

        # Send result back
        comm_world.send((freq, result), dest=0, tag=2)

comm_world.Barrier()
\end{python}

\textbf{Key features of this pattern}:
\begin{itemize}
\item \textbf{Dynamic load balancing}: Master assigns tasks as slaves become available
\item \textbf{Independent domains}: Each slave uses \texttt{MPI.COMM\_SELF} for its own domain
\item \textbf{Asynchronous communication}: \texttt{MPI.ANY\_SOURCE} allows receiving from any slave
\item \textbf{Scalability}: Works with any number of slave processes
\end{itemize}

\textbf{Running this example}:
\begin{verbatim}
mpirun -n 11 run-escript mt_master_slave.py
\end{verbatim}

This runs with 1 master and 10 slaves, solving 10 frequencies efficiently. If more slaves than tasks are available, some slaves will be idle.

A complete working example (\texttt{MT\_master\_slave.py}) is provided in the \texttt{doc/examples/usersguide/} directory.

\section{Multi-Physics Example: Thermo-Mechanical Coupling}

This example demonstrates time-dependent coupling of thermal and mechanical physics using DataCoupler for inter-domain communication. The thermal solver computes temperature evolution while the mechanical solver calculates thermal stresses. The mechanical model runs one time-step behind to use the temperature from the previous step (staggered coupling).

\begin{python}
from mpi4py import MPI
from esys.escript import *
from esys.escript.linearPDEs import LinearPDE
from esys.escript.util import identityTensor4
from esys.ripley import Rectangle
# Import DataCoupler and MPIDomainArray from escript
from esys.escript import MPIDomainArray, DataCoupler

world_comm = MPI.COMM_WORLD
world_rank = world_comm.Get_rank()
world_size = world_comm.Get_size()

# Create domain array: 2 domains (thermal and mechanical)
num_domains = 2
domain_array = MPIDomainArray(numDomains=num_domains, comm=world_comm)

my_domain_idx = domain_array.getDomainIndex()
domain_comm = domain_array.getDomainComm()
domain_rank = domain_comm.Get_rank()
is_thermal = (my_domain_idx == 0)

# Create DataCoupler for inter-domain communication
coupler = DataCoupler(domain_array)

# Create domain on domain-specific communicator
domain = Rectangle(n0=80, n1=80, l0=1.0, l1=1.0, comm=domain_comm)
x = domain.getX()

# Time stepping parameters
dt = 0.00025   # Time step
n_steps = 200  # Number of time steps

if is_thermal:
    # THERMAL SOLVER: Transient heat equation
    # du/dt = kappa * nabla^2(u) + Q

    pde = LinearPDE(domain)
    kappa = 1.0  # Thermal diffusivity
    pde.setValue(A=kappa*kronecker(2), D=1/dt)

    # Initial temperature: hot spot in center
    T = 20.0 + 100.0*exp(-50*((x[0]-0.5)**2 + (x[1]-0.5)**2))
    T_old = T

    for step in range(n_steps):
        # Heat source (decaying with time)
        Q = 10.0 * exp(-0.1*step*dt) * \
            exp(-50*((x[0]-0.5)**2 + (x[1]-0.5)**2))

        # Backward Euler: (T - T_old)/dt = kappa*nabla^2(T) + Q
        pde.setValue(Y=T_old/dt + Q)
        T = pde.getSolution()

        # Send temperature field to mechanical solver using DataCoupler
        T_data = interpolate(T, Function(domain))
        coupler.send(T_data, dest_domain_index=1, tag=step)

        T_max = Lsup(T)
        if domain_rank == 0 and step % 10 == 0:
            print(f"Thermal step {step}: T_max = {T_max:.2f}")
        T_old = T

else:
    # MECHANICAL SOLVER: Thermal stress (one step behind)
    # Solve: -div(sigma) = 0, sigma = E*epsilon - alpha*E*Delta_T

    pde = LinearPDE(domain, numEquations=2)
    E = 70e9        # Young's modulus (Pa)
    nu = 0.3        # Poisson's ratio
    alpha = 23e-6   # Thermal expansion coefficient (1/K)
    T0 = 20.0       # Reference temperature

    # Elastic stiffness tensor (plane stress)
    # C_ijkl = lambda * delta_ij * delta_kl + mu * (delta_ik * delta_jl + delta_il * delta_jk)
    lam = E*nu/(1-nu**2)       # Lame's first parameter (plane stress)
    mu = E/(2*(1+nu))           # Shear modulus

    # Construct rank-4 elastic stiffness tensor
    d = 2  # 2D
    C = lam * kronecker(d).reshape(d,1,d,1) * kronecker(d).reshape(1,d,1,d) + \
        mu * (identityTensor4(d) + identityTensor4(d).transpose(0,1,3,2))

    pde.setValue(A=C)
    # Fixed displacement on left edge
    pde.setValue(q=whereZero(x[0])*[1,1], r=[0.0, 0.0])

    for step in range(0, n_steps):
        # Receive temperature from thermal solver using DataCoupler
        T_data = coupler.receive(Function(domain), source_domain_index=0, tag=step)

        # Thermal stress (prestress due to thermal expansion)
        Delta_T = T_data - T0
        sigma_thermal = (lam + 2./3.*mu) * alpha * Delta_T * kronecker(2)

        # Set thermal stress as prestress (X coefficient)
        pde.setValue(X=sigma_thermal)

        # Solve for displacement
        u = pde.getSolution()
        u_max = Lsup(length(u))
        if domain_rank == 0 and step % 10 == 0:
            print(f"Mechanical step {step}: u_max = {u_max:.2e} m")

world_comm.Barrier()
\end{python}

\textbf{Key features of this coupled simulation}:
\begin{itemize}
\item \textbf{DataCoupler}: Uses high-level DataCoupler API for inter-domain communication
\item \textbf{MPIDomainArray}: Manages 2D Cartesian communicator topology with domain and subdomain communicators
\item \textbf{Parallel physics}: Both solvers run simultaneously on separate communicators
\item \textbf{Correct physics}: Uses proper rank-4 elastic tensor and prestress formulation
\item \textbf{Realistic model}: Thermal expansion causes mechanical deformation
\item \textbf{Scalability}: Each physics can use different number of processes
\end{itemize}

\textbf{Running this example}:
\begin{verbatim}
# Standard mode (only rank 0 output visible):
run-escript -p 8 thermo_mechanical.py

# Verbose mode (see output from both domains):
run-escript -p 8 -a thermo_mechanical.py
\end{verbatim}

This uses 4 processes for thermal domain and 4 for mechanical domain. The \texttt{-a} flag enables viewing output from all MPI ranks, allowing you to see progress messages from both the thermal and mechanical solvers.

\section{Master-Slave Pattern for Embarrassingly Parallel Problems}

The master-slave pattern is ideal for \textbf{embarrassingly parallel} problems where many independent computations must be performed with different parameters. One master process distributes tasks to multiple slave processes, which compute results independently and return them to the master.

This example demonstrates solving a magnetotelluric (MT) problem for multiple frequencies in parallel. The MT method uses electromagnetic induction to image subsurface electrical resistivity. Each frequency requires solving a complex-valued PDE, but the solutions are independent, making this an ideal master-slave problem.

\subsection{Domain Creation Strategy}

Each slave process creates its own domain on \texttt{MPI.COMM\_SELF} (a communicator containing only that process). This allows:
\begin{itemize}
\item \textbf{Independent execution}: Each slave solves its problem without MPI collective operations
\item \textbf{OpenMP parallelization}: Each domain can use OpenMP threads for shared-memory parallelism
\item \textbf{Efficiency}: Domain is created once and reused for all frequencies assigned to that slave
\end{itemize}

\textbf{Key code structure} from \texttt{MT\_master\_slave.py}:

\begin{python}
from mpi4py import MPI
from esys.escript import *
from esys.finley import Rectangle

comm_world = MPI.COMM_WORLD
rank = comm_world.Get_rank()
size = comm_world.Get_size()

# Create domain once on COMM_SELF (single MPI rank with OpenMP)
# Each process creates its own independent domain
domain = Rectangle(n0=NE0, n1=NE1, l0=L0, l1=L1, comm=MPI.COMM_SELF)

def solve_MT_problem(frequency):
    """Solve MT problem for given frequency using pre-created domain."""

    # Set up complex-valued PDE for magnetic field
    pde = LinearSinglePDE(domain, isComplex=True)
    pde.setValue(D=1j*2*np.pi*frequency*Mu0)

    # Define resistivity distribution (anomaly + background)
    rho = rho_b * (1-m) + rho_a * m  # m is anomaly mask
    pde.setValue(A=rho*np.eye(2))

    # Boundary condition: Hx = 1 at top surface
    pde.setValue(q=mD, r=1)

    # Solve for magnetic field
    Hx = pde.getSolution()

    # Calculate impedance and apparent resistivity
    Ey = rho * grad(Hx)[1]
    Zyx = Ey / Hx
    rho_a = 1./(2*np.pi*frequency*Mu0) * abs(Zyx)**2

    return x_positions, rho_a, phase

def master_process():
    """Distribute frequencies to slaves and collect results."""
    frequencies = np.logspace(-1, 1, 10)  # 0.1 to 10 Hz

    # Send initial tasks to all slaves
    for slave_rank in range(1, size):
        if tasks_sent < len(frequencies):
            comm_world.send(frequencies[tasks_sent], dest=slave_rank, tag=1)
            tasks_sent += 1

    # Receive results and send new tasks
    while tasks_received < len(frequencies):
        result = comm_world.recv(source=MPI.ANY_SOURCE, tag=2, status=status)
        results.append(result)

        # Send next task if available
        if tasks_sent < len(frequencies):
            comm_world.send(frequencies[tasks_sent], dest=slave_rank, tag=1)

    # Send termination signal
    for slave_rank in range(1, size):
        comm_world.send(None, dest=slave_rank, tag=1)

    plot_results(results)

def slave_process():
    """Receive frequencies, solve, and return results."""
    while True:
        freq = comm_world.recv(source=0, tag=1)
        if freq is None:  # Termination signal
            break

        # Solve MT problem for this frequency
        x_ts, rho_a_ts, phi_ts = solve_MT_problem(freq)

        # Send results back to master
        comm_world.send((freq, x_ts, rho_a_ts, phi_ts), dest=0, tag=2)

# Main execution
if rank == 0:
    master_process()
else:
    slave_process()
\end{python}

\subsection{Performance Considerations}

For embarrassingly parallel problems with relatively small domains:
\begin{verbatim}
OMP_NUM_THREADS=1 mpirun -n 11 python3 MT_master_slave.py
\end{verbatim}

Setting \texttt{OMP\_NUM\_THREADS=1} maximizes parallelism by using more slave processes rather than OpenMP threads. With 11 processes (1 master + 10 slaves), you can solve 10 frequencies simultaneously.

\subsection{Results}

Figure~\ref{fig:mt_master_slave} shows apparent resistivity and phase computed for 10 frequencies (0.1--10~Hz). The conductive anomaly (0.5~$\Omega$m) within the resistive background (100~$\Omega$m) creates a characteristic signature that varies with frequency due to electromagnetic skin depth effects.

\begin{figure}[ht]
\centering
\includegraphics[width=0.95\textwidth]{figures/MT_master_slave_results.png}
\caption{Magnetotelluric apparent resistivity and phase transects computed for 10 frequencies using the master-slave pattern. Lower frequencies penetrate deeper and show stronger anomaly response.}
\label{fig:mt_master_slave}
\end{figure}

Figure~\ref{fig:mt_normalized} shows the normalized resistivity (apparent resistivity divided by background resistivity), clearly revealing the frequency-dependent anomaly signature centered at 41~km.

\begin{figure}[ht]
\centering
\includegraphics[width=0.95\textwidth]{figures/MT_master_slave_normalized.png}
\caption{Normalized resistivity showing frequency-dependent anomaly response. The anomaly signature is strongest at lower frequencies which penetrate deeper into the subsurface.}
\label{fig:mt_normalized}
\end{figure}

\textbf{Key advantages of the master-slave pattern}:
\begin{itemize}
\item \textbf{Load balancing}: Master dynamically assigns work as slaves finish
\item \textbf{Scalability}: Add more slaves to process more parameters in parallel
\item \textbf{Simplicity}: No complex inter-process communication or synchronization
\item \textbf{Fault tolerance}: Easy to detect and reassign failed tasks
\item \textbf{Resource efficiency}: Each slave uses minimal MPI resources (single rank)
\end{itemize}

The complete example code is available in \texttt{doc/examples/usersguide/MT\_master\_slave.py}.

\section{Best Practices}

\subsection{Communicator Management}

Always free custom communicators when you're done with them:
\begin{python}
sub_comm = MPI.COMM_WORLD.Split(color, rank)
# ... use sub_comm ...
sub_comm.Free()  # Important: free when done
\end{python}

Do \textbf{not} free \texttt{MPI.COMM\_WORLD} or communicators returned by \texttt{getMPIComm()}, as these are managed by the system.

\subsection{Synchronization}

Use barriers when you need to synchronize across groups:
\begin{python}
# Synchronize within a group
group_comm.Barrier()

# Synchronize all processes
MPI.COMM_WORLD.Barrier()
\end{python}

\subsection{Error Handling}

Exceptions in one process group won't automatically propagate to others.
Consider using \MPI collective operations to detect and handle errors:

\begin{python}
try:
    # Solve PDE
    solution = pde.getSolution()
    error_code = 0
except Exception as e:
    print(f"Rank {rank} error: {e}")
    error_code = 1

# Check if any process had an error
global_error = group_comm.allreduce(error_code, op=MPI.MAX)
if global_error != 0:
    print(f"Group {group_id} had errors")
\end{python}

\subsection{Load Balancing}

When splitting into groups, ensure each group has sufficient resources:
\begin{python}
# Good: evenly divides processes
num_groups = 4
group_id = rank // (world_size // num_groups)

# Bad: might create groups with different sizes
group_id = rank % num_groups  # Only works if size divisible by num_groups
\end{python}

\section{Performance Considerations}

\subsection{Process Group Size}

Each domain needs enough processes to solve efficiently:
\begin{itemize}
\item Very small groups (1-2 processes) may not parallelize well
\item Very large groups waste resources on small problems
\item Experiment to find the optimal group size for your problem
\end{itemize}

\subsection{Communication Overhead}

Minimize communication between process groups:
\begin{itemize}
\item Use group-local operations when possible
\item Reduce data before sending between groups
\item Consider asynchronous communication (\texttt{isend/irecv})
\end{itemize}

\subsection{Load Balancing}

Ensure all groups have similar workloads:
\begin{itemize}
\item Groups with different problem sizes finish at different times
\item The slowest group determines overall runtime
\item Use dynamic work distribution for highly variable problems
\end{itemize}

\section{Advanced Topics}

\subsection{Custom Topologies}

Create communicators with specific topologies:
\begin{python}
# Cartesian topology for structured grids
dims = [2, 2]  # 2x2 process grid
periods = [False, False]  # Non-periodic boundaries
cart_comm = comm.Create_cart(dims, periods=periods)

# Use in escript domain
domain = Rectangle(n0=100, n1=100, comm=cart_comm)
\end{python}

\subsection{Interoperability}

The communicators work with other \MPI-based Python libraries:
\begin{python}
# Use with mpi4py
comm = domain.getMPIComm()

# Send data using standard MPI
if comm.Get_rank() == 0:
    data = numpy.array([1.0, 2.0, 3.0])
    comm.Send(data, dest=1, tag=11)
\end{python}

\subsection{Fault Tolerance}

Handle communicator-specific failures:
\begin{python}
try:
    solution = pde.getSolution()
    success = 1
except:
    success = 0

# Check if all processes succeeded
all_success = group_comm.allreduce(success, op=MPI.MIN)
if all_success == 0:
    # At least one process failed
    if group_rank == 0:
        print(f"Group {group_id} failed")
\end{python}

\section{Summary}

Using \mpifo with \escript provides:
\begin{itemize}
\item Direct control over parallel resource allocation
\item Standard \MPI programming techniques
\item Flexibility for complex parallel patterns
\item Interoperability with other \MPI libraries
\item Fine-grained performance tuning
\end{itemize}

Key capabilities:
\begin{itemize}
\item Pass custom communicators to domains via \var{comm} parameter
\item Retrieve communicators via \texttt{getMPIComm()}
\item Run ensemble simulations and parameter sweeps
\item Implement multi-physics coupling
\item Use standard \MPI collective operations
\end{itemize}

For more information:
\begin{itemize}
\item \mpifo documentation: \url{https://mpi4py.readthedocs.io/}
\item \MPI standard: \url{https://www.mpi-forum.org/}
\item \escript examples directory: \texttt{doc/examples/}
\end{itemize}
