Ensemble Kalman Filter

Learning Objectives

After completing this chapter, students will be able to

explain why a single deterministic forecast is insufficient for estimating forecast uncertainty,
describe the basic principles of the Ensemble Kalman Filter (EnKF),
understand how the ensemble forecast estimates the forecast-error covariance,
compute the Kalman gain from ensemble statistics and observation-error covariance,
apply the EnKF to assimilate synthetic observations into the Lorenz climate model,
investigate the influence of ensemble size on EnKF performance,
explain the purpose of Incremental Analysis Update (IAU) and how it reduces analysis shocks, and
evaluate how assumed observation uncertainty influences the EnKF analysis.

Concept

The Kalman Filter provides an optimal estimate of a system state by combining information from a model forecast and observations according to their uncertainties. In the previous chapter, we prescribed a constant analysis weight to combine the forecast and observations. Although this approach demonstrates the basic idea of data assimilation, it assumes that the forecast uncertainty is known and remains constant throughout the simulation.

In practice, forecast uncertainty evolves continuously because of the nonlinear dynamics of the atmosphere and ocean. Small differences in the initial conditions grow with time, causing the forecast uncertainty to increase or decrease depending on the flow state. Consequently, a fixed analysis weight cannot represent the changing reliability of the model forecast.

The Ensemble Kalman Filter (EnKF) estimates forecast uncertainty directly from an ensemble of model simulations. Each ensemble member begins from a slightly different initial condition and evolves independently. The ensemble spread provides an estimate of the forecast-error covariance, which determines how much weight should be given to the model forecast relative to the observations. When the ensemble spread is small, the forecast is considered more reliable and receives greater weight. Conversely, when the ensemble spread is large, the observations receive greater influence during the analysis.

Another practical issue is that the standard EnKF applies the entire analysis increment instantaneously at each observation time. This abrupt adjustment can produce unrealistic high-frequency oscillations, commonly referred to as analysis shocks. To reduce these shocks, many operational forecasting systems apply the Incremental Analysis Update (IAU), which distributes the analysis increment gradually over a short assimilation window. Although the total analysis increment remains unchanged, the smoother adjustment improves the dynamical balance of the model state.

In this chapter, we first implement the EnKF using a three-member ensemble, then investigate how increasing the ensemble size improves the estimate of forecast uncertainty. Next, we introduce the IAU method to produce smoother analyses, and finally examine how the assumed observation uncertainty influences the EnKF performance.

Model/Equations

Suppose that the forecast ensemble consists of \(N\) members,

\[ \mathbf{x}_1^f,\; \mathbf{x}_2^f,\; \ldots,\; \mathbf{x}_N^f . \]

The ensemble mean is

\[ \overline{\mathbf{x}}^f = \frac{1}{N} \sum_{i=1}^{N} \mathbf{x}_i^f. \]

The forecast perturbation matrix is constructed as

\[ \mathbf{X}' = \left[ \mathbf{x}_1^f-\overline{\mathbf{x}}^f, \; \mathbf{x}_2^f-\overline{\mathbf{x}}^f, \; \cdots, \; \mathbf{x}_N^f-\overline{\mathbf{x}}^f \right]. \]

The forecast-error covariance is estimated from the ensemble,

\[ \mathbf{P}^f = \frac{1}{N-1} \mathbf{X}' \mathbf{X}'^{T}. \]

Unlike the prescribed covariance used in the previous chapter, this covariance evolves naturally with the ensemble and changes throughout the simulation.

The Kalman gain is computed as

\[ \mathbf{K} = \mathbf{P}^f \mathbf{H}^{T} \left( \mathbf{H} \mathbf{P}^f \mathbf{H}^{T} + \mathbf{R} \right)^{-1}, \]

where

\(\mathbf{H}\) is the observation operator, and
\(\mathbf{R}\) is the observation-error covariance matrix.

Each ensemble member is updated independently,

\[ \mathbf{x}_i^{a} = \mathbf{x}_i^{f} + \mathbf{K} \left( \mathbf{y}_i - \mathbf{H} \mathbf{x}_i^{f} \right), \]

where \(\mathbf{y}_i\) is a randomly perturbed observation. Updating each member individually allows the ensemble spread to be maintained after the analysis.

The Incremental Analysis Update (IAU) applies the analysis increment,

\[ \delta\mathbf{x} = \mathbf{x}^{a} - \mathbf{x}^{f}, \]

gradually over an assimilation window containing \(N_{\rm IAU}\) model time steps. The increment applied during each model time step is

\[ \Delta\mathbf{x} = \frac{\delta\mathbf{x}} {N_{\rm IAU}}. \]

Rather than introducing an abrupt adjustment at the observation time, the IAU continuously nudges the model toward the analyzed state, producing a smoother and more dynamically balanced evolution.

Finally, the observation-error covariance is

\[ \mathbf{R} = \begin{bmatrix} \sigma_X^2 & 0 & 0\\ 0 & \sigma_Y^2 & 0\\ 0 & 0 & \sigma_Z^2 \end{bmatrix}. \]

Increasing the observation error increases \(\mathbf{R}\), reduces the Kalman gain, and causes the EnKF to rely more heavily on the model forecast than on the observations.

Sample Code

The sample programs in this section reuse the Model_src-v3.0 framework developed in the previous exercises.

The control simulation provides the true climate state in the perfect-model framework. Synthetic observations are generated by adding random observation errors to the control simulation. An ensemble of initial conditions is generated by perturbing the synthetic observation at the initial analysis time. The Ensemble Kalman Filter (EnKF) then uses the evolving ensemble to estimate the forecast-error covariance, calculate the Kalman gain, and update each ensemble member when observations become available.

The Incremental Analysis Update (IAU) method is subsequently combined with the EnKF to distribute each analysis increment gradually over several model time steps, producing smoother transitions between the forecast and analysis states.

Download the source-code archive,

Model_src-v3.0.tar.gz

and extract it using

tar -xzvf Model_src-v3.0.tar.gz

This creates the directory

Model_src-v3.0/
├── figures/
│   ├── fig-timeseries_obs.py
│   └── fig-timeseries_CTL_EnKF-template.py
├── utils/
│   ├── get_observation.py
│   ├── create_initial_obs-EnKF.py
│   ├── Lmodel_EnKF-v3.py
│   └── Lmodel_ENKF-IAU-v3.py
├── Lmodel_climate-v3.py
└── run_model_climate-v3.sh

The baseline experiments use the following configuration:

Control simulation:          t = 0–100
Observation period:          t = 45–100
Observation interval:        OBS_DT = 1.0
Baseline observation error:  OERR = 2.5
Initial analysis time:       t = 45
Evaluation period:           t = 50–100

The EnKF experiments use three, five, and ten ensemble members:

ENKF03:  NENS = 3
ENKF05:  NENS = 5
ENKF10:  NENS = 10

The source files used in this chapter are summarized below.

File	Description
`run_model_climate-v3.sh`	Defines the experiment settings, prepares the model program, and executes the Lorenz climate model. It is modified during the exercises to run the EnKF and EnKF–IAU experiments.
`Lmodel_climate-v3.py`	Performs numerical integration of the Lorenz climate model and generates the control simulation representing the true climate state.
`utils/get_observation.py`	Generates synthetic observations by adding random Gaussian observation errors to the control simulation.
`utils/create_initial_obs-EnKF.py`	Creates an ensemble of initial conditions from a synthetic observation by adding random perturbations to the observed Lorenz variables.
`utils/Lmodel_EnKF-v3.py`	Performs Lorenz ensemble integration and applies the Ensemble Kalman Filter at each observation time. The program estimates the forecast-error covariance, calculates the Kalman gain, and updates each ensemble member.
`utils/Lmodel_ENKF-IAU-v3.py`	Combines the Ensemble Kalman Filter with the Incremental Analysis Update (IAU) method. Each EnKF analysis increment is distributed gradually over several model time steps.
`figures/fig-timeseries_obs.py`	Plots the true Lorenz state together with the synthetic observations.
`figures/fig-timeseries_CTL_EnKF-template.py`	Provides a template for comparing the control simulation with the individual EnKF members and ensemble mean. It also calculates the ensemble-mean RMSE.

During the exercises, the original source programs are progressively configured to conduct several data assimilation experiments.

First, a three-member EnKF experiment is developed using

Lmodel_EnKF-v3.py

The ensemble size is then increased to five and ten members to examine how ensemble size affects the estimation of the forecast-error covariance and the performance of the analysis.

Next, the standard EnKF model is replaced with

Lmodel_ENKF-IAU-v3.py

to distribute the analysis increment gradually over an IAU window and produce smoother transitions around the observation times.

Finally, a second synthetic observation dataset is generated using

OERR = 5.0

Both observation-error sensitivity experiments assimilate the same file,

obs_err50_dt10.nc

and begin from the same ten-member initial ensemble. The experiments differ only in the observation-error standard deviation assumed by the EnKF:

EXP1:  OBSERR_X = OBSERR_Y = OBSERR_Z = 2.5
EXP2:  OBSERR_X = OBSERR_Y = OBSERR_Z = 5.0

Additional experiment directories, shell scripts, restart files, figures, and analysis programs are developed throughout the exercises and are therefore not included in the original Model_src-v3.0 source-code archive.

Exercise

Step 1. Generate the Control Simulation and Baseline Synthetic Observations

In this exercise, we first generate the control (CTL) simulation, which represents the true evolution of the Lorenz climate system. The control simulation serves as the reference state for evaluating the Ensemble Kalman Filter (EnKF) experiments developed later in this chapter.

Synthetic observations are then created by adding random observation errors to the control simulation. Compared with the previous chapter, observations are available less frequently and contain larger observational errors, providing a more realistic test for the EnKF.

The baseline observation experiment uses

OERR   = 2.5
OBS_DT = 1.0

The observation file is named

obs_err25_dt10.nc

where

err25    OERR = 2.5
dt10     OBS_DT = 1.0

This naming convention allows additional observation files with different observation errors and observation intervals to be stored in the same directory.

Create the Working Directories

Create the chapter and control-simulation directories,

mkdir Chapter10_2
cd Chapter10_2

mkdir CTL
cd CTL

Copy the model source files,

cp ../../Model_src-v3.0/*.* .

The directory should initially contain

CTL/
├── Lmodel_climate-v3.py
└── run_model_climate-v3.sh

Run the Control Simulation

Open

run_model_climate-v3.sh

and configure the experiment as

REST=False
TSTR=0
TEND=101
TSTEP=1

Run the control simulation,

bash run_model_climate-v3.sh

The model automatically creates the required output and restart directories and integrates the control simulation over

t = 0–100

Generate the Baseline Synthetic Observations

Copy the observation-generation script,

cp ../../Model_src-v3.0/utils/get_observation.py .

Open

get_observation.py

and configure

outfile_nc = "obs_err25_dt10.nc"

TSTR = 0
TEND = 100

OBS_TSTR = 45.0
OBS_TEND = 100.0

OERR = 2.5
OBS_DT = 1.0

Run

python get_observation.py

This creates

obs_err25_dt10.nc

The EnKF experiments will begin at

t = 45

The period

t = 45–50

is treated as analysis spin-up, while the evaluation period is

t = 50–100

Visualize and Verify the CTL Simulation

Return to the chapter directory,

cd ../

mkdir Figure
cd Figure

Copy the figure script,

cp ../../Model_src-v3.0/figures/fig-timeseries_obs.py .

Modify

FDIR = "../CTL/out/"
OBSFILE = "../CTL/obs_err25_dt10.nc"

TSTR = 40
TEND = 100

Run

python fig-timeseries_obs.py

Verify that

the CTL trajectory is continuous,
observations begin at t = 45,
observations fluctuate around the true state,
observations are separated by OBS_DT = 1.0, and
the larger observation error produces greater scatter around the true trajectory than in the previous chapter.

Verify the Results

After completing this step, the directory should resemble

Chapter10_2/
├── CTL/
│   ├── out/
│   ├── rest/
│   ├── obs_err25_dt10.nc
│   ├── get_observation.py
│   ├── Lmodel_climate-v3.py
│   └── run_model_climate-v3.sh
│
└── Figure/
    └── fig-timeseries_obs.py

The CTL simulation represents the true evolution of the Lorenz climate system. The file obs_err25_dt50.nc provides the baseline synthetic observations used in the EnKF experiments. In the following exercises, these observations will be assimilated by an ensemble of model forecasts to estimate the evolving forecast-error covariance and compute the Kalman gain.

Step 2. Apply the Ensemble Kalman Filter

In this exercise, we apply the Ensemble Kalman Filter (EnKF) to the Lorenz climate system. Unlike the previous chapter, where the analysis weight was prescribed, the EnKF estimates the analysis weight objectively from the ensemble forecast-error covariance and the prescribed observation-error covariance.

A three-member ensemble is used in the first experiment:

NENS = 3

The synthetic observations generated in Step 1 use

OERR   = 2.5
OBS_DT = 1.0

The corresponding observation file is

obs_err25_dt10.nc

The ensemble evolves freely from

t = 0–45

The first EnKF analysis is performed at

t = 45

Subsequent analyses are performed every five model time units:

t = 45, 50, 55, ..., 100

The period

t = 45–50

is treated as analysis spin-up, while the period

t = 50–100

is used to evaluate the EnKF analysis.

Create the EnKF Experiment Directory

Return to the chapter directory,

cd ../

Create the EnKF experiment directory,

mkdir ENKF03
cd ENKF03

The directory name

ENKF03

indicates that the experiment uses five ensemble members.

Copy the EnKF source program,

cp ../../Model_src-v3.0/utils/Lmodel_EnKF-v3.py .

Copy the control shell script,

cp ../CTL/run_model_climate-v3.sh run_model_EnKF-v3.sh

Copy the initial-condition generator:

cp ../../Model_src-v3.0/utils/create_initial_obs-EnKF.py .

The directory should initially contain

ENKF03/
├── Lmodel_EnKF-v3.py
├── run_model_EnKF-v3.sh
└── create_initial_obs-EnKF.py

Generate the Initial Ensemble

Unlike the previous chapter, the Ensemble Kalman Filter requires an ensemble of initial conditions rather than a single model restart. The initial ensemble is generated from the synthetic observation at the initial analysis time by adding random perturbations that represent the prescribed observation uncertainty.

Run the program using

python create_initial_obs-EnKF.py --t_initial=45  --ens=3  --seed=1  --amp=2.5  --obsfile=../CTL/obs_err25_dt10.nc

where

--t_initial

specifies the initial analysis time used to generate the ensemble.

--ens

specifies the number of ensemble members.

--seed

specifies the random seed used to generate reproducible ensemble perturbations.

--amp

specifies the standard deviation of the random perturbations added to each observed Lorenz variable when generating the initial ensemble.

--obsfile

specifies the observation file from which the initial condition is read.

If no command-line options are provided, the script uses the default values defined within the program.

The script reads the synthetic observation at

t = 45

adds random perturbations to the observed Lorenz variables, and creates a three-member ensemble.

The perturbations are sampled independently from Gaussian distributions:

X ~ N(Xobs, OBSERR_X²)
Y ~ N(Yobs, OBSERR_Y²)
Z ~ N(Zobs, OBSERR_Z²)

where the observation-error standard deviations are

OBSERR_X = 2.5
OBSERR_Y = 2.5
OBSERR_Z = 2.5

The resulting ensemble is written to the restart directory as

rest/restart.0045.nc

The restart file contains the internal and external model components for each ensemble member:

Xint(ens, time)
Yint(ens, time)
Zint(ens, time)

Xext(ens, time)
Yext(ens, time)
Zext(ens, time)

The external components are initialized to zero because no external forcing is applied in this experiment.

Verify that the restart file has been created:

ls rest/

The output should include

restart.0045.nc

You may also inspect the file structure using

ncdump rest/restart.0045.nc

The ensemble dimension should be

ens = 3

This ensemble provides the initial spread required for the Ensemble Kalman Filter to estimate the forecast-error covariance during the first analysis cycle.

Configure the EnKF Experiment

Open

run_model_EnKF-v3.sh

and configure the experiment as

REST="True"
TSTR="45"
TEND="101"
TSTEP="1"

Use the following output and restart directories,

Keep the external forcing amplitudes at zero,

AMP_X="0.0"
AMP_Y="0.0"
AMP_Z="0.0"

Add the following EnKF settings:

# ============================================================
# Ensemble Kalman Filter setting
# ============================================================

NENS="3"

OBSFILE="../CTL/obs_err25_dt10.nc"

OBSERR_X="2.5"
OBSERR_Y="2.5"
OBSERR_Z="2.5"

where

NENS

specifies the number of ensemble members, and

OBSERR_X
OBSERR_Y
OBSERR_Z

specify the observation-error standard deviations for the three Lorenz variables.

Change the model source from

SMODEL="Lmodel_climate-v3.py"

to

SMODEL="Lmodel_EnKF-v3.py"

Finally, add the following replacement commands to the sed section of the shell script:

-e "s|%OBSFILE|$OBSFILE|g" \
-e "s|%NENS|$NENS|g" \
-e "s|%OBSERR_X|$OBSERR_X|g" \
-e "s|%OBSERR_Y|$OBSERR_Y|g" \
-e "s|%OBSERR_Z|$OBSERR_Z|g" \

Run the EnKF Experiment

Run

bash run_model_EnKF-v3.sh

The model automatically creates the required output and restart directories and integrates the five ensemble members over

t = 0–100

Before the first observation, the ensemble evolves freely. Beginning at

t = 45

the EnKF estimates the forecast-error covariance from the ensemble and computes the Kalman gain automatically. The analysis is then performed every

OBS_DT = 1.0

model time units until

t = 100.

Verify the EnKF Results

To visualize the EnKF experiment, use the figure template provided with the model source code.

Return to the chapter directory:

cd ../

Move to the Figure directory:

cd Figure

Copy the template and rename it for the three-member EnKF experiment:

cp ../../Model_src-v3.0/figures/fig-timeseries_CTL_EnKF-template.py  fig-timeseries_CTL_EnKF03.py

Open

fig-timeseries_CTL_EnKF03.py

and confirm that the model output directories are defined as

# Model output directories
FDIR_CTL = "../CTL/out/"
FDIR_ENKF = "../ENKF03/out/"

The control simulation is plotted over

TSTR_CTL = 40
TEND_CTL = 100

while the EnKF simulation is plotted over

TSTR_ENKF = 45
TEND_ENKF = 100

The period from

t = 45–50

is treated as the EnKF spin-up period. Therefore, the ensemble-mean RMSE is evaluated over

TSTR_RMSE = 50
TEND_RMSE = 100

Run the figure script:

python fig-timeseries_CTL_EnKF03.py

The script reads all control and EnKF output files using

xr.open_mfdataset()

and automatically identifies the ensemble dimension. The terminal output should contain information similar to

Ensemble dimension: ens
Xens dimensions   : ('ens', 'time')
Xens shape        : (...)
Members           : [0 1 2]

The script then calculates

the ensemble mean,
the ensemble spread (standard deviation), and
the RMSE of the ensemble mean relative to the control simulation.

The RMSE is evaluated separately for the three Lorenz variables over

t = 50–100

The terminal output should contain results similar to

--------------------------------
EnKF ensemble-mean RMSE
--------------------------------
Evaluation period: 50–100
RMSE X = ...
RMSE Y = ...
RMSE Z = ...
--------------------------------

The script produces a three-panel figure showing

the control simulation (black),
the individual EnKF members (blue), and
the EnKF ensemble mean (orange).

The dashed vertical line indicates the beginning of the EnKF experiment at

t = 45

and the dotted vertical line marks the beginning of the RMSE evaluation period at

t = 50.

The figure is automatically saved as

fig-timeseries_CTL_EnKF03.png

This figure provides the first quantitative evaluation of the EnKF experiment by comparing the ensemble evolution and the ensemble-mean RMSE against the control simulation.

Step 3. Investigate the Effect of Ensemble Size

In this exercise, we investigate how the number of ensemble members influences the performance of the Ensemble Kalman Filter (EnKF). The three-member experiment developed in Step 2 is used as the reference experiment.

Two additional experiments are created:

ENKF05

using five ensemble members, and

ENKF10

using ten ensemble members.

Increasing the ensemble size generally improves the estimate of the forecast-error covariance and leads to a more stable Kalman gain.

Create the Five-Member EnKF Experiment

Return to the chapter directory:

cd ../

Create the experiment directory:

mkdir ENKF05

Copy the EnKF source files from the three-member experiment:

cp ENKF03/*.* ENKF05

Move to the new experiment directory:

cd ENKF05

Generate a five-member initial ensemble:

python create_initial_obs-EnKF.py --t_initial=45 --ens=5 --seed=1 --amp=2.5 --obsfile=../CTL/obs_err25_dt10.nc

Open

run_model_EnKF-v3.sh

and change

NENS="3"

to

NENS="5"

Run the EnKF experiment:

bash run_model_EnKF-v3.sh

Create the Ten-Member EnKF Experiment

Return to the chapter directory:

cd ../

Create the experiment directory:

mkdir ENKF10

Copy the EnKF source files:

cp ENKF03/*.* ENKF10

Move to the experiment directory:

cd ENKF10

Generate a ten-member initial ensemble:

python create_initial_obs-EnKF.py --t_initial=45 --ens=10 --seed=1 --amp=2.5 --obsfile=../CTL/obs_err25_dt10.nc

Open

run_model_EnKF-v3.sh

and change

NENS="3"

to

NENS="10"

Run the EnKF experiment:

bash run_model_EnKF-v3.sh

Compare the EnKF Experiments

Move to the Figure directory:

cd ../Figure

Create the figure scripts:

cp fig-timeseries_CTL_EnKF03.py fig-timeseries_CTL_EnKF05.py
cp fig-timeseries_CTL_EnKF03.py fig-timeseries_CTL_EnKF10.py

For the five-member experiment, edit

fig-timeseries_CTL_EnKF05.py

and change

FDIR_ENKF = "../ENKF03/out/"

to

FDIR_ENKF = "../ENKF05/out/"

Run

python fig-timeseries_CTL_EnKF05.py

For the ten-member experiment, edit

fig-timeseries_CTL_EnKF10.py

and change

FDIR_ENKF = "../ENKF03/out/"

to

FDIR_ENKF = "../ENKF10/out/"

Run

python fig-timeseries_CTL_EnKF10.py

The scripts calculate the ensemble-mean RMSE and produce the figures

fig-timeseries_CTL_EnKF05.png

and

fig-timeseries_CTL_EnKF10.png

Compare the results from the three experiments:

ENKF03
ENKF05
ENKF10

Pay particular attention to

the ensemble spread,
the ensemble-mean trajectory,
the RMSE of the ensemble mean, and
the stability of the EnKF analyses.

As the ensemble size increases, the forecast-error covariance is generally estimated more accurately, resulting in a more stable Kalman gain and improved analysis. However, because the Lorenz system is nonlinear and chaotic, the improvement is not necessarily monotonic at every analysis time.

Step 4. Apply the EnKF with Incremental Analysis Update

In this step, we combine the Ensemble Kalman Filter with the Incremental Analysis Update (IAU).

In the standard EnKF experiment, the complete analysis increment is applied instantaneously at each observation time. In the EnKF–IAU experiment, the same analysis increment is distributed gradually over several model time steps.

This gradual application produces a smoother transition from the forecast state to the analysis state.

The ten-member EnKF experiment developed in Step 3 is used as the base configuration.

Create the EnKF–IAU Experiment Directory

Return to the chapter directory:

cd ../

Create the EnKF–IAU experiment directory:

mkdir ENKF10_IAU

Copy the EnKF experiment files from ENKF10:

cp ENKF10/run_model_EnKF-v3.sh ENKF10_IAU/

Copy the EnKF–IAU model source code:

cp ../../Model_src-v3.0/utils/Lmodel_EnKF-IAU-v3.py ENKF10_IAU/.

Move to the new experiment directory, create restart directory, and copy the restart file:

cd ENKF10_IAU
mkdir rest
cp ../ENKF10/rest/restart.0045.nc ./rest/

The directory should initially contain files similar to

ENKF10_IAU/
├── Lmodel_EnKF-IAU-v3.py
├── run_model_EnKF-v3.sh
└── rest/
    └── restart.0045.nc

Configure the EnKF–IAU Experiment

Open

run_model_EnKF-v3.sh

The shell-script settings remain the same as those used for the ENKF10 experiment:

REST="True"
TSTR="45"
TEND="101"
TSTEP="1"

Confirm that the ensemble and observation settings are

NENS="10"

OBSFILE="../CTL/obs_err25_dt10.nc"

OBSERR_X="2.5"
OBSERR_Y="2.5"
OBSERR_Z="2.5"

Keep the external forcing amplitudes at zero:

AMP_X="0.0"
AMP_Y="0.0"
AMP_Z="0.0"

Only the model source needs to be changed.

Change

SMODEL="Lmodel_EnKF-v3.py"

to

SMODEL="Lmodel_EnKF-IAU-v3.py"

No additional shell-script settings are required because the IAU parameters are defined inside

Lmodel_ENKF-IAU-v3.py

For example, the number of IAU steps is specified as

IAU_NSTEP = 15

and the IAU window length is calculated as

IAU_WINDOW = IAU_NSTEP * DT

With

DT = 0.01

the total IAU window length is

0.15 model time units

Run the EnKF–IAU Experiment

Run:

bash run_model_EnKF-v3.sh

The program reads the ten-member initial ensemble from

rest/restart.0045.nc

and integrates the ensemble from

t = 45–100

At each observation time, the program

calculates the EnKF analysis,
determines the member-specific analysis increment,
distributes the increment over the IAU window, and
gradually updates the model state.

The model prints information similar to

EnKF increment calculated for t=50.00
IAU begins at t=...
IAU window length = 0.15
IAU completed for observation at t=50.00

The exact values depend on the observation time and IAU configuration.

Verify the EnKF–IAU Results

After the experiment finishes, move to the Figure directory:

cd ../Figure

Copy the ten-member EnKF figure script:

cp fig-timeseries_CTL_EnKF10.py fig-timeseries_CTL_EnKF10_IAU.py

Open

fig-timeseries_CTL_EnKF10_IAU.py

and change the EnKF output directory from

FDIR_ENKF = "../ENKF10/out/"

to

FDIR_ENKF = "../ENKF10_IAU/out/"

Run:

python fig-timeseries_CTL_EnKF10_IAU.py

The figure is saved as

fig-timeseries_CTL_EnKF10_IAU.png

The terminal also reports the ensemble-mean RMSE for the EnKF–IAU experiment.

Compare the results from

ENKF10

and

ENKF10_IAU

Pay particular attention to

the smoothness of the analysis transitions,
the ensemble spread,
the ensemble-mean trajectory,
the RMSE of the ensemble mean, and
the response of the Lorenz variables near each analysis time.

The standard EnKF applies the full increment instantaneously, while the EnKF–IAU experiment distributes the same increment gradually. Therefore, the EnKF–IAU trajectory should generally show smoother transitions around the analysis times.

After completing this step, the chapter directory should resemble

Chapter10_2/
├── CTL/
├── ENKF03/
├── ENKF05/
├── ENKF10/
├── ENKF10_IAU/
│   ├── out/
│   ├── rest/
│   │   └── restart.0045.nc
│   ├── create_initial_obs-EnKF.py
│   ├── Lmodel_ENKF-IAU-v3.py
│   └── run_model_EnKF-v3.sh
│
└── Figure/
    ├── fig-timeseries_CTL_EnKF10.py
    ├── fig-timeseries_CTL_EnKF10.png
    ├── fig-timeseries_CTL_EnKF10_IAU.py
    └── fig-timeseries_CTL_EnKF10_IAU.png

This experiment demonstrates how the IAU method can produce a smoother transition between the forecast and analysis states by gradually applying the EnKF analysis increment.

Step 5. Examine the Impact of the Assumed Observation Error

In this final exercise, we investigate how the assumed observation error influences the performance of the Ensemble Kalman Filter with Incremental Analysis Update (EnKF–IAU).

The synthetic observations are generated using an observation-error standard deviation of

σobs = 5.0

Both experiments assimilate exactly the same observations and start from exactly the same ensemble initial conditions. The only difference is the observation error assumed by the EnKF.

Experiment	Observation file	Assumed observation error
EXP1	`obs_err50_dt10.nc`	2.5
EXP2	`obs_err50_dt10.nc`	5.0

This experimental design isolates the influence of the observation-error covariance matrix used by the EnKF.

Create a New Observation Dataset

First, generate a synthetic observation dataset with an observation-error standard deviation of

OERR = 5.0

Move to the control simulation directory:

cd ../CTL

Open the observation-generation script (get_observation.py) and modify

OERR = 2.5

to

OERR = 5.0

Set the output filename to

OBSFILE = "obs_err50_dt10.nc"

Keep the observation interval unchanged:

OBS_DT = 1.0

Run the script:

python get_observation.py

Confirm that the new observation file has been created:

ls obs_err50_dt10.nc

Create EXP1

Return to the chapter directory:

cd ../

Copy the complete EnKF–IAU experiment:

cp -r ENKF10_IAU EXP1

Move to the experiment directory:

cd EXP1

Remove the previous model output:

rm -rf out

Keep only the initial restart file:

find rest -type f ! -name "restart.0045.nc" -delete

Open

run_model_EnKF-v3.sh

Use the new observation file:

OBSFILE="../CTL/obs_err50_dt10.nc"

Keep the EnKF observation-error settings as

OBSERR_X="2.5"
OBSERR_Y="2.5"
OBSERR_Z="2.5"

Although the observations contain an error standard deviation of 5.0, the EnKF assumes that the observations are more accurate (2.5).

Run the experiment:

bash run_model_EnKF-v3.sh

Create EXP2

Return to the chapter directory:

cd ../

Copy the completed experiment:

cp -r EXP1 EXP2

Move to the new directory:

cd EXP2

Remove the copied model output:

rm -rf out

The restart file

rest/restart.0045.nc

is retained, ensuring that both experiments begin from exactly the same ensemble initial conditions.

Open

run_model_EnKF-v3.sh

Keep the observation file unchanged:

OBSFILE="../CTL/obs_err50_dt10.nc"

Modify only the assumed observation error:

OBSERR_X="5.0"
OBSERR_Y="5.0"
OBSERR_Z="5.0"

All other model settings remain identical to EXP1.

Run the experiment:

bash run_model_EnKF-v3.sh

Verify the Results

Move to the Figure directory:

cd ../Figure

Create the plotting script for EXP1:

cp fig-timeseries_CTL_EnKF10_IAU.py fig-timeseries_CTL_EXP1.py

Change

FDIR_ENKF = "../ENKF10_IAU/out/"

to

FDIR_ENKF = "../EXP1/out/"

Run

python fig-timeseries_CTL_EXP1.py

Next, create the plotting script for EXP2:

cp fig-timeseries_CTL_EXP1.py fig-timeseries_CTL_EXP2.py

Change

FDIR_ENKF = "../EXP1/out/"

to

FDIR_ENKF = "../EXP2/out/"

Run

python fig-timeseries_CTL_EXP2.py

The scripts calculate the ensemble-mean RMSE and produce

fig-timeseries_CTL_EXP1.png

and

fig-timeseries_CTL_EXP2.png

Compare the Two Experiments

Compare the RMSE values and the time series from

Experiment	Observation file	Assumed observation error
EXP1	`obs_err50_dt10.nc`	2.5
EXP2	`obs_err50_dt10.nc`	5.0

Record the RMSE values.

Experiment	RMSE X	RMSE Y	RMSE Z
EXP1
EXP2

Pay particular attention to

the magnitude of the EnKF analysis increments,
the agreement between the ensemble mean and the control simulation,
the ensemble spread, and
the RMSE of the ensemble mean.

Because both experiments assimilate exactly the same observations and begin from exactly the same initial ensemble, any differences are caused solely by the assumed observation-error covariance used by the EnKF.

In EXP1, the EnKF assumes that the observations are more accurate than they actually are. Consequently, the Kalman gain becomes relatively large, and the filter gives more weight to the observations.

In EXP2, the assumed observation error matches the actual observation uncertainty. The Kalman gain becomes smaller, producing weaker analysis increments and placing more weight on the model forecast.

This experiment demonstrates one of the fundamental principles of data assimilation: the performance of the EnKF depends not only on the observations themselves but also on how accurately their uncertainty is represented.

Key Points

The Ensemble Kalman Filter (EnKF) estimates the forecast-error covariance from an ensemble of model simulations rather than prescribing a fixed analysis weight.
Small perturbations in the initial conditions generate an ensemble whose spread represents the evolving forecast uncertainty.
The Kalman gain is computed from the forecast-error covariance and the observation-error covariance, allowing the analysis to adapt automatically to changing uncertainty.
Increasing the number of ensemble members generally improves the estimate of the forecast-error covariance and produces a more stable analysis.
The quality of the EnKF depends on both the ensemble size and the representation of forecast uncertainty.
The Incremental Analysis Update (IAU) distributes the EnKF analysis increment gradually over several model time steps instead of applying it instantaneously.
The IAU reduces analysis shocks and produces smoother transitions between the forecast and analysis states.
The assumed observation-error covariance directly influences the Kalman gain. Larger observation errors decrease the weight assigned to observations and increase the reliance on the model forecast.
Effective data assimilation requires a realistic representation of both forecast uncertainty and observation uncertainty.
The EnKF provides the foundation for many operational atmospheric and oceanic data assimilation systems used in numerical weather prediction and Earth system forecasting.

Homework

The baseline experiment in this chapter uses the following configuration:

Ensemble members:        NENS = 10
Observation interval:    OBS_DT = 1.0
Observation error:       OERR = 2.5
Assimilation method:     EnKF with IAU
Simulation period:       t = 45–100

Choose one of the following experiments (or propose your own modification with instructor approval). Compare your results with the baseline ENKF10_IAU experiment.

Option 1. Effect of Observation Frequency

Investigate how the observation interval influences the EnKF analysis.

For example,

OBS_DT = 2.0

or

OBS_DT = 5.0

Generate a new observation file, rerun the EnKF experiment, and compare the ensemble-mean RMSE with the baseline experiment.

Option 2. Effect of Ensemble Size

Investigate how increasing the ensemble size influences the EnKF performance.

For example,

NENS = 20

or

NENS = 50

Generate a new ensemble using create_initial_obs-EnKF.py, rerun the EnKF experiment, and compare the ensemble spread and RMSE with the baseline experiment.

Option 3. Effect of Observation Error

Investigate how the assumed observation uncertainty influences the analysis.

For example,

OERR = 1.0

or

OERR = 10.0

Generate a new observation dataset and evaluate how different observation uncertainties affect the Kalman gain and the EnKF analysis.

Option 4. Longer Simulation

Extend the experiment to investigate the long-term behavior of the EnKF.

For example,

Control simulation:  t = 0–200
EnKF simulation:     t = 45–200

Examine whether the EnKF maintains a stable analysis over a longer integration period.

Option 5. Your Own Experiment

Design your own EnKF experiment by modifying one model or data assimilation parameter not listed above.

Possible examples include

changing the initial analysis time,
modifying the perturbation amplitude used to generate the initial ensemble,
changing the IAU window length,
changing the observation period, or
combining two of the experiments listed above.

Submit

Submit the following:

One figure comparing your experiment with the baseline ENKF10_IAU simulation.
A brief discussion (approximately 200–300 words) describing - the purpose of the experiment, - the modifications you made, - the differences you observed, and - why you think those differences occurred based on EnKF theory.