Chapter 8.2 Observations and Perfect-Model Experiments

---

Learning Objectives

By the end of this section, you should be able to

• explain the concept of an observation and why observations contain uncertainty;
• distinguish between the true climate state and an observed value;
• explain the concept of a perfect-model experiment;
• generate synthetic observations from a climate model simulation; and
• investigate how observation frequency and observation errors influence the representation of the true climate state.

---

Concept

Observation

In the previous section, we investigated how tiny differences in the initial condition can grow rapidly in a chaotic system. However, before we can study how to predict the future evolution of the climate state, we first need to understand how the climate state is observed.

An observation is a measurement of a physical quantity, such as temperature, pressure, or wind speed. Observations provide information about the climate state, but they are never perfectly accurate.

For example, suppose we measure the temperature at a weather station and obtain

\[ 20^\circ\mathrm{C}. \]

This value does not necessarily mean that the exact temperature is precisely \(20.000\ldots^\circ\mathrm{C}\). The actual temperature may be closer to

\[ 20.1^\circ\mathrm{C}, \]

or perhaps

\[ 20.13^\circ\mathrm{C}, \]

or

\[ 20.15^\circ\mathrm{C}. \]

The question is: How many digits can we trust?

Every observing system has a finite measurement accuracy. The last reported digit represents the uncertainty of the observation. Additional digits generally do not contain meaningful information because they are smaller than the observational uncertainty.

Therefore, an observation is not the exact climate state. Instead, it is an estimate of the climate state with some uncertainty.

We define the true climate state as the actual physical state of the climate system at a given time and location. In reality, the true climate state is never known exactly because no observing system can measure every variable with infinite accuracy and resolution.

An observation can therefore be expressed as

\[ X_{\mathrm{obs}}(t) = X_{\mathrm{true}}(t) + \varepsilon(t), \]

where

• \(X_{\mathrm{true}}\) is the true climate state,
• \(X_{\mathrm{obs}}\) is the observed value, and
• \(\varepsilon\) is the observation error.

The observation error represents the difference between the measured value and the unknown true climate state.

---

Perfect-Model Experiment

Because the true climate state is unknown, it is difficult to evaluate how accurately observations represent the real system. A useful approach is to create an idealized experiment in which the true climate state is known.

This approach is called a perfect-model experiment.

In a perfect-model experiment, the numerical model is assumed to represent the real physical system exactly. The model simulation is therefore treated as the true climate state. Synthetic observations are then created by sampling the model output and adding artificial observation errors.

For example, if the model produces a temperature of

\[ 20.13^\circ\mathrm{C}, \]

we define this value as the true climate state. A synthetic observation may then be generated as

\[ 20.13^\circ\mathrm{C}+\varepsilon, \]

where \(\varepsilon\) represents the uncertainty of the observing system.

The advantage of a perfect-model experiment is that the true climate state is known. We can therefore investigate how observation errors and observation frequency influence our ability to represent the underlying climate system.

In this section, we use the climate model introduced previously and regard the model simulation as the true climate state. Synthetic observations are generated from this known state, providing a simplified framework for studying observations before introducing data assimilation methods.

---

Model / Equation

Perfect-Model Experiment

Throughout this section, we continue to use the same climate-model framework introduced in Section 7.1. The external forcing remains zero,

AMP_X="0.0"
AMP_Y="0.0"
AMP_Z="0.0"

so that all variability originates from the nonlinear Lorenz dynamics.

The perfect-model experiment extends the climate-model workflow by generating synthetic observations from the model simulation.

Experiment configuration
        │
        ▼
Model initialization
        │
        ▼
Time integration
        │
        ├── Internal tendency (Lorenz model)
        ├── External forcing tendency
        ▼
Update model state
        │
        ├── Restart files
        ├── Model output (True climate state)
        ▼
Generate synthetic observations
        │
        ├── Observation interval
        ├── Random observation error
        ▼
Observation dataset
        │
        ▼
Diagnostics and visualization

The first part of the workflow is identical to that used in previous sections. After the model simulation is completed, the model output is regarded as the true climate state. Synthetic observations are then generated by sampling the true climate state at prescribed observation times and adding random observation errors.

Let the true climate state be

\[ X_{\mathrm{true}}(t). \]

A synthetic observation is defined as

\[ X_{\mathrm{obs}}(t) = X_{\mathrm{true}}(t) + \varepsilon(t), \]

where

• \(X_{\mathrm{true}}\) is the true climate state,
• \(X_{\mathrm{obs}}\) is the observed value,
• \(\varepsilon\) is the observation error.

The observation error is assumed to follow a Gaussian distribution,

\[ \varepsilon \sim \mathcal{N}(0,\sigma^2), \]

where \(\sigma\) denotes the observation-error standard deviation.

In addition to observation errors, observations are available only at discrete times. By changing the observation interval and the observation-error standard deviation, we can investigate how the quantity and quality of observations influence our understanding of the true climate state.

The objective of this section is to generate synthetic observations from the model simulation and examine how observation frequency and observation errors affect the representation of the true climate state.

---

Sample Code

The sample programs in this section reuse the Model_src-v3.0 framework developed in Section 7.1. The climate model first generates the true climate state, and synthetic observations are then created by sampling the model output and adding random observation errors.

Download the source code archive,

• Model_src-v3.0.tar.gz

and extract it using

tar -xzvf Model_src-v3.0.tar.gz

This creates the directory

Model_src-v3.0/
├── figures/
│   ├── fig-timeseries_climate.py
│   └── fig-timeseries_obs.py
├── utils/
│   └── get_observation.py
├── Lmodel_climate-v3.py
└── run_model_climate-v3.sh

The individual source files used in this section are available below.

File	Description
run_model_climate-v3.sh	Defines the experiment settings and executes the climate model.
Lmodel_climate-v3.py	Performs numerical integration of the Lorenz climate model to generate the true climate state.
get_observation.py	Samples the model output at prescribed observation times and generates synthetic observations by adding random observation errors.
fig-timeseries_obs.py	Reads the true climate state and the synthetic observations and visualizes them together.

No modifications are made to the climate model itself in this section. Instead, the model output is regarded as the true climate state, and synthetic observations are generated by specifying

• the observation interval,
• the observation-error standard deviation, and
• the random-number seed.

The generated observations are stored in

observation.nc
├── time
├── Xtrue
├── Ytrue
├── Ztrue
├── Xobs
├── Yobs
└── Zobs

where Xtrue, Ytrue, and Ztrue represent the true climate state sampled at the observation times, while Xobs, Yobs, and Zobs are the corresponding synthetic observations after random observation errors have been added.

The observation.nc file will be used in the next section to introduce data assimilation, where observations are combined with model forecasts to estimate the true climate state.

---

Exercise

In this exercise, we generate synthetic observations from the Lorenz climate model using a perfect-model experiment. The model simulation is regarded as the true climate state, and observations are created by sampling the model output and adding random observation errors.

---

Step 1. Control Simulation

In a perfect-model experiment, the numerical model is regarded as the true climate system. Therefore, we first generate the true climate state using the Lorenz climate model.

Create a directory for the control simulation,

mkdir CTL
cd CTL

Copy the model source code,

cp ../Model_src-v3.0/*.* .

Open run_model_climate-v3.sh and set

REST="False"

TSTR="0"
TEND="70"

AMP_X="0.0"
AMP_Y="0.0"
AMP_Z="0.0"

DEBUG_OUTPUT="False"

Run the model,

./run_model_climate-v3.sh

or

bash run_model_climate-v3.sh

The resulting model output will be regarded as the true climate state throughout this chapter.

Questions

• Why can the model simulation be regarded as the true climate state in a perfect-model experiment?
• What assumption distinguishes a perfect-model experiment from a real observing system?

---

Step 2. Generate Synthetic Observations

The control simulation produced in the previous section is regarded as the true climate state.

Copy the observation-generation program,

cp ../Model_src-v3.0/utils/get_observation.py .

Open get_observation.py.

The observation settings are

OERR = 1.0

SEED = 150

OBS_TSTR = 55.0
OBS_TEND = 70.0
OBS_DT = 1.0

where

OERR is the observation-error standard deviation, SEED specifies the random-number seed, OBS_TSTR and OBS_TEND define the observation period, and OBS_DT is the observation interval.

Run the program,

python get_observation.py

The synthetic observations are stored in

observation.nc
├── time
├── Xtrue
├── Ytrue
├── Ztrue
├── Xobs
├── Yobs
└── Zobs

Inspect the output file,

ncdump -h observation.nc

Questions

• Why does the observation file contain both the true climate state and the observations?
• Which variables would be available in a real observing system?

---

Step 3. Visualize and Evaluate the Observations

Copy the plotting program,

cp ../Model_src-v3.0/figures/fig-timeseries_obs.py .

Run the program,

python fig-timeseries_obs.py

The default figure compares the sampled true state (cross marks connected by line segments) with the synthetic observations (open circles connected by line segments). Both quantities are available only at the observation times.

Although the observations do not exactly match the sampled true state because of random observation errors, they should exhibit similar temporal variations.

Next, evaluate the observation quality by calculating the Root Mean Square Error (RMSE) and the Anomaly Correlation Coefficient (ACC).

Add the following functions near the beginning of fig-timeseries_obs.py.

import numpy as np

def calc_rmse(obs, true):

    rmse = np.sqrt(np.mean((obs - true)**2))

    return rmse


def calc_acc(obs, true):

    acc = np.corrcoef(obs, true)[0,1]

    return acc

After reading the observation file, calculate the verification statistics.

# ============================================================
# Observation verification
# ============================================================
rmse_X = calc_rmse(Xobs, Xtrue)
rmse_Y = calc_rmse(Yobs, Ytrue)
rmse_Z = calc_rmse(Zobs, Ztrue)

acc_X = calc_acc(Xobs, Xtrue)
acc_Y = calc_acc(Yobs, Ytrue)
acc_Z = calc_acc(Zobs, Ztrue)

print("Observation verification")
print("------------------------")

print("RMSE")
print(f"X = {rmse_X:8.4f}")
print(f"Y = {rmse_Y:8.4f}")
print(f"Z = {rmse_Z:8.4f}")

print("")

print("ACC")
print(f"X = {acc_X:8.4f}")
print(f"Y = {acc_Y:8.4f}")
print(f"Z = {acc_Z:8.4f}")

Run the program again,

python fig-timeseries_obs.py

The RMSE measures the average magnitude of the observation errors, while the ACC measures how well the temporal variations of the observations agree with those of the sampled true state.

Finally, compare the observations with the complete model trajectory.

Uncomment the section that reads the model output,

fin = xr.open_mfdataset(
    FDIR + "out.*.nc",
    combine="by_coords"
)

time = fin.time.sel(time=slice(TSTR, TEND))

Xmdl = fin.X.sel(time=slice(TSTR, TEND))
Ymdl = fin.Y.sel(time=slice(TSTR, TEND))
Zmdl = fin.Z.sel(time=slice(TSTR, TEND))

Then uncomment the model trajectory in each panel,

ax[0].plot(time, Xmdl, color="gray", linewidth=1.0,
           label="Model trajectory")

ax[1].plot(time, Ymdl, color="gray", linewidth=1.0)

ax[2].plot(time, Zmdl, color="gray", linewidth=1.0)

Run the program again,

python fig-timeseries_obs.py

The gray curve represents the complete climate trajectory, while the colored lines indicate only the values sampled at the observation times. Notice that the observations provide information only at discrete times and therefore cannot fully resolve the continuous evolution of the climate state between observations.

Questions

• How well do the observations reproduce the sampled true state?
• What do the RMSE and ACC indicate about the observation quality?
• Which features of the model trajectory are captured well by the observations?
• Which features are missed because the observations are available only at discrete times?
• How might increasing the observation frequency improve the representation of the true climate state?

---

Step 4. Change the Observation Frequency

In the previous exercise, observations were generated every

OBS_DT = 1.0

and saved as

outfile_nc = "observation.nc"

This experiment serves as the reference case.

Now investigate how the observation frequency influences our ability to represent the true climate state.

First, reduce the observation frequency by changing

OBS_DT = 2.0

outfile_nc = "observation_low_FQ.nc"

Run the observation-generation program,

python get_observation.py

Next, create a new plotting script,

cp fig-timeseries_obs.py \
   fig-timeseries_obs_low_FQ.py

Open fig-timeseries_obs_low_FQ.py and change

OBSFILE = "observation.nc"

to

OBSFILE = "observation_low_FQ.nc"

Run the program,

python fig-timeseries_obs_low_FQ.py

Next, increase the observation frequency by changing

OBS_DT = 0.1

outfile_nc = "observation_high_FQ.nc"

Run the observation-generation program again,

python get_observation.py

Create another plotting script,

cp fig-timeseries_obs.py \
   fig-timeseries_obs_high_FQ.py

Open fig-timeseries_obs_high_FQ.py and change

OBSFILE = "observation.nc"

to

OBSFILE = "observation_high_FQ.nc"

Run the program,

python fig-timeseries_obs_high_FQ.py

Compare the three experiments.

Experiment	Observation interval	Observation file	Figure
Default	OBS_DT = 1.0	observation.nc	fig-timeseries_obs.py
Low frequency	OBS_DT = 2.0	observation_low_FQ.nc	fig-timeseries_obs_low_FQ.py
High frequency	OBS_DT = 0.1	observation_high_FQ.nc	fig-timeseries_obs_high_FQ.py

Pay particular attention to

• the number of observations,
• the ability of the observations to capture the continuous model trajectory,
• the RMSE, and
• the ACC.

Questions

• How does changing the observation interval affect the number of observations?
• Which observation frequency best captures the variability of the true climate state?
• Why does the low-frequency experiment miss some features of the model trajectory?
• Does increasing the observation frequency substantially change the RMSE or ACC? Why or why not?
• What are the advantages and disadvantages of increasing the observation frequency?

---

Step 5. Change the Observation Error

In the previous experiments, the observation-error standard deviation was

OERR = 1.0

and the default observation file was saved as

outfile_nc = "observation.nc"

Now investigate how observation error influences the quality of the synthetic observations.

First, reduce the observation error by changing the settings in get_observation.py to

OERR = 0.5
OBS_DT = 1.0

outfile_nc = "observation_low_ERR.nc"

Run the observation-generation program,

python get_observation.py

Next, create a new plotting script,

cp fig-timeseries_obs.py \
   fig-timeseries_obs_low_ERR.py

Open fig-timeseries_obs_low_ERR.py and change

OBSFILE = "observation.nc"

to

OBSFILE = "observation_low_ERR.nc"

Run the program,

python fig-timeseries_obs_low_ERR.py

Next, increase the observation error by changing the settings in get_observation.py to

OERR = 3.0
OBS_DT = 1.0

outfile_nc = "observation_high_ERR.nc"

Run the observation-generation program again,

python get_observation.py

Create another plotting script,

cp fig-timeseries_obs.py \
   fig-timeseries_obs_high_ERR.py

Open fig-timeseries_obs_high_ERR.py and change

OBSFILE = "observation_high_ERR.nc"

Run the program,

python fig-timeseries_obs_high_ERR.py

Compare the three experiments.

Experiment	Observation error	Observation file	Figure script
Default	OERR = 1.0	observation.nc	fig-timeseries_obs.py
Low error	OERR = 0.5	observation_low_ERR.nc	fig-timeseries_obs_low_ERR.py
High error	OERR = 3.0	observation_high_ERR.nc	fig-timeseries_obs_high_ERR.py

Pay particular attention to

• the scatter of observations around the sampled true state,
• the RMSE,
• the ACC, and
• the difference between observation frequency and observation accuracy.

Questions

• How does decreasing the observation error affect the observations?
• How does increasing the observation error affect the observations?
• How do RMSE and ACC change as OERR changes?
• Which experiment provides the most accurate observations?
• What is the difference between increasing observation frequency and improving observation accuracy?

---

Key Points

• In a perfect-model experiment, the model simulation is treated as the true climate state.
• Synthetic observations are generated by sampling the true climate state at selected times and adding random observation errors.
• Observations are not the exact true state. They provide incomplete and uncertain information about the climate system.
• Observation frequency controls how often information is available in time.
• Observation error controls how accurately the observations represent the true climate state.
• RMSE measures the average magnitude of the observation error.
• ACC measures how well the observations capture the temporal variability of the sampled true state.
• Increasing observation frequency helps capture more details of the model trajectory, but it does not necessarily reduce random observation error.
• Reducing observation error improves the accuracy of each observation and generally reduces RMSE and improves ACC.
• These concepts provide the basis for later forecast experiments, ensemble prediction, and data assimilation.

---

Homework

In this homework, you will examine how observation error influences the quality of synthetic observations over a longer simulation period.

Task

Extend the control model simulation to model time 100.

In run_model_climate-v3.sh, set

TEND="100"

Run the control simulation again,

./run_model_climate-v3.sh

or

bash run_model_climate-v3.sh

Next, generate three observation datasets using the same observation frequency but different observation-error standard deviations.

In get_observation.py, use

OBS_DT = 0.2

Then generate the following three experiments.

Experiment	Observation error	Output file	Figure script
Low error	OERR = 0.5	observation_low_ERR.nc	fig-timeseries_obs_low_ERR.py
Default error	OERR = 1.0	observation.nc	fig-timeseries_obs.py
High error	OERR = 3.0	observation_high_ERR.nc	fig-timeseries_obs_high_ERR.py

For each experiment, run

python get_observation.py

Then create the corresponding figure by modifying the OBSFILE setting in the plotting script.

For example,

OBSFILE = "observation_low_ERR.nc"

Run the plotting script,

python fig-timeseries_obs_low_ERR.py

Repeat the same procedure for the default-error and high-error experiments.

Questions

Answer the following questions briefly.

1. How does increasing OERR change the scatter of the observations around the sampled true state?
2. How does RMSE change as OERR increases?
3. How does ACC change as OERR increases?
4. Which observation-error experiment best represents the true climate state?
5. Why does increasing observation frequency not fully solve the problem of observation error?

Submission

Submit the following three figures:

• fig-timeseries_obs_low_ERR.png
• fig-timeseries_obs.png
• fig-timeseries_obs_high_ERR.png

Also submit brief answers to the homework questions.

---