Introduction to CPT for SubSeasonal Forecasting

(1)

Introduction to CPT for SubSeasonal Forecasting

NOAA’s CPC International Desks

NOAA-USAID 11ITWCVP - Ankara, Turkey, 15 – 26 April 2019

(2)

Climate Prediction Tools (CPT) Software

https://iri.columbia.edu/our-expertise/climate/tools/cpt/

What's it all about?

 Software providing packages (Both Windows and Linux) for :

 S

easonal Climate Forecasting

Model Validation

Forecast Verification

Model Skill assessment

 Uses ASCII input files

 Save Output in various formats

 Data : ASCII, binary

 Graphics : jpeg, png, …

(3)

Usual actions with CPT

Seasonal Climate Forecasting

Statistical forecast : - Predictor is an observed field of a variable during an earlier time - Target is a forecast of rainfall in a later season

Downscalling and Forecast corrections : - Predictor is a dynamical model prediction - Target is a “corrected” forecast.

Forecast Verification

Model validation and Skill assessment

(4)

Schematic showing possible forecasting actions with CPT

X is observed earlier predictors, such as the field of governing SST

1. Observational predictor design

1. is a purely statistical forecast system

2. Is a dynamical forecast corrected by a statistical adjustment

X is dynamical model prediction pattern around a region of interest

2. Model MOS design

Y is an earlier observed pattern over the region of interest

Forecast pattern

over the region

of interest

(5)

Climate Predictability Tool

Predictors

Predictands

Predictions

(6)

Building “empirical” models

Observations (X and Y)

(7)

Model Output Statistics (MOS)

Model X and

Observation Y

(8)

Raw Model Output (Predictors) Observation (Predictand)

Multi-model Corrected Probabilistic Prediction

Statistical post- processing

(Model Output Statistics)

• No spatial-pattern based

• Spatial-pattern based

Multiple models

(9)

CPT and MOS : what is it about?

 Downscaling – the translation of a forecast to a spatial and/or temporal resolution that is finer than that of the original forecast.

 Corrected forecast – detection and correction of models such systematic errors; training on a history of model hindcasts and the corresponding observations

Model Hindcast (Historical forecast) hindcast around

the region of interest Model Hindcast (Historical

forecast) hindcast around the region of interest Historical observations over

the region of interest Historical observations over

the region of interest

Current model forecast around the region of interest

Corrected forecast over the region of

interest Corrected forecast over the region of

interest

CPT CPT Skill maps

(pearson correlation)

Skill maps

(pearson

correlation)

(10)

Quick view on the S2S projects and

data

(11)

S2S Prediction Project

(12)

IRI/LDEO Climate Data Library

http://iridl.ldeo.columbia.edu

(13)

IRI/LDEO Climate Data Library

(14)

IRI/LDEO Climate Data Library

(15)

S2S Data Available

(16)

S2S Data Available

(17)

S2S Data Available

(18)

S2S Data Available

(19)

S2S Prediction Project

(20)

https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.ECMF/.reforecas t/.perturbed/.sfc_precip/.tp/

S2S Data Available

(21)

?

1 10 20 30 60 80 90 Forecast lead time (days) Weather

prediction 1~14 days

Seasonal prediction 1~7 months

Subseasonal prediction

2 weeks~2 months

Subseasonal predictions contribute to fill the gap between weather and seasonal time scales

• Improve forecast skill and understanding on the subseasonal to seasonal timescale with special emphasis on high-impact weather events

• Promote the initiative’s uptake by operational centers and exploitation by the applications community

• Capitalize on the expertise of the weather and climate research communities to address issues of importance to the Global Framework for Climate Services

Objectives

The Subseasonal to Seasonal Projects

(22)

The international S2S Prediction Project

http://s2sprediction.net/

Updated in real time, but data has a 3 week delay

• Not very helpful for operational activities

• Valid mainly for research applications More details will be provided by Caio Coelho during his talks on Thursday 25 April 2019.

• The S2S Project

• Hands-On S2S

Assessing the S2S Predcition Project

• ECMWF data portal :

http://apps.ecmwf.int/datasets/data/s2s/

• CMA S2S Archiving Data Center: http://s2s.cma.cn/index

• IRI Data library (IRIDL):

http://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/

• NCEP Realtime ftp server : ftp.cpc.ncep.noaa.gov/S2S/S2SRT

(23)

l0Dh , T* ³ * 0 J -Ely l ‘ ²0, ²0l8, E8E6d○r I· *· MEñ○G: ,1,’C D6D6 .ibr6ry f○r 2²2

The Subseasonal Experimental (SubX) Project

https://cpo.noaa.gov/Meet-the-Divisions/Earth-System-Science-and-Modeling/MAPP/Research-to-Operations-and-Ap plications/Subseasonal-Experiment

 Collecting and serving data both internally at CPC for use by operational forecasters and for the external community via the IRI data library.

 Providing a baseline verification particularly for the weeks 3-4 temperature and precipitation probability forecasts

 Evaluating the skill of individual model systems

 Investigating multi-model combinations including selecting suitable models, optimizing the design of the system, and evaluation of the prediction products

 Enhancing communications between operational forecasts and the model forecast producers

 Participation in the NOAA/MAPP S2S Task Force

Objectives Prediction systems contributing to SubX

Data are available here : http://iridl.ldeo.columbia.edu/SOURCES/.Models/.SubX/

The Sub-X Project will be presented by Michelle L’heureux (NOAA/CPC) on Thursday 25 April 2019.

(24)

The tools presented are based on rainfall data from

Model Time Range Resolution

Forecasts Re-forecasts (Hindcasts)

Ens. size Frequency Type Rfc period Rfc freq Ens size

CFSv2 Days 0.5- 43.5 1° x 1° 4 Daily, every 6 hrs Fixed 1999 - 2017 Daily, every 6hrs 1

 NCEP Climate Forecast System model version (CFSv2) by SubX via IRI Data Library :

 For validation and verification, observations data also come from IRIDL.

The tools are designed to ingest precipitation data from :

 CPC Unified Gauge-Based Analysis

 African Rainfall Climatology Version (ARC2)

The future release of the tools will include the temperature.

(25)

Practicing the forecast correction -

A very quick look under the hood

(26)

Background :

On the common usage of forecast output

Exceedance

QPF Probabilistic

NCEP CFS 10days forecasts are initiated on Feb 11, 2019 at 00 UTC. Valid period : 12 – 21 Feb 2019

Skill map of the above forecast

(27)

• Dynamical models often have systematic errors:

- bias in the mean - bias in the amplitude - bias in the shape of the anomaly pattern

• Detecting and Correcting such systematic errors is a possible way out:

Transform the raw output in forecasts that are better fit to users needs.

Background :

When dealing with models, be aware that:

(28)

Schematic of a probabilistic weather and climate forecast using initial condition

uncertainties.

Julia Slingo, and Tim Palmer Phil. Trans. R. Soc. A 2011;369:4751-4767

(29)

Schematic of ensemble prediction system on the forecast timescales

Julia Slingo, and Tim Palmer Phil. Trans. R. Soc. A 2011;369:4751-4767

(a) the impact of model biases

(b) climate variability and climate change.

(30)

Background :

Schematic description of Models Calibration

Model Hindcast (Historical

forecast) Model Hindcast (Historical

forecast) Historical observations

Historical observations

Calibrated forecast Calibrated

forecast

Calibration approaches :

 Nudging

 Change Factor

 Quantile Mapping

 Linear Regression

 …

Obs. = TF (model)

TF( )

Current forecast

Identify the Transfer Function between

obs and model

Apply the Transfert Function to model

forecast

Notes: Calibration rely on ,

 the quality of avalible observation

 Bias correction methods you used

 Model himself

(31)

CPT and MOS :

A way out of the troubled waters of correction – How?

Model Hindcast (Historical forecast) hindcast around

the region of interest Model Hindcast (Historical

forecast) hindcast around the region of interest Historical observations over

the region of interest Historical observations over

the region of interest

Current model forecast around the region of interest

Corrected forecast over the region of

interest Corrected forecast over the region of

interest

CPT CPT Skill maps

(pearson correlation)

Skill maps (pearson correlation)

Four embedded calibration methods, all based on the linear

regression:

CCA, PCR , MLR , GCM

(32)

CCA, PCR and GCM : What is it about?

• CCA : Canonical Correlation Analysis

– Prediction of Spatial patterns i.e. Grid points of the predictand (obs.) variable are predicted simultaneously by predicting spatial patterns.

• PCR : Principal Components Regression

– Grid points of the predictand (obs.) variable are predicted individually using the Principal components of the predictor(model) variable.

• GCM : Comparing model output with Observed data (station or Gridded)

– Here predictor(model) and predictand (obs.) have to be on the same area. The nearest grid point or interpolation (from predictor to predictand ) will be used for comparison.

(33)

Linear regression based

(34)

Raw vs Calibrated Forecast

Calibration in action (1)

NCEP CFS 10days forecasts are initiated on Feb 11, 2019 at 00 UTC. Valid period : 12– 21 Feb 2019

• How is the skill score for raw and calibrated forecast? Any improvement from the calibration?

• How are the anomalies patterns ? Compare to raw forecast, how big is the difference?

• Look the area where the pattern seem similar, they can be very helpful for your final decision.

 Using the ppt drawing tools, draw your below-and above average polygons.

 Use brownish line color for below- average and greenish line color for above-average polygon

(35)

Verification – observed anomaly

Calibration in action (1)

NCEP CFS 10days forecasts are initiated on Feb 11, 2019 at 00 UTC. Valid period : 12– 21 Feb 2019

(36)

Calibration in action (2)

NCEP CFS week1 forecasts are initiated on Feb 11, 2019 at 00 UTC. Valid period : 12– 18 Feb 2019

(37)

Calibration in action (3)

NCEP CFS week2 forecasts are initiated on Feb 11, 2019 at 00 UTC. Valid period : 19 – 25 Feb 2019

(38)

Forecast Calibration : Key Message

• Dynamical models often have systematic errors:

– bias in the mean

– bias in the amplitude

– bias in the shape of the anomaly pattern

• Detect and correct such systematic errors,

– we need to compare history of model hindcasts

and the corresponding observations

(39)

>Dealing with calibrated forecast

• How is the skill score for raw and calibrated forecast? Any improvement from the calibration?

• How are the anomalies patterns ? Compare to raw forecast, how big is the difference?

• Look the area where the pattern seem similar, they can be very helpful for your final decision.

• Before we take your decision, it would be good idea to look at as many products as possible for target period

–The projected statut of MJO – Circulation anomalies – Exceedance

Probability maps, …

(40)

Hands on CPT :

Producing Corrected Subseasonal Forecast

Using CPT in GCM mode

Using CPT in CCA mode

Using CPT in PCR mode

Cross analysis of the corrected and raw forecasts to produced a consensus outlook map

ftp://203.158.131.96/pub/data/Turkey/day2

04/16/2019 11:59AM 895,153 model_fcst_week2.tsv

04/16/2019 12:00PM 17,079,911 model_hdcst_week2.tsv

04/16/2019 12:00PM 40,113,220 obs_hist_week2.tsv

(41)

Hand on Tools : Get the data ready and open CPT

• Demo : Corrected S2S rainfall forecast

– Initiation date : 11 Feb 2019

– Target period : week1 (

Lead 1 to 7 here 22-28 Feb 2019

) week2 (

Lead 8 to 14 here 01-07 Mar 2019

) – Input data (in CPT format)

• Model hindcast : reforecasts over 20 years (1999-2018) of Total precipitation for 22-28 Feb and for 01-07 Mar

• Model forecast : Total precipitation for 22-28 Feb ,

• Historical observation: CPC Unified historical records (12 years : 1999-

2010) of total precipitation for 22-28 Feb and for 01-07 Mar

(42)

Using CPT in GCM Mode :

(43)

Selecting the Analysis Various choices under the tab

View

(44)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

1 2 3

(45)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(46)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

choose the spatial domain for model (predictor), this should be a domain fitting your region of interest.

Set the X domain latN = 0

latS = 40S lonW = 22W lonE = 55E

1

2

3

(47)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(48)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

CPT opens a browser, which by default looks for data in:

You can search for data from any other directory.

(49)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

Set the Y domain latN = 0

latS = -40 lonW = -22 lonE = 55

1 3 2

choose the spatial domain for observation (predictand), put the geographical coordinates of your region of interest.

(50)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(51)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(52)

SETTING THE TRAINING PERIOD

Make sure to set the start year to the same values.

By default CPT usually starts the analysis from the first years in the X and Y files; note that these years could be different. You would normally set them equal to the latest year in the two files.

You have to specify the length of the training period.

Same start year for hindcast and observation :

Set the length of the training period accordingly

(53)

Check the Synchronous predictor box,

SETTING ANALYSIS

OPTIONS

(54)

Make sure you leave the zero box unchecked

SETTING ANALYSIS

OPTIONS

(55)

Cross-Validation windows

SETTING ANALYSIS

OPTIONS

(56)

For Cross-Validation windows take at least 3 if working with rainfall (1 might be good enough for temperature)

Cross-Validation windows

SETTING ANALYSIS

OPTIONS

(57)

Tailoring – setting output to be a two categories probabilities map (1)

SETTING ANALYSIS OPTIONS

By default, CPT is set so that the boundaries between the categories are the terciles of the climatogical distribution.

For Subseasonal forecast, it’s recommended to deliver the results in terms of two categories forecast.

(58)

SETTING ANALYSIS OPTIONS

Setting the cli

By default CPT usually starts the analysis from the first years in the X and Y files; note that these years could be different. You would normally set them equal to the latest year in the two files.

Tailoring – setting output to be a two categories probabilities map (2)

By default, CPT is set so that the boundaries between the categories are the terciles of the climatogical distribution.

For Subseasonal forecast, it’s recommended to deliver the results in terms of two categories forecast.

(59)

RUNNING CPT

Then you can run the analysis:

Actions => Calculate => Cross-validated

(60)

RUNNING CPT

Look at Goodness Index -- Kendall’s tau -- value (0.1 and above values are wanted, 0.4 described very good skill for the model)

CPT begins the specified analysis in a new “Results Window”. Here you can see the steps of the analysis and of the optimization procedure.

(61)

Results : skill maps

The menu Tools => Validation => Skill Maps displays a page will skill scores,

(62)

Results : Skill Maps

Pearson correlation

(63)

Calibrated Forecast

The menu Tools => Forecasts => Maps => Probabilistic displays two category maps

(64)

Calibrated Forecast (cnt’d)

(65)

Using CPT in CCA Mode :

(66)

Selecting the Analysis Various choices under the tab

View

(67)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

1 2 3

(68)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

(69)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

choose the spatial domain (for predictor) over which you want to perform your EOF or CCA analysis. In general the domain is known in advance through experience.

latS = -8

lonW = 104E lonE = 139E

1 3 2

(70)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

Set the X maximum mode to 3

(71)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(72)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

(73)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

latS = -2

lonW = 114E lonE = 129E

1 3 2

(74)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

Set the Y maximum mode to 3

(75)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y) 3. Forecast data (X)

Set the X maximum mode to 3Set the CCA

maximum mode to 3

(76)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(77)

Upload data files

1. Hindcast data (X)

2. Historical Obs data (Y)

3. Forecast data (X)

(78)

SETTING THE TRAINING PERIOD

Make sure to set the start year to the same values.

By default CPT usually starts the analysis from the first years in the X and Y files; note that these years could be different. You would normally set them equal to the latest year in the two files.

You have to specify the length of the training period.

Same start year for hindcast and observation :

Set the length of the training period accordingly

(79)

Check the Synchronous predictor box,

SETTING ANALYSIS

OPTIONS

(80)

Make sure that the zero box is also check SETTING ANALYSIS

OPTIONS

(81)

Cross-Validation windows

SETTING ANALYSIS

OPTIONS

(82)

For Cross-Validation windows take at least 3 if working with rainfall (1 might be good enough for temperature)

Cross-Validation windows

SETTING ANALYSIS

OPTIONS

(83)

RUNNING CPT

Then you can run the analysis:

Actions => Calculate => Cross-validated

(84)

RUNNING CPT

Look at Goodness Index -- Kendall’s tau -- value (0.1 and above values are wanted, 0.4 described very good skill for the model)

CPT begins the specified analysis in a new “Results Window”. Here you can see the steps of the analysis and of the optimization procedure.

(85)

Results : skill maps

The menu Tools => Validation => Skill Maps displays a page will skill scores,

(86)

Results : Skill Maps

Pearson correlation

(87)

Calibrated Forecast

The menu Tools => Forecasts => Maps => Probabilistic displays two category maps

(88)

Calibrated Forecast (cnt’d)

(89)

Some possible actions

• Methods comparaison : GCM vs CCA vs PCR

• Playing with domain size of the predictor

• Playing with lead time : week2, week34

• Playing with initiation date

• Getting data from the sources (demo with the sh script

provide)

(90)

CCA PCR GCM

NCEP Week1 Calibrated forecast

init date : 11Nov2018 valid from 12Nov-!8Nov 2018

(91)

CCA PCR GCM

NCEP Week1 Calibrated forecast

init date : 11Nov2018 valid from 12Nov-!8Nov 2018

(92)

CCA PCR GCM

NCEP Week1 Calibrated forecast

init date : 02Dec2018 valid from 03-09Dec 2018

(93)

CCA PCR GCM