1
How Does NCEP/CPC Make
Operational Monthly and Seasonal Forecasts?
Huug van den Dool (CPC)
ESSIC, February, 23, 2011
2
Assorted Underlying Issues
• Jin Huang - R2O - CTB
• Which tools are used…
• How are tools combined???
• CFSv1 CFSv2
• Dynamical vs Empirical Tools
• Skill of tools and OFFICIAL
• How easily can a new tool be included?
• US, yes, but occasional global perspective
3
Menu of CPC predictions:
• 6-10 day (daily)
• Week 2 (daily)
• Monthly (monthly + update)
• Seasonal (monthly)
• Other (hazards, drought monitor, drought outlook,
MJO, UV-index, degree days, POE, SST) (some are
‘briefings’)
• Informal forecast tools (too many to list)
• http://www.cpc.ncep.noaa.gov/products/predictions/90d ay/tools/briefing/index.pri.html
4
5
EXAMPLE
P U B L I C L Y I S S U E D
From an internal CPC Briefing package 6
7
EMP EMP EMP EMP
DYN EMP
DYN CON CON
N/A
88
SMLR CCA OCN
LAN
LFQ
(15 CASES: 1950, 54, 55, 56, 64, 68, 71, 74, 75, 76, 85, 89, 99, 00, 08) OLD-OTLK
CFSV1
ECP IRI
ECA CON
9
Element
US-T US-P SST US-soil moisture Method:
CCA X X X
OCN X X
CFS X X X X
SMLR X X
ECCA X X
Consolidation X X X Constr Analog X X X X
Markov X
ENSO Composite X X
Other (GCM) models (IRI, ECHAM, NCAR, CDC etc):
X X
CCA = Canonical Correlation Analysis OCN = Optimal Climate Normals
CFS = Climate Forecast System (Coupled Ocean-Atmosphere Model) SMLR = Stepwise Multiple Linear Regression
CON = Consolidation
10
Long Lead Predictions of US Surface
Temperature using Canonical Correlation Analysis. Barnston(J.Climate, 1994, 1513)
Predictor - Predictand Configuration
Predictors Predictand
* Near-global SSTA
* N.H. 700mb Z * US sfc T
* US sfc T
four predictor “stacked” fields one predictand period
4X652=2608 predictors 102 locations
Data Period 1955 - last month
11
12
13
About OCN. Two contrasting views:
- Climate = average weather in the past - Climate is the ‘expectation’ of the future
30 year WMO normals: 1961-1990; 1971-2000 etc
OCN = Optimal Climate Normals: Last K year average. All seasons/locations pooled: K=10 is optimal (for US T).
Forecast for Jan 2012
= (Jan02+Jan03+... Jan11)/10. – WMO-normal plus a skill evaluation for some 50+ years.
Why does OCN work?
1) climate is not constant (K would be infinity for constant climate) 2) recent averages are better
3) somewhat shorter averages are better (for T)
see Huang et al 1996. J.Climate. 9, 809-817.
14
OCN has become the bearer of most of the skill, see also EOCN
method (Peng et al)
15
16
G H C N - C A M S F A N 2 0 0 8
17
Major Verification Issues
• ‘a-priori’ verification (used to be rare)
• After the fact (fairly normal)
18
Source Peitao Peng
After the fact…..
19
(Seasonal) Forecasts are useless unless accompanied by a reliable a-
priori skill estimate.
Solution: develop a 50+ year track record for each tool. 1950-present.
(Admittedly we need 5000 years)
20
Consolidation
21
--- OUT TO 1.5 YEARS ---
22
OFFicial Forecast(element, lead, location, initial month) =
a * A + b * B + c * C +
…
Honest hindcast required 1950-present.
Covariance (A,B), (A,C), (B,C), and
(A, obs), (B, obs), (C, obs) allows solution for a,
b, c (element, lead, location, initial month)
CFS skill 1982-2003
2324
Fig.7.6: The skill (ACX100) of forecasting NINO34 SST by the CA method for the period 1956-2005. The plot has the target season in the horizontal and the lead in the vertical. Example: NINO34 in rolling seasons
2 and 3 (JFM and FMA) are predicted slightly better than 0.7 at lead 8 months. An 8 month lead JFM forecast is made at the end of April of the previous year. A 1-2-1 smoothing was applied in the vertical to
reduce noise.
CA skill 1956-2005
25
M. Peña Mendez and H. van den Dool, 2008:
Consolidation of Multi-Method Forecasts at CPC.
J. Climate, 21, 6521–6538.
Unger, D., H. van den Dool, E. O’Lenic and D.
Collins, 2009: Ensemble Regression.
Monthly Weather Review, 137, 2365-2379.
(1) CTB, (2) why do we need ‘consolidation’?
26
27
(Delsole 2007)
28
UR MMA COR
RI RIM RIW
Climo
29
3CVRE
SEC
SEC and CV
30
31
Bayesian Multimodel Strategies
Linear regression leads to unstable weights for small sample sizes.
Methods for producing more stable estimates have been proposed by van den Dool and Rukhovets (1994), Kharin and Zwiers (2002), Yun et al. (2003), and Robertson et al. (2004).
These methods are special cases of a Bayesian method, each distinguished by a different set of prior assumptions (DelSole 2007).
Some reasonable prior assumptions:
R:0 Weights centered about 0 and bounded in magnitude (ridge regression)
R:MM Weights centered about 1/K (K = # models) and bounded in magnitude R:MM+R Weights centered about an optimal value and bounded in magnitude R:S2N Models with small S2N (signal-to-noise) ratio tend to have small weights LS Weights are unconstrained (ordinary least squares)
From Jim Kinter (Feb 2009)
32
If the multimodel strategy is carefully cross validated, then the simple mean beats all other investigated multimodel
strategies.
Since Bayesian methods involve additional empirical
parameters, proper assessment requires a two-deep cross validation procedure. This can change the conclusion
about the efficacy of various Bayesian priors.
Traditional cross validation procedures are biased and incorrectly indicate that Bayesian schemes beat a simple mean.
From Jim Kinter (Feb 2009)
33
34
No SEC, (no CV required), ‘raw’) 35
36
Good to know:
• If a model/methods has no systematic
error (by design), leave it alone, please do not apply systematic error correction
where none is needed.
• There is more (need for) statistics in dynamical models and their post-
processing than in any statistical method
37
38
39
40
See also:
O’Lenic, E.A., D.A. Unger, M.S.
Halpert, and K.S. Pelman, 2008:
Developments in Operational Long-Range Prediction at CPC.
Wea. Forecasting, 23, 496–515.
41
Empirical tools can be
comprehensive! (Thanks to
reanalysis, among other things).
And very economic.
42
SST Z500
Precip T2m
CA
43
SST Z500
Precip T2m
CFS
Source: Wanqiu Wang
44
Is CPC in good shape to admit more tools???
Even if these tools come from the outside???
Does the outside have the right
expectations???
45
extra
46
Dynamics> Empiricism Symbiosis
• Successive generations of Reanalyses are produced
because NWP exists, the desire for initial conditions etc, but empirical prediction methods are one of the main
beneficiaries of having all this data.
• Early empirical methods have served as an example (for anybody to follow) to produce (honest) hindcasts, a-priori skill assessment, cross validation etc
• Consolidation (of tools) is “color-blind” relative to questions of dynamical or empirical origin
47
Progress in Empirical Methods
• More data to work with (1 year per year)
• More (and global) data to work with (Re- analyses). Oceans, land, atmosphere
(note the symbiotic relationship with modeling here)
• New empirical methods
• New applications
Inch-by-inch methods Revolutionary changesAlways be ready for surprises
48
Why are empirical Methods competitive with dynamical
methods (in seasonal prediction)??
• Linearity (define)
• A system with skill in <= 3EDOFs is functionally linear
• See Chapter 10 of book
49
