SolarFlare Communications

(1)

10GBASE

10GBASE - - T Tutorial T Tutorial

IEEE 802.3 IEEE 802.3 Kauai, Hawaii Kauai, Hawaii

November 11, 2002 November 11, 2002

SolarFlare Communications

(2)

SolarFlare

Agenda Agenda

• Introduction, Cabling & Challenges -

George Zimmerman, Ph.D.

CEO & CTO Founder

• Implementation & Performance -

Bill Jones, Ph.D.

Director, Systems Engr.

(3)

10G on UTP Possible or Not?

• The Problem

• Characterization vs. Specification

• Cabling & Impairments

• Limitations

• Capacity

– How to pick the right bandwidth?

• Challenges

(4)

SolarFlare

Applications Overview Applications Overview

• 10-Gb/s Ethernet connections </= 100m

• Utilize installed base of structured Cat 5e UTP

• Upgrade from 1000BASE-T

10GBase-T Transceiver

11/4/2002

MAC

10GBase-T

Transceiver

^MAC

Cat5e UTP (4 pairs)

up to 90m

Wallplate Patch Panelor

up to 5m up to 5m

(5)

Ethernet Evolution Ethernet Evolution

Speed

Time

New Model Shannon

AFE+DSP MIMO/Multiuser Channel-Optimized

• 10GBASE-T

•1000BASE-T

>>1 bit/s/Hz

“Divide & Conquer”

DSP, Coded Conventional Wisdom

Shannon

1 bit/s/Hz Analog

Linear

• 10BASE-T

Perceived Shannon •100BASE-T

>1 bit/s/Hz Analog & Digital

Non-Linear Perceived Shannon

(6)

SolarFlare

What Makes Shannon Limits?

• NOT modulation-specific

• Signal Attenuation ( assumed usable bandwidth)

• Assumed irreducible noise sources

– Background – Crosstalk

• Crosstalk from other pairs in our sheath

• Alien crosstalk – coming from other bundled 4-pair sheaths

– Device noise from transceiver

• Change the assumptions & change the limit!

– (to a point…)

(7)

Channel Impairments Channel Impairments

R

H T

Y B R I

R D

T ^H_Y

B R I D

R

H T

Y B R I

R D

T ^H_Y

B R I D

R

H T

Y B R I

R D

T ^H_Y

B R I D

R

H T

Y B R I

R D

T ^H_Y

B R I D

NEXT14 NEXT12

NEXT13

FEXT14 FEXT13 FEXT12

Far Echo Near Echo

Alien Crosstalk, EMI

(8)

SolarFlare

Characterization vs. Specification Characterization vs. Specification

• Cat 5/5e cable must be high quality with minor structural variations to meet TIA-568 requirements

• 100 MHz (or 250 MHz) “limit” imposed by TIA qualification requirements

– not the physical limitations of the cable

• Cable properties stable beyond 500 MHz

– depends mainly on transmission line geometry and construction materials

• Minor structural variations and connector

discontinuities affect channel transmission, but not

catastrophically

(9)

Cat 5e Channel: Insertion Loss Cat 5e Channel: Insertion Loss

Measured Cat 5e 100 Meter Channel Insertion Gain at 20 C

0 50 100 150 200 250 300 350 400 450 500

-60 -50 -40 -30 -20 -10 0

Frequency (MHz)

Insertion gain (dB)

Manufacturer A Manufacturer B Manufacturer C Cat 5e limit Extended limit

(10)

SolarFlare

Cat 5e Channel: NEXT Cat 5e Channel: NEXT

0 100 200 300 400 500 600

-80 -70 -60 -50 -40 -30 -20 -10 0

Measured Pair-to-pair NEXT Coupling into Cat 5e Pair 1

Frequency (MHz)

Insertion gain (dB)

NEXT12 NEXT13 NEXT14 Cat 5e limit Extended limit

(11)

Cat 5e Channel: FEXT Cat 5e Channel: FEXT

0 50 100 150 200 250 300 350 400 450 500

-100 -90 -80 -70 -60 -50 -40

Measured Pair-to-pair FEXT Coupling into Cat 5e Pair 1

Frequency (MHz)

Insertion gain (dB)

FEXT12 FEXT13 FEXT14

(12)

SolarFlare

Alien NEXT Alien NEXT

0 50 100 150 200 250 300 350 400 450 500

-80 -70 -60 -50 -40 -30 -20

Frequency (MHz)

Insertion gain (dB)

Adjacent connector Non-adjacent connector Cat 5e NEXT limit

Extended limit Single (4-pair cable) disturber, 40 meter length unbundled

Cat 5e Power Sum Alien NEXT vs. Patch Panel Position

(13)

EMI EMI - - Emitted Emitted

• >100 MHz on Cat 5e can meet FCC Class A

~12 dBm launch power limitation

0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 8 0 0

2 0 2 5 3 0 3 5 4 0 4 5 5 0 5 5 6 0

W o rst-C a se R a d ia te d E m issio n s a t 3 M e te rs - C a t 5 e U T P

F req u e n c y (M H z )

Electric field (dBuV/m)

-8 0 d B m /H z sig n a l F C C C la ss A lim it F C C C la ss B lim it

(14)

SolarFlare

Overall Environment Overall Environment

• Sources require significant cancellation

– Extensions from 1000BASE-T

– Significantly greater NEXT + FEXT + Equalization challenge

-150 -140 -130 -120 -110 -100 -90 -80 -70

0 100 200 300 400 500 600

Frequency (MHz)

Power Spectral Density (dBm/Hz)

XmtPSD RcvPSD SelfNEXT SelfFEXT Alien NEXT

(15)

Strawman

Strawman Improvements Improvements

• Baseline Requirements:

~40 dB Echo & NEXT Cancellation, ~20 dB FEXT Cancellation Alien NEXT suppression for crowded installations

Received Signal & Residual Noise Terms

-160.00 -150.00 -140.00 -130.00 -120.00 -110.00 -100.00 -90.00 -80.00

0 50 100 150 200 250 300 350 400

Frequency (MHz)

PSD ref to Input (dBm/Hz)

-20.0 -10.0 0.0 10.0 20.0 30.0 40.0 50.0 60.0

SNR (dB)

Rcv Sig Res Echo Res NEXT Res FEXT Res ANEXT Bkgnd SNR

(16)

SolarFlare

Capacity Capacity

• 100m Cat 5e, with cancellers

• 14.4 Gbps on 100m at 10dBm launch, 600 MHz

• 10 Gbps @ 8.9 dB Margin, 430 MHz bandwidth

Bit Capacity

0 2 4 6 8 10 12 14 16 18 20

0 50 100 150 200 250 300 350 400

Frequency (MHz)

Bits/Sec/Hz

BitCap (Bkgd) Bit Cap (all residuals)

(17)

Conclusion:

It CAN be done, but HOW?

• Bandwidth required 400-500 MHz

• 40+ dB Echo & NEXT reduction

• 20+ dB FEXT reduction

• 10-12 dBm launch power

• > 8 bits (ENOB) signal processing

– A/D performance, or analog noise performance if analog circuits used

• Shannon limits say “Not Impossible”, just hard!

– It’s up to us engineers!

(18)

SolarFlare

Realizing 10GBASE

Realizing 10GBASE - - T T

• Addressing communication system challenges

• Modern signal processing algorithms

• Low power, high speed digital circuit design

• High linearity, wideband analog circuit

design

(19)

Communication System Challenges Communication System Challenges

• High frequency multiple twisted pair media characterization – Line attenuation, NEXT, FEXT, Alien Xtalk & EMI

– Cat 5e specification out to 100MHz

• Sufficient for 1000BASE-T

– Utilizing frequencies beyond cable’s initial intended objective is not new

– Case in point: xDSL

• Installation designed for 20kHz max

• Measurements converted for use in system evaluation – No assumptions or short cuts taken

– Scaled to worse case specifications (when they exist)

(20)

SolarFlare

Line Code Selection Line Code Selection

• Pulse Amplitude Modulation (PAM)

• Evolution Of 1000BASE-T

– Builds on proven technology

• Lower AFE requirements

– De-stressing an already stressed requirement

• Utilizing an optimal DFE achieves capacity

(21)

PAM PAM - - 10 Coding 10 Coding

• Given the characteristics of the channel/disturbers, capacity is maximized with an analog bandwidth around 400MHz

• 10Gbps is achieved with a baud rate of 833MHz and 12 bits/baud or 3 bits/pair (4 pairs)

– Minimum requirement of PAM8 for uncoded operation

• PAM9 may be sufficient for Ethernet control symbols

• PAM10 needed for both control and Trellis coding

• Extension of 1000BASE-T

– 4D, 8-state Trellis code (one dimension per pair)

– 6 dB coding gain relative to uncoded 10PAM

(22)

SolarFlare

Comparison With 1000BASE

Comparison With 1000BASE - - T T

High-Performance FEXT cancellation

No specified FEXT cancellation

High-Performance NEXT cancellation

Moderate NEXT cancellation

833 Mbaud, ~400 MHz used bandwidth

125 Mbaud, ~80 MHz used bandwidth

Full duplex echo-cancelled transmission

10-level with Trellis code across pairs

5-level with Trellis code across pairs

Multilevel coded PAM signaling (3- bits/symbol)

Multilevel coded PAM signaling (2- bits/symbol)

Straw Man 10GBASE-T

1000BASE-T

(23)

PCS SER & BER Straw Man Goal PCS SER & BER Straw Man Goal

Slicer Input SNR (dB)

1000BASE-T Reqm’t

19 20 21 22 23 24 25 26 27

10^-14 10^-12 10^-10 10^-8 10^-6 10^-4 10^-2 10⁰

Coded PAM10 simulation SER Coded PAM10 Theory SER Coded PAM10 Theory BER

25.3 dB

(24)

SolarFlare

PMD Performance Straw Man Goal PMD Performance Straw Man Goal

• TSB 67 Channel

10GBASE-T Transceiver

11/7/2002

MAC

10GBASE-T

Transceiver

^MAC

Cat 5e UTP (4 pairs)

up to 90m

Wallplate Patch Panelor

up to 5m up to 5m

• Consider an aggregate slicer SNR of 25.3 dB with

five dominating noise terms requires individually

around 32 dB SNR

(25)

Channel Impairments

Channel Impairments - - ISI ISI

• Pre- & Post-Cursor Interference from limited Bandwidth

• Post-Cursor Dominates (>100 terms)

• Feedforward & Decision Feedback Equalization Solution

FeedForward Equalizer

Decision Feedback Equalizer Slicer

_

⁺

(26)

SolarFlare

Echo Echo

• Full duplex needed for limited BW

• Compromise hybrid for Tx/Rx isolation

• Impedance mismatches require residual echo cancellation

• > 40 dB rejection

(27)

NEXT NEXT

• High-level interference from transmitters

• Very long response time

• > 40dB NEXT cancellation

wireline Tx Channel 0

Rx Channel 1 (target rcvr)

Next Coupling

wireline

Rx Channel 0 (victim rcvr)

Tx Channel 1 (interfering xmtr)

(28)

SolarFlare

FEXT Impairment FEXT Impairment

• Pre- and post-cursor elements of interference

• Based on an equal-level FEXT (ELFEXT) model

• Uncompensated in 1000BASE-T

• Must be cancelled in 10GBASE-T

• > 20 dB cancellation

wireline Tx Channel 0

Tx Channel 1 (interfering xmtr)

Fext Coupling

wireline

Rx Channel 0 (victim rcvr)

Rx Channel 1 (target rcvr)

chan elf

fext

H H

H = ⋅

(29)

Challenging Implementation Challenging Implementation

• A new approach to problem solving needed to meet SNR requirement (>25.3 dB)

• Efficient reuse of resources in MIMO modeling

• Utilization of parallel structures

(30)

SolarFlare

Traditional Signal Processing Traditional Signal Processing

SISOSISO SISOSISO

T₄ T₃

T₂ T₁

++++ ++++ ++++ ++++ R₄

• Echo & NEXT cancellation

• 16 Single Input Single Output (SISO) processing elements (scalar filters)

• With canceller taps on the order of several hundred

10 Tera Operations (TOps)!

(31)

Modern Signal Processing Modern Signal Processing

• Echo & NEXT Cancellation

R₄ T₄

T₃ T₂

MIMO

T₁

• One Multiple Input Multiple Output (MIMO) processing element (matrix filter)

• Exploits correlation to reduce interference common to all received channels

• Enables massive reuse of computing resources

(32)

SolarFlare

Modern Signal Processing Modern Signal Processing

• Data recovery & Fext cancellation

d₁

d₂

d₃

d₄ T₁

T₂

T₃

T₄

R₁

R₂

R₃

R₄ H₁

F₂₁ F₃₁ F₄₁

MIMO CHANNEL

MUD d₁

d₂

d₃

d₄

• Multiuser Detector (MUD) of MIMO channel provides

simultaneous data decisions & interference removal

(33)

Parallelizing FIRs Parallelizing FIRs

• One high rate N tap filter

( )

⁽ ⁾

)

(z H₀ z² z ¹H₁ z²

H = + ⁻

x(n)

H(z)

y(n)

H₀(z)

H₁(z)

H₀(z) Z^-1

x(n)

Z^-1

+

+ +

Z^-1

y(n)

2

2 • Good for clock limited or high speed applications

• Four half rate N/2 tap filters

• Equivalent number of operations per unit time

(34)

SolarFlare

Efficient Parallelization Efficient Parallelization

H₀(z)

H₀(z)+H₁(z)

H₀(z) Z^-1

x(n)

Z^-1

+

+ +

Z^-1

y(n)

2

2 + _ +

_

• Four filters reduced to three

• 25% improvement in efficiency

•Greater efficiency with greater parallelism

(35)

Digital Circuit Straw Man Goals Digital Circuit Straw Man Goals

• Puts total DSP requirements at 1.5 Tera Operations (TOps)

• Quad 1000BASE-T requires 1.0 TOps

150% increase in possible aggregation with 50%

increase in complexity today!

• CMOS technology

(36)

SolarFlare

Analog Circuit Straw Man Goals Analog Circuit Straw Man Goals

• Transmitter: DAC & Line driver

– >40 dB Linearity

– 450 MHz Bandwidth

• Receiver: Hybrid, LNA & ADC

– >8 bits ENOB – 833 MSPS

• PLL & Clock recovery

– 833 MHz

• CMOS technology

(37)

ISI Impairment

ISI Impairment - - After Equalization After Equalization

0 0.5 1 1.5 2 2.5 3 3.5 4

x 10

⁸

-70

-60 -50 -40 -30 -20 -10 0 10

Frequency

Spectrum (dB)

Equalized Signal

Residual ISI

• FFE scales to

produce unit variance hard decisions

T _x MAG Cat 5e cable MAG R _x ENFX EQ

(38)

SolarFlare

ISI Impairment

ISI Impairment - - Symbol Stream Symbol Stream

a)

b)

0 1 2 3 4 5 6 7 8 9 10

x 10⁴ -2

-1 0 1 2

a) Rx Far End Signal b) Slicer Input Vs. time

0 1 2 3 4 5 6 7 8 9 10

x 10⁴ -2

-1 0 1 2

MAG Cat 5e cable MAG R _x ENFX EQ

a b

T _x

(39)

Echo Impairment

Echo Impairment - - After Cancellation After Cancellation

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 10⁸ -100

-80 -60 -40 -20 0 20

Frequency

Spectrum (dB)

Rx Echo

Cancelled Echo

Cat 5e cable

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 x 10

⁸

-100

-80 -60 -40 -20 0 20

Frequency

Spectrum (dB)

Rx far end signal

Rx echo

(a) (b)

a b

T _x MAG MAG R _x ENFX EQ

(40)

SolarFlare

Echo Impairment

Echo Impairment - - SNR @ Detector SNR @ Detector

0 0.5 1 1.5 2 2.5 3 3.5 4

x 10

⁸

-70

-60 -50 -40 -30 -20 -10 0 10

Frequency

Spectrum (dB)

Equalized Signal

Residual Echo

• FFE scales to

produce unit variance hard decisions

T _x MAG Cat 5e cable MAG R _x ENFX EQ

(41)

NEXT Impairment

NEXT Impairment - - After Cancellation After Cancellation

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 10⁸ -140

-120 -100 -80 -60 -40 -20 0 20

Frequency

Spectrum (dB)

Rx Next(3)

Cancelled Next(3)

4.5

8

(b)

T _x MAG Cat 5e cable MAG R _x ENFX EQ

a b

0 0.5 1 1.5 2 2.5 3 3.5 4 x 10 -140

-120 -100 -80 -60 -40 -20 0 20

Frequency

Spectrum (dB)

Rx far end signal

Rx Next(3)

(a)

(42)

SolarFlare

NEXT Impairment

NEXT Impairment - - SNR @ Detector SNR @ Detector

T _x MAG MAG R _x ENFX EQ

0 0.5 1 1.5 2 2.5 3 3.5 4

x 10

⁸

-70

-60 -50 -40 -30 -20 -10 0 10

Frequency

Spectrum (dB)

Equalized Signal

Residual Next(3) Cat 5e cable

• FFE scales to

produce unit variance

hard decisions

(43)

FEXT Impairment

FEXT Impairment - - After Cancellation After Cancellation

Cat 5e cable

(a) (b)

a b

T _x MAG MAG R _x ENFX EQ

(44)

SolarFlare

FEXT Impairment

FEXT Impairment - - SNR @ Detector SNR @ Detector

• FFE scales to

produce unit variance hard decisions

T _x MAG Cat 5e cable MAG R _x ENFX EQ

(45)

Total

Total Slicer Slicer SNR SNR

T _x MAG Cat 5e cable MAG R _x ENX EQ

0 0.5 1 1.5 2 2.5 3 3.5 4

x 10

⁸

-60

-50 -40 -30 -20 -10 0 10

Frequency

Spectrum (dB)

Equalized Signal

Total Noise

Total Noise Power Goal (-25.3) • FFE scales to

produce unit variance

hard decisions

(46)

SolarFlare

Eye Diagram Eye Diagram

Slicer input, including all noise sources vs. time

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

(47)

Sequenced Startup Sequenced Startup

N E , P F D O N E

D O N E F

E , T N

S L A V E M A S T E R

N = E C H O /N E X T C anceller C o nverge nce T = Tim in g A cq uisition P = P h ase A d justm en t F = FE X T C anceller C o nverge nce