`Server Memory Architecture:
`Capacity, Performance, Reliability, and
`Longevity
`
`Pete Vogt
`Principal Engineer
`Intel Corp
`
`February 18, 2004
`
`1
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Agenda
`(cid:121) Server Capacity Problem and Solution
`(cid:121) Performance
`(cid:121) Reliability
`(cid:121) Longevity
`
`2
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`Intel Platform Memory Technology Roadmap
`DDR3
`DDR3
`DDR2 667/800
`
`DDR2 400/533
`
`FB-DIMM
`
`DDR2
`DDR
`
`RDRAM*
`
`SDRAM
`
`DDR266
`
`PC800
`
`DDR333/400
`
`PC1066
`
`PC133
`
`200420042004
`
`2004
`
`2005200520052005
`
`
`
`
`200520052006
`
`2005
`
`(cid:121) DDR2 400/533 support in all main IA segments in 2004
`– with DDR flexibility
`(cid:121) FB-DIMM new server interconnect in 2005
`(cid:121) Low Power SDRAM moving to low power DDR
`(cid:121) RDRAM still used in specific applications
`
`*Other names and brands may be claimed as the property of others
`3
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`FB-DIMM New Server Interconnect
`in 2005
`Why is FB-DIMM Needed in the
`Future?
`
`4
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`DRAM Bit Density; Mb/dev
`2500
`
`2000
`
`DRAM Device Trends
`(cid:121) DRAM bit density follows
`Moore’s law and
`increases 2X every ~2
`years
`
`1500
`
`1000
`
`500
`
`(cid:121) DRAM data rates driven
`by CPU performance
`increase 2X every
`generation (3.35 years)
`
`0
`
`99
`
`00
`
`01
`
`02
`
`07
`06
`05
`04
`03
`Rule of thumb - Source: Pete Vogt
`
`DRAM Data Rate; Mbps
`1200
`
`1000
`
`800
`
`600
`
`400
`
`200
`
`0
`
`99
`
`00
`
`01
`
`07
`06
`05
`04
`03
`02
`Projected historical data - Source: Pete Vogt
`
`DRAM Devices Continue to Scale
`
`5
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`Devices per Channel Trend
`(cid:121) Existing “stub-bus”
`architecture has
`impedance discontinuities
`that effect signal integrity
`
`Stub-bus Topology
`
`DRAM
`
`DRAM
`
`DRAM
`
`DRAM
`
`Mem
`Ctrl
`
`Impedance
`Discontinuities
`
`(cid:121) As the data rate goes up
`the number of devices on
`the channel is going down
`
`Stub-bus Devices/Channel
`160
`140
`120
`100
`80
`60
`40
`20
`0
`
`99
`
`00
`
`01
`
`02
`
`07
`06
`05
`04
`03
`Estimated trend - Source: Pete Vogt
`
`Devices per Channel is Decreasing
`
`6
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`Memory Capacity Trends
`(cid:121) Server performance drives
`2X capacity increase
`demand every ~2 years
`
`Capacity Demand; GB
`35
`30
`25
`20
`15
`10
`
`Demand
`
`99
`
`00
`
`01
`
`02
`
`03
`
`05
`
`Stub-bus Capacity/Ch: GB
`32
`28
`24
`20
`16
`12
`
`07
`06
`05
`04
`Rule of thumb - Source: Pete Vogt
`Demand
`
`Gap is
`growing
`
`Supply
`
`99
`
`00
`
`01
`
`02
`
`07
`06
`05
`04
`03
`Computed trend - Source: Pete Vogt
`
`048
`
`(cid:121) Stub-bus channel capacity
`(density * devices/ch)
`supply has hit a ceiling
`
`Meeting Demand Requires a Change
`
`7
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`FB-DIMM Eliminates the “Stubs”
`(cid:121) FB-DIMM buffers the DRAM
`data pins from the channel
`and uses point-to-point
`links to eliminate the stub
`bus
`
`DRAM
`
`Buffer
`
`DRAM
`
`Buffer
`
`DRAM
`
`Buffer
`
`Mem
`Ctrl
`
`Point-to-Point Links
`
`DRAM
`
`Buffer
`
`FB-DIMM
`
`(cid:121) FB-DIMM capacity scales
`throughout DDR2 & DDR3
`generations
`
`FB-DIMM Capacity/Ch: GB
`35
`30
`25
`20
`15
`10
`
`99
`
`00
`
`01
`
`02
`
`07
`06
`05
`04
`03
`FB-DIMM Capability - Source: Pete Vogt
`
`05
`
`FB-DIMM Meets the Capacity Demand
`
`8
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`FB-DIMM Solution Details
`
`DDR2 connector
`with unique key
`
`Commodity
`DRAMs
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`Buffer
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`Buffer
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`Buffer
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`Up to 8
`DIMMs
`• • •
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`Buffer
`
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`DRAMDRAM
`
`Serial signaling
`similar to
`PCI-Express
`
`10
`
`14
`
`Memory
`Controller
`
`SMBus
`
`CLK
`Source
`
`Common clock
`source
`
`SMbus access to buffer
`registers
`
`9
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`FB-DIMM Channel Pin Count
`
`Data Path to DIMMs
`Data Path from DIMMs
`Total High-Speed Signals
`Power
`Ground
`Shared Pins (clocks,
`calibration, PLL pwr, test)
`Total Pins
`
`Diff Signals
`10
`14
`
`Pins
`20
`28
`
`48
`6
`12
`~ 3
`
`~ 69
`
`(cid:121) Compare with ~240 pins for DDR2 channel
`
`FB-DIMM Pin Count is 1/3rd of DDR2
`
`10
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`Routing Comparison
`Direct DDR2 Registered DIMMs:
`FB-DIMMs:
`1 Channel, 2 Routing Layers with 3rd layer
`2 Channels, 2 Routing Layers
`required for power
`(includes power delivery)
`Serpentine
`routing is
`complicated
`and uses up
`a lot of
`board area
`
`Fewer
`signals and
`no trace
`length
`matching
`minimizes
`board area
`
`FB-DIMM: Fewer Layers, Less Routing Area
`
`11
`
`Source: Intel Enterprise Architecture Group
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Riser Card
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`~12 inches
`
`DIMM
`
`Logic
`Analyzer
`Riser
`
`LAI buffer
`
`Problem & Solution
`
`~12 inches
`
`Expansion and Visibility
`(cid:121) Repeaters support
`flexible system
`packaging and
`memory riser cards
`
`Repeater
`
`Memory
`Controller
`
`(cid:121) Logic Analyzer Interface
`provides visibility of
`high-speed bus activity
`for system analysis
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`Memory
`Controller
`
`FB-DIMM is a Versatile Solution
`
`12
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Problem & Solution
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`DIMM
`
`FB-DIMM
`Memory
`Controller
`
`Capacity Comparison
`(cid:121) 24x capacity
`– 8GB vs. 192GB
`(cid:121) ~4x bandwidth
`– ~10GB/s vs. ~40GB/s
`(cid:121) ~Lower pin count
`– ~480 vs. ~420
`
`DDR2
`Memory
`Controller
`
`DIMM
`DIMM
`
`DIMM
`DIMM
`
`8GB with 1Gb x4 DRAMs
`~10GB/s of BW w/DDR2-800
`(only 2 ranks per channel)
`
`192GB with 1Gb x4 DRAMs
`~40 GB/s of BW w/DDR2-800
`(2 ranks per DIMM)
`
`FB-DIMM Solves the Server Memory Capacity Problem
`
`13
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`FB-DIMM Performance
`
`14
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`FB-DIMM
`Memory
`Controller
`
`DIMM
`DIMM
`
`DIMM
`DIMM
`
`DDR2
`Memory
`Controller
`
`Performance
`
`Entry Configuration Comparison
`(cid:121) Equal capacity
`– 8GB vs. 8GB
`(cid:121) Better throughput
`– ~6.5GB/s vs. ~8.2GB/s
`(cid:121) Lower pin count
`– ~480 vs. ~280
`
`DIMM
`
`DIMM
`
`DIMM
`
`DIMM
`
`8GB with 1Gb x4 DRAMs
`~6.5GB/s of throughput w/DDR2-800
`(only 2 ranks per channel)
`
`8GB with 1Gb x4 DRAMs
`~8.2GB/s of throughput w/DDR2-800
`(1 rank per DIMM)
`
`Better Throughput with 200 Fewer Pins
`
`15
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Performance
`Entry Configuration Comparison
`
`FB-DIMM provides higher
`sustained throughput
`because of the increased
`number of channels
`
`Inherent serialization
`delay, decreases as
`data rate goes up
`
`FB-DIMM has lower latency
`at higher throughput
`
`2 DDR2 800 channels, 2 ranks per channel
`4 FB-DIMM 800 channels, 1 single rank DIMM per channel
`
`Source: Intel
`
`0
`
`2
`
`4
`
`6
`8
`10
`12
`Sustained Memory Subsystem Throughput (GB/s)
`
`14
`
`16
`
`18
`
`16
`
`Average Memory Read Latency
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Performance
`
`Mid-range Config Comparison
`(cid:121) 4x capacity
`– 8GB vs. 32GB
`(cid:121) ~2.5x throughput
`– ~6.5GB/s vs. ~16.5GB/s
`(cid:121) Lower pin count
`– ~480 vs. ~280
`
`DIMM
`DIMM
`
`DIMM
`DIMM
`
`DIMM
`DIMM
`
`DIMM
`DIMM
`
`FB-DIMM
`Memory
`Controller
`
`4 DIMMs per channel
`doesn’t work
`
`DIMM
`DIMM
`DIMM
`DIMM
`
`DIMM
`DIMM
`DIMM
`DIMM
`
`DDR2
`Memory
`Controller
`
`8GB with 1Gb x4 DRAMs
`~6.5GB/s of throughput w/DDR2-800
`(only 2 ranks per channel)
`
`32GB with 1Gb x4 DRAMs
`~16.5GB/s of throughput w/DDR2-800
`(2 ranks per DIMM)
`
`FB-DIMM Provides 4x the Capacity
`
`17
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Performance
`Mid-range Config Comparison
`
`Latency crossover
`point is earlier
`
`FB-DIMM provides very high
`sustained throughput due to
`simultaneous reads/writes
`
`Slightly larger delay for
`the additional buffer
`
`FB-DIMM has consistent latency
`across a wide throughput range
`Source: Intel
`
`2 DDR2 800 channels, 2 ranks per channel
`4 FB-DIMM 800 channels, 2 dual rank DIMMs per channel
`
`0
`
`2
`
`4
`
`6
`8
`10
`12
`Sustained Memory Subsystem Throughput (GB/s)
`
`14
`
`16
`
`18
`
`18
`
`Average Memory Read Latency
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Performance
`
`Features That Reduce Latency
`(cid:121) Fast pass-through-path in the IO cells
`– The data is not stored and forwarded
`(cid:121) DRAM synchronized to channel clock
`– No asynchronous clock boundary crossings
`(cid:121) Simultaneous Reads and Writes
`– Writes do not block read accesses
`(cid:121) Elimination of read-to-read dead time
`– Reads from different DIMMs have no dead time between
`data transfers
`
`New Channel Features Mitigate Buffer Latency
`
`19
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`FB-DIMM RAS Features
`
`20
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Reliability
`
`Channel Reliability Targets
`(cid:121) Intel’s server goal is >100 years (1142 FITs) per
`Silent Data Corruption
`– The memory channels receive a small portion of this
`total FIT budget
`(cid:121) FB-DIMM is architected for exceptional Silent Data
`Corruption prevention
`– Channel data is protected by a strong CRC
`– Per channel segment SDC FIT rate <0.10 (1,142,000
`years) to support even the highest-RAS servers
`
`21
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Reliability
`
`Improved RAS Features
`(cid:121) CRC protection for commands and data
`(cid:121) Optional bit widths and CRC coverage to cover
`wide range of applications
`(cid:121) Transient bit error detection and retry
`(cid:121) Bit lane fail-over “correction”
`(cid:121) Pass-through-path for high availability
`(cid:121) Hot Add while the channel is active
`(cid:121) Error registers in the buffer for improved fault
`isolation
`Comprehensive RAS Features Deliver a Reliable Solution
`
`22
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`FB-DIMM Longevity
`
`23
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Longevity
`
`FB-DIMM Momentum
`(cid:121) Industry is aligned around FB-DIMM
`– Server OEM suppliers recognize the need for a long-term
`buffered DRAM memory solution
`– DRAM & DIMM vendors committed to delivering FB-
`DIMM products: Samsung, Elpida, Infineon, Micron,
`Hynix, Nanya, Kingston, Smart
`– Complete set of tools for system development
`(cid:121) FB-DIMM is a long-term strategic direction
`– Smooth transition from DDR2 to DDR3 using the same
`connectors and topology
`– Industry investment in the technology leveraged over
`multiple generations
`
`24
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Longevity
`
`FB-DIMM Standardization
`(cid:121) Intel working with JEDEC to establish
`FB-DIMM as an industry standard
`(cid:121) Seven specifications being developed
`– FB-DIMM: Architecture & Protocol
`– FB-DIMM: High-speed Signaling
`– FB-DIMM: Connector (variation of DDR2)
`– FB-DIMM: Module
`– FB-DIMM: Advanced Memory Buffer (AMB)
`– FB-DIMM: SPD EPROM
`– FB-DIMM: Test & Verification
`
`25
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Longevity
`
`FB-DIMM Technology Roadmap
`(cid:121) DDR2 is the first technology intercept
`– Needed part-way through the DDR2 lifetime
`(cid:121) All DDR3 data rates supported
`
`?
`
`FB-DIMM DDR3
`DDR3
`
`FB-DIMM DDR2
`DDR2
`
`DDR
`
`01
`
`02
`
`03
`
`04
`
`05
`
`06
`
`07
`
`08
`
`09
`
`26
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Longevity
`
`FB-DIMM Cost Effectiveness
`(cid:121) Cost competitive environment
`– Multiple buffer vendors and DIMM suppliers
`(cid:121) Eliminates memory expansion hubs
`– Motherboard not burdened with hubs
`(cid:121) Reduces motherboard routing area
`– Only 24 pairs, no serpentine trace matching
`(cid:121) Enables reduced PCB layer count
`– FB-DIMM can be routed in one signal layer
`
`FB-DIMM is a cost effective long-term technology
`
`27
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Summary
`
`Summary
`(cid:121) FB-DIMM solves the long-term server memory
`capacity problem
`(cid:121) Buffer latency is managed with new channel
`features
`(cid:121) Comprehensive RAS features deliver a reliable
`solution
`(cid:121) FB-DIMM is a cost effective long-term technology
`
`FB-DIMM is The Server Memory DIMM
`of the Future
`
`28
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Fully Buffered DIMM Server Memory
`Architecture
`
`Pete Vogt
`
`Intel Corporation
`
`Any Questions?
`
`Please remember to turn in
`Please remember to turn in
`your session survey form.
`your session survey form.
`
`29
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Definitions
`(cid:121) AMB: Advanced Memory Buffer – used to
`implement FB-DIMM
`(cid:121) FB-DIMM: Fully Buffered DIMM
`(cid:121) FIT: Failure in Time (failures in 1 billion hours)
`(cid:121) JEDEC: Joint Electronic Devices Engineering
`Council
`(cid:121) RAS: Reliability/Availability/Serviceability
`(cid:121) RDIMM: Registered DIMM
`(cid:121) SDC: Silent Data Corruption
`
`31
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Fully Buffered DIMM Architecture
`
`(cid:121) Session Outline:
`(cid:121) Server Memory Capacity Problem & Solution
`– Description of the capacity problem facing servers as data rates go up. Graph of
`DRAMs/channel vs. data rate. Description of the high-level FB-DIMM architecture
`that addresses this problem utilizing commodity DRAM devices.
`– Key point: FB-DIMM solves the Server Memory Capacity problem
`(cid:121) FB-DIMM Performance Features
`– Description of architecture features to mitigate the buffer latency; synchronous
`operation, fast pass through path, simultaneous reads/writes, & elimination of
`read-to-read dead time. Loaded latency graph illustrates result.
`– Key point: Buffer latency is managed with new channel features
`(cid:121) FB-DIMM Reliability/Availability/Serviceability (RAS) Features
`– Highlight RAS features in FB-DIMM including bit lane fail-over, reduced pin
`counts, high-availability pass through path, deterministic behavior, strong
`channel CRC protection.
`– Key point: Comprehensive RAS features deliver a reliable solution
`(cid:121) Long-term Server Memory Solution
`– Describe the longevity of FB-DIMM from DDR2-533 to DDR3-1600, indicate
`multiple buffer vendor sources, working within JEDEC to establish as a standard.
`– Key point: FB-DIMM is a cost effective long-term technology
`
`Pete Vogt
`
`32
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`
`
`Title:
`Fully Buffered DIMM (FB-DIMM) Server Memory Architecture:
`Capacity, Performance, Reliability, and Longevity
`Abstract:
`What you’ll get from this session:
`– Trend data of the server memory capacity problem
`– Loaded latency curves illustrating FB-DIMM architecture
`features to mitigate inherent buffer delay
`– List of mechanisms that address FB-DIMM reliability
`concerns
`– Roadmap of multiple generations of DRAM support
`
`33
`
`Netlist Ex 2059
`Samsung v Netlist
`IPR2022-00996
`
`