Memory Systems
`Cache, DRAM, Disk
`
`Bruce Jacob
`University of Maryland at College Park
`
`Spencer W. Ng
`Hitachi Global Storage Technologies
`
`David T. Wang
`MetaRAM
`
`With Contributions By
`Samuel Rodriguez
`Advanced Micro Devices
`
`AMSTERDAM (cid:129) BOSTON (cid:129) HEIDELBERG LONDON
`NEW YORK (cid:129) OXFORD (cid:129) PARIS (cid:129) SAN DIEGO
`SAN FRANCISCO (cid:129) SINGAPORE (cid:129) SYDNEY (cid:129) TOKYO
`Morgan Kaufmann is an imprint of Elsevier
`
`Ml c:◄·
`
`NOII.IIAN
`
`KAUFMANN
`
`~UBLIIHEII.I
`
`ELSEVIER
`
`IPR2018-01600
`
`EXHIBIT
`2076
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 1
`
`

`

`Publisher
`Acquisitions Editor
`Publishing Services Manager
`Senior Production Editor
`Developmental Editor
`Assistant Editor
`Cover Design
`Text Design
`Composition
`Interior printer
`Cover printer
`
`Denise E.M. Penrose
`Chuck Glaser
`George Morrison
`Paul Gottehrer
`Nate McFadden
`Kimberlee Honjo
`Joanne Blank
`Dennis Schaefer
`diacriTech
`Maple-Vail Book Manufacturing Group
`Phoenix Color
`
`Morgan Kaufmann Publishers is an imprint of Elsevier.
`30 Corporate Drive, Suite 400, Burlington, MA 01803, USA
`This book is printed on acid-free paper.
`© 2008 by Elsevier Inc. All rights reserved.
`Designations used by companies to distinguish their products are often claimed as trademarks or registered trademarks. In all
`instances in which Morgan Kaufmann Publishers is aware of a claim, the product names appear in initial capital or all capital
`letters. Readers, however, should contact the appropriate companies for more complete information regarding trademarks and
`registration.
`No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic,
`mechanical, photocopying, scanning, or otherwise—without prior written permission of the publisher.
`Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone:
`(+44) 1865 843830, fax: (+44) 1865 853333, E-mail: permissions@elsevier.com. You may also complete your request online
`via the Elsevier homepage (http://elsevier.com), by selecting “Support & Contact” then “Copyright and Permission” and then
`“Obtaining Permissions.”
`
`Library of Congress Cataloging-in-Publication Data
`Application submitted
`
`ISBN: 978-0-12-379751-3
`For information on all Morgan Kaufmann publications,
`visit our Web site at www.mkp.com or www.books.elsevier.com
`Printed in the United States of America
`08 09 10 11 12 5 4 3 2 1
`
`Working together to grow
`libraries in developing countries
`www.elsevier.com | www.bookaid.org | www.sabre.org
`t?.?i,~,!~?i Sabre Foundation
`ELSEVIER
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 2
`
`

`

`DRAM Memory
`System Organization
`
`the basic building
`examine
`Previous chapters
`blocks of DRAM devices and signaling
`issues that
`constrain
`the transmission
`and subsequent
`storage
`of data into the DRAM devices. In this chapter, basic
`terminologies and building blocks of DRAM memory
`systems are described. Using the building blocks
`described
`in the previous chapters,
`the text in this
`chapter examines the construction, organization, and
`operation of multiple DRAM devices in a larger mem(cid:173)
`ory system. This chapter covers the terminologies
`and topology, as well as the organization of various
`types of memory modules.
`
`10. 1 Conventional Memory System
`The number of storage bits contained
`in a given
`DRAM device is constrained by the manufacturing
`process technology, the cell size, the array efficiency,
`and the effectiveness of the defect-cell
`remapping
`mechanism
`for yield enhancement. As the manu(cid:173)
`facturing process technology advances
`in line with
`Moore's Law, the number of storage bits contained
`in
`a given DRAM device doubles every few years. How(cid:173)
`ever, the unspoken corollary to Moore's Law states
`that software written by software companies
`in the
`Pacific Northwest and elsewhere will automatically
`expand
`to fill available memory
`in a given system.
`Consequently,
`the number of storage bits contained
`in a single DRAM device at any given instance
`in
`time has been and will continue
`to be inadequate
`to serve as the main memory
`for most computing
`platforms with the exception of specialty embedded
`systems.
`
`In the past few decades, the growth rate of DRAM
`device storage capacity has roughly paralleled
`the
`growth rate of the size of memory systems for desk(cid:173)
`top computers, workstations, and servers. The paral(cid:173)
`lel growth rates have dictated system designs in that
`multiple DRAM devices must be connected
`together
`to form memory systems in most computing plat(cid:173)
`forms. In this chapter, the organization of different
`multi-chip DRAM memory
`systems and different
`interconnection
`strategies deployed for cost and per(cid:173)
`formance concerns are explored.
`In Figure 10.1, multiple DRAM devices are intercon(cid:173)
`nected together to form a single memory system that
`is managed by a single memory controller. In modern
`computer systems, one or more DRAM memory con(cid:173)
`trollers (DMCs) may be contained
`in the processor
`package or integrated
`into a system controller
`that
`resides outside of the processor package. Regardless
`of the location of the DRAM memory controller, its
`functionality
`is to accept read and write requests to
`a given address in memory, translate
`the request to
`one or more commands
`to the memory system, issue
`those commands
`to the DRAM devices in the proper
`sequence and proper timing, and retrieve or store data
`on behalf of the processor or 1/0 devices in the sys(cid:173)
`tem. The internal structures of a system controller are
`examined in a separate chapter. This chapter focuses
`on the organization of DRAM devices in the context of
`multi-device memory systems.
`
`10.2 Basic Nomenclature
`The organization of multiple DRAM devices into a
`memory system can impact the performance of the
`
`409
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 3
`
`

`

`410 Memory Systems: Cache, DRAM, Disk
`
`Channel?
`
`Rank? Bank?
`
`Row?
`
`Column?
`
`Channel Address=?
`Rank Address= ?
`Bank Address= ?
`Row address=?
`Column Address =?
`
`Data - -
`GMemor~)
`-
`----
`
`Controller
`
`Command
`Sequence
`
`FIGURE 10.1: Multiple DRAM devices connected to a processor through a DRAM memory controller.
`
`memory system in terms of system storage capac(cid:173)
`ity, operating data rates, access latency, and sustain(cid:173)
`It is therefore of great
`able bandwidth characteristics.
`importance
`that the organization of multiple DRAM
`devices into larger memory systems be examined in
`detail. However, the absence of commonly accepted
`nomenclature has hindered the examination of DRAM
`memory-system
`organizations. Without a common
`basis of well-defined nomenclature,
`technical articles
`and data sheets sometimes succeed
`in introducing
`confusion
`rather
`than clarity into discussions on
`DRAM memory systems. In one example, a technical
`data sheet for a system controller used the word bank
`in two bulleted items on the same page to mean two
`different things. In this data sheet, one bulleted item
`proclaimed
`that the system controller could support
`6 banks (of DRAM devices). Then, several bulleted
`items later, the same data sheet stated that the same
`system controller could support SD RAM devices with
`4 banks. In a second example, an article in a well(cid:173)
`respected
`technical journal examined
`the then-new
`i875P system controller
`from Intel and proceeded
`to discuss the performance advantage of the system
`controller due to the fact that the i875P system con(cid:173)
`troller could control 2 banks of DRAM devices (it can
`control two entire channels).
`the word bank was used
`In these two examples,
`to mean three different
`things. While the meaning
`
`of the word bank can be inferred from the context
`in each case, the overloading and repeated use of
`the word
`introduces
`unnecessary
`confusion
`into
`discussions about DRAM memory systems. In this
`section,
`the usage of channel, rank, bank, row, and
`column is defined, and discussions
`in this and sub(cid:173)
`sequent chapters will conform
`to the usage in this
`chapter.
`
`10.2.1 Channel
`Figure 10.2 shows three different system control(cid:173)
`lers with slightly different configurations of the DRAM
`memory system. In Figure 10.2, each system con(cid:173)
`troller has a single DRAM memory controller (DMC),
`and each DRAM memory controller controls a single
`channel of memory. In the example labelled as the
`typical system controller, the system controller con(cid:173)
`trols a single 64-bit-wide channel. In modern DRAM
`memory systems, commodity DRAM memory mod(cid:173)
`ules are standardized with 64-bit-wide data busses,
`and the 64-bit data bus width of the memory mod(cid:173)
`ule matches
`the data bus width of the typical per(cid:173)
`sonal computer system controller. 1 In the example
`labelled as Intel i875P system controller, the system
`controller connects to a single channel of DRAM with
`a 128-bit-wide data bus. However, since commod(cid:173)
`ity DRAM modules have 64-bit-wide data busses,
`
`1Commodity memory modules designed for error correcting memory systems are standardized with a 72-bit-wide data bus.
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 4
`
`

`

`Chapter 10 DRAM MEMORY SYSTEM ORGANIZATION 411
`
`the i875P system controller requires matching pairs
`of 64-bit wide memory modules to operate with the
`128-bit-wide data bus. The paired-memory module
`configuration of the i875P is often referred
`to as a
`dual channel configuration. However, since there is
`only one memory controller, and since both memory
`modules operate
`in lockstep
`to store and retrieve
`data through the 128-bit-wide data bus, the paired(cid:173)
`memory module configuration
`is, logically, a 128-bit(cid:173)
`wide single channel memory system. Also, similar to
`SD RAM and DDR SD RAM memory systems, standard
`
`Direct RDRAM memory modules are designed with
`16-bit-wide data busses, and high-performance
`sys(cid:173)
`tem controllers
`that use Direct RDRAM, such as the
`Intel i850 system controller, use matched pairs of
`Direct RDRAM memory modules
`to form a 32-bit(cid:173)
`wide channel that operates in lockstep across the two
`physical channels of memory.
`that use a single
`In contrast
`to system controllers
`DRAM memory controller to control the entire mem(cid:173)
`ory system, Figure 10.3 shows that the Alpha EV7 pro(cid:173)
`cessor and the Intel i925x system controller each have
`
`"Typical"
`system controller
`
`DMC
`
`q4
`'
`
`DOR
`
`One "physical channel" of 64 bit width
`
`One DMC: One logical 64 bit wide channel
`
`Intel i875P
`system controller
`
`DMC
`I---"'-<,,---~~
`
`64~
`Two "physical channels" of 64 bit wide busses
`t2S ~ ~
`One DMC: One logical 128 bit wide channel
`64 ~
`
`Intel i850
`system controller
`
`16 ~
`Two "physical channels" of 16 bit width
`~2 ~ ~
`
`DMC ,__ ..... ,----
`One DMC: One logical 32 bit wide channel
`~
`
`FIGURE 10.2: Systems with a single memory controller and different data bus widths.
`
`DMC
`
`DMC
`
`64
`'
`"
`64
`'
`"
`
`I DDR2
`I
`
`I DDR2
`I
`
`I
`
`I
`
`0 (/)
`ti:l (/)
`..c Q)
`c.. (.)
`<i'. e
`c..
`0 I'--
`a..>
`IW
`
`64
`
`DMC
`
`DMC
`
`64
`
`D-RDRAM
`I D-RDRAM I
`D-RDRAM
`
`D-RDRAM
`
`D-RDRAM
`
`D-RDRAM
`
`D-RDRAM
`
`D-RDRAM
`
`Q)
`
`...
`e
`Xe
`L!) 8
`&! E
`ID u5
`-
`>,
`.!: (/)
`
`--
`
`Q)
`
`Two Channels: 64 bit wide per channel
`
`Two Channels: 64 bit wide per channel
`
`FIGURE 10.3: Systems with two independent memory controllers and two logical channels of memory.
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 5
`
`

`

`412 Memory Systems: Cache, DRAM, Disk
`
`control
`that independently
`two DRAM controllers
`data busses. 2 The use of indepen(cid:173)
`64-bit-wide
`dent DRAM memory controllers can lead to higher
`sustainable
`bandwidth
`characteristics,
`since
`the
`narrower channels
`lead to longer data bursts per
`cacheline
`request, and
`the various
`inefficiencies
`dictated by DRAM-access protocols can be better
`amortized. As a result, newer system controllers are
`often designed with multiple memory controllers
`despite the additional die cost.
`Modern memory
`systems with one DRAM
`memory
`controller
`and multiple physical chan(cid:173)
`nels of DRAM devices such as those illustrated
`in
`Figure 10.2 are typically designed with the physi(cid:173)
`cal channels operating
`in lockstep with respect
`to
`each other. However, there are two variations
`to the
`single-controller-multiple-physical-channel
`con(cid:173)
`figuration. One variation of the single-controller(cid:173)
`multiple-physical-channel
`configuration
`is
`that
`some system controllers, such as the Intel i875P sys(cid:173)
`tem controller, allow the use of mismatched pairs of
`memory modules in the different physical channels.
`In such a case, the i875P system controller operates
`
`in an asymmetric mode and independently
`controls
`the physical channels of DRAM modules. However,
`since there is only one DRAM memory controller, the
`multiple physical channels of mismatched memory
`modules cannot be accessed concurrently, and only
`one channel of memory can be accessed at any given
`instance
`in time. In the asymmetric configuration,
`the maximum
`system bandwidth
`is the maximum
`bandwidth of a single physical channel.
`A second
`variation
`of
`the
`single-controller(cid:173)
`multiple-physical-channel
`configuration
`can be
`found
`in high-performance
`FPM DRAM memory
`systems that were designed prior to the emergence of
`SD RAM-type DRAM devices that can burst out multi(cid:173)
`ple columns of data with a given column access com(cid:173)
`mand. Figure 10.4 illustrates a sample timing diagram
`of a column access in an SDRAM memory system.
`Figure 10.4 shows that an SDRAM device is able to
`return a burst of multiple columns of data for a single
`column access command. However, an FPM DRAM
`device
`supported
`neither
`single-access-multiple(cid:173)
`burst capability nor the ability to pipeline multiple
`column access commands. As a result, FPM DRAM
`
`C~
`datal
`
`I
`I I I I I I
`
`I
`
`I
`
`_,
`cmd_._l ___,_ __
`I tc~s ~u~st [e rlgtrl
`
`I
`
`I
`
`SDRAM memory bursts multiple
`columns of data (2) for each column
`access command (1 ).
`
`FPM DRAM returns one column of
`data (2) for each column access
`command (1 ). Column accesses
`cannot be pipelined. Solution: stagger
`column accesses to different physical
`channels of FPM DRAM devices
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`2
`
`I
`
`I
`
`data!
`--------'"'-<1'--
`1
`I
`I
`I
`
`I
`
`I
`
`I
`
`I
`I
`I
`'-------..,----
`tcAS
`
`DMC
`
`72
`
`FPM
`
`FPM
`
`FPM
`
`FPM
`
`FIGURE 10.4: High-performance memory controllers with four channels of interleaved FPM DRAM devices.
`
`2Ignoring additional bitwidths used for error correction and cache directory.
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 6
`
`

`

`Chapter 10 DRAM MEMORY SYSTEM ORGANIZATION 413
`
`to retrieve
`devices need multiple column accesses
`the multiple columns of data for a given cacheline
`access, column accesses that cannot be pipelined
`to
`a single FPM DRAM device.
`One solution deployed to overcome the shortcom(cid:173)
`ings of FPM DRAM devices is the use of multiple
`FPM DRAM channels operating
`in an interleaved
`fashion. Figure 10.4 also shows how a sophisticated
`FPM DRAM controller can send multiple column
`accesses
`to different physical channels of memory
`so that the data for the respective column accesses
`appears on the data bus in consecutive cycles. In this
`configuration,
`the multiple FPM DRAM channels can
`provide the sustained
`throughput
`required
`in high(cid:173)
`performance workstations
`and servers before
`the
`appearance of modern synchronous DRAM devices
`that can burst through multiple columns of data in
`consecutive cycles.
`
`10.2.2 Rank
`Figure 10.5 shows a memory system populated
`with 2 ranks of DRAM devices. Essentially, a rank of
`is a "bank" of one or more DRAM devices
`memory
`that operate in lockstep in response
`to a given com(cid:173)
`mand. However, the word bank has already been
`used to describe the number of independent DRAM
`arrays within a DRAM device. To lessen the confu(cid:173)
`sion associated with overloading
`the nomenclature,
`the word rank is now used to denote a set of DRAM
`devices that operate in lockstep to respond to a given
`command
`in a memory system.
`Figure 10.5 illustrates a configuration of 2 ranks
`of DRAM devices in a classical DRAM memory sys(cid:173)
`tem topology. In the classical DRAM memory system
`topology, address and command busses are con(cid:173)
`nected to every DRAM device in the memory system,
`
`address and command
`
`1-------,
`
`I
`
`data
`16
`
`data
`16
`
`data
`16
`
`data us
`16
`
`DMC
`
`chip-select O
`chip-select 1
`
`FIGURE 10.5: Memory system with 2 ranks of DRAM devices.
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 7
`
`

`

`414 Memory Systems: Cache, DRAM, Disk
`
`CKE
`
`CS#
`
`WE#_
`
`CU< ....
`-
`-
`-
`-
`-
`
`CAS#
`
`RAS#
`
`control
`logic
`
`"O
`C
`ti:l Q)
`E -o
`
`0 Q)
`(.) "O
`
`E 8 -7
`I refresh I
`counter
`
`mode
`register
`
`' l
`
`addr
`bus
`·I address~
`~ register I
`
`row
`--
`addr
`~ mux ~I
`
`rrnM
`1row
`I row
`row
`
`latch &
`decoder
`
`J,
`
`_ I bank
`- 1 control
`
`■ -
`address ~-
`
`s
`p •
`.... --
`
`■
`
`-
`
`I I I I I I_
`DRAM-
`Array =
`
`•
`• Y•
`sense amp array •
`in n
`
`■ y
`
`y ■
`
`' 1 '
`
`Ba nkO
`
`nk 1
`- Ba
`
`nk 2
`Ba
`
`nk 3
`Ba
`
`aata out I
`register
`'
`I data 1/0 I
`'
`data in I
`register
`
`-
`-
`
`address
`counter
`
`-
`
`read data latch
`write drivers
`
`. --j]]J
`' ' ' '
`
`column
`decoder
`
`FIGURE 10.6: SDRAM device with 4 banks of DRAM arrays internally.
`
`but the wide data bus is partitioned and connected
`to different DRAM devices. The memory control(cid:173)
`ler in this classical system topology then uses chip(cid:173)
`select signals to select the appropriate
`rank of DRAM
`devices to respond to a given command.
`In modern memory
`systems, multiple DRAM
`devices are commonly grouped
`together
`to provide
`the data bus width and capacity required by a given
`memory
`system. For example, 18 DRAM devices,
`each with a 4-bit-wide data bus, are needed in a given
`rank of memory
`to form a 72-bit-wide data bus. In
`contrast, embedded
`systems that do not require as
`much capacity or data bus width typically use fewer
`devices in each rank of memory-sometimes
`as few
`as one device per rank.
`
`10.2.3 Bank
`As described previously, the word bank had been
`used to describe a set of independent memory arrays
`inside of a DRAM device, a set of DRAM devices
`that collectively act
`in response
`to commands,
`
`and different physical channels of memory. In this
`chapter, the word bank is only used to denote a set of
`independent memory arrays inside a DRAM device.
`Figure 10.6 shows an SDRAM device with 4 banks
`of DRAM arrays. Modern DRAM devices contain mul(cid:173)
`tiple banks so that multiple, independent
`accesses to
`different DRAM arrays can occur in parallel. In this
`design, each bank of memory
`is an independent
`array that can be in different phases of a row access
`cycle. Some common
`resources, such as 1/0 gating
`that allows access to the data pins, must be shared
`between different banks. However, the multi-bank
`architecture allows commands such as read requests
`to different banks to be pipelined. Certain commands,
`such as refresh commands, can also be engaged in
`multiple banks in parallel. In this manner, multiple
`banks can operate
`independently
`or concurrently
`depending on the command. For example, multiple
`banks within a given DRAM device can be activated
`independently
`from each other-subject
`to the power
`constraints of the DRAM device that may specify how
`closely such activations can occur in a given period of
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 8
`
`

`

`Chapter 10 DRAM MEMORY SYSTEM ORGANIZATION 415
`
`time. Multiple banks in a given DRAM device can also
`be precharged or refreshed in parallel, depending on
`the design of the DRAM device.
`
`10.2.4 Row
`In DRAM devices, a row is simply a group of stor(cid:173)
`age cells that are activated in parallel in response
`to
`a row activation command.
`In DRAM memory sys(cid:173)
`tems that utilize the conventional
`system topology
`such as SDRAM, DDR SDRAM, and DDR2 SDRAM
`memory systems, multiple DRAM devices are typi(cid:173)
`cally connected
`in parallel in a given rank of mem(cid:173)
`ory. Figure 10.7 shows how DRAM devices can be
`connected
`in parallel to form a rank of memory. The
`effect of DRAM devices connected as ranks of DRAM
`devices that operate in lockstep is that a row activa(cid:173)
`tion command will activate the same addressed row
`in all DRAM devices in a given rank of memory. This
`arrangement means that the size of a row-from
`the
`perspective of the memory controller-is
`simply the
`size of a row in a given DRAM device multiplied by
`
`the number of DRAM devices in a given rank, and a
`DRAM row spans across the multiple DRAM devices
`of a given rank of memory.
`A row is also referred to as a DRAM page, since a
`row activation command
`in essence caches a page
`of memory at the sense amplifiers until a subse(cid:173)
`quent precharge command
`is issued by the DRAM
`memory controller. Various schemes have been pro(cid:173)
`posed to take advantage oflocality at the DRAM page
`level. However, one problem with the exploitation
`of locality at the DRAM page level is that the size of
`the DRAM page depends on the configuration of the
`DRAM device and memory modules, rather than the
`architectural page size of the processor.
`
`10.2.5 Column
`In DRAM memory systems, a column of data is
`the smallest addressable unit of memory. Figure 10.8
`illustrates
`that, in memory systems such as SD RAM
`and DDRx 3 SDRAM with topology similar
`to the
`memory system illustrated
`in Figure 10.5, the size of
`
`DRAM devices arranged in parallel in a given rank ~--------
`
`■
`
`■
`
`-
`
`I
`
`I
`I
`I
`I
`I_
`DRAM -
`Array =
`
`■
`
`■
`
`-
`
`I
`I
`I
`I
`I_
`DRAM -
`Array =
`
`• • •
`
`■
`
`■
`
`-
`
`I
`I
`I
`I
`I_
`DRAM -
`Array =
`
`■
`
`-~
`•
`-~
`•v• ~~ ■ y ■
`V
`
`1~11•
`sense amp a
`
`•
`
`senj.e amp array
`
`-~
`•
`• Y•
`"'-
`I ~i
`
`IY •
`
`sense amp array
`
`•
`
`I'
`■
`
`one row spanning multiple DRAM devices
`
`FIGURE 10.7: Generic DRAM devices with 4 banks, 8196 rows, 512 columns per row, and 16 data bits per column.
`
`3DDRx denotes DDR SDRAM and evolutionary DDR memory systems such as DDR2 and DDR3 SDRAM memory systems,
`inclusively.
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 9
`
`

`

`416 Memory Systems: Cache, DRAM, Disk
`
`DRAM devices arranged in parallel in a given rank
`
`•••
`
`SDRAM memory systems: width of data bus = column size
`
`FIGURE 10.8: Classical DRAM system topology; width of data bus equals column size.
`
`a column of data is the same as the width of the data
`bus. In a Direct RD RAM device, a column is defined
`as 16 bytes of data, and each read command
`fetches
`a single column of data 16 bytes in length from each
`physical channel of Direct RD RAM devices.
`A beatis simply a data transition on the data bus. In
`SD RAM memory systems, there is one data transition
`per clock cycle, so one beat of data is transferred per
`clock cycle. In DDRx SDRAM memory systems, two
`data transfers can occur in each clock cycle, so two
`beats of data are transferred
`in a single clock cycle.
`The use of the beat terminology avoids overloading
`the word cycle in DDRx SD RAM devices.
`In DDRx SDRAM memory systems, each column
`access command
`fetches multiple columns of data
`depending on the programmed
`burst
`length. For
`example, in a DDR2 DRAM device, each memory read
`command
`returns a minimum of 4 columns of data.
`The distinction between a DDR2 device returning a
`minimum burst length of 4 beats of data and a Direct
`RD RAM device returning a single column of data over
`8 beats is that the DDR2 device accepts the address of
`a specific column and returns the requested columns
`in different orders depending on the programmed
`behavior of the DRAM device. In this manner, each
`column is separately addressable. In contrast, Direct
`RDRAM devices do not reorder data within a given
`burst, and a 16-byte burst from a single channel of
`
`Direct RDRAM devices is transmitted
`treated as a single column of data.
`
`in order and
`
`10.2.6 Memory System Organization: An
`Example
`Figure 10.9 illustrates a DRAM memory system with
`4 ranks of memory, where each rank of memory consists
`of 4 devices connected in parallel, each device contains
`4 banks of DRAM arrays internally, each bank contains
`8192 rows, and each row consists of 512 columns of
`data. To access data in a DRAM-based memory sys(cid:173)
`tem, the DRAM memory controller accepts a physical
`address and breaks down the address into respective
`address fields that point to the specific channel, rank,
`bank, row, and column where the data is located.
`Although Figure 10.9
`illustrates
`a uniformly
`organized memory system, memory system orga(cid:173)
`nizations of many computer
`systems, particularly
`end-user
`configurable
`systems, may be typically
`non-uniformly organized. The reason that the DRAM
`memory systems organizations
`in many computer
`systems are typically non-uniform
`is because most
`computer
`systems are designed
`to allow end-users
`to upgrade
`the capacity of the memory system by
`inserting and removing commodity memory mod(cid:173)
`ules. To support memory capacity upgrades by the
`end-user, DRAM controllers have to be designed
`to
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 10
`
`

`

`Chapter 10 DRAM MEMORY SYSTEM ORGANIZATION 417
`
`bank ID= 1
`
`column ID= 0x187
`
`Memory System
`
`FIGURE 10.9: Location of data in a DRAM memory system.
`
`to different configurations of DRAM
`flexibly adapt
`devices and modules
`that the end-user could place
`into the computer system. This support
`is provided
`for through the use of address range registers whose
`functionality
`is examined separately
`in the chapter
`on memory controllers.
`
`10.3 Memory Modules
`The first generations of computer systems allowed
`end-users
`to increase memory capacity by providing
`sockets on the system board where additional DRAM
`devices could be inserted. The use of sockets on the
`system board made sense in the era where the price
`of DRAM devices was quite expensive relative to the
`cost of the sockets on the system board. In these
`early computer
`systems, system boards were typi(cid:173)
`cally designed with sockets that allowed end-users
`to remove and insert individual DRAM devices, usu(cid:173)
`ally contained
`in dual in-line packages
`(DIPs). The
`process of memory upgrade was cumbersome
`and
`
`difficult, as DRAM devices had to be individually
`removed and inserted
`into each socket. Pins on the
`DRAM devices may have been bent and not visu(cid:173)
`ally detected as such. Defective DRAM chips were
`difficult to locate, and routing of sockets for a large
`memory system required
`large surface areas on the
`system board. Moreover, it was physically possible
`to place DRAM devices in the wrong orientation
`in
`the socket-180°
`from the intended placement. Cor(cid:173)
`rect placement with proper orientation depended on
`clearly labelled sockets, clearly labelled devices, and
`an end-user
`that paid careful attention while insert(cid:173)
`ing the devices into the sockets. 4 The solution to the
`problems associated with memory upgradability was
`the creation and use of memory modules.
`Memory modules are essentially miniature system
`boards that hold a number of DRAM devices. Mem(cid:173)
`ory modules provide an abstraction at the module
`interface so that different manufacturers
`can manu(cid:173)
`facture memory upgrades for a given computer sys(cid:173)
`tem with different DRAM devices. DRAM memory
`
`4The author of this text can personally attest to the consequences of inserting chips into sockets with incorrect orientation.
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 11
`
`

`

`418 Memory Systems: Cache, DRAM, Disk
`
`modules also reduce the complexity of the memory
`upgrade process. Instead of the removal and inser(cid:173)
`tion of individual DRAM chips, memory upgrades
`with modules containing multiple DRAM chips can
`be quickly and easily inserted into and removed from
`a module socket. The first generations of memory
`modules
`typically consisted of specially created,
`system-specific memory modules
`that a given com(cid:173)
`puter manufacturer used in a given computer
`sys(cid:173)
`tem. Over the years, memory modules have obtained
`a level of sophistication, and they are now specified
`as a part of the memory-system definition process.
`
`10.3.1 Single In-line Memory Module (SIMM)
`In the late 1980s and early 1990s, the personal
`computer
`industry
`first standardized
`on the use
`of 30-pin SIMMs and then
`later moved
`to 72-pin
`SIMMs. SIMMs, or Single In-line Memory Modules,
`are referred to as such due to the fact that the con(cid:173)
`tacts on either side of the bottom of the module are
`electrically identical.
`to 8 or 9
`A 30-pin SIMM provides interconnects
`signals on the data bus, as well as power, ground,
`address,
`command,
`and chip-select
`signal
`lines
`between
`the system board and the DRAM devices.
`A 72-pin SIMM provides interconnects
`to 32 to 36 sig(cid:173)
`nals on the data bus in addition to the power, ground,
`address, command, and chip-select signal lines. Typ(cid:173)
`ically, DRAM devices on a 30 pin, 1 Megabyte SIMM
`collectively provide a 9-bit, parity protected data bus
`interface to the memory system. Personal computer
`systems in the late 1980s typically used sets of four
`matching 30-pin SIMMs to provide a 36-bit-wide
`memory interface to support parity checking by the
`memory controller. Then, as the personal computer
`system moved
`to support memory
`systems with
`wider data busses, the 30- pin SIMM was replaced by
`72-pin SIMMs in the early 1990s.
`
`10.3.2 Dual In-line Memory Module (DIMM)
`In the late 1990s, as the personal computer
`indus(cid:173)
`try transitioned
`from FPM/EDO DRAM to SDRAM,
`72-pin SIMMs were, in turn, phased out in favor of
`Dual In-line Memory Modules (DIMMs). DIMMs
`are physically
`larger
`than SIMMs and provide a
`
`64- or 72-bit-wide data bus interface to the memory
`system. The difference between a SIMM and a DIMM
`is that contacts on either side of a DIMM are elec(cid:173)
`trically different. The electrically different contacts
`allow a denser routing of electrical signals from the
`system board through the connector
`interface to the
`memory module.
`for the commodity
`Typically, a DIMM designed
`desktop market contains
`little more than the DRAM
`devices and passive resistor and capacitors. These
`DIMMs are not buffered on either the address path
`from the memory controller
`to the DRAM devices
`or the datapath between the DRAM devices and the
`memory controller. Consequently,
`these DIMMs are
`also referred to as UnbujferedDIMMs (UDIMMs).
`
`10.3.3 Registered Memory Module (RDIMM)
`To meet the widely varying requirements of sys(cid:173)
`tems with end-user configurable memory systems,
`memory modules of varying capacity and timing
`characteristics are needed
`in addition
`to the typical
`UDIMM. For example, workstations and servers typi(cid:173)
`cally require larger memory capacity than those seen
`for the desktop computer systems. The problem asso(cid:173)
`ciated with large memory capacity memory modules
`is that the large number of DRAM devices in a mem(cid:173)
`ory system tends to overload the various multi-drop
`busses. The large number of DRAM devices, in turn,
`creates the loading problem on the various address,
`command, and data busses.
`Registered Dual
`In-line Memory Modules
`(RDIMMs) alleviate the issue of electrical loading of
`large numbers of DRAM devices in a large memory
`system through
`the use of registers that buffer the
`address and control signals at the interface of the
`memory module. Figure 10.10 illustrates
`that regis(cid:173)
`tered memory modules use registers at the interface
`of the memory module
`to buffer the address and
`control signals. In this manner,
`the registers greatly
`reduce the number of electrical loads that a memory
`controller must drive directly, and the signal inter(cid:173)
`connects in the memory system are divided into two
`separate segments: between
`the memory controller
`and the register and between the register and DRAM
`devices. The segmentation allows timing characteris(cid:173)
`tics of the memory system to be optimized by limiting
`
`PATENT OWNER DIRECTSTREAM, LLC
`EX. 2090, p. 12
`
`

`

`Chapter 10 DRAM MEMORY SYSTEM ORGANIZATION 419
`
`the number of electrical loads, as well as by reduc(cid:173)
`ing the path lengths of the critical control signals in
`individual segments of the memory system. However,
`the drawback to the use of the registered latches on a
`memory module is that t