United States Patent [19J
`Bruce et al.
`
`[54] EXPANDABLE FLASH-MEMORY MASS(cid:173)
`STORAGE USING SHARED BUDDY LINES
`AND INTERMEDIATE FLASH-BUS
`BETWEEN DEVICE-SPECIFIC BUFFERS
`AND FLASH-INTELLIGENT DMA
`CONTROLLERS
`
`[75]
`
`Inventors: Ricardo H. Bruce, Union City;
`Rolando H. Bruce, South San
`Francisco; Earl T. Cohen, Fremont, all
`of Calif.
`
`[73] Assignee: Bit Microsystems, Inc., Fremont, Calif.
`
`[21] Appl. No.: 939,601
`
`[22] Filed:
`
`Sep. 29, 1997
`
`Related U.S. Application Data
`
`[63] Continuation-in-part of Ser. No. 918,203, Aug. 25, 1997.
`Int. Cl.6
`..................................................... GllC 13/00
`[51]
`[52] U.S. Cl. ................................ 365/185.33; 365/189.01;
`365/218
`[58] Field of Search ......................... 365/189.01, 230.01,
`365/185.33, 218
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,752,871
`5,297,148
`5,379,401
`5,388,083
`5,432,748
`5,448,577
`5,479,638
`5,500,826
`5,509,134
`5,513,138
`5,524,231
`5,530,828
`
`6/1988 Sparks ................................ 365/189.01
`3/1994 Harari et al. ........................... 371/10.2
`1/1995 Robinson et al.
`...................... 395/425
`2/1995 Assar et al. ............................. 365/218
`7/1995 Hsu et al.
`.......................... 365/230.01
`9/1995 Wells et al. ............................ 371/10.1
`12/1995 Assar et al. ............................. 395/430
`3/1996 Hsu et al.
`.......................... 365/230.01
`4/1996 Fandrich et al. ........................ 395/430
`4/1996 Manabe et al. .................... 365/185.33
`6/1996 Brown ..................................... 395/428
`6/1996 Kaki et al. .............................. 395/430
`
`I 1111111111111111 11111 111111111111111 111111111111111 lllll 111111111111111111
`US005822251A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,822,251
`Oct. 13, 1998
`
`5,535,328
`5,559,956
`5,568,439
`5,572,466
`5,594,883
`5,602,987
`5,603,001
`5,606,529
`5,619,470
`5,627,783
`5,640,349
`
`7/1996 Harari et al. ....................... 395/182.05
`9/1996 Sukegawa .......................... 395/182.06
`10/1996 Harari ..................................... 365/218
`11/1996 Sukegawa .......................... 365/185.33
`1/1997 Pricer ...................................... 395/440
`2/1997 Harari et al. ....................... 395/182.06
`2/1997 Sukegawa et al. ..................... 395/430
`2/1997 Honma et al. ..................... 365/230.01
`4/1997 Fukumoto ............................... 365/228
`5/1997 Miyauchi ........................... 365/185.33
`6/1997 Kakinuma et al. ................ 365/185.33
`
`Primary Examiner-Terrell W. Fears
`Attorney, Agent, or Firm-Stuart T. Auvinen
`
`[57]
`
`ABSTRACT
`
`A flash-memory system is expandable. Rather than directly
`connecting individual flash-memory chips to a controller,
`flash buffer chips are used. Each flash buffer chip can
`connect to four banks of flash-memory chips. Chip enables
`for individual chips in a bank are generated from an address
`sent to the flash buffer chips. Two flash-specific DMA
`controllers are provided, each with four DMA state
`machines for controlling the four banks of flash-memory
`chips attached to a flash buffer chip. This allows for four(cid:173)
`way interleaving. Two flash buses connect the two DMA
`controllers to flash buffer chips. The flash bus has a narrow
`byte-wide interface to send command, address, and data
`bytes from the DMA controller to the flash buffer chips.
`These command, address, and data bytes are then passed
`through the flash buffer chip to the flash-memory chips. Two
`additional command signals on the flash bus are used to
`select and control the flash buffer chips. Busy signals from
`all flash-memory chips in a bank are connected together, and
`the four busy signals from the four banks are time(cid:173)
`multiplexed to a single common busy line for the flash bus.
`The four DMA state machines each monitor one period of
`the busy line, allowing four flash operations to be monitored
`at a time, even though only one data transfer can occur
`across the flash bus.
`
`24 Claims, 8 Drawing Sheets
`
`FLASH
`BUFFER
`CHIP M
`
`FLASH
`BUFFER
`CHIP M
`
`FLASH
`BUS-A
`
`10
`
`18
`
`FLASH
`BUFFER
`CHIP M
`
`FLASH
`BUFFER
`CHIP M
`
`FLASH
`BUS-B
`
`FLASH
`FLASH
`BUFFER
`BUFFER
`CHIP M
`CHIP M
`,------ ----------- ---------,
`
`I
`FLASH-
`:
`: SPECIFIC
`I
`DMA 12
`:
`I•-••-• - • • • • • • • • • • • • • • • • • • -1
`
`I
`:
`FLASH-
`:
`SPECIFIC
`I
`DMA 1§ 50:
`
`DRAM
`CACHE
`~
`
`LOCAL BUS
`
`SCSI
`CTLR
`
`LOCAL
`PROC
`MEM 'll
`
`LOCAL
`PROC 2§
`
`30
`
`ROM
`
`~
`
`34
`
`36
`
`INTEL-1006
`8,010,740
`
`

`

`~
`Ul
`N
`....
`N
`N
`00
`....
`Ul
`
`00
`0
`....,
`:
`[
`'Jl
`
`~ \0
`~
`~
`p.
`o
`
`00
`
`~ =
`~
`~
`r'1
`d •
`
`-
`
`DEVICES
`MEM
`FLASH
`
`BSY
`
`I
`
`BSY
`
`CHIP 141
`BUFFER
`FLASH
`
`FIG. 1
`
`CHIP 141-----1--,....--
`BSY _...:~
`
`L---~:::r-~--~
`
`BSY
`
`..,_ ___ ....
`1---1----..J
`
`FLASH
`
`18
`
`BUS
`FLASH
`
`SPECIFIC t-------------' BUFFER
`FLASH-
`
`16
`
`OMA
`
`BSY
`
`,....-..
`
`BSY
`
`-11111-----1
`CHIP 141--------1
`BUFFER
`
`10
`
`~
`
`F~~H
`
`SPECIFIC 1-----~----1--1 FLASH
`
`BSY
`
`--~=r-~---1
`CHIP 14.,__ __ __..-----.....-
`BUFFER
`FLASH
`
`BSY
`
`.,._ ___ _.
`
`I I
`I
`
`I
`
`12
`
`I
`
`22 I
`
`FLASH-I
`
`OMA
`
`

`

`~
`Ul
`N
`....
`N
`N
`00
`....
`Ul
`
`00
`0 ....,
`N
`~ ....
`'JJ. =(cid:173)~
`
`00
`\0
`'"""'
`\0
`'"""' ~~
`!""'"
`I")
`0
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`~ < 36
`
`HOST
`
`.......
`
`BSY ~
`
`~ 20
`
`-
`
`~ 20 -
`
`BSY ~
`
`~ 20 -
`
`-
`
`BSY ~
`
`-
`
`-
`
`34
`~
`
`CTLR
`SCSI
`
`FIG. 2
`30
`1
`
`28
`
`ROM
`
`PROC 26 -
`
`LOCAL
`
`MEM 27 -
`
`PROC
`LOCAL
`
`LOCAL BUS
`
`CHIP 14
`BUFFER
`FLASH
`
`-
`
`CHIP 14
`BUFFER
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`18
`"'
`
`10
`"'
`
`BUS-A
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`BUS-B
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`-
`
`20 §i'
`~ BSY -
`
`BSY
`
`20 §r
`~
`
`-
`
`-
`
`20 §r
`~ BSY -
`
`-
`
`-
`
`-
`
`-
`
`,------------------------------.
`
`-1
`
`16 so:
`
`OMA
`
`12
`
`OMA
`
`.. _______ ------------· ----------
`
`SPECIFIC
`FLASH-
`
`SPECIFIC
`FLASH-
`
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`32
`CACHE
`DRAM
`
`

`

`~
`Ul
`N
`....
`N
`N
`00
`....
`Ul
`
`~
`
`00
`0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`00
`\0
`'"""'
`\0
`'"""' ~~
`!""'"
`I")
`0
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`50
`
`FIG. 3
`
`--
`
`1'
`
`42
`
`41
`
`FIFO B
`
`-
`
`FIFO A
`
`-
`
`-
`
`44
`
`PARITY
`
`-
`
`DRAM CACHE
`TO/FROM
`
`-
`
`---------------------------------
`
`"' CHK
`ECG
`-I> FLASH BUS-B
`
`18
`)
`
`"'
`95 CHK
`ECC
`
`85 97
`
`-
`
`r--,.
`
`10
`)
`
`FLASH BUS-A
`
`83
`~
`--t>
`....
`
`-
`
`ECC GEN
`
`93)
`
`CMOS
`
`~
`
`91
`
`ECG GEN -
`
`CMOS
`
`.
`
`.
`
`48
`
`LOGIC
`CTL
`BUS-B
`
`40"-BANK3 SM
`
`40"-BANK2 SM
`
`40"-BANK1SM
`
`40"-BANKO SM
`---------------------------------
`---------------------------------
`40" BANK3 SM
`46
`
`--
`
`LOGIC
`CTL
`BUS-A
`
`40" BANK2 SM
`
`40" BANK1SM
`
`40" BANKO SM
`
`12
`
`

`

`~
`Ul
`N
`....
`N
`N
`00
`....
`Ul
`
`00
`0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`,i;;..
`
`00
`\0
`'"""'
`\0
`'"""' ~~
`!""'"
`I")
`0
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`ALE0:1
`
`• CHIPS
`1--1 76 I WP0:1 MEM
`FLASH
`WE0:4 TO
`
`•
`
`RE0:4 ..
`
`___,J
`74 -
`
`...a 72
`
`CE0:7
`
`I LOCKr
`
`I
`
`64
`
`t' 54
`
`2-BIT CNTR
`
`.I GROUP
`62"'1 START
`
`µ
`I
`
`BUSY0
`
`__,_
`
`BUSY3
`BUSY2
`
`56
`
`~t:= BUSY1
`
`FACE
`INTER(cid:173)
`BUS
`FLASH
`
`..,
`..
`
`FBCMD0:1
`
`RESET
`
`WP
`
`14
`
`FB-~SY 1
`
`-... 79 I DATA0:7
`...
`
`,_
`
`781 CLE0~1
`
`-
`
`H
`
`52
`
`FIG. 4
`
`7~=--te
`
`FBDATA0:7 ~
`
`

`

`~
`Ul
`N
`....
`N
`N
`00
`....
`Ul
`
`00
`0 ....,
`Ul
`~ ....
`'JJ. =(cid:173)~
`
`00
`\0
`'"""'
`\0
`'"""' ~~
`!""'"
`I")
`0
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`TIME
`
`FIG. 5
`
`OMA 12
`
`SPECIFIC
`FLASH(cid:173)
`
`I Bsvo I BSY 1 I Bsv2 I BSY3 I Bsvo I
`
`82~
`
`88~
`
`86~
`
`84~
`
`82~
`
`BUSY3
`
`FB-BSY
`
`20 1120 1120 1120 1120 1120 1120 1120
`
`20 1120 1120 1120 1120 1120 1120 1120
`
`56
`
`120 1120 1120 1120 1120 1120 1120 1120
`
`20 1120 1120 1120 1120 1120 1120 1120
`
`BUSY0
`
`I
`: /
`14
`~-------, I
`
`FIG. 6
`
`CHIP 14
`BUFFER
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 13, 1998
`Oct. 13, 1998
`
`Sheet 6 of 8
`Sheet 6 of 8
`
`5,822,251
`5,822,251
`
`YAHLOOAY
`
`QWOYSHLO
`
`(9
`LL
`
`LIVMCGd
`
`QWOLYVLS
`
`LHYVLSOAY
`
`AsVdsdsY
`
`ASvesAy1¢d
`
`ASddauy
`
`quoswy
`
`Ce
`
`ASYM
`
`QNOdT
`
`GQW903y
`
`Carvin
`
` Vivddd
`
`IVLSO3Y
`
`QYLVLS
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 13, 1998
`Oct. 13, 1998
`
`Sheet 7 of 8
`Sheet 7 of 8
`
`5,822,251
`5,822,251
`
`8Old
`
`co
`(9
`LL
`
`AOOINN
`
`0
`~
`l)
`::I:
`
`~ LJ..
`
`Cl) n::
`0
`:5 w
`
`Cl)
`
`O1sYdav
`
`0
`....J
`Cl) n::
`0
`<(
`
`0
`~
`Cl)
`0::
`0
`<(
`
`::I:
`Cl) n::
`0
`<(
`
`ASVYsA1AS
`
`QWOsASVYyS
`
`w
`z
`0
`0
`
`IHOWOLAS,OTQWOLAS
`
`Vivdlas
`
`LIVMAYM
`
`VIVGYM
`
`90d14S
`
`QW990ud
`
`0
`~
`l)
`C)
`0
`0:: a..
`
`
`
`
`
`
`
`

`

`~
`Ul
`N
`~
`QC
`....
`Ul
`
`00
`0 ....,
`00
`~ ....
`'JJ. =(cid:173)~
`
`00
`\0
`'"""'
`\0
`'"""' ~~
`!""'"
`I")
`0
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`I
`I
`
`FLASH BUS-B
`
`18
`~
`
`FLASH BUS-A
`
`I
`
`I
`
`92
`
`CHIP 14
`BUFFER
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`BUFFER
`FLASH
`
`CHIP 14
`BUFFER
`FLASH
`
`I
`I '10
`-------·----------~-------
`90
`---I
`CHIP 14
`
`FLASH-I
`
`OMA 16
`
`SPECIFIC
`
`OMA ~
`
`I
`
`SPECIFIC
`FLASH-
`
`FIG. 9
`
`,-------------------------
`
`,-------------------------
`
`

`

`5,822,251
`
`1
`EXPANDABLE FLASH-MEMORY MASS(cid:173)
`STORAGE USING SHARED BUDDY LINES
`AND INTERMEDIATE FLASH-BUS
`BETWEEN DEVICE-SPECIFIC BUFFERS
`AND FLASH-INTELLIGENT DMA
`CONTROLLERS
`
`RELATED APPLICATION
`
`This application is a continuation-in-part of the
`co-pending application for "Unified Re-Map and Cache(cid:173)
`Index Table with Dual Write-Counters for Wear-Leveling of
`Non-Volatile Flash RAM Mass Storage", filed Aug. 25,
`1997, U.S. Ser. No. 08/918,203.
`
`BACKGROUND OF THE INVENTION
`
`15
`
`2
`completed. The busy signal allows a local processor to
`continue with other tasks while the flash-memory integrated
`circuit chip performs the flash operation. Flash operations
`generally cannot be stopped once initiated. Erase operations
`5 are the exception-they can be suspended and resumed later,
`but generally only one erase operation per flash-memory
`chip can be active or suspended at a time.
`Busy signals complicate expansion, as the local processor
`must receive all the busy signals. As more flash-memory
`10 devices are added to the system, the local processor must
`receive additional busy signals from the additional chips.
`While all the busy signals can be connected together, this
`can limit performance as parallel flash operations driving the
`same busy line can interfere with each other, requiring that
`the local processor read status registers on all the flash chips
`when any device asserts the common busy line. Connecting
`all the busy signals together generally also means that the
`busy line will stay asserted (busy) as long as any device is
`busy-this limits parallelism by not providing separate
`indications of when each flash-memory chip has completed
`an operation.
`DMA for Data Transfers
`Direct-memory access (DMA) has been used to facilitate
`data transfers. While DMA is efficient for transfers of raw
`data to a memory, flash-memory chips also require com(cid:173)
`mand and address sequences to set up the relatively long
`flash operations. DMA is not well-suited to transfer
`addresses and commands since it is designed to transfer long
`strings of data beginning at a starting address through an
`30 ending address.
`These command and address sequences are inputted to the
`flash-memory chips through shared data 1/0 pins, and gen(cid:173)
`erally require special control signals to be asserted to
`distinguish whether the shared data 1/0 pins contain data,
`35 address, or command information. Often only 8 data 1/0
`pins are provided, limiting transfers to one byte of a
`command, address, or data per cycle. Thus one or two cycles
`are needed for each command and two or three cycles are
`needed for each address inputted to the flash-memory chips.
`It is desired to modify a DMA controller so that com-
`mands and addresses can also be input to a flash-memory
`chip over the shared data/address/command 1/0 pins. It is
`also desired to allow for expansion of storage capacity of a
`flash-memory system. A high-performance flash-memory
`45 system performing flash operations in parallel is desirable,
`even when expansion flash chips are added.
`SUMMARY OF THE INVENTION
`An expandable flash-memory system has a host interface
`that receives requests for access of flash memory. A flash(cid:173)
`specific direct-memory access (DMA) controller controls
`access of the flash memory. The flash-specific DMA con(cid:173)
`troller generates a sequence of command bytes followed by
`address bytes to initiate a data transfer from the flash
`memory.
`A flash bus is coupled to the flash-specific DMA control(cid:173)
`ler. The flash bus has shared lines that transmit the sequence
`of command bytes and address bytes from the flash-specific
`DMA controller. The shared lines also transmit data bytes
`from the flash-specific DMAcontroller for the host interface.
`Flash buffer chips are coupled to the flash bus. They
`generate control signals that control flash-memory chips.
`The flash buffer chips pass the sequence of command bytes
`and address bytes from the flash-specific DMA controller to
`65 the flash memory.
`A plurality of flash-memory chips are arranged in banks
`that share some of the control signals from a flash buffer
`
`20
`
`1. Field of the Invention
`This invention relates to non-volatile memory storage
`systems, and more particularly to expansion of flash
`memory systems.
`2. Description of the Related Art
`Hard disks have been the traditional storage medium in
`computer systems. More recently these hard disks are being
`replaced by non-volatile semiconductor memory. An array
`of non-volatile random-access memories (NVRAM's) or 25
`non-volatile flash memories substitutes for the hard-disk
`storage. These memory devices use electrically-erasable
`programmable read-only-memory (EEPROM) technology
`for storage cells. Floating polysilicon gates in these storage
`cells retain charge and state when power is lost, providing
`non-volatile storage. These flash-memory systems are fre(cid:173)
`quently used as a mass-storage replacement for a hard disk
`on a personal computer and are thus sometimes referred to
`as a flash "disk", even though no rotating disks are used.
`Hard disks suffer from their inherent fixed size, since
`expansion of the storage capacity of a hard disk is not
`possible. An entire new disk assembly is needed to expand
`disk storage. Like DRAM memory, flash memory has the
`potential for expansion. However, current flash-memory
`systems have not fully realized the potential for expandabil- 40
`ity. Rigid bus architectures often restrict expansion of flash(cid:173)
`memory systems. Like hard disks, the expansion of these
`flash-memory systems is limited.
`Slow Flash Operations
`Flash EEPROM chips are divided into pages and blocks.
`A 64 Mbit flash chip typically has 512-byte pages which
`happens to match the sector size for IDE and SCSI hard
`disks. Rather than writing to or reading from just one word
`in the page, the entire page must be read or written at the
`same time; individual bytes cannot be written. Thus flash(cid:173)
`memory operations are inherently slow since an entire page
`must be read or written.
`Each page must be cleared of any previous data before
`being written; clearing is accomplished by a flash erase 55
`cycle. An entire block of pages (typically 16 pages) is erased
`at once. A block of 16 pages must be erased together, while
`all 512 bytes on a page must be written together. Erase is
`therefore significantly slower than read or write operations.
`Interleaving on a page basis increases the effective block 60
`size since several banks are grouped together. For example,
`a 4:1 page-interleaved system has an effective block size of
`4 blocks. The larger block size can further increase erase
`time. Interleaving is preferably done on a block basis that
`doesn't increase the effective block size.
`A busy signal is usually provided by each flash-memory
`chip to indicate when the read, write, or erase operation has
`
`50
`
`

`

`5,822,251
`
`4
`A plurality of other flash-memory chips are arranged into
`banks. The flash-memory chips are non-volatile semicon(cid:173)
`ductor memory chips that retain data when power is lost.
`Other flash buffer chips are coupled to the first flash bus.
`5 Each flash buffer chip is coupled to a different plurality of
`banks of the flash-memory chips.
`A second flash-specific DMA controller is coupled to the
`local processor. It generates command, address, and data
`sequences to a second flash-memory chip in a format
`required by the second flash-memory chip. A second flash
`bus is coupled to the second flash-specific DMA controller.
`It transfers the data, addresses, and commands over shared
`address/data/command lines. A second flash buffer chip is
`coupled to the second flash bus. It sends the data, addresses,
`and commands from the shared lines of the second flash bus
`to the second flash-memory chip. Other flash buffer chips are
`coupled to the second flash bus. Each flash buffer chip is
`coupled to a different plurality of banks of the flash-memory
`chips. Thus two flash-specific DMA controllers control
`access of flash-memory chips through flash buffer chips
`connected by two flash buses.
`
`15
`
`20
`
`3
`chip. The flash-memory chips are non-volatile
`semiconductor-memory chips that retain data when power is
`no longer applied.
`The flash bus accepts additional flash-memory chips
`attached through the flash buffer chips to expand a storage
`capacity of the expandable flash-memory system. Thus the
`flash-memory chips are buffered by the flash buffer chips to
`the flash bus and the flash-specific DMA controller.
`In further aspects of the invention the flash bus has
`command lines to indicate transmission of a select code to 10
`the flash buffer chips. The flash buffer chips each have a
`select decoder that receives the select code. Each flash buffer
`chip generates a plurality of chip-enable control signals to
`the flash-memory chips. One of the chip-enable control
`signals is activated to enable a selected flash-memory chip
`while other chip-enable control signals disable other flash(cid:173)
`memory chips. Thus the chip-enable control signals are
`generated by the flash buffer chips from the select code from
`the flash-specific DMA controller.
`In further aspects of the invention each flash-memory chip
`outputs a busy signal to indicate when a flash operation is in
`progress within the flash-memory chip. All busy signals
`from flash-memory chips in a bank that is connected to a
`flash buffer chip are connected to a common bank-busy line.
`The bank-busy line for each bank is input to the flash buffer
`chip.
`The flash buffer chip also has a busy combining means
`that receives the bank-busy signals from all banks of flash(cid:173)
`memory chips attached to the flash buffer chip. It combines
`the bank-busy signals into a merged busy signal output to the 30
`flash bus. Thus busy signals from individual flash-memory
`chips are combined by the flash buffer chips.
`In still further aspects the merged busy signals from all
`flash buffer chips on the flash bus are connected to a
`common flash-bus busy line. The flash-specific DMA con- 35
`troller monitors the common flash-bus busy line to deter(cid:173)
`mine when the flash operation is in progress in the flash(cid:173)
`memory chip. Thus all busy signals are combined into a
`single busy line to the flash-specific DMA controller.
`In further aspects of the invention a second flash bus is 40
`coupled to a second flash-specific DMAcontroller. A second
`plurality of flash buffer chips are coupled to the second flash
`bus. Each flash buffer chip in the second plurality generates
`control signals and buffers command and address bytes from
`the second flash bus to additional banks of flash buffer chips. 45
`Thus the second flash bus accesses the additional banks of
`flash buffer chips.
`In other aspects of the invention a flash-memory storage
`peripheral has a host interface to a host system and a local
`processor that controls the flash-memory storage peripheral. 50
`A read-only memory (ROM) is coupled to the local proces(cid:173)
`sor. It stores routines for execution by the local processor.
`The routines include wear-leveling routines for re-mapping
`data from over-used or faulty memory blocks to unused
`memory blocks.
`A cache is coupled to the local processor. It temporarily
`stores data from the host in volatile memory that loses data
`when power is disconnected. A first flash-specific DMA
`controller is coupled to the local processor. It generates
`command, address, and data sequences to a first flash- 60
`memory chip in a format required by the first flash-memory
`chip. A first flash bus is coupled to the first flash-specific
`DMA controller. It transfers data, addresses, and commands
`over shared address/data/command lines. A first flash buffer
`chip is coupled to the first flash bus. It sends the data, 65
`addresses, and commands from the shared lines of the first
`flash bus to the first flash-memory chip.
`
`25
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram of a large flash-memory system
`with dual DMA controllers driving dual flash busses.
`FIG. 2 is an expandable flash-disk system using dual
`DMA and dual flash busses.
`FIG. 3 is a diagram of an ASIC chip containing dual DMA
`controllers modified for controlling flash memories.
`FIG. 4 is a diagram of a flash buffer chip for an expand(cid:173)
`able flash-memory system.
`FIG. 5 is a timing diagram of a time-multiplexed busy
`signal.
`FIG. 6 highlights the busy signals from the flash-memory
`chips to the flash-specific DMA controller.
`FIG. 7 is a state diagram of the flash-specific DMA state
`machines.
`FIG. 8 is a state diagram of the flash bus state machine.
`FIG. 9 illustrates expansion cards in a flash memory
`system.
`
`DETAILED DESCRIPTION
`
`The present invention relates to an improvement in flash(cid:173)
`memory systems. The following description is presented to
`enable one of ordinary skill in the art to make and use the
`invention as provided in the context of a particular applica(cid:173)
`tion and its requirements. Various modifications to the
`preferred embodiment will be apparent to those with skill in
`the art, and the general principles defined herein may be
`applied to other embodiments. Therefore, the present inven(cid:173)
`tion is not intended to be limited to the particular embodi-
`55 ments shown and described, but is to be accorded the widest
`scope consistent with the principles and novel features
`herein disclosed.
`An expandable flash-memory system uses multiple levels
`of busses. The flash-memory chips are buffered from an
`intermediate flash bus by flash buffer chips that combine
`busy signals from an array of flash-memory chips. Flash(cid:173)
`specific direct-memory access (DMA) drives commands,
`addresses, and data out over the flash bus.
`FIG. 1 is a block diagram of a large flash-memory system
`with dual DMA controllers driving dual flash busses. High(cid:173)
`level requests from a local processor are sent to flash
`memory over bus 22. Each request is translated into a
`
`

`

`5,822,251
`
`25
`
`30
`
`5
`
`5
`sequence of commands, address bytes, and data transfers by
`one of the flash-specific DMA controllers 12, 16. DMA
`controllers 12, 16 transfer not only data, as in traditional
`DMA, but also generate the appropriate command and
`address bytes that are transmitted over the same lines of the
`flash bus as the data.
`Commands, address bytes, and data bytes are transferred
`from DMA controller 12, 16 to flash-memory chips 20 over
`flash busses 10, 18, which have 8 multiplexed data/address/
`command lines. In addition to the 8 shared data/address/
`command lines, flash busses 10, 18 also have 2 lines for an
`encoded command. This 2-bit encoded command is not sent
`to flash-memory chips 20, but is instead used to select and
`control flash buffer chips 14. Thus each flash bus has an 8-bit
`portion destined for flash buffer chips 14 or for flash(cid:173)
`memory chips 20, and a 2-bit portion sent only for flash 15
`buffer chips 14.
`DMA controller 12 drives flash bus 10, while DMA
`controller 16 drives second flash bus 18. Flash busses 10, 18
`can operate at the same time, allowing flash operations to be
`initiated and processed in parallel. Each flash buffer chip 14 20
`can control several banks of flash-memory chips 20. Each
`bank can be separately accessed, allowing many flash opera(cid:173)
`tions to be performed in parallel.
`The busy signals from all flash-memory chips in a bank
`are connected together, so only one flash-memory chip in a
`bank can be active at a time. However, each flash buffer chip
`14 receives separate busy signals from each separate bank;
`in FIG. 1, each flash buffer chip 14 has two banks attached
`and receives two separate busy signals. A preferred embodi-
`ment has four banks attached to each flash buffer chip 14.
`Each bank has eight flash memory chips.
`An expandable architecture is provided by flash busses
`10, 18. Additional banks of flash-memory chips can be
`added to existing flash buffer chips, or modules of flash- 35
`memory chips with a flash buffer chip can be plugged into
`a flash bus. The flash bus is an intermediate bus that
`facilitates expansion since any number of flash buffer chips
`can be added.
`Performance is enhanced because two DMA controllers 40
`are provided, able to launch two new flash operations at
`once. Once launched, flash operations are monitored by the
`flash buffer chips, which receive the busy lines from the
`flash-memory chips. Thus the DMA controller can launch
`additional flash operations to other flash-memory chips. This
`inherent parallelism improves performance by allowing
`multiple flash operations to be performed at the same time.
`Indeed, multiple flash operations can be launched or com(cid:173)
`pleted at the same time since two DMA controllers are
`provided.
`Expandable Flash-Disk System-FIG. 2
`FIG. 2 is an expandable flash-disk system using dual
`DMA and dual flash busses. Banks of flash-memory chips 20
`connect to flash buffer chips 14, which are connected to flash
`busses 10, 18. Expansion occurs when additional flash buffer 55
`chips are connected to flash busses 10, 18. Since relatively
`few flash buffer chips are connected to each flash bus,
`loading from additional chips is not critical. A great many
`flash-memory chips can be added since each flash buffer
`chip can connect to four banks of 8 flash-memory chips, a 60
`total of 32 flash-memory chips per flash buffer chip.
`Flash specific DMA controllers 12, 16 are preferably
`contained in a single Application-Specific Integrated Circuit
`(ASIC) 50. ASIC 50 connects DMA controllers 12, 16 to
`local bus 30. Local bus 30 connects DRAM cache 32, local 65
`processor 26, and small-computer system interface (SCSI)
`controller 34 to DMA controllers 12, 16.
`
`6
`Requests from host 36 are received by SCSI controller 34
`and driven onto local bus 30. Local processor 26 responds
`to these host requests by storing host data into DRAM cache
`32 for writes, or reading data from flash-memory chips 20 or
`from DRAM cache 32 for reads. ROM 28 contains firmware
`code of routines that execute on local processor 26 to
`respond to host requests. Other system-maintenance routines
`are stored on ROM 28, such as wear-leveling and copy-back
`routines. DRAM cache 32 is preferably under firmware
`10 control by local processor 26, and thus the local processor's
`local memory 27 and DRAM cache 32 may share the same
`physical DRAM chips.
`Accesses of flash-memory chips 20 by local processor 26
`are initiated by local processor 26 sending a high-level
`command to one of flash-specific DMA controllers 12, 16.
`These DMA controllers then generate sequences of com-
`mand and address bytes, and transfer data. Many sequences
`may be needed, such as for block reads and writes. A block
`read requires that many page read sequences be performed,
`each sequence generally sending command and address
`bytes to the flash-memory chips through the flash buffer
`chips. Some flash-memory chips also have a sequential read
`mode where command and address bytes need only be sent
`for the first page in a sequence.
`The flash-memory chips being read, written, or erased
`remain busy for some period of time after the command and
`address bytes are sent. The flash buffer chips combine and
`send the busy signals from multiple flash-memory chips to
`the DMA controller. Once the busy signal is de-asserted,
`indicating that the flash-memory chip is ready, the DMA
`controller continues the operation, perhaps reading the data
`or moving on to the next page in the block. Finally, once all
`flash operations have finished, the DMAcontroller signals to
`local processor 26 that the high-level flash operation is
`complete.
`Flash-Specific DMA-FIG. 3
`FIG. 3 is a diagram of an ASIC chip containing dual DMA
`controllers modified for controlling flash memories. Data
`received from flash busses 10, 18 is buffered by FIFOs 41,
`42 before being sent to the DRAM cache. Having separate
`FIFOs 41, 42 for flash busses 10, 18 allows for independent
`bus operation. One flash bus can be reading from flash(cid:173)
`memory chips while the other is writing to flash-memory
`45 chips. Flash data is always sent to and from the DRAM
`cache rather than directly from the host to the flash memory.
`Parity circuit 44 adds or checks parity bits for storage in the
`DRAM cache. FIFOs 41, 42 also buffer data from the host
`or DRAM cache before it is written to the flash memory over
`50 one of flash busses 10, 18.
`Flash bus 10 is controlled by flash-specific DMA control(cid:173)
`ler 12, while flash bus 18 is controlled by flash-specific
`DMA controller 16. Each DMA controller 12, 16 contains
`four state machines 40, one for each of four banks of
`flash-memory chips. Having four state machines allows four
`flash-memory chips to be accessed at once for each flash
`bus, for a total of eight flash accesses. The data can be
`interleaved among the four banks for each flash buffer chip,
`and also interleaved among chips connected to the two flash
`busses. Separate state machines 40 allow the eight flash
`accesses to be in different stages of completion.
`Flash bus 10 is driven by bus control logic 46 in DMA
`controller 12, while flash bus 18 is driven by bus control
`logic 48 in DMA controller 16. Bus control logic 46, 48
`generates command and address bytes or enables reading
`and writing of FIFOs 41, 42 in response to states entered by
`state machines 40. Arbitration logic (not shown) determines
`
`

`

`5,822,251
`
`10
`
`7
`which state machine 40 currently controls each flash bus,
`since only one transfer at a time can be performed on a flash
`bus.
`Multiplexers or muxes 83, 85 select address bytes and
`commands from bus control logic 46, 48, or data from FIFOs
`41, 42. This allows both commands and data to be sent over
`the same flash bus.
`Error-correcting-code (ECC) generators 91, 93 and ECC
`checkers 95, 97 connect to flash busses 10, 18. A block code,
`such as a Reed-Solomon code, is used to protect entire pages
`of data stored in the flash-memory chips and to guard against
`more than just simple, single-bit errors. When data is being
`transferred to the flash-memory chips, ECC generators 91,
`93 generate ECC, and when data is being transferred from
`the flash-memory chips, ECC checkers 95, 97 check the
`ECC and determine if there were any errors, either in the
`flash-memory chips themselves, or possibly in the interface
`to or from them. It is possible to design a combined ECC
`generator/checker unit so that one ECC unit can be used per
`flash bus.
`Flash Buffer Chip-FIG. 4
`FIG. 4 is a diagram of a flash buffer chip for an expand(cid:173)
`able flash-memory system. Flash buffer chips allow a rela(cid:173)
`tively large number of flash-memory chips to be added to the
`system without excessively loading the flash bus. Busy
`signals are combined by the flash buffer chips so that fewer
`busy signals need to be input to the DMA controllers.
`Flash buffer chip 14 connects to a flash bus to receive
`commands, addresses, and data from the flash-specific DMA
`controller. The 8-bit data/address/command portion of the
`flash bus is passed through to output logic 79 and then on to
`the flash-memory chips. The 2-bit select and command
`portion of the flash bus are decoded by flash-bus interface
`70, which includes state machines to keep track of multi(cid:173)
`cycle command sequences.
`The flash buffer chips also serve as protocol converters,
`taking the flash bus protocol and converting it to the protoco