GRTD60-34 l 3 8
`
`PATENT
`
`IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
`
`APPLICATION FOR UNITED STATES PATENT
`
`LIFETlME MIXED LEVEL NON-VOLATILE MEMORY SYSTEM
`
`Inventor:
`
`G. R. Mohan Rao — Allen, Texas
`
`Attorneys:
`Munck Wilson Mandala, LLP
`PO. Drawer 800889
`
`Dallas, Texas 75380
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
`
`LIFETHVIE MIXED LEVEL NON-VOLATILE MEMORY SYSTEM
`
`CROSS-REFERENCE TO RELATED APPLICATIONS
`
`[0001]
`
`This application is a Continuation of US. Patent Application No. US. Application
`
`No. 14/950,553 filed on November 24, 2015, entitled LIFETIME MIXED LEVEL NON-
`
`VOLATILE MEMORY SYSTEM (Atty. Dkt. No. GRTD-32800), which published on June 2,
`
`2016, as US. Application Publication No. 2016-0155496, now US. Patent No. 9,997,240 issued
`
`June 12, 2018, which is incorporated by reference in its entirety. US. Application No.
`
`14/950,553 is a Continuation of US. Patent Application No. 14/525,411, filed October 28, 2014,
`
`entitled LIFIEL'I'IMES NIIXED LEVEL NON~VOLA'E‘ILE MEMORY SYS'EIEIVI, which published
`
`on October 1, 2015, as US. Publication No. 2015-0278013, now US. Patent No. 9,196,385,
`
`issued on November 24, 2015, (Atty. Dkt. No GRID—32620). Application No. l4/525,4ll is a
`
`Division of US. Patent Application No. 13/455,267, filed April 25, 2012, published on January
`
`24, 2013, as US. Publication No. 2013-0021846, now US. Patent No. 8,891,298, issued on
`
`November 18, 2014, entitled LIFETHVIE MIXED LEVEL NON-VOLATILE MEMORY
`
`SYSTEM (Atty. Dkt. No. GRTD-326l9). Application No. 13/455,267 claims the benefit of US.
`
`Provisional Application No. 61/509,257,
`
`filed July 19, 2011, entitled LIFETIME MIXED
`
`LEVEL NAND FLASH SYSTEM (Atty. Dkt. No. GRTD-32624). Patent Nos. 9,997,240,
`
`9,196,385, and 8,891,298 and Patent Application Publication Nos. 2015-0278013 and 2013-
`
`0021846 are hereby incorporated by reference in their entirety.
`
`This application also
`
`incorporates by reference the complete disclosure of US. Patent Application No. 12/256,362,
`
`filed October 22, 2008, published on April 30, 2009, as US. Publication No. 2009-0109787, now
`
`US. Patent No. 7,855,916, issued on December 21, 2010, entitled NONVOLATILE MEMORY
`
`SYSTEMS WITH ElVfl3EDDED FAST READ AND WRITE MEMORIES (Atty. Dkt. No.
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`GRTD-32614). This application also incorporates by reference the complete disclosure of US.
`
`Patent Application No. 12/915,177, filed October 29, 2010, published on March 10, 2011, as
`
`US. Publication No. 2011-0060870, now US. Patent No. 8,194,452, issued on June 5, 2012,
`
`entitled NONVOLATILE MEMORY SYSTEMS WITH ElVfl3EDDED FAST READ AND
`
`WRITE MEMORIES (Atty. Dkt. No. 32615).
`
`TECHNICAL FIELD
`
`[0002]
`
`This application relates to a system and method for providing reliable storage through
`
`the use of non-volatile memories and, more particularly, to a system and method of increasing
`
`the reliability and lifetime of a NAND flash storage system, module, or chip through the use of a
`
`combination of single-level cell (SLC) and multi-level cell (MLC) NAND flash storage without
`
`substantially raising the cost of the NAND flash storage system. The memory in a total non-
`
`volatile memory system may contain some SRAM (static random-access memory), DRAM
`
`(dynamic RAM), RRAM (resistive RAM), PCM (phase change memory), MAGRAM (magnetic
`
`random-access memory), NAND flash, and one or more HDDs (hard disk drives) when storage
`
`of the order of several terabytes is required. The SLC non-volatile memory can be flash, PCM,
`
`RRAM, MAGRAM or any other solid-state non-volatile memory as long as it has endurance that
`
`is superior to that of MLC flash, and it provides for data access speeds that are faster than that of
`
`MLC flash or rotating storage media (e.g., HDDs).
`
`

`

`GRTD60-3413 8
`
`BACKGROUND
`
`PATENT
`
`[0003]
`
`Non-volatile memories provide long-term storage of data. More particularly, non-
`
`volatile memories can retain the stored data even when not powered. Magnetic (rotating) hard
`
`disk drives (HDD) dominate this storage medium due to lower cost compared to solid state disks
`
`(SSD). Optical (rotating) disks, tape drives and others have a smaller role in long-term storage
`
`systems. SSDs are preferred for their superior performance (fast access time), mechanical
`
`reliability and ruggedness, and portability. Flash memory, more specifically NAND flash, is the
`
`dominant SSD medium today.
`
`[0004]
`
`RAM, PCM, MAGRAM and others, wi111ikely play a larger role in the future, each
`
`of them having their own advantages and disadvantages. They may ultimately replace flash
`
`memories, initially for use as a "write buffer" and later to replace "SLC flash" and "MLC flash."
`
`MLC NAND flash is a flash memory technology using multiple levels per cell to allow more bits
`
`to be stored using the same number of transistors. In SLC NAND flash technology, each cell can
`
`eXist in one of two states, storing one bit of information per cell. Most MLC NAND flash
`
`memory has four possible states per cell, so it can store two bits of information per cell.
`
`[0005]
`
`These semiconductor technology driven "flash alternatives," i.e., RRAM, PCM,
`
`MAGRAM and others, have several advantages over any (SLC or MLC) flash because they: 1)
`
`allow data to be written over eXisting data (without prior erase of existing data), 2) allow for an
`
`erase of individual bytes or pages (instead of having to erase an entire block), and 3) possess
`
`superior endurance (1,000,000 write-erase cycles compared to typical 100,000 cycles for SLC
`
`flash and less than 10,000 cycles for MLC flash).
`
`[0006]
`
`HDDs have several platters. Each platter contains 250-5,000 tracks (concentric
`
`circles). Each track contains 64 to 256 sectors. Each sector contains 512 bytes of data and has a
`
`unique "physical (memory) address." A plurality of sectors is typically combined to form a
`
`"logical block" having a unique "logical address." This logical address is the address at which the
`
`logical block of physical sectors appears to reside from the perspective of an executing
`
`application program. The size of each logical block and its logical address (and/or address
`
`ranges/boundaries)
`
`is optimized for
`
`the particular operating system (OS) and software
`
`

`

`GRTD60-34 l 3 8
`
`PATENT
`
`applications executed by the host processor. A computer OS organizes data as "files." Each file
`
`may be located (stored) in either a single logical block or a plurality of logical blocks, and
`
`therefore, the location of files typically traverses the boundaries of individual (physical) sectors.
`
`Sometimes, a plurality of files has to be combined and/ or modified, which poses an enormous
`
`challenge for the memory controller device of a non-volatile memory system.
`
`[0007]
`
`SSDs are slowly encroaching on the HDD space and the vast majority of NAND flash
`
`in enterprise servers utilizes a SLC architecture, which further comprises a NAND flash
`
`controller and a flash translation layer (FTL). NAND flash devices are generally fragmented into
`
`a number of identically sized blocks, each of which is further segmented into some number of
`
`pages. It should be noted that asymmetrical block sizes, as well as page sizes, are also acceptable
`
`within a device or a module containing devices. For example, a block may comprise 32 to 64
`
`pages, each of which incorporates 2 - 4 Kbit of memory. In addition, the process of writing data
`
`to a NAND flash memory device is complicated by the fact that, during normal operation of, for
`
`example, single-level storage (SLC) , erased bits (usually all bits in a block with the value of ‘1’)
`
`can only be changed to the opposite state (usually ‘0’) once before the entire block must be
`
`erased. Blocks can only be erased in their entirety, and, when erased, are usually written to ‘1’
`
`bits. However, if an erased block is already there, and if the addresses (block, page, etc.) are
`
`allowed, data can be written immediately, if not, a block has to be erased before it can be written
`
`to.
`
`[0008]
`
`FTL is the driver that works in conjunction with an existing operating system (or, in
`
`some embedded applications, as the operating system) to make linear flash memory appear to the
`
`system like a disk drive, i.e., it emulates a HDD. This is achieved by creating "virtual" small
`
`blocks of data, or sectors, out of fiash's large erase blocks and managing data on the flash so that
`
`it appears to be "write in place" when in fact it is being stored in different locations in the flash.
`
`FTL further manages the flash so that there are clean/ erased places to store data.
`
`[0009]
`
`Given the limited number of writes that individual blocks within fiash devices can
`
`tolerate, wear leveling algorithms are used within the flash devices (as firmware commonly
`
`known as FTL or managed by a controller) to attempt to ensure that "hot" blocks, i.e., blocks that
`
`are frequently written, are not rendered unusable much faster than other blocks. This task is
`
`

`

`GRTD60-34 l 3 8
`
`PATENT
`
`usually performed within a flash translation layer. In most cases, the controller maintains a
`
`lookup table to translate the memory array physical block address (PBA) to the logical block
`
`address (LBA) used by the host system. The controller's wear-leveling algorithm determines
`
`which physical block to use each time data is programmed, eliminating the relevance of the
`
`physical location of data and enabling data to be stored anywhere within the memory array and
`
`thus prolonging the service life of the flash memory. Depending on the wear-leveling method
`
`used, the controller typically either writes to the available erased block with the lowest erase
`
`count (dynamic wear leveling); or it selects an available target block with the lowest overall
`
`erase count, erases the block if necessary, writes new data to the block, and ensures that blocks
`
`of static data are moved when their block erase count is below a certain threshold (static wear
`
`leveling).
`
`[0010]
`
`MLC NAND flash SSDs are slowly replacing and/ or coeXisting with SLC NAND
`
`flash in newer SSD systems. MLC allows a single cell to store multiple bits, and accordingly, to
`
`assume more than two values, i.e., ‘0’ or ‘1’. Most MLC NAND flash architectures allow up to
`
`four (4) values per cell, i.e., ‘00’, ‘Ol’, ‘lO’, or ‘ll’. Generally, MLC NAND flash enjoys greater
`
`density than SLC NAND flash, at the cost of a decrease in access speed and lifetime (endurance).
`
`It should be noted, however, that even SLC NAND flash has a considerably lower lifetime
`
`(endurance) than rotating magnetic media (e.g., HDDs), being able to withstand only between
`
`50,000 and 100,000 writes, and MLC NAND flash has a much lower lifetime (endurance) than
`
`SLC NAND flash, being able to withstand only between 3,000 and 10,000 writes. As is well
`
`known in the art, any “write” or “program” to a block in NAND flash (floating gate) requires an
`
`“erase” (of a block) before “write.”
`
`[0011]
`
`Despite its limitations, there are a number of applications that lend themselves to the
`
`use of MLC flash. Generally, MLC flash is used in applications where data is read many times
`
`(but written few times) and physical size is an issue. For example, flash memory cards for use in
`
`digital cameras would be a good application of MLC flash, as MLC can provide higher density
`
`memory at lower cost than SLC memory.
`
`[0012]
`
`When a non-volatile storage system combines HDD, SLC and MLC (setting aside
`
`volatile memory for buffering, caching etc) in a single (hybrid) system, new improvements and
`
`

`

`GRTD60-34 l 3 8
`
`PATENT
`
`solutions are required to manage the methods of writing data optimally for improved life time
`
`(endurance) of flash memory. Accordingly, various embodiments of a NAND flash storage
`
`system that provides long lifetime (endurance) storage at low cost are described herein.
`
`[0013]
`
`The following description is presented to enable one of ordinary skill in the art to
`
`make and use the disclosure and is provided in the context of a patent application and its
`
`requirements. Various modifications to the preferred embodiment and the generic principles and
`
`features described herein will be readily apparent to those skilled in the art. Thus, the present
`
`disclosure is not intended to be limited to the embodiments shown, but is to be accorded the
`
`widest scope consistent with the principles and features described herein.
`
`

`

`GRTD60-34 l 3 8
`
`SUMMARY
`
`PATENT
`
`[0014]
`
`According to one embodiment of the present disclosure, there is provided a system
`
`for storing data which comprises at least one MLC nonvolatile memory module (hereinafter
`
`referred to as "MLC module") and at least one SLC non-volatile memory module (hereinafter
`
`referred to as "SLC module"), each module comprises a plurality of individually erasable blocks.
`
`The data storage system according to one embodiment of the present disclosure further
`
`comprises a controller for controlling both the at least one MLC module and the at least one SLC
`
`module. In particular, the controller maintains an address map comprising a list of individual
`
`logical address ranges each of which maps to a similar range of physical addresses within either
`
`the at least one MLC module or the at least one SLC module. After each write to (flash) memory,
`
`the controller conducts a data integrity check to ensure that the data was written correctly. When
`
`the data was not written correctly, the controller modifies the table so that the range of addresses
`
`on which the write failed is remapped to the next available range of physical addresses within the
`
`at least one SLC module. The SLC module can be (NAND) flash, PCM, RRAM, MAGRAM or
`
`any other solid-state non-volatile memory as long as it has endurance that is superior to that of
`
`MLC flash, and it provides for data access speeds that are faster than that of MLC flash or
`
`rotating storage media (e.g., HDDs).
`
`[0015]
`
`According to another embodiment of the present disclosure,
`
`there is provided a
`
`system for storing data which comprises a controller that is further adapted to determine which
`
`of the blocks of the plurality of the blocks in the MLC and SLC non-volatile memory modules
`
`are accessed most frequently and wherein the controller segregates those blocks that receive
`
`frequent writes into the at least one SLC non-volatile memory module and those blocks that
`
`receive infrequent writes into the at least one MLC nonvolatile module.
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0016]
`
`The present disclosure will he more folly understood by reference to the following
`
`detailed description of one or more preferred embodiments when read in conjunction with the
`
`accompanying drawings, in which like reference characters refer to like parts throughout the
`
`Views and in which:
`
`[0017]
`
`FIG. l is a block diagram or" a computer system incorporating one embodiment of the
`
`present disclosure;
`
`[0018]
`
`FIGS 2A and 2B are drawings depicting a,
`
`translation table/address map in
`
`accordance with one embodiment of the present disclosure;
`
`[0019]
`
`FIGS. 3A and BB are a flow chart iiiustrating an exemplary method for use in
`
`implementing one embodiment of the present disclosure; and
`
`[0020]
`
`FIG. 4 is a hiocl; diagram depicting one embodiment of the present disclosure for
`
`ii'nplementetion Withi n a NAND flash inodnie.
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`DETAILED DESCRIPTION
`
`[0021]
`
`The present disclosure is directed to the reliable storage of data in read and write
`
`memory, and, in particular, to the reliable storage of data in non—volatile memory, such as, for
`
`example, NAND flash Generally, and in particular regard to YARD flash memory, two separate
`
`banks of NAND flash are maintained lay a controller. Cine hank contains economical Mil:
`
`VAVD flash, while a second hank contains high endurance SLC NAND flash. The controller
`
`conducts a data integrity test after every write. if a particular address range fails a data integrity
`
`test, the address range is reinapped from MLC NAND flash to SLC NAND flash. As the SLC
`
`NAND flash is used to boost
`
`the lifetime (endurance) of the storage system,
`
`it can he
`
`considerably lesser in amount than the MLC NAND flash. For example, a system may set SLC
`
`YAVD flash equal to l25% or 25% of MLC NAND flash (total non—volatile memory storage
`
`space=MLC+ SLC‘).
`
`[0022]
`
`'l‘urning to the Figures and to FlG,
`
`l in particular, a computer system ltl depicting
`
`one embodiment of the present disclosure is shown. A processor l2 is coupled to a device
`
`controller l4, such as a chi pset, using a data link well lrnown in the art, such as a parallel bus or
`
`pacltet~hased link. The device controller l4 provides interface functions to the processor l2. in
`
`some computer systems, the device controller l4 may he an integral part of the (host) processor
`
`12, The device controller l4 provides a number of input/output ports l6 and 18, such as, for
`
`example, serial ports (e. g, USB ports and Pirewire ports) and networlr ports (eg, Ethernet ports
`
`and 802ll “Wisl’i” ports}. The device controller l4 may also control a bank of, for example,
`
`DRAM '20. In addition, the device controller l4 controls access to one or more disks '24, such as,
`
`for example, a rotating magnetic dislr, or an optical disk, as well as two or more types of NAND
`
`flash memory. One type of NAND flash memory is a MLC NAND flash memory module '26.
`
`Another type of NAND flash memory is a SLC NAND flash memory module 28,
`
`[0023]
`
`The device controller l4 maintains a translation table/address map which may include
`
`address translations for all devices in the computer system. Nonetheless, the discussion in the
`
`present disclosure will he limited only to NAND flash memory modules, in particular, the device
`
`controller
`
`l4 maintains a translation table that maps logical computer system addresses to
`
`physical addresses in each one of the MLC— and SLC—NAND flash memory modules 4.6 and 28,
`
`10
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`respectively, As MLC‘ flash memory is less expensive than SLC flash memory, on a cost per hit
`
`basis,
`
`the translation table will initially map all
`
`logical NAND flash addresses to the Ml.,C
`
`NAND flash memory module 26, The address ranges within the translation table will assume
`
`some minimum quantum, such as, for example, one block, although a smaller size, such as one
`
`page could he used, it“ the NAND tlash has the capability of erasing the smaller size quantum.
`
`[0024]
`
`A “rcad~modit‘y—writc” scheme is used to write data to the NAND flash, Data to he
`
`written to NAND tlash is maintained in DRAM 20. After each write to an address within a
`
`particular address range, the device controller li-‘l will—as time permits—perform a read on the
`
`address range to ensure the integrity of the written data. ll" a data integrity test fails, the address
`
`range is remapped from the MLC NAND flash memory module 26 to the next available address
`
`range in the SLC NAVD flash memory module 28.
`
`[0025]
`
`FIGS. 2A and 2B illustrate one embodiment of a translation table/address map of the
`
`present disclosure.
`
`ln Figure. 2A, a list of logical address ranges (RU—RN)
`
`is translated to
`
`physical address ranges. As illustrated, all of the logical address ranges are translated to hlochs
`
`on the MLC NAND fl ash memory module 26. However,
`
`through the application of a data
`
`integrity verification check (explained in more detail below) it is determined that, for example,
`
`address range R2 con‘esponds to t‘ailed quanta of data stored in block 2 of the M16 NAND ll ash
`
`memory module 26. FIG. 28 shows the quanta of data which failed the data integrity verification
`
`check (see PlG. 2A) rernapped to the next available range of physical addresses within the SLC
`
`NAND flash memory module 28, in this example, SL Z/ilocl: O.
`
`[0026]
`
`FlGS, 3A and 3B are a tlow chart illustrating a method for utilizing a NAND tlash
`
`memory system incorporating one embodiment of the present disclosure. The method begins in a
`
`step lOO, when a command to write a quantum of data stored in DRAM to a. particular location in
`
`NAND flash memory is received ln step l02, the quantum of data is read from DRAM into
`
`memory within the device controller (which acts as the memory controller). In step lOZl, both the
`
`logical address range and the NAND flash physical address range to which the quantum of data
`
`is to be written, is read into memory of the device controller. In step lilo, the quantum of data to
`
`be written is combined with the contents of the NAND flash memory. in step lOS, the NAND
`
`flash physical address range to be written is erased. ln step llii‘r, the combined data is written to
`
`11
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`the appropriate NAND flash physical address range.
`
`in step l lit the NAND flash physical
`
`address range that was written in step l l O is read into device controller rnem ory.
`
`[0027]
`
`The flowchart continues in Fifi. 3B. in step lldL the NAND flash physical address
`
`range that was read into device controller rnemory is compared with the retained data
`
`representing the comhination of the previous contents of the physical address range and the
`
`quantum of data to he written. ln step l l6, if the retained data matches the newly stored data in
`
`the 'NAND flash memory, the write was a success, and the method exits in step lltl. l-lowever, if
`
`the retained data does not match the newly stored data in the NAND flash memory, the method
`
`executes step thl, which identities the next quantum of availahle SLC NAND flash memory
`
`addresses. in step l22, a check is made to determine if additional SLC NAND flash memory is
`
`available, and, if not, the NAND flash memory system is marked as failed, prompting a system
`
`alert step l24. However, if additional SLC NAND flash memory is available, the failed NAND
`
`flash physical address range is remapped to the next available quantum of SlC NAND tlash
`
`memory in step 126. Execution then returns to step l l 0, where the write is repeated.
`
`[0028]
`
`Another application of one embodiment of the present disclosure, not depicted in any
`
`of the drawings, is to allocate “hot” blocks; i.e., those blocks that receive frequent writes, into the
`
`SLC NAND flash memory module 28, while allocating “cold” blocks; ie, those blocks that only
`
`receive infrequent writes,
`
`into the MLC NAND flash memory module 26. This could he
`
`accomplished within the device controller l4 described above, which could simply maintain a.
`
`count of those blocks that are accessed (written to) most frequently, and, on a periodic hasis,
`
`such as, for example, every lGOO writes, or every l0,000 writes, transfer the contents of those
`
`blocks into the SLC 'NAND flash memory module 28.
`
`[0029]
`
`Fill. 4 depicts another embodiment of the present disclosure. The embodiment is
`
`entirely resident within a NAND flash module :30, in particular, a standard NAND flash interface
`
`52 is managed by flash translation layer (FTL) logic Sill. The flash translation layer (FTL) Sill
`
`manages two NAME) flash memory hanks 56 and 58, wherehy memory hank :36 comprises a.
`
`plurality of MLC NAME flash memory modules 60a and a plurality of SLC NAND flash
`
`memory modules 62a. Memory hanlc 58 comprises a plurality of MlC NAND flash memory
`
`modules 60b and a plurality of SLC NAND flash memory modules 62h.
`
`12
`
`

`

`GRTD60-3413 8
`
`PATENT
`
`[0030]
`
`This embodiment of the present disclosure could function similarly to the system
`
`level embodiment discussed earlier with reference to FlGS 1-3 B, but the control functions, such
`
`as maintenance of the translation table/address map ("FEES 2A and 28:), could he conducted
`
`within the flash translation layer (FTL) 54 instead ol‘in a device controller l4.
`
`[0031]
`
`Bmhodinrents of the present disclosure relate to a system and method of increasing
`
`the reliability and lifetime of a NAND flash storage system? module? or chip through the use of a
`
`combination or" multi~level cell (MLC) and single-level cell (SLC) NAND flash storage The
`
`above description is presented to enable one of ordinary skill in the art to make and use the
`
`disclosure and is provided in the context of a patent application and its requirements. While this
`
`disclosure contains descriptions with reference to certain illustrative aspects,
`
`it will he
`
`understood that these descriptions shall not he construed in a limiting sense, Rather, various
`
`changes and modifications can he made to the illustrative embodiments without departing from
`
`the true spirit, central characteristics and scope of the disclosure, including those combinations of
`
`features that are individually disclosed or claimed herein. Furthermore? it will he appreciated that
`
`any such changes and modifications will he recognized by those skilled in the art as an
`
`equivalent to one or more elements of the following claims, and shall he covered hy such claims
`
`to the fullest extent permitted by law.
`
`13
`
`