I 1111111111111111 11111 1111111111 11111 11111 111111111111111 111111111111111111
`US009997240B2
`
`c12) United States Patent
`Rao
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 9,997,240 B2
`*Jun. 12, 2018
`
`(54) LIFETIME MIXED LEVEL NON-VOLATILE
`MEMORY SYSTEM
`
`(71) Applicant: Greenthread, LLC, Dallas, TX (US)
`
`(72)
`
`Inventor: G. R. Mohan Rao, Allen, TX (US)
`
`(73) Assignee: Greenthread, LLC, Dallas, TX (US)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`12/2010 Rao
`7,855,916 B2
`8,825,941 B2 * 9/2014 Moshayedi ......... G06F 11/1441
`711/103
`
`(Continued)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days. days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`OTHER PUBLICATIONS
`
`Intel, Understanding the Flash Translation Layer (FTL) Specifica(cid:173)
`tion, Dec. 1998.
`
`(Continued)
`
`(21) Appl. No.: 14/950,553
`
`(22) Filed:
`
`Nov. 24, 2015
`
`(65)
`
`Prior Publication Data
`
`US 2016/0155496 Al
`
`Jun. 2, 2016
`
`Related U.S. Application Data
`
`(60) Continuation of application No. 14/525,411, filed on
`Oct. 28, 2014, now Pat. No. 9,196,385, which is a
`(Continued)
`
`(51)
`
`Int. Cl.
`GllC 16104
`GllC 11156
`
`(2006.01)
`(2006.01)
`(Continued)
`
`(52) U.S. Cl.
`CPC ...... GllC 1115635 (2013.01); G06F 1111068
`(2013.01); G06F 1111072 (2013.01);
`(Continued)
`( 58) Field of Classification Search
`CPC ........ GllC 11/5621; GllC 2211/5641; GllC
`2029/1806; GllC 29/50004; GllC 29/76;
`(Continued)
`
`Primary Examiner - Harry W Byrne
`Assistant Examiner - R Lance Reidlinger
`(74) Attorney, Agent, or Firm - Munck Wilson Mandala,
`LLP
`
`(57)
`
`ABSTRACT
`
`A controller for managing at least one MLC non-volatile
`memory module and at least one SLC non-volatile memory
`module. The flash controller is adapted to determine if a
`range of addresses listed by an entry and mapped to said at
`least one MLC non-volatile memory module fails a data
`integrity test. In the event of such a failure, the controller
`remaps said entry to an equivalent range of addresses of said
`at least one SLC non-volatile memory module. The flash
`controller is further adapted to determine which of the
`blocks in the MLC and SLC non-volatile memory modules
`are accessed most frequently and allocating those blocks that
`receive frequent writes to the SLC non-volatile memory
`module and those blocks that receive infrequent writes to the
`MLC non-volatile memory module.
`
`10 Claims, 5 Drawing Sheets
`
`-- -
`
`Wo
`
`......_
`
`:
`
`-. 62b
`
`m
`
`I
`
`Interface
`
`----- "
`
`------52
`
`60a_,,..
`
`1-----
`I
`
`e--
`
`62a __,,,.,.
`
`5 6 ---
`
`"-
`
`-----
`
`Micron Ex. 1005, p. 1
`Micron v. Vervain
`IPR2021-01549
`
`

`

`US 9,997,240 B2
`Page 2
`
`Related U.S. Application Data
`
`division of application No. 13/455,267, filed on Apr.
`25, 2012, now Pat. No. 8,891,298.
`
`(60) Provisional application No. 61/509,257, filed on Jul.
`19, 2011.
`
`(51)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`Int. Cl.
`GllC 16134
`G06F 12102
`G06F 11110
`GllC 29152
`GllC 29100
`GllC 16116
`(52) U.S. Cl.
`CPC ...... G06F 1210246 (2013.01); GllC 1115621
`(2013.01); GllC 1115678 (2013.01); GllC
`16116 (2013.01); GllC 1613495 (2013.01);
`GllC 29152 (2013.01); GllC 29176 (2013.01);
`G06F 2212/7202 (2013.01); GllC 2211/5641
`(2013.01)
`
`( 58) Field of Classification Search
`CPC . GllC 29/765; G06F 11/1068; G06F 11/1072
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`8,891,298 B2 * 11/2014 Rao.
`
`GllC 16/3495
`365/148
`9,196,385 B2 * 11/2015 Rao.
`GllC 16/3495
`2009/0268513 Al
`10/2009 De Ambroggi et al.
`2009/0307418 Al
`12/2009 Chen et al.
`2009/0327591 Al* 12/2009 Moshayedi
`
`G06F 11/1441
`711/103
`
`2010/0058018 Al
`2010/0172179 Al*
`
`3/2010 Kund et al.
`7/2010 Gorobets ............ G06F 12/0246
`365/185.09
`2010/0325352 Al* 12/2010 Schuette ............... G06F 3/0613
`711/103
`
`2011/0060870 Al
`2011/0271043 Al
`
`3/2011 Rao
`11/2011 Segal et al.
`
`OTHER PUBLICATIONS
`
`Tae-Sun Chung et al., A Survey of Flash Translation Layer, Journal
`of Systems Architecture 55, pp. 332-343, 2009.
`Seagate Technology LLC, The Transition to Advanced Format 4K
`Sector Hard Drives, Apr. 2010.
`Roberto Bez et al., Introduction to Flash Memory, Proceedings of
`the IEEE, vol. 91, No. 4, Apr. 2003.
`Dingsong Wei et al., WAFTL: A Workload Adaptive Flash Trans(cid:173)
`lation Layer with Data Partition, IEEE 27th Symposium on Massive
`Storage Systems and Technologies (MSST), May 23-27, 2011.
`Taeho Kgil et al., Improving NAND Flash Based Disk Caches,
`International Symposium on Computer Architecture, Copyright
`2008 IEEE.
`Samsung Electronics Co., Ltd. ,7th International Symposium on
`Advanced Gate Stack Technology, RRAM Technology from an
`Industrial Perspective, Process Development Tearn/RRAM PJT
`In-Gyu Baek, Sep. 2010.
`Paolo Pavan et al., Flash Memory Cells-An Overview, Proceed(cid:173)
`ings of the IEEE, vol. 85, No. 8, Aug. 1997.
`Yoshihisa Iwata et al., A High-Density NAND EEPROM with
`Block-Page Programming for Microcomputer Applications, IEEE
`Journal of Solid-State Circuits, vol. 25, No. 2, Apr. 1990.
`Tae-Sung Jung et al., A 117-mm2 3.3-V Only 128-Mb Multilevel
`N AND Flash Memory for Mass Storage Applications, IEEE Journal
`of Solid-State Circuits, vol. 31, No. 11, Nov. 1996.
`
`Masayoshi Ohkawa et al., A 98 mm2 Die Size 3.3V 64-Mb Flash
`Memory with FN-NOR Type Four-Level Cell, IEEE Journal of
`Solid-State Circuits, vol. 31, No. 11, Nov. 1996.
`Masaki Momodomi et al., An Experimental4-Mbit CMOS
`EEPROM with a NANO-Structured Cell, IEEE Journal of Solid(cid:173)
`State Circuits, vol. 24, No. 5, Oct. 1989.
`Ken Takeuchi et al., A Multipage Cell Architecture for High-Speed
`Programming Multilevel NAND Flash Memories, IEEE Journal of
`Solid-State Circuits, vol. 33, No. 8, Aug. 1998.
`Shigeru Atsumi et al., A Channel-Erasing 1.8-V-Only 32-Mb NOR
`Flash EEPROM with a Bitline Direct Sensing Scheme, IEEE
`Journal of Solid-State Circuits, vol. 35, No. 11, Nov. 2000.
`Taehee Cho et al., A Dual-Mode NAND Flash Memory: 1-Gb
`Multilevel and High-Performance 512-Mb Single-Level Modes,
`IEEE Journal of Solid-State Circuits, vol. 36, No. 11, Nov. 2001.
`Douglas J. Lee et al., Control Logic and Cell Design for a 4K
`NVRAM, IEEE Journal of Solid-State Circuits, vol. SC-18, No. 5,
`Oct. 1983.
`Duane H. Oto et al., High-Voltage Regulation and Process Consid(cid:173)
`erations for High-Density 5 V-Only E2PROM's, IEEE Journal of
`Solid-State Circuits, vol. SC-18, No. 5, Oct. 1983.
`Gheorghe Samachisa et al., A 128K Flash EEPROM Using Double(cid:173)
`Polysilicon Technology, IEEE Journal of Solid-State Circuits, vol.
`SC-22, No. 5, Oct. 1987.
`Nelson Duann, Silicon Motion, Inc., Flash Memory Summit, SIC &
`MIC Hybrid, Santa Clara, CA, Aug. 2008.
`Simona Boboiia et al., Write Endurance in Flash Drives: Measure(cid:173)
`ments and Analysis, Usenix Conference on File and Storage Tech(cid:173)
`nologies, San Jose, CA, Feb. 2010.
`Simona Boboiia et al. Write Endurance in Flash Drives: Measure(cid:173)
`ments and Analysis, Handout at Unsenix Conference on File and
`Storage Technologies, San Jose, CA, Feb. 2010.
`Silicon Systems, Increasing Flash SSD Reliability, StorageSearch.
`com, Apr. 2005.
`Ismael Chang Ghalimi, Intalio, An lntalio White Paper, Cloud
`Computing is Memory Bound, May 2010.
`Hynix, 32Gb NAND Flash, HY27UKO8BGFM, Product Descrip(cid:173)
`tion Sheet, Feb. 2007.
`Chris Evans, Consultant with Langton Blue, SearchStorage.co.UK,
`Enterprise MLC: How flash vendors are boosting MLC write
`endurance, Jun. 3, 2011.
`Jesung Kim et al., A Space-Efficient Flash Translation Layer for
`Compactflash Systems, IEEE Transactions on Consumer Electron(cid:173)
`ics, vol. 48, No. 2, May 2002.
`Garth Goodson et al., Design Tradeoffs in a Flash Translation Layer,
`HPCA West 2010 (High Perf Comp Arch Conference, Bangalore,
`India.
`Abdul Rub Aamer Mohammed, Improving Hot Data Identification
`for Hybrid SLC/MLC Device; CSci 8980-Advanced Storage
`Systems, Spring 2009.
`Yang Hu, Achieving Page-Mapping FTL Performance at Block(cid:173)
`Mapping FTL Cost by Hiding Address Translation, 26th IEEE
`Symposium on Massive Storage Systems and Technologies (MSST)
`May 3-7, 2010.
`Clinton w. Smullen, IVet al., Accelerating Enterprise Solid-State
`Disks with Non-Volatile Merge Caching, 2010 International Green
`Computing Conference, Aug. 15-18, 2010.
`Monolithic 3D, Inc. Introducing our monolithic 3D resistive
`memory architecture, http://www.monolithic3d.com/2/post/2011/
`06/introducing-our-3d-resistive-memory-architecture.htrnl, Jun. 27,
`2011.
`Song Jiang et al., S-FTL: An Efficient Address Translation for Flash
`Memory by Exploiting Spatial Locality, Proceedings of the MSST
`2011, May 2011.
`Greg Atwood et al., Intel Strata Flash TM Memory Technology
`Overview, Intel Technology Journal Q4 1997.
`Moinuddin K. Qureshi et al., Morphable Memory System: A Robust
`Architecture for Exploiting Multi-Level Phase Change Memories,
`International Symposium on Computer Architecture, Saint-Malo,
`France, Jun. 19-23, 2010.
`Abhishek Rajimwale et al., Block Management in Solid-State
`Devices, Usenix Conference, Jun. 14-19, 2009.
`
`Micron Ex. 1005, p. 2
`Micron v. Vervain
`IPR2021-01549
`
`

`

`US 9,997,240 B2
`Page 3
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Brendan Gregg et al., Sun Storage 7000 Unified Storage System
`L2ARC: Second Level Adaptive Replacement Cache, Oracle White
`Paper-Sun Storage 7000 Unified Storage System L2ARC, May
`2010.
`Chunqiang Tang, FVD: a High-Performance Virtual Machine Image
`Format for Cloud, USENIX Conference Jun. 2011.
`Anand Lal Shimpi, AnandTech, The Crucial m4 (Micron C400)
`SSD Review, Mar. 31, 2011.
`Anand Lal Shimpi, AnandTech, The Intel SSD 320 Review: 25nm
`G3 is Finally Here, Mar. 28, 2011.
`Micron Technology, Inc., TN-29-42: Wear-Leveling Techniques in
`NAND Flash Devices Introduction, Oct. 2008.
`
`* cited by examiner
`
`Micron Ex. 1005, p. 3
`Micron v. Vervain
`IPR2021-01549
`
`

`

`U.S. Patent
`
`Jun.12,2018
`
`Sheet 1 of 5
`
`US 9,997,240 B2
`
`10
`
`20
`
`I
`
`L-- 12
`
`Processor
`
`14
`
`I
`
`16
`
`V
`
`I/0
`
`DRAM
`
`Device
`Controller
`
`18
`
`/
`
`I/0
`
`Disk(s)
`
`24
`/ / /
`
`(e.g., rotating 1nedia-
`n1agnetic or opncal}
`
`MLC
`flash
`
`/
`
`26
`
`SLC
`flash
`
`~
`28
`
`FIG. 1
`
`Micron Ex. 1005, p. 4
`Micron v. Vervain
`IPR2021-01549
`
`

`

`U.S. Patent
`
`Jun.12,2018
`
`Sheet 2 of 5
`
`US 9,997,240 B2
`
`PHYSICAL
`LOGICAL
`ADDRESS RANGE ADDRESS RANGE
`
`RO
`
`MLC/Block 0
`
`Failed Data
`Integrity Test
`
`-------1'
`
`MLC/Block 1
`Rl
`,----==------- ............ -....- .... =------- I
`MLC/Block 2 i
`R2
`! ,_..., _____ ..., ___ ..., ---------=----
`
`!
`
`R~3
`
`R4
`
`RN
`
`MLC/Block 3
`
`MLC/Block 4
`
`MLC/Block N
`
`FIG.2A
`
`LOGlCAL
`ADDRESS RANGE
`
`PHYSICAL
`ADDRESS RANGE:
`
`RO
`
`MLC/Block 0
`
`I"""""' .... -
`!
`l
`
`Rl
`MLC/Block 1
`...... ""'""""" ... .,. -------------- I
`.,., -
`SLC/Block 0 I
`R2
`, .................... ...,_,...,_,... __ -------------~
`I
`R:3
`MLC/Block 3
`
`Remapping to SLC
`flash module
`
`-----i,
`
`R4
`
`RN
`
`MLC/Block 4
`
`MLC/Block N
`
`FIG.2B
`
`Micron Ex. 1005, p. 5
`Micron v. Vervain
`IPR2021-01549
`
`

`

`U.S. Patent
`
`Jun.12,2018
`
`Sheet 3 of 5
`
`US 9,997,240 B2
`
`Begin
`
`100
`
`Read data quantum
`from DRAM into memory
`of device controller
`
`/
`
`102
`
`Read logical address
`range and NAND flash
`physical address range
`to which data quanturn
`is to be written into
`memory of device
`controller
`
`l
`
`104
`
`/
`
`Combine contents of
`NAND flash memmy
`wlth data quantum to be
`written
`
`,,,. 106
`
`Erase NAND flash
`physical address range
`
`Write combined data to
`appropriate NAND flash
`physical range
`
`Read NAND Flash
`physical address range
`into device controller
`m.ernory
`
`108
`
`110
`
`112
`
`FIG. 3A
`
`Micron Ex. 1005, p. 6
`Micron v. Vervain
`IPR2021-01549
`
`

`

`U.S. Patent
`
`Jun.12,2018
`
`Sheet 4 of 5
`
`US 9,997,240 B2
`
`Compare Data Written to
`NAND FLASH Physical
`Address Range to Data Read
`from NAND FLASH Physical
`Address Range
`
`114
`
`120
`
`Identify next
`quantum of
`available SLC
`NAND flash
`
`122
`
`Remap NAND flash
`physical range to
`next available SLC
`NAND flash
`
`126
`
`Success
`
`118
`
`System
`Failure
`
`/
`
`124
`
`FIG. 3B
`
`Micron Ex. 1005, p. 7
`Micron v. Vervain
`IPR2021-01549
`
`

`

`U.S. Patent
`
`Jun.12,2018
`
`Sheet 5 of 5
`
`US 9,997,240 B2
`
`I
`I
`I
`I
`I
`I
`I
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`I
`i
`
`l
`
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`
`-
`
`60a
`
`---
`
`'
`~ '
`'
`'
`'
`MLC
`
`------------------------
`
`I
`I
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`MLC
`
`i
`i
`
`i
`
`I
`
`MLC
`
`'
`' ~
`'
`'
`
`I
`I
`I
`I
`I
`I
`I
`--
`I
`I
`--
`I
`---------
`--·
`
`-
`
`-
`
`...__
`
`60b
`
`62b
`
`58
`
`•---------N ____ M ________
`
`I --
`
`SLC
`
`I
`I
`I
`I
`I
`I
`
`I
`I
`
`'
`
`I
`I
`I
`I
`
`SLC
`
`I
`I
`
`I
`
`I
`
`I
`
`I
`
`~ I
`
`SLC
`
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`
`'
`'
`'
`
`I
`
`SLC
`
`--
`-
`I
`-----
`--
`
`62a
`
`56
`
`54 ' ------
`
`-
`
`FTL
`
`Interface
`
`FIG. 4
`
`Micron Ex. 1005, p. 8
`Micron v. Vervain
`IPR2021-01549
`
`

`

`US 9,997,240 B2
`
`1
`LIFETIME MIXED LEVEL NON-VOLATILE
`MEMORY SYSTEM
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a Continuation of U.S. patent appli(cid:173)
`cation Ser. No. 14/525,411, filed Oct. 28, 2014, published on
`Oct. 1, 2015, as U.S. Publication No. 2015-0278013, now
`U.S. Pat. No. 9,196,385, issued on Nov. 24, 2015, entitled
`"LIFETIME MIXED LEVEL NON-VOLATILE MEMORY
`SYSTEM". Application Ser. No. 14/525,411 is a Division of
`U.S. patent application Ser. No. 13/455,267, filed Apr. 25,
`2012, published on Jan. 24, 2013, as U.S. Publication No.
`2013-0021846, now U.S. Pat. No. 8,891,298, issued on Nov.
`18, 2014, entitled "LIFETIME MIXED LEVEL NON(cid:173)
`VOLATILE MEMORY SYSTEM". Application Ser. No.
`13/455,267 claims benefit of U.S. Provisional Application
`No. 61/509,257, filed Jul. 19, 2011, entitled "LIFETIME
`MIXED LEVEL NAND FLASH SYSTEM". U.S. Pat. Nos.
`9,196,385 and 8,891,298 and Patent Application Publication
`Nos. 2015-0278013 and 2013-0021846 are hereby incorpo(cid:173)
`rated by reference in their entirety. This application also
`incorporates by reference the complete disclosure of U.S.
`patent application Ser. No. 12/256,362, filed Oct. 22, 2008,
`published on Apr. 30, 2009, as U.S. Publication No. 2009-
`0109787, now U.S. Pat. No. 7,855,916, issued on Dec. 21,
`2010, entitled "NONVOLATILE MEMORY SYSTEMS
`WITH EMBEDDED FAST READ AND WRITE MEMO(cid:173)
`RIES". This application also incorporates by reference the
`complete disclosure of U.S. patent application Ser. No.
`12/915,177, filed Oct. 29, 2010, published on Mar. 10, 2011,
`as U.S. Publication No. 2011-0060870, now U.S. Pat. No.
`8,194,452, issued on Jun. 5, 2012, entitled "NONVOLA(cid:173)
`TILE MEMORY SYSTEMS WITH EMBEDDED FAST
`READ AND WRITE MEMORIES".
`
`TECHNICAL FIELD
`
`5
`
`2
`long-term storage systems. SSDs are preferred for their
`superior performance ( fast access time), mechanical reliabil(cid:173)
`ity and ruggedness, and portability. Flash memory, more
`specifically NAND flash, is the dominant SSD medium
`today.
`RRAM, PCM, MAGRAM and others, will likely play a
`larger role in the future, each of them having their own
`advantages and disadvantages. They may ultimately replace
`flash memories, initially for use as a "write buffer" and later
`10 to replace "SLC flash" and "MLC flash." MLC NAND flash
`is a flash memory technology using multiple levels per cell
`to allow more bits to be stored using the same number of
`transistors. In SLC NAND flash technology, each cell can
`exist in one of two states, storing one bit of information per
`15 cell. Most MLC NAND flash memory has four possible
`states per cell, so it can store two bits of information per cell.
`These semiconductor technology driven "flash alterna(cid:173)
`tives," i.e., RRAM, PCM, MAGRAM and others, have
`several advantages over any (SLC or MLC) flash because
`20 they: 1) allow data to be written over existing data (without
`prior erase of existing data), 2) allow for an erase of
`individual bytes or pages (instead of having to erase an
`entire block), and 3) possess superior endurance (1,000,000
`write-erase cycles compared to typical 100,000 cycles for
`25 SLC flash and less than 10,000 cycles for MLC flash).
`HDDs have several platters. Each platter contains 250-5,
`000 tracks ( concentric circles). Each track contains 64 to 256
`sectors. Each sector contains 512 bytes of data and has a
`unique "physical (memory) address." A plurality of sectors
`30 is typically combined to form a "logical block" having a
`unique "logical address." This logical address is the address
`at which the logical block of physical sectors appears to
`reside from the perspective of an executing application
`program. The size of each logical block and its logical
`35 address (and/or address ranges/boundaries) is optimized for
`the particular operating system (OS) and software applica(cid:173)
`tions executed by the host processor. A computer OS orga(cid:173)
`nizes data as "files." Each file may be located (stored) in
`either a single logical block or a plurality of logical blocks,
`40 and therefore, the location of files typically traverses the
`boundaries of individual (physical) sectors. Sometimes, a
`plurality of files has to be combined and/or modified, which
`poses an enormous challenge for the memory controller
`device of a non-volatile memory system.
`SSDs are slowly encroaching on the HDD space and the
`vast majority ofNAND flash in enterprise servers utilizes a
`SLC architecture, which further comprises a NAND flash
`controller and a flash translation layer (FTL). NAND flash
`devices are generally fragmented into a number of identi-
`50 cally sized blocks, each of which is further segmented into
`some number of pages. It should be noted that asymmetrical
`block sizes, as well as page sizes, are also acceptable within
`a device or a module containing devices. For example, a
`block may comprise 32 to 64 pages, each of which incor-
`55 porates 2-4 Kbit of memory. In addition, the process of
`writing data to a NAND flash memory device is complicated
`by the fact that, during normal operation of, for example,
`single-level storage (SLC), erased bits (usually all bits in a
`block with the value of '1 ') can only be changed to the
`60 opposite state (usually 'O') once before the entire block must
`be erased. Blocks can only be erased in their entirety, and,
`when erased, are usually written to '1' bits. However, if an
`erased block is already there, and if the addresses (block,
`page, etc.) are allowed, data can be written immediately; if
`65 not, a block has to be erased before it can be written to.
`FTL is the driver that works in conjunction with an
`existing operating system (or, in some embedded applica-
`
`This application relates to a system and method for
`providing reliable storage through the use of non-volatile
`memories and, more particularly, to a system and method of
`increasing the reliability and lifetime of a NAND flash
`storage system, module, or chip through the use of a
`combination of single-level cell (SLC) and multi-level cell 45
`(MLC) NAND flash storage without substantially raising the
`cost of the NAND flash storage system. The memory in a
`total non-volatile memory system may contain some SRAM
`(static random-access memory), DRAM (dynamic RAM),
`RRAM (resistive RAM), PCM (phase change memory),
`MAGRAM (magnetic random-access memory), NAND
`flash, and one or more HDDs (hard disk drives) when
`storage of the order of several terabytes is required. The SLC
`non-volatile memory can be
`flash, PCM, RRAM,
`MAGRAM or any other solid-state non-volatile memory as
`long as it has endurance that is superior to that ofMLC flash,
`and it provides for data access speeds that are faster than that
`of MLC flash or rotating storage media (e.g., HDDs).
`
`BACKGROUND
`
`Non-volatile memories provide long-term storage of data.
`More particularly, non-volatile memories can retain the
`stored data even when not powered. Magnetic (rotating)
`hard disk drives (HDD) dominate this storage medium due
`to lower cost compared to solid state disks (SSD). Optical
`(rotating) disks, tape drives and others have a smaller role in
`
`Micron Ex. 1005, p. 9
`Micron v. Vervain
`IPR2021-01549
`
`

`

`US 9,997,240 B2
`
`3
`tions, as the operating system) to make linear flash memory
`appear to the system like a disk drive, i.e., it emulates a
`HDD. This is achieved by creating "virtual" small blocks of
`data, or sectors, out of flash's large erase blocks and man(cid:173)
`aging data on the flash so that it appears to be "write in 5
`place" when in fact it is being stored in different locations in
`the flash. FTL further manages the flash so that there are
`clean/erased places to store data.
`Given the limited number of writes that individual blocks
`within flash devices can tolerate, wear leveling algorithms 10
`are used within the flash devices (as firmware commonly
`known as FTL or managed by a controller) to attempt to
`ensure that "hot" blocks, i.e., blocks that are frequently
`written, are not rendered unusable much faster than other
`blocks. This task is usually performed within a flash trans- 15
`lation layer. In most cases, the controller maintains a lookup
`table to translate the memory array physical block address
`(PBA) to the logical block address (LBA) used by the host
`system. The controller's wear-leveling algorithm determines
`which physical block to use each time data is progranmied, 20
`eliminating the relevance of the physical location of data and
`enabling data to be stored anywhere within the memory
`array and thus prolonging the service life of the flash
`memory. Depending on the wear-leveling method used, the
`controller typically either writes to the available erased 25
`block with the lowest erase count (dynamic wear leveling);
`or it selects an available target block with the lowest overall
`erase count, erases the block if necessary, writes new data to
`the block, and ensures that blocks of static data are moved
`when their block erase count is below a certain threshold 30
`(static wear leveling).
`MLC NAND flash SSDs are slowly replacing and/or
`coexisting with SLC NAND flash in newer SSD systems.
`MLC allows a single cell to store multiple bits, and accord(cid:173)
`ingly, to assume more than two values; i.e., 'O' or '1 '. Most 35
`MLC NAND flash architectures allow up to four ( 4) values
`per cell; i.e., '00', '01', '10', or '11'. Generally, MLC
`NAND flash enjoys greater density than SLC NAND flash,
`at the cost of a decrease in access speed and lifetime
`(endurance). It should be noted, however, that even SLC 40
`NAND flash has a considerably lower lifetime (endurance)
`than rotating magnetic media (e.g., HDDs), being able to
`withstand only between 50,000 and 100,000 writes, and
`MLC NAND flash has a much lower lifetime (endurance)
`than SLC NAND flash, being able to withstand only 45
`between 3,000 and 10,000 writes. As is well known in the
`art, any "write" or "program" to a block in NAND flash
`(floating gate) requires an "erase" (of a block) before
`''write.''
`Despite its limitations, there are a number of applications
`that lend themselves to the use of MLC flash. Generally,
`MLC flash is used in applications where data is read many
`times (but written few times) and physical size is an issue.
`For example, flash memory cards for use in digital cameras
`would be a good application of MLC flash, as MLC can
`provide higher density memory at lower cost than SLC
`memory.
`When a non-volatile storage system combines HDD, SLC
`and MLC (setting aside volatile memory for buffering,
`caching etc) in a single (hybrid) system, new improvements 60
`and solutions are required to manage the methods of writing
`data optimally for improved life time (endurance) of flash
`memory. Accordingly, various embodiments of a NAND
`flash storage system that provides long lifetime (endurance)
`storage at low cost are described herein.
`The following description is presented to enable one of
`ordinary skill in the art to make and use the disclosure and
`
`65
`
`DETAILED DESCRIPTION
`
`The present disclosure is directed to the reliable storage of
`data in read and write memory, and, in particular, to the
`
`4
`is provided in the context of a patent application and its
`requirements. Various modifications
`to
`the preferred
`embodiment and the generic principles and features
`described herein will be readily apparent to those skilled in
`the art. Thus, the present disclosure is not intended to be
`limited to the embodiments shown, but is to be accorded the
`widest scope consistent with the principles and features
`described herein.
`
`SUMMARY
`
`According to one embodiment of the present disclosure,
`there is provided a system for storing data which comprises
`at least one MLC nonvolatile memory module (hereinafter
`referred to as "MLC module") and at least one SLC non(cid:173)
`volatile memory module (hereinafter referred to as "SLC
`module"), each module comprises a plurality of individually
`erasable blocks. The data storage system according to one
`embodiment of the present disclosure further comprises a
`controller for controlling both the at least one MLC module
`and the at least one SLC module. In particular, the controller
`maintains an address map comprising a list of individual
`logical address ranges each of which maps to a similar range
`of physical addresses within either the at least one MLC
`module or the at least one SLC module. After each write to
`(flash) memory, the controller conducts a data integrity
`check to ensure that the data was written correctly. When the
`data was not written correctly, the controller modifies the
`table so that the range of addresses on which the write failed
`is remapped to the next available range of physical addresses
`within the at least one SLC module. The SLC module can be
`(NAND) flash, PCM, RRAM, MAGRAM or any other
`solid-state non-volatile memory as long as it has endurance
`that is superior to that of MLC flash, and it provides for data
`access speeds that are faster than that of MLC flash or
`rotating storage media (e.g., HDDs).
`According to another embodiment of the present disclo(cid:173)
`sure, there is provided a system for storing data which
`comprises a controller that is further adapted to determine
`which of the blocks of the plurality of the blocks in the MLC
`and SLC non-volatile memory modules are accessed most
`frequently and wherein the controller segregates those
`blocks that receive frequent writes into the at least one SLC
`non-volatile memory module and those blocks that receive
`infrequent writes into the at least one MLC nonvolatile
`module.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present disclosure will be more fully understood by
`reference to the following detailed description of one or
`more preferred embodiments when read in conjunction with
`50 the accompanying drawings, in which like reference char(cid:173)
`acters refer to like parts throughout the views and in which:
`FIG. 1 is a block diagram of a computer system incor(cid:173)
`porating one embodiment of the present disclosure;
`FIGS. 2A and 2B are drawings depicting a translation
`55 table/address map in accordance with one embodiment of
`the present disclosure;
`FIGS. 3A and 3B are a flow chart illustrating an exem(cid:173)
`plary method for use in implementing one embodiment of
`the present disclosure; and
`FIG. 4 is a block diagram depicting one embodiment of
`the present disclosure for implementation within a NAND
`flash module.
`
`Micron Ex. 1005, p. 10
`Micron v. Vervain
`IPR2021-01549
`
`

`

`US 9,997,240 B2
`
`5
`reliable storage of data in non-volatile memory, such as, for
`example, NAND flash. Generally, and in particular regard to
`NAND flash memory, two separate banks of NAND flash
`are maintained by a controller. One bank contains economi(cid:173)
`cal MLC NAND flash, while a second bank contains high 5
`endurance SLC NAND flash. The controller conducts a data
`integrity test after every write. If a particular address range
`fails a data integrity test, the address range is remapped from
`MLC NAND flash to SLC NAND flash. As the SLC NAND
`flash is used to boost the lifetime (endurance) of the storage
`system, it can be considerably lesser in amount than the
`MLC NAND flash. For example, a system may set SLC
`NAND flash equal to 12.5% or 25% of MLC NAND flash
`(total non-volatile memory storage space=MLC+SLC).
`Turning to the Figures and to FIG. 1 in particular, a
`computer system 10 depicting one embodiment of the pres(cid:173)
`ent disclosure is shown. A processor 12 is coupled to a
`device controller 14, such as a chipset, using a data link well
`known in the art, such as a parallel bus or packet-based link.
`The device controller 14 provides interface functions to the
`processor 12. In some computer systems, the device con(cid:173)
`troller 14 may be an integral part of the (host) processor 12.
`The device controller 14 provides a number of input/output
`ports 16 and 18, such as, for example, serial ports (e.g., USB
`ports and Firewire ports) and network ports (e.g., Ethernet
`ports and 802.11 "Wi-Fi" ports). The device controller 14
`may also control a bank of, for example, DRAM 20. In
`addition, the device controller 14 controls access to one or
`more disks 24, such as, for example, a rotating magnetic
`disk, or an optical disk, as well as two or more types of
`NAND flash memory. One type of NAND flash memory is
`a MLC NAND flash memory module 26. Another type of
`NAND flash memory is a SLC NAND flash memory module
`28.
`The device controller 14 maintains a translation table/ 35
`address map which may include address translations for all
`devices in the computer system. Nonetheless, the discussion
`in the present disclosure will be limited only to NAND flash
`memory modules. In particular, the device controller 14
`maintains a translation table that maps logical computer
`system addresses to physical addresses in each one of the
`MLC- and SLC-NAND flash memory modules 26 and 28,
`respectively. As MLC flash memory is less expensive than
`SLC flash memory, on a cost per bit basis, the translation
`table will initially map all logical NAND flash addresses to
`the MLC NAND flash memory module 26. The address
`ranges within the translation table will assume some mini(cid:173)
`mum quantum, such as, for example, one block, although a
`smaller size, such as one page could be used, if the NAND
`flash has the capability of erasing the smaller size quantum.
`A "read-modify-write" scheme is used to write data to the
`NAND flash. Data to be written to NAND flash is main(cid:173)
`tained in DRAM 20. After each write to an address within
`a particular address range, the device controller 14 will-as
`time permits-perform a read on the address range to ensure
`the integrity of the written data. If a data integrity test fails,
`the address range is remapped from the MLC NAND flash
`memory module 26 to the next available address range in the
`SLC NAND flash memory module 28.
`FIGS. 2A and 2B illustrate one embodiment of a trans- 60
`lation table/address map of the present disclosure. In FIG.
`2A, a list of logical address ranges (RO-RN) is translated to
`physical address ranges. As illustrated, all of the logical
`address ranges are translated to blocks on the MLC NAND
`flash memory module 26. However, through the application
`of a data integrity verification check ( explained in more
`detail below) it is determined that, for example, address
`
`6
`range R2 corresponds to failed quanta of data stored in block
`2 of the MLC NAND flash memory module 26. FIG. 2B
`shows the quanta of data which failed the data integrity
`verification check (see FIG. 2A) remapped to the next
`available range of physical addresses within the SLC NAND
`flash memory module 28, in this example, SLC/block 0.
`FIGS. 3A and 3B are a flow chart illustrating a method for
`utilizing a NAND flash memory system incorporating one
`embodiment of the present disclosure. The method begins in
`10 a step 100, when a command to write a quantum of data
`stored in DRAM to a particular location in NAND flash
`memory is received. In step 102, the quantum of data is read
`from DRAM into memory within the device controller
`15 (which acts as the memory controller). In step 104, both the
`logical address range and the NAND flash physical address
`range to which the quantum of data is to be written, is read
`into memory of the device controller. In step 106, the
`quantum of data to be written is combined with the contents
`20 of the NAND flash memory. In step 108, the NAND flash
`physical address range to be written is erased. In step 110,
`the combined data is written to the appropriate NAND flash
`physical address r