USOO8554968B1
`
`(12) United States Patent
`Onufryk et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,554.968 B1
`Oct. 8, 2013
`
`(54) INTERRUPTTECHNIQUE FOR A
`NONVOLATILE MEMORY CONTROLLER
`
`(75) Inventors: Peter Z. Onufryk, Flanders, NJ (US);
`Jayesh Patel, San Jose, CA (US); Ihab
`Jaser, San Jose, CA (US)
`
`(73) Assignee: PMC-Sierra, Inc., Sunnyvale, CA (US)
`
`( c ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 261 days.
`(21) Appl. No.: 13/052,388
`(22) Filed:
`Mar 21, 2011
`O
`O
`Related U.S. Application Data
`(60) Provisional application No. 61/374,242, filed on Aug.
`16, 2010.
`
`(2006.01)
`
`(51) Int. Cl.
`G06F I3/24
`(52) U.S. Cl.
`USPC .............................. 710/260; 710/48; 710/269
`(58) Field of Classification Search
`USPC ...................... 710/48, 50, 260, 269; 711/156
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,875,343 A * 2/1999 Binford et al. ................ T10,263
`7.620,784 B2 11/2009 Panabaker
`
`5/2010 Yoshida et al.
`7,708,195 B2
`8,244.946 B2 * 8/2012 Gupta et al. .................. T10,263
`2008/0256280 A1* 10, 2008 Ma. ................................ T10,263
`2008/0320214 A1 12/2008 Ma et al.
`2009, OO773O2 A1
`3/2009 Fukuda
`2010.0185808 A1
`7, 2010 Yu et al.
`2010/0262979 A1 10, 2010 Borchers et al.
`2011/O161678 A1
`6, 2011 Niwa
`OTHER PUBLICATIONS
`
`NVM Express, Revision 1.0b; Intel Corporation; pp. 103-106 and
`110-114; Jul 12, 2011.*
`NVM Express, Revision 1.0; Intel Corporation; Mar. 1, 2011.
`* cited by examiner
`Primary Examiner — Glenn AAuve
`(74) Attorney, Agent, or Firm — Kenneth Glass; Stanley J.
`Pawlik; Glass & Associates
`
`ABSTRACT
`(57)
`A nonvolatile memory controller processes a nonvolatile
`memory command and generates a completion status for the
`nonvolatile memory command. The nonvolatile memory con
`troller transmits the completion status to a host processing
`unit for storage in a completion queue of the host processing
`unit. An interrupt manager in the nonvolatile memory con
`troller determines the completion queue contains an unproc
`essed completion status and generates an interrupt message
`packet. The nonvolatile memory controller transmits the
`interrupt message packet to the host processing unit for trig
`gering an interrupt in the host processing unit and alerting the
`host processing unit to the unprocessed completion status.
`
`19 Claims, 17 Drawing Sheets
`
`100-
`
`
`
`110
`Host
`Processing
`Unit
`
`105
`Nonvolatile Mcmory Controller
`
`115
`Communication
`Network
`
`120
`Host Controller
`Interface
`
`125
`Network
`Module
`
`130
`Control Module
`300
`
`
`
`Storage
`Controller
`
`140
`NWM
`Storage
`Device
`
`Processor
`
`160
`Controller Memory
`315
`-- Buffer
`Mcmory
`
`*
`
`320,
`| Memory |
`Module :
`(optional)
`- - - - - -
`
`I
`
`- - - - - - -
`325
`Memory
`Hi Interface -
`| (optional):
`
`- - - - - - -
`
`
`Ex.1001 / Page 1 of 34Ex.1001 / Page 1 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`US 8,554.968 B1
`
`SI
`
`JOSS000IJ.
`
`Ç?T
`
`W – OOI
`
`
`
`
`Ex.1001 / Page 2 of 34Ex.1001 / Page 2 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`
`Ex.1001 / Page 3 of 34Ex.1001 / Page 3 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`
`Ex.1001 / Page 4 of 34Ex.1001 / Page 4 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 4 of 17
`
`US 8,554.968 B1
`
`
`
`150
`Processor
`
`400
`Processing Unit
`
`405
`Message
`Network
`Interface
`
`410
`Data
`Network
`Interface
`
`To/From Message
`Network 200
`
`To/From Data
`Network 205
`
`415
`Instruction
`Memory
`
`FIG. 4
`
`
`Ex.1001 / Page 5 of 34Ex.1001 / Page 5 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 5 Of 17
`
`US 8,554.968 B1
`
`
`
`110
`Host Processing Unit
`
`500
`Computing
`Processor
`
`510
`Communication
`Interface
`
`To/From Communication
`Nctwork 115
`
`515
`Memory System
`
`520
`Submission
`Qucuc
`
`525
`Completion
`Qucuc
`
`FIG. 5
`
`
`Ex.1001 / Page 6 of 34Ex.1001 / Page 6 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 6 of 17
`
`US 8,554.968 B1
`
`
`
`
`
`520 -
`
`605
`
`610
`
`525 -
`
`705
`
`710
`
`600
`Storage Location
`
`600
`Storage Location
`
`600
`Storage Location
`
`600
`Storage Location
`
`600
`Storage Location
`
`FIG. 6
`
`700
`Storage Location
`
`700
`Storage Location
`
`700
`Storage Location
`
`700
`Storage Location
`
`700
`Storage Location
`
`FIG. 7
`
`
`Ex.1001 / Page 7 of 34Ex.1001 / Page 7 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 7 Of 17
`
`US 8,554.968 B1
`
`
`
`120
`Host Controller Interface
`
`To/From Communication
`Network 115
`
`son
`k Manager
`
`To Message Network 200
`
`To/From Communication
`Network 115
`
`s
`C
`ompletion
`Manager
`
`To/From Message Network 200
`
`FIG. 8
`
`
`Ex.1001 / Page 8 of 34Ex.1001 / Page 8 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`US 8,554.968 B1
`
`
`
`6 " OIH
`
`
`Ex.1001 / Page 9 of 34Ex.1001 / Page 9 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`
`Ex.1001 / Page 10 of 34Ex.1001 / Page 10 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 10 of 17
`
`US 8,554.968 B1
`
`
`
`ÇI I SJOAA13N.
`
`
`
`
`
`JoIIoJquoO Joñeue W jdn JonuI
`
`SZI I
`
`[ [ "OIH
`
`07 I I
`
`
`Ex.1001 / Page 11 of 34Ex.1001 / Page 11 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 11 of 17
`
`US 8,554.968 B1
`
`
`
`FIG. 12
`
`1125
`Interrupt Vector State
`
`1300
`Interrupt Event Indicators
`
`1310
`Empty
`Queue
`Indicator
`
`1315
`Aggregate
`Doorbell
`Update Status
`
`1320
`Aggregate
`Event Count
`
`1325
`Aggregate
`Time
`Indicator
`
`1400
`Aggregate
`Time
`Counter
`
`Timer
`Active
`Indicator
`
`FIG. 14
`
`
`Ex.1001 / Page 12 of 34Ex.1001 / Page 12 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 12 of 17
`
`US 8,554.968 B1
`
`
`
`112)
`Configuration Register
`
`1500
`Interrupt Vector
`Identificr
`
`Event
`Aggregation
`Threshold
`
`Event
`Aggregation
`Enable
`
`1620
`Time
`Aggregation
`Threshold
`
`1625
`Time
`Aggregation
`Enable
`
`FIG 16
`
`
`Ex.1001 / Page 13 of 34Ex.1001 / Page 13 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 13 of 17
`
`US 8,554.968 B1
`
`Update interrupt event indicators of
`interrupt vector States based on
`completion queue cvent states
`
`Update interrupt statuses of interrupt
`vector States based on interrupt event
`indicators
`
`Selectively generate interrupt message
`packcts based on interrupt statuscs
`
`
`
`Start
`
`Update interrupt event
`indicators of interrupt vector
`states based on completion
`ueue eVent StateS
`C
`
`Update interrupt statuses of
`interrupt vector States based on
`interrupt cvent indicators
`
`1710
`
`17ns.
`
`---------------
`
`Selectively generate interrupt
`message packets based on
`interrupt statuses
`----------------
`
`1715
`
`FIG. 18
`
`
`Ex.1001 / Page 14 of 34Ex.1001 / Page 14 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 14 of 17
`
`US 8,554.968 B1
`
`1705 -
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Select completion queue State
`
`Identify interrupt vector state associated with
`completion queue state
`
`Completion
`queue empty?
`
`Add event count of completion queue State to
`aggregate event count of interrupt vector State
`
`Clear empty queue indicator of interrupt vector state
`
`Doorbell
`update status
`Set?
`
`Clear aggregate doorbell update Status of interrupt
`VectOr State
`
`
`
`Clear completion queue event indicators of
`completion queue State
`
`
`
`
`Ex.1001 / Page 15 of 34Ex.1001 / Page 15 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 15 Of 17
`
`US 8,554.968 B1
`
`Start
`
`2040
`Clear aggregate event count
`
`2005
`
`
`
`
`
`
`
`Empty
`queue indicator
`Set?
`
`
`
`
`
`Aggregate
`doorbell update
`status set?
`
`2045
`
`2050
`Clear interrupt status
`
`2055
`Update aggregate time
`COunter
`
`2060
`Update time stamp
`
`Aggregate
`time threshold
`reached?
`
`
`
`Update time stamp
`
`Set interrupt status
`
`|
`
`A.
`
`A.
`
`Set timer active
`indicator
`
`Set interrupt
`Status
`
`-------------
`
`
`
`
`
`
`
`
`
`2090
`Additional
`interrupt vector
`State?
`
`N
`
`End
`
`FIG. 20
`
`Clear aggregate time counter
`
`2080
`Set empty queue indicator
`
`2085
`Set aggregate doorbell
`update status
`
`
`Ex.1001 / Page 16 of 34Ex.1001 / Page 16 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 16 of 17
`
`US 8,554.968 B1
`
`
`
`
`
`Select interrupt vector state
`
`
`
`
`
`
`
`
`
`Empty
`queue indicator
`Set?
`
`2110
`
`
`
`
`
`
`
`
`
`
`
`Event
`aggregation
`threshold
`reached?
`
`Set interrupt status
`
`
`
`
`
`Clear aggregate event count
`
`2130
`Set empty queue indicator
`
`
`
`
`
`2145
`
`
`
`Additional
`interrupt vector
`State?
`
`2135
`Clear aggregate event count
`2140
`Clear interrupt status
`
`FIG 21
`
`
`Ex.1001 / Page 17 of 34Ex.1001 / Page 17 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 17 Of 17
`
`US 8,554.968 B1
`
`1715 -
`
`
`
`
`
`
`
`
`
`
`
`Select interrupt vector state
`
`2205
`
`Interrupt
`Status Set?
`
`Generate interrupt message packet for triggering
`interrupt in host processing unit
`
`Send interrupt message packet to host processing
`unit
`
`
`
`
`
`Clear interrupt status in interrupt vector State
`
`2230
`
`
`
`Additional
`interrupt vector
`State?
`
`
`
`
`Ex.1001 / Page 18 of 34Ex.1001 / Page 18 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`1.
`INTERRUPTTECHNIQUE FOR A
`NONVOLATILE MEMORY CONTROLLER
`
`US 8,554,968 B1
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application claims benefit of U.S. provisional patent
`application Ser. No. 61/374,242 filed Aug. 16, 2010 and
`entitled “Non Volatile Memory Host Controller Interface
`Device,” which is incorporated herein by reference in its
`entirety.
`
`10
`
`BACKGROUND
`
`15
`
`Nonvolatile Memory Express (NVMe) is a standard defin
`ing a register interface, command set, and feature set for
`high-performance Peripheral Component Interconnect
`Express (PCIe) based solid-state drives (SSDs). An NVMe
`interface includes a register interface defined by the NVMe
`standard and allows a host computer to communicate with a
`non-volatile memory Subsystem such as a Flash storage
`device. Typically, the NVMe interface is implemented as a
`stand-alone Peripheral Component Interconnect (PCI)
`device.
`In a typical computing system including an NVMe inter
`face, a host computer provides nonvolatile memory com
`mands to a non-volatile memory Subsystem including the
`NVMe interface. In turn, the NVMe interface processes the
`nonvolatile memory commands to manage data in a non
`volatile memory device of the non-volatile memory sub
`system. Although the NVMe standard specifies a register set
`and a standard command set for designing an NVMe inter
`face, the NVMe standard leaves other implementation details
`open to a designer of the non-volatile memory Subsystem.
`
`25
`
`30
`
`35
`
`SUMMARY
`
`In various embodiments, a nonvolatile memory controller
`includes a processor and an interrupt manager. The processor
`processes a nonvolatile memory command. Additionally, the
`processor generates a completion status based on the non
`Volatile memory command for storage of the completion
`status in a completion queue of a host processing unit. The
`interrupt manager determines the completion queue in the
`host processing unit includes an unprocessed completion sta
`tus. Further, the interrupt manager generates an interrupt mes
`sage packet for triggering an interrupt in the host processing
`unit. In this way, the nonvolatile memory controller alerts the
`host processing unit to the unprocessed completion status in
`the completion queue. Because the interrupt manager alerts
`the host processing unit of the unprocessed completion status
`in the completion queue, the host processing unit need not
`include resources for monitoring the completion queue for an
`unprocessed completion status.
`A nonvolatile memory controller, in accordance with one
`embodiment, includes an interrupt manager. The interrupt
`manager is configured to determine a completion queue in a
`host processing unit contains an unprocessed completion sta
`tus. The interrupt manager is further configured to generate a
`completion queue State for indicating the occurrence of a
`completion queue event associated with the completion
`queue, generate an interrupt vector state based on the comple
`tion queue State, determine the completion queue of the host
`processing unit contains an unprocessed completion status
`based on the interrupt vector State, and generate an interrupt
`message packet for triggering an interrupt in the host process
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`ing unit to alert the host processing unit of the unprocessed
`completion status in the completion queue.
`A nonvolatile memory controller, in accordance with one
`embodiment, includes a completion queue State memory, an
`interrupt vector State memory, and an interrupt manager con
`troller. The interrupt manager controller is coupled to the
`completion queue State memory and the interrupt vector state
`memory. The completion queue State memory is configured
`to store completion queue States corresponding to completion
`queues in a host processing unit. The interrupt vector state
`memory is configured to store interrupt vector states corre
`sponding to interrupt vectors in the host processing unit. The
`interrupt manager controller is configured to map at least one
`of the completion queue States to an interrupt vector state. The
`interrupt manager is further configured to generate the inter
`rupt vector state based on each completion queue State
`mapped to the interrupt vector state. Additionally, the inter
`rupt manager controller is configured to determine a comple
`tion queue corresponding to a completion queue State mapped
`to the interrupt vector State includes an unprocessed comple
`tion status based on the interrupt vector state. Further, the
`interrupt manager controller is configured to generate an
`interrupt message packet for triggering an interrupt in the host
`processing unit to alert the host processing unit of the unproc
`essed completion status in the completion queue.
`A method, in accordance with one embodiment, includes
`generating a completion queue State by a nonvolatile memory
`controller for indicating the occurrence of a completion
`queue event associated with a completion queue in a host
`processing unit. The method further includes generating an
`interrupt vector state by the nonvolatile memory controller
`based on the completion queue state. The interrupt vector
`state is associated with an interrupt vector in the host process
`ing unit. Additionally, the method includes determining by
`the nonvolatile memory controller based on the interrupt vec
`tor state that the completion queue contains an unprocessed
`completion status. The method also includes generating an
`interrupt message packet by the nonvolatile memory interface
`for triggering an interrupt in the host processing unit to alert
`the host processing unit of the unprocessed completion status
`in the completion queue.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The accompanying drawings are included to provide a
`further understanding of the invention, and are incorporated
`in and constitute a part of this specification. The drawings
`illustrate embodiments of the invention, and together with the
`description, serve to explain the principles of the invention.
`FIG. 1 is a block diagram of a computing system, in accor
`dance with an embodiment of the present invention.
`FIG. 2 is a block diagram of a network module, in accor
`dance with an embodiment of the present invention.
`FIG. 3 is a block diagram of a computing system, in accor
`dance with an embodiment of the present invention.
`FIG. 4 is a block diagram of a processor, in accordance with
`an embodiment of the present invention.
`FIG. 5 is a block diagram of a host processing unit, in
`accordance with an embodiment of the present invention.
`FIG. 6 is a block diagram of a Submission queue, in accor
`dance with an embodiment of the present invention.
`FIG. 7 is a block diagram of a completion queue, in accor
`dance with an embodiment of the present invention.
`FIG. 8 is a block diagram of a host controller interface, in
`accordance with an embodiment of the present invention.
`FIG. 9 is a block diagram of a Submission manager, in
`accordance with an embodiment of the present invention.
`
`Ex.1001 / Page 19 of 34Ex.1001 / Page 19 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`3
`FIG. 10 is a block diagram of a completion manager, in
`accordance with an embodiment of the present invention.
`FIG. 11 is a block diagram of an interrupt manager, in
`accordance with an embodiment of the present invention.
`FIG. 12 is a block diagram of a completion queue State, in
`accordance with an embodiment of the present invention.
`FIG. 13 is a block diagram of an interrupt vector state, in
`accordance with an embodiment of the present invention.
`FIG. 14 is a block diagram of an aggregate time indicator,
`in accordance with an embodiment of the present invention.
`FIG. 17 is a flow chart for a method of alerting a host
`processing unit of an unprocessed completion status, in
`accordance with an embodiment of the present invention.
`FIG. 18 is a flow chart for a method of alerting a host
`processing unit of an unprocessed completion status, in
`accordance with an embodiment of the present invention.
`FIG. 19 is a flow chart for a portion of a method of alerting
`a host processing unit of an unprocessed completion status, in
`accordance with an embodiment of the present invention.
`FIG. 20 is a flow chart for a portion of a method of alerting
`a host processing unit of an unprocessed completion status, in
`accordance with an embodiment of the present invention.
`FIG. 21 is a flow chart for a portion of a method of alerting
`a host processing unit of an unprocessed completion status, in
`accordance with an embodiment of the present invention.
`FIG.22 is a flow chart for a portion of a method of alerting
`a host processing unit of an unprocessed completion status, in
`accordance with an embodiment of the present invention.
`
`DETAILED DESCRIPTION
`
`In various embodiments, a nonvolatile memory controller
`includes an interrupt manager that identifies a completion
`queue containing an unprocessed completion status in a host
`processing unit. The interrupt manager generates an interrupt
`message packet for generating an interrupt in the host pro
`cessing unit. Moreover, the interrupt is associated with the
`completion queue containing the unprocessed completion
`status. In this way, the interrupt manager alerts the host pro
`cessing unit to the unprocessed completion status in the
`completion queue.
`FIG. 1 illustrates a computing system 100, in accordance
`with an embodiment of the present invention. The computing
`system 100 includes a host processing unit 110, a communi
`cation network 115, a nonvolatile memory controller 105, and
`a nonvolatile memory device 140. The communication net
`work 115 is coupled (e.g., connected) to the host processing
`unit 110 and the nonvolatile memory controller 105. Addi
`tionally, the nonvolatile memory controller 105 is coupled
`(e.g., connected) to the nonvolatile memory device 140. In
`various embodiments, the nonvolatile memory device 140 is
`a flash storage device and the nonvolatile memory controller
`105 is a flash controller.
`In various embodiments, the computing system 100 is a
`desktop computer, a server, a computer workstation, or the
`like. In some embodiments, the computing system 100 is a
`portable electronic device, such as a portable computer, a
`mobile phone, a digital camera, a media player, a personal
`digital assistant, a pager, a global positioning system, or the
`like.
`The communication network 115 facilitates communica
`tion between the host processing unit 110 and the nonvolatile
`memory controller 105. For example, the communication
`network 115 may be a packet communication network, Such
`as a Peripheral Component Interconnect Express (PCIe) net
`work. The nonvolatile memory controller 105 manages data
`stored in the nonvolatile memory device 140 and communi
`
`US 8,554,968 B1
`
`4
`cates with the host processing unit 110 through the commu
`nication network 115 for transferring data between the host
`processing unit 110 and the nonvolatile memory device 140.
`In various embodiments, the nonvolatile memory controller
`105 processes nonvolatile memory commands generated by
`the host processing unit 110 for controlling operation of the
`nonvolatile memory controller 105. In some embodiments,
`the host processing unit 110 generates Enterprise Non-Vola
`tile Memory Host Control Interface (eNVMHCI) commands
`from the host processing unit 110 and the nonvolatile memory
`controller 105 processes the eNVMHCI commands to man
`age operation of the nonvolatile memory controller 105.
`In various embodiments, the nonvolatile memory control
`ler 105 includes a host controller interface 120, a network
`module 125, a control module 130, a processor module 145,
`and a controller memory 160. The network module 125 is
`coupled (e.g., connected) to the host controller interface 120,
`the control module 130, the processor module 145, and the
`controller memory 160. Additionally, the host controller
`interface 120 is coupled (e.g., connected) to the communica
`tion network 115, and the control module 130 is coupled (e.g.,
`connected) to the nonvolatile memory device 140. Further
`more, the processor module 145 includes processors 150
`coupled (e.g., connected) to the network module 125. In these
`embodiments, each of the host controller interface 120, the
`control module 130, the processors 150, and the controller
`memory 160 is a functional unit of the nonvolatile memory
`controller 105.
`In various embodiments, each of the host controller inter
`face 120, the control module 130, the processor module 145,
`and the controller memory 160 is source node or a destination
`node of the nonvolatile memory controller 105. In this way,
`each functional unit of the nonvolatile memory controller 105
`may be a source node or a destination node. In some embodi
`ments, one or more of the host controller interface 120, the
`control module 130, the processor module 145, and the con
`troller memory 160 is both a source node and a destination
`node of the nonvolatile memory controller 105. In this way, a
`functional unit of the nonvolatile memory controller 105 may
`be both a source node and a destination node.
`The host controller interface 120 facilitates communica
`tion between the communication network 115 and the func
`tional units of the nonvolatile memory controller 105 through
`the network module 125. The control module 130 manages
`data in the nonvolatile memory device 140. For example, the
`control module 130 may read data from the nonvolatile
`memory device 140 and write data into the nonvolatile
`memory device 140.
`The controller memory 160 stores data being transferred
`from the host processing unit 110 to the nonvolatile memory
`device 140. Additionally, the controller memory 160 stores
`data being transferred from the nonvolatile memory device
`140 to the host processing unit 110. In this way, the controller
`memory 160 is an intermediate storage location for tempo
`rarily storing data being transferred between the host process
`ing unit 110 and the nonvolatile memory device 140. In vari
`ous embodiments, the controller memory 160 includes a
`random access memory (RAM), such as a static random
`access memory (SRAM) or a dynamic random access
`memory (DRAM).
`In various embodiments, the host processing unit 110 gen
`erates nonvolatile memory commands and stores the nonvola
`tile memory commands in one or more Submission queues
`contained in the host processing unit 110. The host controller
`interface 120 determines that a submission queue in the host
`processing unit 110 contains a nonvolatile memory com
`mand, generates a request packet, and transmits the request
`
`Ex.1001 / Page 20 of 34Ex.1001 / Page 20 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`US 8,554,968 B1
`
`10
`
`15
`
`40
`
`45
`
`5
`packet to the host processing unit 110 through the communi
`cation network 115. In response to the request packet gener
`ated by the host controller interface 120, the host processing
`unit 110 generates a completion packet including a nonvola
`tile memory command contained in a Submission queue and
`transmits the completion packet to the host controller inter
`face 120 of the nonvolatile memory controller 105 through
`the communication network 115.
`The host controller interface 120 retrieves the completion
`packet including the nonvolatile memory command, gener
`ates a request message packet including the nonvolatile
`memory command based on the request packet, and provides
`the request message packet to the network module 125. In
`turn, the network module 125 routes the request message
`packet to the processor module 145. The processor module
`145 processes the nonvolatile memory command in the
`request message packet, generates a completion message
`packet including a completion status (e.g., a completion
`entry) based on the request message packet, and provides the
`completion message packet to the network module 125. The
`network module 125 routes the completion message packet to
`the host controller interface 120.
`The host controller interface 120 generates a request packet
`including the completion status based on the completion mes
`sage packet and transmits the request packet to the host pro
`25
`cessing unit 110 through the communication network 115. In
`turn, the host processing unit 110 stores the completion status
`of the request message packet received from the host control
`ler interface 120 in a completion queue contained in the host
`processing unit 110. Additionally, the host processing unit
`30
`110 processes the completion status to determine the status of
`processing the nonvolatile memory command (i.e., a process
`ing status).
`In various embodiments, a processor module 145 pro
`cesses the nonvolatile memory command in a request mes
`35
`sage packet by generating additional request message pack
`ets, each of which includes a command. The processor 150
`provides the request message packets to the network module
`125. In turn, the network module 125 routes each of the
`request message packets received from the processor module
`145 to a functional unit of the nonvolatile memory controller
`105 identified in the request message packet. In this way, the
`processor 150 functions as a source node and the functional
`unit receiving the request message packet functions as a des
`tination node.
`The functional unit receiving a request message packet
`from the processor 150 through the network module 125
`processes the command in the request message packet, gen
`erates a completion message packet including a completion
`status based on the request packet, and provides the comple
`tion message packet to the network module 125. The comple
`tion status of the completion message packet indicates a status
`of the command processed by the functional unit (i.e., a
`processing status). The network module 125 routes the
`completion message packet to the processor 150.
`In various embodiments, the processor module 145 pro
`cesses the nonvolatile memory command in the request mes
`sage packet received from the host controller interface 120 by
`generating a request message packet including a data transfer
`command for transferring data between the host processing
`unit 110 and the controller memory 160, and generating
`another request message packet including a data transfer
`command for transferring data between the controller
`memory 160 and the nonvolatile memory device 140. In this
`way, the processor module 145 generates request message
`packets for transferring data in a piecemeal manner between
`the host processing unit 110 and the nonvolatile memory
`
`50
`
`55
`
`60
`
`65
`
`6
`device 140 based on the request message packet including the
`nonvolatile memory command.
`In some cases, each functional unit receiving a request
`message packet from the processor module 145 generates one
`or more request data packets based on the request message
`packet. Each of the request data packets is a request for
`transferring data between functional units of the nonvolatile
`memory controller 105 through the network module 125 or
`transferring data between the host processing unit 110 and the
`host controller interface 120. For example, the control mod
`ule 130 may generate request data packets and provide the
`request data packets to the network module 125 for routing to
`the controller memory 160. In this example, the controller
`memory 160 generates completion packets including data
`stored in the controller memory 160 and provides the comple
`tion packets to the network module 125 for routing to the
`control module 130.
`In various embodiments, the nonvolatile memory control
`ler 105 is implemented in an integrated circuit of an integrated
`circuit device which may include an integrated circuit pack
`age containing the integrated circuit. In some embodiments,
`the nonvolatile memory controller 105 is implemented in a
`single integrated circuit die. In other embodiments, the non
`volatile memory controller 105 is implemented in more than
`one integrated circuit die of an integrated circuit device which
`may include a multichip package containing the integrated
`circuit die.
`In some embodiments, the host processing unit 110, the
`communication network 115, and the nonvolatile memory
`controller 105 are mounted to a printed circuit board. In these
`embodiments, the host processing unit 110 and the nonvola
`tile memory controller 105 are directly coupled to the com
`munication network 115. For example, the host processing
`unit 110 and the nonvolatile memory controller 105 may be
`directly connected to the communication network 115
`through wires in the printed circuit board.
`FIG. 2 illustrates the network module 125, in accordance
`with an embodiment of the present invention. The network
`module 125 includes a message network 200 and a data
`network 205. Each of the message network 200 and the data
`network 205 is coupled (e.g., connected) to the host controller
`interface 120, the control module 130, the processor module
`145, and the controller memory 160.
`The message network 200 routes message packets, such as
`request message packets and completion message packets,
`between functional units of the nonvolatile memory control
`ler 105. In various embodiments, the message network 200
`routes message packets among functional units of the non
`volatile memory controller 105 by using an inter-processor
`communication (IPC) protocol.
`The data network 205 routes data packets, such as data
`request packets and data completion packets, between func
`tional units of the nonvolatile memory controller 105. In
`various embodiments, the data network 205 routes data pack
`ets among functional units of the nonvolatile memory con
`troller 105 by using an inter-processor communication (IPC)
`protocol.
`FIG. 3 illustrates a computing system 100, in accordance
`with an embodiment of the present invention. In this embodi
`ment, the control module 130 includes a storage controller
`300. The storage controller 300 is coupled (e.g., connected) to
`the network module 125 and the nonvolatile memory device
`140. Furthermore, the controller memory 160 includes a
`buffer memory 315 coupled to the network module 125. Each
`of the storage controller 300 and the buffer memory 315 is a
`functional unit of the nonvolatile memory controller 105.
`
`Ex.1001 / Page 21 of 34Ex.1001 / Page 21 of 34
`
`Sandisk Technologies, Inc. et alSandisk Technologies, Inc. et al
`
`

`

`US 8,554,968 B1
`
`10
`
`15
`
`7
`The storage controller 300 manages data in the nonvolatile
`memory device 140. For example, the storage controller 300
`may