(cid:9) (cid:9)
`
`[54] REMOTE DATA SHADOWING USING A
`[54] REMOTE DATA SHADOWING USING A
`MULTIMODE INTERFACE TO
`MULTlMODE INTERFACE To
`DYNAMICALLY RECONFIGURE CONTROL
`DYN AMIC ALLY RECONFIGURE CONTROL
`LINK-LEVEL AND COMMUNICATION
`LINK-LEVEL AND COMMUNICATION
`LINK-LEVEL
`LINK-LEVEL
`
`[75] Inventors: Roger G. Hathorn; Bret W. Holley;
`[75] Inventors: Roger G. Hathorn; Bret W_ Holley;
`James L. Iskiyan; William F. Micka;
`James L. Iskiyan; William F. Micka;
`Asim R. Qureshi, all of Tucson, Ariz.
`Asim R. Qureshi, all of Tucson, Ariz.
`
`[73] Assignee: International Business Machines
`[73] Assignee: International Business Machines
`Corporation, Armonk, N.Y.
`Corporation’ Armonk’ NY‘
`
`[21] Appl. No.: 203,248
`[21] APPL NO-I 2031248
`[22] Filed:
`Mar. 1,1994
`Mar. 1, 1994
`[22] Filed: (cid:9)
`
`........................................ .. G06F 15/02
`[51] Int. Cl.6
` GO6F 15/02
`[51] Int. C1.6
` 395/861; 395/200.09; 364/222.2;
`[52] U.S. Cl.
`[52] US. Cl. ............... .. 395/861; 395/200.09; 364/2222;
`364/242.94; 364/2425; 364/DIG. 1
`364/242.94; 364/242.5; 364/DIG. 1
` 364/952, 200;
`[58] Field of Search (cid:9)
`[58] Field of Search ................................... .. 364/952, 200;
`395/650, 145, 600, 575, 20001, 20002,
`395/650, 145, 600, 575, 200.01, 200.02,
`20009, 321, 361
`200.09, 821, 861
`
`[56]
`[56] (cid:9)
`
`References Cited
`References Cited
`
`_
`
`U.S. PATENT DOCUMENTS
`U'S' PATENT DOCUMENTS
` 364/900
`5,058,056 10/1991 Hammer et a1. ...................... .. 364/900
`5,058,056 10/1991 Hammer et al. (cid:9)
` 395/575
`5,089,958 2/1992 Horton et al. (cid:9)
` 395/575
`5,155,845 10/1992 Beal et al. (cid:9)
`5,155,845 10/1992 Beal et a1. ............................ .. 395/575
`
`5,089,958
`
`2/1992 Horton et a1. . . . . . . .
`
`. . . . . .. 395/575
`
`United States Patent [19]
`United States Patent [19]
`Hathorn et al.
`Hathorn et al.
`
`lllllllllllllllllllllllllllllllllllllIlllllllllllllllllllllllllllllllllllll
`11111 11111111 111 11111 111119101,11111109111 1101 11111111 11111
`USOO5574950A
`5,574,950
`[1 1] Patent Number: (cid:9)
`[11] Patent Number:
`5,574,950
`Nov. 12, 1996
`[45] Date of Patent: (cid:9)
`[45] Date of Patent:
`Nov. 12, 1996
`
` 395/600
`5,404,508
`4/1995 Konrad et al. (cid:9)
`5,404,508
`4/1995 Konrad et a1. ........................ .. 395/600
` 395/275
`5/1995 Elliott (cid:9)
`5,420,988
`5,420,988
`5/1995 Elliott ........... ..
`395/275
`8/1995 Shomler et al. (cid:9)
` 395/180
`5,446,871
`5,446,871
`8/1995 Shomler et a1. ...................... .. 395/180
`FOREIGN PATENT DOCUMENTS
`FOREIGN PATENT DOCUMENTS
`
` GO6F 13/10
`0212425 3/1987 European Pat. Off. (cid:9)
`0212425 3/1987 European Pat. O?i ...... .. G06F 13/10
`0359384 3/1990 European Pat. Off. (cid:9)
` GO6F 11/10
`0359384 3/1990 European Pat. Off. ...... .. G06F 11/10
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`_
`“Escon I/U Interface” SA22-7202y-0l Enterprise Systems
`"Escon I/U Interface" SA22-7202-01 Enterprise Systems
`Architecture/390.
`AIChit6CI11I6/390
`Primary Examiner-—Thornas C. Lee
`Primary Examiner—Thomas C. Lee
`Assistant Examiner—Terance J. Stanton
`Assistant Examiner—Terance J. Stanton
`Attorney, Agent, or Firm—F1oyd E. Anderson; Robert M.
`Attorney, Agent, or Finn—Floyd E. Anderson; Robert M.
`Sullivan
`Sulhvan
`ABSTRACT
`[57] (cid:9)
`[57]
`ABSTRACT
`A remote copy system incorporates dynamically modifiable
`Aremote copy system incorporates dynamically modi?able
`ports on the storage controllers such that those ports can
`ports on the storage controllers such that those ports can
`operate either as a control unit link-level facility or as a
`operate either as a control unit link-level facility or as 9
`channel link-level facility. When configured as a channel
`channel 1ink~level facility. When con?gured as a channel
`link-level facility, a primary storage controller can appear as
`link-level facility, a primary storage controller can appear as
`a host processor to a secondary storage controller. Using
`a host processor to a secondary storage controller. Using
`dynamic switches coupled between primary and secondary
`dynamic switches coupled between primary and secondary
`sites, fewer ESCON communication links are required since
`sites, fewer ESCON communication links are required since
`the ESCON communication links can function either as a
`the ESCON communication links can function either as a
`channel or as storage controller communication link.
`channel or as Storage controller Communication link.
`
`17 Claims, 9 Drawing Sheets
`17 Claims, 9 Drawing Sheets
`
`s00
`\
`
`, s60
`
`_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. _. .Z ._ _ _ _. ._ ...
`
`/ 301
`
`PRIMARY
`
`HOST
`
`i
`
`l
`'
`'
`:
`I
`'
`:
`l
`'
`'
`1
`:
`'
`'
`i
`
`/ s22
`
`/ 323:
`
`|
`_
`STORAGE
`STORAGE — I
`DASD
`DASD |
`CONTROLLER
`VOLA I
`VOL A
`CONTROLLER
`_ VOL 5 :
`VOL B
`l
`l
`I
`|
`STORAGE
`1
`STORAGE —-
`DASD I
`CONTROLLER
`:
`CONTROLLER
`1
`|
`324 _ \azs ___:\_3g6J
`
`|
`l
`
`|
`l
`
`1
`l
`
`I _ _ _ _ _ _ _ _ _ _ _
`
`_ _ _ _ _ _ _ _ _ _ _ _ _ _. J _ _ - _ ._ _.
`
`, 310
`
`312\
`
`| 311
`‘
`:
`|
`: SECONDARY
`
`HOST
`
`'
`l
`|
`l
`'
`:
`1
`I
`'
`'
`'
`
`333:
`/332
`l
`STORAGE — :
`STORAGE
`R
`DASD I
`CONTROLLER
`CONTROLLE __ Vol-A :
`
`'
`|
`l
`
`|
`l
`'
`I
`1
`|
`l
`l
`l
`|
`i
`STORAGE ——
`STORAGE
`CONTROLLER
`DASD I
`CONTROLLER
`VOL B’
`l
`1
`1
`l
`
`DHPN-1005
`Dell Inc. vs. Electronics and Telecommunications, IPR2013-00635
`Page 1 of 18
`
`(cid:9)
`

`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996
`Nov. 12, 1996 (cid:9)
`
`Sheet 1 of 9
`Sheet 1 of 9 (cid:9)
`
`5,574,950
`5,574,950
`
`FIG. 1
`
`PRIOR ART
`PRIOR ART
`
`1 10 /
`110
`
`HOST
`HOST
`
`1 o
`\
`130
`130
`\
`PORTS
`PORTS
`A - D
`A - 0
`
`120
`120
`1| mun n CHANNELS
`II'IIIIIII'III
`CHANNELS
`/ 121 \
`121
`/ \
`13o
`/ 130
`
`161
`161
`/
`
`PORTS
`PORTS
`E - H
`E - H
`
`PORTS
`PORTS
`A - D
`A - 0
`
`PORTS
`PORTS
`E - H
`E - H
`
`125
`125 H
`
`STORAGE STORAGE
`STORAGE
`STORAGE
`PATH 0
`PATH 1
`PATH O\/PATH 1
`
`L
`
`140
`140
`
`J A s 339
`JAS
`
`CACHE 145
`CACHE ?é
`
`NVS 170
`
`W8 1Z9
`
`VPD
`VPD
`VPD
`vpo
`\ /
`95 96
`95 96
`
`STORAGE STORAGE
`STORAGE
`STORAGE
`PATH 2
`PATH 3
`PATH 2\/PATH a
`M
`140
`140
`
`75
`75
`
`DIRECT ACCESS
`DIRECT ACCESS
`STORAGE DEVICE
`STORAGE DEVICE
`
`DHPN-1005 / Page 2 of 18
`
`(cid:9)
`

`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996 (cid:9)
`Nov. 12, 1996
`
`2 Sheet 2 of 9
`Sheet 2 of 9 (cid:9)
`
`5,574,950
`5,574,950
`
`200
`20o
`\
`
`FIG. 2
`
`'
`
`, 260
`/ 260
`
`I - - - - - - - _ - _ - - _ - _ - _ _ - _ - - - - _ - _ - - _ - _ J _ - _ _ - -.
`
`202
`201 (cid:9)
`/ 201 / 202
`
`241
`241
`
`PRIMARY
`
`205
`
`/ 222
`/ 222
`
`CONTROLLER
`
`'
`
`221 (cid:9)
`221\
`A
`250
`2
`B STORAGE
`STORAGE
`205 l g CONTROLuER
`T
`=
`"I A
`’
`B STORAGE
`STORAGE
`C CONTROLLER
`CONTROLLER
`D
`
`l
`
`2 I
`/ 223
`/ 2sl
`|
`I
`DASD :
`DASD
`:
`1:
`|
`|
`I
`DAS[) |
`DASD
`I
`I
`l
`\ 226 I
`4-7726
`
`T
`
`242
`
`_“
`
`243
`243
`
`224 \ 225
`224 (cid:9)
`
`241 <
`
`248
`
`, 210
`270
`
`I
`
`|
`:
`:
`:
`=
`'
`:
`|
`I |
`:
`:
`
`I
`l
`I
`I
`I
`I
`I
`I
`I
`I
`l
`I
`I
`l
`I
`I
`I
`I
`I
`
`I _ _ _ _ _ _ _ _ _ .._ _
`
`_ _
`
`212
`211 (cid:9)
`211 \ 212
`
`_ _ _ _ _ _ _ _ _ _ _ J _ _ _ _ - _
`
`232
`/ 232
`231 (cid:9)
`/ 231 /
`A
`
`233'
`233
`/
`/
`
`SECONDARY
`SECONDARY
`
`215 ,
`
`STORAGE
`B STORAGE
`g CONTROLLER
`CONTROLLER
`
`DASD
`
`HOST
`HOST
`
`DYNAMIC|
`SWITCH
`
`I
`
`I
`|
`
`i
`|
`
`I
`|
`
`244
`244
`
`245
`
`STORAGE
`
`CONTROLLER
`
`DAS D
`DASD
`
`234/ 235/
`
`DHPN-1005 / Page 3 of 18
`
`

`
`l
`:
`:
`'
`l I
`l
`:
`|
`I
`:
`l
`I
`— t
`343
`: __________ _
`
`/ s01
`/ 301 (cid:9)
`
`302
`
`321
`
`350
`
`341
`
`305
`
`PRIMARY
`PRIMARY
`
`HOST
`HOST
`
`349
`
`_
`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996 (cid:9)
`Nov. 12, 1996
`
`Sheet 3 0f 9
`Sheet 3 of 9 (cid:9)
`
`5,574,950
`5,574,950
`
`FIG. 3
`
`/ 360
`
`_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.. ._ J ._ _ _. .... ._ _.
`
`/ s22
`/ 322
`STORAGE
`STORAGE
`CONTROLLER|
`CONTROLLER
`
`l
`323
`/ 32al
`/
`l
`I
`DASD :
`DASD
`vOL A
`VOL A
`VOL B :
`VOL B
`l
`'
`'
`i
`l
`DASD
`DASD w
`:
`1 l
`324 \ 325 1---7726
`s24 _ \325 _ __ >336)
`
`l
`|
`
`i
`1
`
`l
`|
`
`STORAGE
`STORAGE
`CONTROLLER;
`CONTROLLER
`
`l - - - _ - - - _ _ - -
`
`s12
`312
`
`: a11>\
`311
`:
`|
`: SECONDARY
`SECONDARY
`
`_ _ _ _ - _ _ _ _ _ _ _ _ _ -- J _ _ - _ - -.
`
`331
`
`332
`332
`/
`/
`STORAGE
`B STORAGE
`
`346
`
`CONTROLLER
`C CONTROLLER
`
`370
`, 370
`_z
`
`333
`I
`/saal
`/
`l
`'
`DASD :
`DASD
`VOL A’
`:
`VOL A'
`
`HOST
`HOST
`
`l
`|
`l
`'
`:
`:
`
`i
`'
`
`.
`|
`l
`
`|
`l
`
`344
`
`315 (cid:9)
`
`345
`
`STORAGE
`STORAGE
`CONTROLLER
`CONTROLLER
`
`352/
`
`334/ 335/ 336/
`
`.'
`|
`:
`l
`l
`|
`|
`DASD :
`DASD
`VOL B'
`l
`| l
`
`|
`
`DHPN-1005 / Page 4 of 18
`
`

`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996 (cid:9)
`Nov. 12, 1996
`
`Sheet 4 of 9 (cid:9)
`Sheet 4 0f 9
`
`5,574,950
`5,574,950
`
`REMOTE COPY
`REMOTE COPY
`PATHS INITIAUZATION
`PATHS INITIALIZATION
`
`CONFIGURE PATHS
`CONFIGURE PATHS
`TO SECONDARY
`TO SECONDARY
`
`401
`/
`
`402
`402
`/
`
`(cid:9)1.
`
`FIG. 4
`
`403
`403
`/
`
`LINK ADDRESSES
`LINK ADDRESSES
`SERIAL NUMBERS
`ssi'ggigrgfgg
`SUBSYSTEM ID'S
`:
`
`404
`
`YES
`
`LOGICAL
`PATH BEING
`USED ?
`
`407
`r
`/
`POST ERROR
`POST ERROR
`MESSAGE
`MESSAGE
`
`406
`/
`ESTABLISH LOGICAL PATH
`ESTABLISH LOGICAL PATH
`TO SECONDARY CONTROLLER
`TO SECONDARY CONTROLLER
`
`408
`40a
`/
`
`QUERY STATUS OF
`QUERY STATUS OF
`SECONDARY CONTROLLER
`SECONDARY CONTROLLER
`
`409
`409
`/
`IS
`IS
`SECONDARY CU
`SECONDARY CU
`THE CORRECT
`THE CORRECT
`ONE?
`ONE ?
`
`YES
`
`INDICATE
`INDICATE
`COMPLETION STATUS
`COMPLETION STATUS
`
`NO
`
`410 I
`410
`\
`POST ERROR
`POST ERROR
`MESSAGE
`MESSAGE
`
`DHPN-1005 / Page 5 of 18
`
`

`
`U.S. Patent (cid:9)
`U.S. Patent
`
`Nov. 12, 1996
`Nov. 12, 1996 (cid:9)
`
`Sheet 5 of 9
`Sheet 5 of 9 (cid:9)
`
`5,574,950
`5,574,950
`
`ESTABLISH REMOTE (cid:9)
`ESTABLISH REMOTE
`DUAL COPY
`DUAL CoPY
`
`501
`501
`/
`
`I
`
`‘
`►
`vERIPY PARAMETERS
`VERIFY PARAMETERS
`AND DISCONNECT
`AND DISCONNECT
`
`CONNECT TO
`CONNECT TO
`SECONDARY CU
`SECONDARY CU
`5/04
`504
`
`502
`502
`/
`
`_
`
`:
`/ 503
`
`
`
`NO
`NO
`
`YES
`
`505
`/
`
`OBTAIN SECONDARY
`- OBTAIN SECONDARY
`DEVICE INFORMATION
`DEvICE INPDRMATIDN
`l
`506
`/
`RIMARY
`- RIMARY
`ND SECONDAR
`- ND SECDNDAR
`COMPATIBLE
`COMPATIBLE
`9
`
`NO
`
`507
`507
`
`YES
`SEND COMMAND
`SEND COMMAND
`TO SEC DEVICE 'GO
`T0 SEC DEVICE ‘GO
`DUPLEx PENDING‘
`DUPLEX PENDING'
`
`508
`
`NO
`NO
`
`512
`512
`1
`COPY ALL TRACKS OF DATA
`CDPY ALL TRACKS OF DATA
`FROM PRI DEVICE
`FROM PRI DEVICE
`TO SEC DEVICE
`TD SEC DEVICE
`
`‘
`
`‘
`
`NO
`
`‘V
`
`5/13
`CDPY
`COMPLETE
`
`YES (cid:9)
`
`514
`
`5714
`
`ENTER FULL DUPLEX
`ENTER FULL DUPLEX
`STATE FOR BOTH
`STATE FOR BOTH
`PRI AND SEC DEVICES
`PRI AND SEC DEVICES
`
`FIG. 5
`FlG. 5
`
`YES
`
`SET PRIMARY TO
`SET PRIMARY To
`DUPLEX PENDING
`DUPLEx PENDING
`511
`511
`\
`
`I
`
`510
`510
`/
`POST ERROR I
`POST ERROR
`STATUS
`STATUS
`‘————l
`_
`I
`PRESENT ENDING
`PRESENT ENDING
`STATUS
`STATUS
`I
`
`509
`509
`l
`
`,
`
`DHPN-1005 / Page 6 of 18
`
`

`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996
`Nov. 12, 1996 (cid:9)
`
`Sheet 6 0f 9
`Sheet 6 of 9 (cid:9)
`
`5,574,950
`5,574,950
`
`601
`601
`/
`PROCESS WRITE COMMAND'
`_ PROOEss wRITE COMMAND
`FROM HOST
`FROM HOST
`602
`
`NI
`CHECK ALL
`CHECK ALL
`WRITE I/O FLAG
`wRITE I/O FLAG
`SET
`SH
`
`YES
`YES
`
`FIG 6
`FIG. 6
`
`NO (cid:9)
`
`WRITE DATA TO (cid:9)
`wRITE DATA TO
`CACHE AND DASD
`CACHE AND man
`
`603
`603
`/ _
`I
`
`END
`OF WRITE
`DOMAIN
`
`YES (cid:9)
`
`6
`
`ESTABUSH CONNECTION
`TO sEOONDARY cu
`
`6)”
`SUCCESSFUL
`
`NO
`
`‘2’
`YES
`YES
`wRITE MODIFIED DATA
`TO sEOONDARY Ou i
`
`SUCCESSFUL
`
`YES
`SET CLEAN
`SET CLEAN
`ENDING STATUS
`ENDING STATUS
`
`N0
`
`611
`61 1
`/
`/
`
`612
`612
`\
`
`I
`PRESENT ENDING I
`PRESENT ENDING
`STATUS TO HOST
`STATUS TO HOST
`
`607
`607
`/
`ENTER SUSPENDED STATE
`ENTER SUSPENDED sTATE
`ON PRIMARY DEVICE -
`ON PRIMARY DEvIOE -
`RECORD CHANGES IN NVS
`REOORD CHANGES IN NVS
`
`6/10
`610
`ENTER SUSPENDED STATE
`ENTER SUSPENDED STATE
`ON PRI AND SEC DEVICES -
`7 ON FRI AND sEO DEVICES -
`RECORD CHANGES
`REOORD CHANGES
`
`613
`
`CRITICAL
`VOLUME
`9
`NO
`NO
`
`YES
`
`614
`61 4
`/
`1
`SET FLAG TO UNIT
`SET FLAG TO UNIT
`CHECK ALL WRITE
`CHECK Au. wRITE
`CMDS TO PRI DEVICE
`CMDS TO PRI DEvIOE
`
`615
`515
`\
`SET UNIT CHECK
`sET UNIT CHECK
`IN ENDING STATUS
`IN ENDING STATUS
`J
`
`
`
`DHPN-1005 / Page 7 of 18
`
`(cid:9)
`

`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996 (cid:9)
`Nov. 12, 1996
`
`Sheet 7 of 9
`Sheet 7 of 9 (cid:9)
`
`5,574,950
`5,574,950
`
`FIG. 7
`
`PRIMARY
`PRIMARY
`SENDS ELP FRAME
`SENDS ELP FRAME
`
`SECONDARY
`SECONDARY
`PROCESS ELP FRAME
`PROCESS ELP FRAME
`
`701
`101
`/
`
`702
`702
`/
`
`711
`111
`PRIMARY
`PRIMARY
`/
`SENDS EPC FRAME
`SENDS EPC FRAME
`EPC RESP. EXPECTED ON
`EPC RESP. EXPECTED ON
`
`I
`SECONDARY
`SECONDARY
`PROCESS EPC FRAME
`PROCESS EPC FRAME
`SENDS ACK FRAME
`SENDS ACK FRAME
`
`712
`112
`/
`
`703
`703
`
`[704
`."7°4
`NO
`No ERROR
`ERROR
`
`713
`713
`N0
`
`IS EPC
`ESPONSE 0
`-
`
`714
`714
`/
`END
`
`YES
`YES
`
`7
`SECONDARY
`SECONDARY
`SEND LPE FRAME
`SEND LPE FRAME
`
`PRIMARY
`PRIMARY
`PROCESS LPE FRAME
`PROCESS I.PE FRAME
`SEND RID FRAME
`SEND RID FRAME
`
`705
`"/05
`/
`
`706
`706
`/
`
`707
`SECONDARY
`SECONDARY
`107
`PROCESS RID FRAME
`PROCESS RID FRAME /
`SEND IDR FRAME
`SEND IDR FRAME
`
`PRIMARY
`PRIMARY
`PROCESS IDR FRAME
`PROCESS IDR FRAME
`
`F08
`/708
`
`SERIAL
`NUMBER AND
`
`YES
`
`YES
`YES
`
`SECONDARY
`SECONDARY
`SEND EPC FRAME WITH
`SEND EPC FRAME WITH
`RESPONSE EPC ACTIVE
`RESPONSE EPC ACTIVE
`
`715
`/ 715
`
`I
`
`PRIMARY
`PRIMARY
`PROCESS EPC FRAME
`PROCESS EPC FRAME
`
`716
`/ 11s
`
`717 (cid:9)
`
`IS EPC
`FRAME VAUD
`
`NO
`
`718
`718
`
`ERROR
`ERROR
`
`YES
`
`PRIMARY
`PRIMARY
`SEND ACK FRAME
`SEND ACK FRAME
`
`r719
`/719
`
`720
`120
`SECONDARY
`SECONDARY
`/
`PROCESS ACK FRAME Ir
`PROCESS ACK FRAME
`
`ERROR
`
`721
`(cid:9) 721
`/
`
`END
`END
`
`DHPN-1005 / Page 8 of 18
`
`(cid:9)
`(cid:9)
`

`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996 (cid:9)
`Nov. 12, 1996
`Sheet 8 0f 9
`Sheet 8 of 9 (cid:9)
`FIG. 8
`FIG. 8
`
`5,574,950
`5,574,950
`
`801
`01
`/
`PRIMARY SENDS RQC
`PRIMARY SENDS RQC
`FRAME. AS=0
`FRAME. AS=0
`
`802
`/
`
`SECONDARY CU PROCESSES RQC
`SECONDARY CU PROCESSES RQC
`FRAME. SENDS GRANT FRAME.
`FRAME. SENDS GRANT FRAME.
`
`803
`I
`
`PRIMARY CU PROCESSES GRANT
`PRIMARY CU PROCESSES GRANT
`FRAME. SENDS COMMAND FRAME.
`FRAME. SENDS COMMAND FRAME.
`AS=1.
`AS=1.
`
`v
`
`804
`804
`/
`/
`
`SECONDARY CU PROCESSES
`SECONDARY CU PROCESSES
`COMMAND FRAME. SENDS CMD
`COMMAND FRAME. SENDS CMD
`RESP FRAME.
`RESP FRAME.
`
`DHPN-1005 / Page 9 of 18
`
`

`
`PRIMARY
`PRIMARY
`SEND COMMAND
`SEND COMMAND
`FRAME.
`FRAME.
`END BIT = 0
`END BIT = o
`DATA REQ = 0
`DATA REO = o
`CCW CNT = 0
`cow CNT = 0
`
`ESCON
`ESCON
`ARCH.
`AROR.
`DOES
`DOES
`NOT
`NOT
`ALLOW
`ALLOW
`
`901
`/ 901
`
`ESCON
`ESOON
`ARCH.
`AROR.
`DOES
`DOES
`NOT
`NOT
`ALLOW
`ALLOW
`
`CHAINED COMMAND
`CHAINED COMMAND 91 1
`911
`/
`
`PRIMARY
`PRIMARY
`SEND COMMAND
`SEND OOMMAND
`FRAME.
`FRAME.
`END BIT = 0
`END BIT = o
`DATA REQ = 0
`DATA REO = 0
`CCW CNT = 0
`cow CNT = o
`
`U.S. Patent (cid:9)
`US. Patent
`
`Nov. 12, 1996
`Nov. 12,1996
`
`Sheet 9 of 9
`Sheet 9 of 9 (cid:9)
`FIG. 9
`FIG. 9
`MODIFIED READ COMMAND PROTOCOLS
`MODIFIED READ COMMAND PROTOCOLS
`
`5,574,950
`5,574,950
`
`FIRST COMMAND IN A CHAIN
`FIRST COMMAND IN A CHAIN
`
`912
`/ 912
`/
`902
`902
`SECONDARY
`SECONDARY
`w / W
`PROCESS COMMAND
`PROOESS OOMMAND
`PROOESS COMMAND
`PROCESS COMMAND
`FRAME.
`FRAME.
`FRAME.
`FRAME.
`SEND COMMAND
`SEND COMMAND
`SEND OOMMAND
`SEND COMMAND
`RESPONSE FRAME
`RESPONSE FRAME
`RESPONSE FRAME
`RESPONSE FRAME
`
`'
`
`903
`K903
`
`PRIMARY
`PRIMARY
`PROCESS COMMAND
`PROOESS COMMAND
`RESPONSE FRAME.
`RESPONSE FRAME.
`SEND ACCEPT
`SEND ACCEPT
`COMMAND RESPONSE
`OOMMAND RESPONSE
`FRAME.
`FRAME.
`SEND DATA
`SEND DATA
`REQUEST FRAME.
`REQUEST FRAME.
`END BIT = 0 OR 1
`END BIT = 0 OR 1
`DATA REQ = 1
`DATA REQ = 1
`CCW CNT = XX
`saw CNT = xx
`_
`
`904
`/
`
`SECONDARY
`SEOONDARY
`PROCESS ACCEPT
`PROOESS AOOEPT
`COMMAND RESPONSE
`OOMMAND RESPONSE
`FRAME.
`FRAME.
`PROCESS DATA
`PROOESS DATA
`REQUEST FRAME.
`REOuEST FRAME.
`SEND DATA FRAME(S).
`SEND DATA FRAME(S).
`
`/ 913
`/ 91 S
`
`PRIMARY
`PRIMARY
`PROCESS COMMAND
`PROOESS COMMAND
`RESPONSE FRAME.
`RESPONSE FRAME.
`SEND DATA
`SEND DATA
`REQUEST FRAME.
`REOuEST FRAME.
`END BIT = 0 OR 1
`END BIT = 0 OR 1
`DATA REQ = 1
`DATA REO = 1
`CCW CNT = XX
`CCW CNT = xx
`
`914
`914
`/
`
`SECONDARY
`SEOONDARY
`PROCESS DATA
`PROOESS DATA
`REQUEST FRAME.
`REQUEST FRAME.
`SEND DATA FRAME(S).
`SEND DATA FRAME(S).
`
`dIW (cid:9)
`
`DHPN-1005 / Page 10 of 18
`
`(cid:9)
`(cid:9)
`

`
`5,574,950
`5,574,950
`
`1
`1
`REMOTE DATA SHADOWING USING A
`REMOTE DATA SHADOWING USING A
`MULTIMODE INTERFACE TO
`MULTIIVIODE INTERFACE TO
`DYNAMICALLY RECONFIGURE CONTROL
`DYNAMICALLY RECONFIGURE CONTROL
`LINK-LEVEL AND COMMUNICATION
`LINK-LEVEL AND COMMUNICATION
`LINK-LEVEL
`LINK-LEVEL
`
`HELD OF THE INVENTION
`FIELD OF THE INVENTION
`
`The present invention relates generally to remote data
`The present invention relates generally to remote data
`shadowing, and more particularly, to a system for remote
`shadowing, and more particularly, to a system for remote
`data shadowing wherein communication links are operable
`data shadowing wherein communication links are operable
`in multiple modes.
`in multiple modes.
`
`5
`
`BACKGROUND OF THE INVENTION
`BACKGROUND OF THE INVENTION
`Data processing systems, in conjunction with processing
`Data processing systems, in conjunction with processing
`data, typically are required to store large amounts of data (or
`data, typically are required to store large amounts of data (or
`records), which data can be efficiently accessed, modified,
`records), which data can be e?iciently accessed, modi?ed,
`and re-stored. Data storage is typically separated into several
`and re-stored. Data storage is typically separated into several
`different levels, or hierarchically, in order to provide efficient
`different levels, or hierarchically, in order to provide efficient
`and cost effective data storage. A first, or highest level of
`and cost effective data storage. A ?rst, or highest level of
`data storage involves electronic memory, usually dynamic or
`data storage involves electronic memory, usually dynamic or
`static random access memory (DRAM or SRAM). Elec-
`static random access memory (DRAM or SRAM). Elec
`tronic memories take the form of semiconductor integrated
`tronic memories take the form of semiconductor integrated
`circuits wherein millions of bytes of data can be stored on
`circuits wherein millions of bytes of data can be stored on
`each circuit, with access to such bytes of data measured in
`each circuit, with access to such bytes of data measured in
`nano-seconds. The electronic memory provides the fastest
`nano-seconds. The electronic memory provides the fastest
`access to data since access is entirely electronic.
`access to data since access is entirely electronic.
`A second level of data storage usually involves direct
`A second level of data storage usually involves direct
`access storage devices (DASD). DASD storage, for
`access storage devices (DASD). DASD storage, for
`example, can comprise magnetic and/or optical disks, which
`example, can comprise magnetic and/or optical disks, which
`store bits of data as micrometer sized magnetically or
`store bits of data as micrometer sized magnetically or
`optically altered spots on a disk surface for representing the
`optically altered spots on a disk surface for representing the
`"ones" and "zeros" that make up those bits of the data.
`“ones” and “zeros” that make up those bits of the data.
`Magnetic DASD, includes one or more disks that are coated
`Magnetic DASD, includes one or more disks that are coated
`with remnant magnetic material. The disks are rotatably
`with remnant magnetic material. The disks are rotatably
`mounted within a protected environment. Each disk is
`mounted within a protected environment. Each disk is
`divided into many concentric tracks, or closely spaced
`divided into many concentric tracks, or closely spaced
`circles. The data is stored serially, bit by bit, along each
`circles. The data is stored serially, bit by bit, along each
`track. An access mechanism, known as a head disk assembly
`track. An access mechanism, known as a head disk assembly
`(HDA), typically includes one or more read/write heads, and
`(HDA), typically includes one or more read/write heads, and
`is provided in each DASD for moving across the tracks to
`is provided in each DASD for moving across the tracks to
`transfer the data to and from the surface of the disks as the
`transfer the data to and from the surface of the disks as the
`disks are rotated past the read/write heads. DASDs can store
`disks are rotated past the read/write heads. DASDs can store
`giga bytes of data with the access to such data typically
`giga bytes of data with the access to such data typically
`measured in milli-seconds (orders of magnitudes slower
`measured in milli-seconds (orders of magnitudes slower
`than electronic memory). Access to data stored on DASD is
`than electronic memory). Access to data stored on DASD is
`slower due to the need to physically position the disk and
`slower due to the need to physically position the disk and
`HDA to the desired data storage locations.
`HDA to the desired data storage locations.
`A third or lower level of data storage includes tape and/or
`A third or lower level of data storage includes tape and/or
`tape and DASD libraries. At this storage level, access to data
`tape and DASD libraries. At this storage level, access to data
`is much slower in a library since a robot or operator is
`is much slower in a library since a robot or operator is
`necessary to select and load the needed data storage
`necessary to select and load the needed data storage
`medium. The advantage is reduced cost for very large data
`medium. The advantage is reduced cost for very large data
`storage capabilities, for example, tera-bytes of data storage.
`storage capabilities, for example, tera-bytes of data storage.
`Tape storage is often used for back-up purposes, that is, data
`Tape storage is often used for back-up purposes, that is, data
`stored at the second level of the hierarchy is reproduced for
`stored at the second level of the hierarchy is reproduced for
`safe keeping on magnetic tape. Access to data stored on tape
`safe keeping on magnetic tape. Access to data stored on tape
`and/or in a library is presently on the order of seconds.
`and/or in a library is presently on the order of seconds.
`Having a back-up data copy is mandatory for many
`Having a back-up data copy is mandatory for many
`businesses as data loss could be catastrophic to the business.
`businesses as data loss could be catastrophic to the business.
`The time required to recover data lost at the primary storage
`The time required to recover data lost at the primary storage
`level is also an important recovery consideration. An
`level is also an important recovery consideration. An
`improvement in speed over tape or library back-up, includes
`improvement in speed over tape or library back-up, includes
`dual copy. An example of dual copy involves providing
`dual copy. An example of dual copy involves providing
`additional DASD's so that data is written to the additional
`additional DASD’s so that data is written to the additional
`DASDs (sometimes referred to as mirroring). Then if the
`DASDs (sometimes referred to as mirroring). Then if the
`
`2
`2
`primary DASDs fail, the secondary DASDs can be
`primary DASDs fail, the secondary DASDs can be
`depended upon for data. A drawback to this approach is that
`depended upon for data. A drawback to this approach is that
`the number of required DASDs is doubled.
`the number of required DASDs is doubled.
`Another data back-up alternative that overcomes the need
`Another data back-up alternative that overcomes the need
`to double the storage devices involves writing data to a
`to double the storage devices involves writing data to a
`redundant array of inexpensive devices (RAID) con?gura
`redundant array of inexpensive devices (RAID) configura-
`tion. In this instance, the data is written such that the data is
`tion. In this instance, the data is written such that the data is
`apportioned amongst many DASDs. If a single DASD fails,
`apportioned amongst many DASDs. If a single DASD fails,
`then the lost data can be recovered by using the remaining
`then the lost data can be recovered by using the remaining
`10 data and error correction procedures. Currently there are
`data and error correction procedures. Currently there are
`10
`several different RAID con?gurations available.
`several different RAID configurations available.
`The aforementioned back-up solutions are generally suf
`The aforementioned back-up solutions are generally suf-
`ficient to recover data in the event that a storage device or
`?cient to recover data in the event that a storage device or
`medium fails. These back~up methods are useful only for
`medium fails. These back-up methods are useful only for
`15 device failures since the secondary data is a mirror of the
`device failures since the secondary data is a mirror of the
`primary data, that is, the secondary data has the same
`primary data, that is, the secondary data has the same
`volume serial numbers (VOLSERs) and DASD addresses as
`volume serial numbers (VOLSERs) and DASD addresses as
`the primary data. System failure recovery, on the other hand,
`the primary data. System failure recovery, on the other hand,
`is not available using mirrored secondary data. Hence still
`is not available using mirrored secondary data. Hence still
`further protection is required for recovering data if a disaster
`further protection is required for recovering data if a disaster
`occurs destroying the entire system or even the site, for
`occurs destroying the entire system or even the site, for
`example, earthquakes, ?res, explosions, hurricanes, etc.
`example, earthquakes, fires, explosions, hurricanes, etc.
`Disaster recovery requires that the secondary copy of data be
`Disaster recovery requires that the secondary copy of data be
`stored at a location remote from the primary data. A known
`stored at a location remote from the primary data. A known
`method of providing disaster protection is to back-up data to
`25 method of providing disaster protection is to back-up data to
`tape, on a daily or weekly basis, etc. The tape is then picked
`tape, on a daily or weekly basis, etc. The tape is then picked
`up by a vehicle and taken to a secure storage area usually
`up by a vehicle and taken to a secure storage area usually
`some kilometers away from the primary data location. A
`some kilometers away from the primary data location. A
`problem is presented in this backup plan in that it could take
`problem is presented in this back-up plan in that it could take
`days to retrieve the back-up data, and meanwhile several
`30 days to retrieve the back-up data, and meanwhile several
`hours or even days of data could be lost, or worse, the
`hours or even days of data could be lost, or worse, the
`storage location could be destroyed by the same disaster. A
`storage location could be destroyed by the same disaster. A
`somewhat improved back-up method includes transmitting
`somewhat improved back-up method includes transmitting
`data to a back-up location each night. This allows the data
`data to a back-up location each night. This allows the data
`to be stored at a more remote location. Again, some data may
`to be stored at a more remote location. Again, some data may
`be lost between back-ups since back-up does not occur
`be lost between back-ups since back-up does not occur
`continuously, as in the dual copy solution. Hence, a sub-
`continuously, as in the dual copy solution. Hence, a sub
`stantial data amount could be lost which may be unaccept-
`stantial data amount could be lost which may be unaccept
`able to some users.
`able to some users.
`A back-up solution providing a greater degree of protec-
`A back-up solution providing a greater degree of protec
`tion is remote dual copy which requires that primary data
`tion is remote dual copy which requires that primary data
`stored on primary DASDs be shadowed at a secondary or
`stored on primary DASDs be shadowed at a secondary or
`remote location. The distance separating the primary and
`remote location. The distance separating the primary and
`secondary locations depends upon the level of risk accept-
`secondary locations depends upon the level of risk accept—
`45 able to the user, and for synchronous data communications,
`able to the user, and for synchronous data communications,
`can vary from just across a fire-wall to several kilometers.
`can vary from just across a ?re-wall to several kilometers.
`The secondary or remote location, in addition to providing
`The secondary or remote location, in addition to providing
`a back-up data copy, must also have enough system infor-
`a back-up data copy, must also have enough system infor
`mation to take over processing for the primary system
`mation to take over processing for the primary system
`50 should the primary system become disabled. This is due in
`should the primary system become disabled. This is due in
`part because a single storage controller does not write data
`part because a single storage controller does not write data
`to both primary and secondary DASD strings at the primary
`to both primary and secondary DASD strings at the primary
`and secondary sites. Instead, the primary data is stored on a
`and secondary sites. Instead, the primary data is stored on a
`primary DASD string attached to a primary storage control-
`primary DASD string attached to a primary storage control
`ler while the secondary data is stored on a secondary DASD
`ler while the secondary data is stored on a secondary DASD
`string attached to a secondary storage controller.
`string attached to a secondary storage controller.
`Remote dual copy falls into two general categories, syn-
`Remote dual copy falls into two general categories, syn
`chronous and asynchronous. Synchronous remote copy
`chronous and asynchronous. Synchronous remote copy
`allows sending primary data to the secondary location and
`allows sending primary data to the secondary location and
`con?rming the reception of such‘ data before ending a
`60 confirming the reception of such data before ending a
`primary DASD input/output (I/O) operation (providing a
`primary DASD input/output (I/0) operation (providing a
`channel end (CE)/device end (DE) to the primary host).
`channel end (CE)/device end (DE) to the primary host).
`Synchronous remote copy, therefore, slows the primary
`Synchronous remote copy, therefore, slows the primary
`DASD I/O response time while waiting for secondary con-
`DASD 110 response time while waiting for secondary con
`65 firmation. Primary I/O response delay is increased propor-
`?rmation. Primary I/O response delay is increased propor
`65
`tionately with the distance between the primary and second-
`tionately with the distance between the primary and second
`ary systems—a factor that limits the remote distance to tens
`ary systems—a factor that limits the remote distance to tens
`
`20
`
`35
`5
`3
`
`40 (cid:9)
`
`55
`55
`
`DHPN-1005 / Page 11 of 18
`
`

`
`5,574,950
`5,574,950
`
`4
`4
`More recently introduced data disaster recovery solutions
`More recently introduced data disaster recovery solutions
`include remote dual copy wherein data is backed-up not only
`include remote dual copy wherein data is backed-up not only
`remotely, but also continuously. In order to communicate
`remotely, but also continuously. In order to communicate
`duplexed data synchronously from one host processor to
`duplexed data synchronously from one host processor to
`5 another host processor, or from one storage controller to
`another host processor, or from one storage controller to
`another storage controller, or some combination thereof,
`another storage controller, or some combination thereof,
`expensive communication links are required for connecting
`expensive communication links are required for connecting
`each host processor and/or storage controller. Such commu-
`each host processor and/or storage controller. Such commu
`nication links, include, for example, Enterprise Systems
`nication links, include, for example, Enterprise Systems
`Connection (ESCON) ?ber optic links providing serial com
`10 Connection (ESCON) fiber optic links providing serial com-
`munication paths extending tens of kilometers.
`munication paths extending tens of kilometers.
`In a typical remote dual copy system, there may exist
`In a typical remote dual copy system, there may exist
`multiple primary processors connected, by multiple serial or
`multiple primary processors connected, by multiple serial or
`parallel communication links, to multiple primary storage
`parallel communication links, to multiple primary storage
`controllers, each having