AQTAA
`US005412801A
`.
`(11) Patent Number:
`5,412,801
`119
`United States Patent
`
`
`de Remeretal. May 2, 1995 [45] Date of Patent:
`
`[75]
`
`[56]
`
`[54] GAP RECOVERY FOR OFF-SITE DATA
`OTHER PUBLICATIONS
`STORAGE AND RECOVERY SYSTEMS
`Burkes, D. L.,
`“Design Approaches for Real-Time
`Inventors:
`James C. de Remer, Sausalito;
`‘ransaction Processing Remote Site Recovery,” Re-
`:
`.
`search Report. IBM 1990
`Thomas D. Childers, Mill Valley;
`POT,
`|
`.
`.
`James T. Flesher, San Francisco, all
`E-Net Geneal Information Manual, published prior to
`of Calif
`Jul. 1987, pp. 1-5.
`.
`E-Net | Installation and Operations Guide, Release 1.1.
`[73] Assignee: E-Net, San Francisco, Calif.
`E-Net General Information brochure, pp. i-4.
`:
`Brochures Entitled E-Net1, E-Net2, E-Net4, E-Net7,
`[21] Appl. No.: 466,508
`published prior to Sep. 27, 1988; E-Net Corporation.
`{22] Filed:
`Jan. 17, 1990
`Disaster Recovery, Business Software Review, Apr.
`(SQ Mint, CVS coccsesecssasenesestesensenssenee GUE 12/16
`.
`.
`.
`;
`$52] WSS. CM, oeececcecccccssstectccssesssstee 395/575; 395/600;
`371/10.1; 371/29.1; 371/65; 364/DIG.1; OeRkey literature, published prior to
`364/243; 364/243.2; 364/265; 364/266.5;
`Sun ard Recove y Services Information Bulletin,
`pub-
`364/282.1; 364/285; 364/285.1; 364/285.2 ee none 3 apes
`»P
`
`[58] Field of Search............... 395/575, 600; 371/10.1, 1989,>PARES.P + Hy
`
`
`
`371/29.1, 65, 16.5,
`Primary Examiner—Paul V. Kulik
`Attorney, Agent, or Firm—Christie, Parker & Hale
`References Cited
`[57]
`ABSTRACT
`U.S. PATENT DOCUMENTS
`‘methodforcreating a controlrecordforusein future
`4,053,752 10/1977 DeJobin et ale sessssssssscsseee 395/575
`
`4,084,231 11/1978 Capozzi et al..|395/425 machine recoveryof gaps in a completeseries ofjournal
`
`3/1985 Gawlick etal.
`4,507,751
`395/575
`data formed by a computing machine from a complete
`...ccccscssecseesesstesssseenees 395/600
`4,646,229
`2/1987 Boyle
`series of transactional data. Copy is formedofthe series
`
`4,686,620
`8/1987 Neg..........-+.
`395/600
`of journal data for transmission to a remote site. A plu-
`4,710,870 12/1987 Blackwell et al
`395/575
`rality of data group records are formed. Each data
`
`Patten %ion9 Matema et al.
`3057on
`group record identifies a different boundary of each of
`Atte eeceeeee
`814,
`;
`;
`:
`;
`4945474 1/1990 Elliott et al.
`395/875
`a plurality of data groups in the series of journal data.
`-
`The presence is detected of a gap in the copy of the
`
`3,010,560 4/1991 Janney et al
`377/20
`journal data for transmission. One of the data group
`5,036,518
`7/1991 T.
`we
`2
`:
`:
`go pean
`records is created for a boundary of each gap.
`371/83
`
`al.
`5,043,866
`8/1991 Myre, Jr. et
`395/600
`5,067,107 11/1991 Wade .......,
`395/300
`
`2/1992 Horton etal. oes 395/575
`5,089,958
`
`49 Claims, 23 Drawing Sheets
`
`
`
`SCAN SCF FOR DATA GROUP WITH
`“GAP ESTABLISHED” FLAG SET, AND
`
`"GAP RECOVERY SUBMITTED” FLAG
`
`UNSET.
`
`
`
`DATA GROUPS
`FOUND?
`
`GAP
`DATA WRITTEN
`TO TAPE ?
`
`
`
` EsSUBMIT JCL FILE
`
`
`ANY
`
`MORE DATA
`
`GROUPS TO
`PROCESS
`2
`
`
`
`
`APPLE 1013
`
`APPLE 1013
`
`1
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 1 of 23
`
`5,412,801
`
`4LISWYLNAO LavOd
`
`SSdViOLLSANSVA
`
`JIOSNOO
`
`A1EVvAOWAY
`dOLVYadO
`
`AXOOT9
`
`| S
`
`IWNINSSL
`
`AWILWie
`
`43aSnNSWEG
`
`
`
`
`
`
`
` SANAdviJAYSIC;-9v
`
`LYZl
`SS
`
`el
`
`
`
`NoOI93y||NOISSY
`
`qVv
`
`SWVesNIVN
`
`
`
`SLISALOWSY
`
`LOLA
`
`@NOISY
`
`ONINNAY
`
`2
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 2 of 23
`
`5,412,801
`
`G3aNddNSYds
`
`JANLOSTSS
`
`mses—iLINILADSHETILA03|/NOVOSETWaa||[SMITE
`OFONITIVNSNOL
`
`QP
`
`INYGJdvL
`SNILNOY
`JOS(S919)
`
`
`WVuD0"dNIVWSWC
`vivd||viva
`901||3Sve
`LASLas
`
`LIX3
`
`8d
`
`tL
`
`3
`
`
`
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 3 of 23
`
`5,412,801
`
`oO:
`
`SNOILVO
`
`
`
`
`
`
`
`NIVA|HOVLLY)“oninayymWud0ud—YOLVYSNIOLLJN-3
`
`danddns
`
`YOLdAYONGA
`
`
`no1aWo9sHOV66anid|LaSWOOS
`
`NJOWODS(NIVL3)(NUMLA)
`GYdSWOOS7
`__HOVLyL_SS
`
`
`
`
`
`OQNOIDSY
`
`4
`
`
`
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 4 of 23
`
`5,412,801
`
`fF|
`ViVGSLIM
`(-SW4CL3)
`WNaNnor
`
`SIG
`
`Oct
`
`ONIAIBOSY
`
`SSIYdgv
`
`40vVdS
`
`
`
`ve
`
`yaSIdOL
`
`BAIIO3Y
`
`JONLNOOD
`
`SVL
`
`(-OUNWLS)
`
`VNOIDSY
`
`LG
`
`FAI3054
`
`WNanor
`
`Syo018
`
`(AD3u13)
`
`9S
`
`5
`
`
`
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 5 of 23
`
`5,412,801
`
`E-NET 1 REGION INITIALIZATION FLOW
`
`CLOSE ALL OPEN DATA GROUP RECORDS WHICH ARE
`OPEN IN SCF AND LABEL SUCH RECORDS FOR GAP
`RECOVERY.
`
`204
`
`CREATE ECVT IN CSA
`
`206
`
`208
`
`SET E-NET 1 ACTIVE(CSA FLAG)
`
`MESSAGE TO OPERATOR:
`
`210
`“E—NET 1 ACTIVE .
`
`Cano
`FIG.6
`DBMS INTERFACE INITIALIZATION FLOW
`
`
`IS
`
`THE E-—NET 4
`
`SEND REGION ACTIVE
`ECVTACTV= 1
`
`
`
`224
`
`
`
`220
`
`
`
`ASK USER:
`RETRY OR FORCE ?
`
`
`
`SERT A “GAP”
`IN
`RECORD IN
`SCF
`
`
`
`
`
`
`6
`
`

`

`US. Patent
`
`May 2, 1995
`
`Sheet 6 of 23
`
`5,412,801
`
`DBMS-QUEUE—VTAM TRANSFER
`;
`DBMS WRITE
`66
`ROUTINE CALLED
`
`FIG.7
`
`237
`
`30
`
`| | |
`
`,
`USER—SUPPLIED SELECTIVE
`of
`EXIT_PROCESSING
`ee E-NET 1 EXIT ROUTINE!
`
`238
` NO
`SOME
`SELECTED
`
`RECORDS
`:
`?
`YES
`
`
`
`
`|
`
`: | | |
`
`
`
`
`
`MOVE WRITE BLOCK TO
`FIFO QUEUE
`
`
`NORMAL DBMS
`WRITE ROUTINE
`
`Cex oy
`
`7
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`TWNIDINO
`
`ViVd
`
`LSGHYOOsY-L
`
`
`
`HLONATSWad
`
`5,412,801
`
`Sheet 7 of 23
`
` 9I¢vlcEle
`
`dWVISSNIL NOISSAYdNOO#dnOHYod43GO9AdALHLONA1
`
`
`
`
`
`
`
`
`
`
`
`
`
`4JYO4S9EsONaAO3SVIVdNISISOQYOO034QYOOsHY
`
`co
`
`8
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 8 of 23
`
`5,412,801
`
`FIC.9A
`STATE 1
`
`1OOFIFO QUEUE 102
`
`FIG.9B
`STATE 2
`
`FIG.9C
`STATE 3
`
`1OOFIFO QUEUE
`1OOFIFO QUEUE
`
`
`FIG.9D
`STATE 4
`
`1OOFIFO QUEUE
`
`9
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 9 of 23
`
`5,412,801
`
`FIG.10A
`
`Cuan
`202
`CHECK SCF FOR OPEN
`RECORDS; START GAP
`RECOVERY IF NECESSARY
`
`
`
`505
`
`503
`COMQFUCNYES
`LAG, SELT0
`?
`FIG.10C
`NO
`—
`STATE=1
`
`504
`
`506
`
`
`
`
`
`
`
`
`
`
`TO_FIG.10B
`926
`D
`O
`PILL QUEU
`INCLUDING -UNCONF.
`VTIAM_ BUFFERS.
`
`
`
`TO FIG.10B,10C
`
`10
`
`10
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 10 of 23
`
`5,412,801
`
`FIG.10B
`
`FROM FIG.10A
`3—-CO PROCESS
`
`4—FIFO QUEUE
`
`SPILLING
`
`
`
`
`
`(Q<LT)
`AND(VTAM AVAIL.)
`AND(SPILL FILE
`
`SEND OLDEST SPILL
`FILE TO VIAM
`
`
`
`
`FROM FIG.10A
`
`11
`
`11
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 11 of 23
`
`5,412,801
`
`FIG.10€
`ROUTINES INVOKED WHEN FIFO QUEUE FILLS TO
`THE POINT WHERE AN ADDITIONAL RECORD CANNOT
`BE PLACED ON THE FIFO QUEUE
`
`)FROM FIG.10A
`
`580
`
`582
`
`584
`
`586
`
`588
`
`590
`
`FROM FIG.10A
`
`12
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 12 of 23
`
`5,412,801
`
` ANILNOY
`AMSAOO0SYATIWNOILIGNOO51g9/
`
`
`10°GaLLWaNs
`|
`
`SAILOSTAS
`
`—WNANOP
`
`ONIN
`
`
`
`ergLIOryoL3
`
`(Z1914)
`
`80940d
`C.DZ09
`
`vigZ6$8
`cS400|BN“3)LLCOTA
`
`nD3013dais€or82
`
`
`
`LINENSLINENS<S
`
`
`
`gorAMSAODRYv9
`
`AYVYd!]
`
`13
`
`13
`
`
`
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 13 of 23
`
`5,412,801
`
`START
`EIGRJCLI-600
`
`700
`
`FIG.72
`
`SCAN SCF FOR DATA GROUP WITH
`"CAP ESTABLISHED” FLAG SET, AND
`"GAP RECOVERY SUBMITTED” FLAG
`UNSET
`
`702
`
`704
`
`
`
`ANY
`DATA GROUPS~~NO
`
`FOUND?
`
`YES
`
`
`
`READ DCF TO FIND LOCATION OF
`GAP DATA ON TAPE OR DISK
`
`708
`
`
`
`710
`
`GAP
`
`
`YES|CREATE JCL FILE FOR SINGLE
`DATA WRITTEN
`
`
`GAP RECOVERY AND SAVE
`
`TO TAPE ?
`
`TO DISK
`
`
`716
`
`SET "AWAITING ARCHIVE TAPE”
`aneOr
`FLAG IN DCF
`2
`
`
`720
`YES
`
`
`
`
`
`
`ANY
`MORE DATA
`GROUPS TO
`PROCESS
`?
`
`NO
`
`714
`
`718
`
`SUBMIT JCL FILE
`
`END5722
`
`14
`
`14
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 14 of 23
`
`5,412,801
`
`DESIRED
`TAPE LOADED
`2
`
`652
`
`654
`
`YES
`
`ASK USER TO
`LOAD TAPE
`
`FIG.13
`Cora
`
`
` IS THE
`
`
`READ ARCHIVE DATA
`FROM TAPE
`
`_—656
`
`FILTER DATA WITH SELECTIVE
`LOGGING EXIT
`
`COMPRESS DATA
`
`ENCRYPT DATA IF USER
`SUPPLIES ENCRYPT EXIT
`
`FILL A SPILL FILE
`
`UPDATE SCF
`
`658
`
`660
`
`662
`
`664
`
`668—
`~—-—-—---¥-—-——-—~—
`| £1TIMR WILL INVOKE E1DEQU |
`| AND _E1SPRD AT A REGULAR |
`INTERVAL TO COMPLETE
`PROCESSING
`Lo |
`
`ane”
`
`15
`
`15
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 15 of 23
`
`5,412,801
`
`FIG.14
`
`508
`
`
`
`
`PARAMETERS
`FILE
`
`REGION X
`
` JCL
`
`BACKUP
`LIBRARY
`
`
`
`
`852
`
`854
`
`856
`
`858
`
`860
`
`850
`
`
`DBMS
`E-NET 1
`ARCHIVE
`FORMAT
`
`TAPES
`FORMAT
`
`
`
`862
`
`VARIABLE
`STORAGE
`
`
`
`TAPES
`
`
`16
`
`16
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 16 of 23
`
`5,412,801
`
`FIG.A5A=Gay
`(4)
`1502
`
`
`SET DUP FLAG=OFF.
`
`
`4513|SET SEQ FLAG=FALSE
`SAVE CURRENT
`SET DUPCNTA=0,
`
`
`T—RECORD AS
`SET DUPCNTC=0,
`PREDECESSOR
`CLEAR ID RECORD TABLE
`
`
`
`
`READ ONE T—RECORD
`FROM E-—NET 1
`FORMAT TAPE
`
`1504
`
`1506
`YES
`
`NO
`
`1512
`
`1508
`PRINT STAT
`REPORT
`Cexit
`
`1510
`
`
`NOT FIRST
`
`RECORD, CHECK
`SEQUENCE #
`AND DG4
`
`
`
`PRINT ERROR
`ROPER
`MESSAGE, SET
`SEQUENCE
`ISEQFLAG=TRUE
`
`1516
`
`
`DBMS
`NO
`ID MATCHING
`
`SELECTED
`DBMS
`
`. OUT OF
`SEQUENCE
`
`
`
`17
`
`17
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 17 of 23
`
`5,412,801
`
`1518
`
`1520
`
`.
`FIG. 15B
`
`DECOMPRESS T-RECORD
`
` 1522
`
`NO
`
`
`1S
`CURRENT
`
`
`
`T—RECORD IN FIRST
`DATA GROUP TO BE
`
`
`PROCESSED
`2
`
`
`
`YES
`[|
`OUTPUT
`
`
`
`1526
`
`DUPFLAGS,NO
`ON ?
`ES
`
`1524
`
`(a) YES
`
`1503
`
`[NO
`
`1536
`
`YES
`
`1538:
`
`1540
`
`Kien>
`
`
`nSSET DUPCNTC=1,
`DUPCNIA++,
`OUTPUT
`DUPCNTA++
`DUPCNTC++4
`=
`PRINT MESSAGE:
`” DUPLICATES
`ENCOUNTERED”
`
`1530
`SET DUPFLAG
`
`= OFF 1832
`
`PRINT VALUE
`OF DUPCNT:
`1524
`Co
`
`(4)
`
`.
`
`= oC
`
`18
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 18 of 23
`
`5,412,801
`
`SST AVAILABLE COP:
`
`-FIG.16
`
`QUTPUT | OGIC
`
`FIG. 16
`
`OUTPUT LOGIC
`
`QUTPUT
`SUBROUTINE
`CALLED
`
`1524
`
`1544
`
`13546
`
`1548
`
`T—RECORD
`
`SAVE IDREC
`IN IDREC TABLE
`
`OUTPUT NEEDED
`TIME RECORDS
`(IDMS ONLY)
`
`OUTPUT DATA
`FIELD OF
`
`19
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 19 of 23
`
`5,412,801
`
`soeje
`
`00:9019vAIS
`
`velal5
`
`bv9d]
`
`Ol
`
`94
`
`ONIQN3S
`
`JLIS
`
`JOS
`
`00:9019F
`
`O¢:40SLY
`
`
`
`20
`
`20
`
`
`
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 20 of 23
`
`5,412,801
`
`velaiH
`
`00:90|9FAus1ONIONS
`:ALS
`
`
`
`O£:Z000:9019F
`
`
`
`#¢2:4008:40SLP
`
`OOL
`
`3ANAANO
`
`CLIOITA
`
`o¢6q¢6
`30gBr509
`Z%90S
`|NaNO]
`ONIGNAS
`95:40¥E:40SBb
`
`
`B4016Y
`
`
`
`0¢:2000:9019F
`
`ALS
`
`cl
`
`Ol
`
`92
`
`JOS
`
`SUS
`
`
`
`0¢:4000:90|OF
`
`ve:400:40SL¥
`
`ALIOLA
`
`BC:4016b
`
`21
`
`21
`
`
`
`
`
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 21 of 23
`
`5,412,801
`
`
`
`be:408¢:4016F
`
`él
`
`ONIAIZOSY
`
`
`
`
`
`o¢:£000:9071OPLIS
`
`
`
`ve:L0OF:240SLY
`
`br:40Bf:4016P
`
`re:40SBY
`
`o¢:40SL¥
`
`
`o¢:4000:90719FALIS0ONIGNIAS
`
`7)ALISo¢:4000:9019Y
`
`
`
`be:Z40O£°:40SLt
`
`ONIAISO3Y
`
`ZILH
`
`Ol
`
`OOL
`
`Cann]ALLOLA
`
`
`
`
`
`[misTTDq26Be'40e240SBY
`
`
`
`be:4088:4016Y
`
`ONIGQNAS
`
`JLIS
`
`
`
`o¢:Z4000:901av
`
`
`
`ve:4008:40SLb
`
`
`
`Be:40be:40SBY
`
`beZ0OBE:401Gr
`
`22
`
`22
`
`
`
`
`
`

`

`U.S. Patent
`
`May 2, 1995
`
`Sheet 22 of 23
`
`5,412,801
`
`
`
`0¢:Z000:9019t
`
`ve:Z0OF:40SLP
`
`
`
`BF'L0ve:L0SBF
`
`vy:LO88:401BY
`
`L£v:401OS
`
`cl
`
`ONIAIZ934
`
`ALS
`
`|NANO
`INIGNAS
`
`
`001
`
`HLbOld
`
`94
`
`yo:LZ0OF'40SLv
`
`
`
`8E'40vO:Z0SBY
`
`
`
`by:Z08E:4016%
`
`47:401OG
`
`ALIS
`
`
`
`O£:Z000:9019F
`
`23
`
`23
`
`
`
`
`
`
`

`

`U.S. Patent
`
`May2, 1995
`
`Sheet 23 of 23
`
`5,412,801
`
`ve
`
`SL-OLlAJWVYSNIVWJISJLOWIY
`
`
` AWVd4sNIVASLISWaLNSO
`
`91
`
`ZOBL
`
`~
`
`JANGadVLme
`
`24
`
`SW
`
`T1049
`
`duvMaO4
`
`ALMILA
`
`swga
`
`LIWWYOS
`
`SadVL
`
`SILVINWA)
`(65AdVL
`
`CUWL3GYO4vl'Ol4d33S)
`
`JLXAl3
`
`CiviaqNO41’°9l433S)
`
`24
`
`
`
`
`
`

`

`1
`
`5,412,801
`
`2
`of all changes made to the database. Withoutthis assur-
`ance, a prudent data administrator would abandon the
`entire change log back to the last complete image
`backup. The present invention solves each of these
`shortcomings of the priorart.
`One embodiment of the present invention is known as
`E-NET1 Release 2.0. An earlier version, Release 1.1 of
`E-NETI which is prior art to the present invention,
`solved some of the problems associated with data re-
`covery, but failed to solve most of the problems that
`other priorart also failed to solve. E-NET1 Release 1.1
`could not handle abnormal terminations of the operat-
`ing system at the sendingsite, nor the abnormal termina-
`tion of E-NET1 Release 1.1. Any gaps in the data that
`were created in suchsituations remained as gaps in the
`data. No capability existed within Release 1.1 to re-
`cover a loss of data, and users were required to restore
`the lost data manually. While E-NET1 Release 1.1
`could move log data from a centralsite to a remotesite
`via electronic communicationlines, it required the com-
`taunication lines to be operating and throughput to be
`high enoughso that the sending queue atthe centralsite
`did not overflow, since queue overflow in Release 1.1
`resulted in lost data. E-NET1 Release 2.0 is referred to
`hereafter for brevity as E-NET1.
`
`SUMMARYOF THE INVENTION
`An embodiment of the invention is a method for
`creating a control record for use in future machine
`recovery of gaps in a complete series of journal data
`formed by a computing machine from a completeseries
`of transactional data. The method includes forming a
`copy of the series of journal data for transmission to a
`remote site, forming a plurality of data group records,
`each data group recordidentifying a different boundary
`of each of a plurality of data groups in the series of
`journal data, detecting the presence ofa gap in the copy
`of the journal data for transmission, and creating one of
`said data group records for a boundary of each gap.
`An embodiment of the invention is also a method of
`jogging a complete series of journal data, which are
`created and stored in a computing machine at a central
`site, to a remote site. The method is characterized in
`that there may be a gapin the data created at the central
`site that
`is being received at
`the remote site. The
`method involves copying the journal data as it is being
`stored at the central site, testing for a gap in the copied
`data, creating a data group identifier for a group of data
`in the copied data which in theseries is outside of the
`gap, creating a data group identifier for the gap, and
`transmitting to the remotesite for logging a copy ofany
`of the data including the data in the gap and the data
`group identifiers including the gap data group record.
`An embodiment of the invention is also a method for
`loggingat a remote site for future recovery in the event
`ofa failure at the central site, a copy of a complete series
`of journal data. The journal data is computed at the
`central site from a complete series of transactional data.
`The method is characterized in that there may be a gap’
`in the copy of the journal data received at the remote
`site. The method involves forming a uniqueseries of
`incrementally valued data group identifiers each of
`which identifies a different boundary in the complete
`series of journal data, forming at least one said data
`groupidentifier for such data gap, and transmitting the
`data group identifiers, including such gap data group
`
`40
`
`60
`
`65
`
`GAP RECOVERY FOR OFF-SITE DATA STORAGE
`AND RECOVERY SYSTEMS
`
`FIELD OF THE INVENTION
`
`The current invention is in the field of remotereliable
`database change log duplication.
`BACKGROUND OF THE INVENTION
`
`Since the automation of business operations, data
`managers have worried about how to recover from a
`loss of data, be it from computer malfunction, natural
`disaster, or human error. Typically, businesses make
`image copies of the current state of their data on tape
`and send the tapes to an off-site location for storage in
`the event of a disaster at the on-site location. In the
`event of a total disaster, the business can easily restart
`their operations with the data in the state it was when
`the last off-site backup was made. This approach suffers
`from several drawbacks. Transactions occurring after
`the backup copies are made are lost and have to be
`recovered by some method other than via the off-site
`backups. For some businesses, this reentry would just
`add to the nightmare of the disaster. For example, a
`bank’s automaticteller users or an airline’s agents would
`have to reenter days or weeks worth of on-line activity.
`Full image backups of this type should be made asfre-
`quently as possible to minimize the number of transac-
`tions that would have to be reentered. At a typical
`computersite, backup could be done as often as once a
`day if there is a period during the day in whichlittle or
`no activity is taking place. Minimalactivity periods are
`needed for this scheme, since, in manyinstallations, no
`transactions can be entered while the data is being
`backed up. Having to stop computer operations pres-
`ents a problem to sites that are working 24 hours per
`day. Often, the value of computer equipment is mea-
`sured by the number of hours per day that it can be
`actively operating, so increasing downtime and re-
`source usage for backups decreases the value of the
`equipment. One solution to this problem is that the
`computer system make instantaneous backups. As each
`transaction is created and written to disk,it is also writ-
`ten to tape. While this does keep information more up to
`date and doesn’t require any downtime, the disadvan-
`tage of this approachis that the perceived response time
`of the computer is increased to unacceptable levels
`since the processor must now do twice as much writing
`to computer peripherals. This instantaneous informa-
`tion, commonly called log or journal data, can be used
`in combination with the backups anda utility generally
`provided by the DBMS vendor which would recon-
`struct the current data from the backups and the log
`information which shows each successive changeto the
`database. Forthis to be donereliably requires that every
`single record of a change to the database be known
`since changes later in time are directly dependent on
`changes that were done earlier in time. If earlier
`changes are not recorded, later changes will have a
`tendency to corrupt the data. Generally in a recovery
`process, a data administrator would not use any record
`of changes that occurred after the point where informa-
`tion about the changes had been lost. This is a case
`where having no data is actually preferable to having
`some incomplete data. There is currently no easy way,
`aside from making complete image backupsof the data-
`bases, to insure that the journal of database changes
`which has been movedoff-site is actually a complete set
`
`25
`
`25
`
`

`

`2.
`
`A. UP-TO-DATE BASE DATASET
`B. BACKUP BASE DATASET
`C. CHANGE LOG DATASET
`,
`A. SUCCESSFUL RECREATION OF BASE
`DATA
`B. ATTEMPTED RECREATION OF BASE
`DATA
`3. DCF DATA STRUCTURE
`A. ARCHIVED DATA RECORD STRUC-
`TURE
`B. DCF CONTROL RECORD STRUCTURE
`
`4.
`
`FIG. 1 is a block diagram of a central mainframesite
`and a remote mainframesite;
`FIG.2 is a block diagram of a DBMSregion of main-
`frame main memory;
`A. ECVT STRUCTURE
`FIG.3 is a block diagram of an E-NET1region of
`B. SCOM STRUCTURE
`mainframe main memory;
`5. SCF DATA STRUCTURE
`FIG. 4 is a block diagram of a receiving E-NETI
`A. SCF CONTROL RECORD STRUCTURE
`region of the remote site mainframe main memory;
`B. DBMS CONTROL RECORDS STRUC-
`FIG. 5 is a logic flow diagram illustrating the E-
`TURE
`NET1 regioninitialization;
`C. DATA GROUP RECORDS STRUCTURE
`FIG.6 is a logic flow diagram illustrating DBMS
`6.
`SCF CONTENTS EXAMPLE
`interface initialization;
`7.
`DBMS PARAMETER FILE STRUCTURE
`FIG.7 is a logic flow diagram illustrating the DBMS
`8.
`E-NET1 PARAMETERSFILE EXAMPLE
`write routine;
`9. RCF DATA STRUCTURE
`FIG. 8 is a diagram showing the structure of a T-
`10. RCF CONTENTS EXAMPLE
`record and an example the contents of a T-record;
`11. USER PARAMETERSFILE EXAMPLE
`FIGS. 9A through 9D,are a block diagram showing
`12. DCF/SCF/RCF CONTROL RECORD
`the states of the queue handler;
`STRUCTURE DETAIL
`FIG. 10A is a logic flow diagram illustrating the
`13. GAP RECOVERY CONTROL FILE EXAM-
`queue handler operations while in states 1 and 2;
`PLE
`FIG. 10B is a logic flow diagram illustrating the
`I. DBMS AND MAINFRAME OPERATION,
`queue handler operations while in states 3 and 4;
`GENERALLY
`FIG. 10C is a logic flow diagram illustrating the
`FIG. 1 depicts a typical computersite. All of the
`queue handler operations when the queuefills;
`programs and data discussed subsequently reside in one
`FIG. 11 is a block diagram of the gap recovery re-
`of two physical locations, the central site 10 and the
`gion;
`remote site 12. The central site 10 is where the com-
`FIGS.12 and 13 are logic flow diagrams showing the
`puter activity of interest is occurring, and the remote
`operation of the gap recovery programs;
`site 12 is used to assist in the recovery of the central site
`FIG. 14 is a block diagram of the recovery process;
`in the event of a disaster at the central site. FIG. 1 is a
`FIGS. 15A and 15B are a logic flow diagram (in parts
`bank computer operation and is disclosed by way of
`A, B) showing the flow of the DBMSrecovery pro-
`example, but the scope of the inventionis in no means so
`gram EIEXTC;
`limited. The central site could be a bank data center, a
`FIG.16 is a logic flow diagram of the T-record out-
`bank branch, or a check processing center. In a bank
`put routine;
`branch, the computer, by way of example, keeps track
`FIGS. 17A, 17B, 17C, 17D, 17E, 17F, 17G, and 17H
`of customer’s accounts,so as to allowateller to quickly
`are block diagrams showing usage of spill files; and
`find a customer’s balance, updates customer informa-
`FIG.18 is a block diagram of database shadowing.
`tion when a deposit is made, and updates a customer’s
`DETAILED DESCRIPTION OF THE
`name or address. The central site 10 houses a mainframe
`PREFERRED EMBODIMENTS
`computer (“mainframe”) 16 by way of example, the
`Model 3090 by International Business Machines Corpo-
`INDEX
`ration (IBM). The mainframe 16 has such typical com-
`ponents as main memory 14, for the storage of programs
`and data while the computeris operating; a tape drive
`46 for writing to and reading from removable reels of
`magnetic storage tapes 39; magnetic storage disks and
`disk drives (disk memory) 18, for storage of data and
`programs for retrieval by the mainframe 16 and for
`storage into main memory 14, such disk memory also
`being used for storage of data and programs while the
`computer is not operating; user terminals 42, 43 for
`accepting data and commandsrelating to the manipula-
`
`I. DBMS AND MAINFRAME OPERATION,
`GENERALLY
`Il. GENERAL DESCRIPTION OF OFF-SITE
`DATA STORAGE
`IH. DETAILED DESCRIPTION
`A. NORMAL OPERATION OF SENDING AND
`RECEIVING DATA
`B. SPILL FILE OPERATIONS
`C. GAP CREATION
`D. GAP RECOVERY
`
`5,412,801
`
`4
`E. DBMS RECOVERY
`F. EXAMPLE OF SPILL FILE USAGE
`G. E-NET1 SHUTDOWN
`H. DATABASE SHADOWING OPERATION
`IV. SUMMARY
`V. TABLES
`1.
`
`3
`identifier to the remote site along with the correspond-
`ing data group.
`An embodiment of the invention is also a method of
`using a computing machine for recreating data from
`journal data stored on readable tape at a remote site
`without the use of the database which wasthe source of
`the journal data. The method involves reformatting the
`journal data from such tape into a form usable by a
`roll-forward recovery utility program, detecting dupli-
`cations in such journal data, removing such duplica-
`tions, recording the data without the duplications, and
`processing the data without the duplication with a roll-
`forward recovery utility program to recreate a copy of
`the original data.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`15
`
`20
`
`30
`
`Lee] 5
`
`50
`
`65
`
`26
`
`26
`
`

`

`5,412,801
`
`15
`
`20
`
`6
`5
`tion of data contained in the mainframe 16; an operator
`(formerly published by Cullinet Software, Inc.) (Rev.
`console 44, for receiving messages on screen 44a and
`0.0, 9/1986, Order #TDDB-010010000), for CICS in
`issuing commands on keyboard 44¢relating to the oper-
`CICS General Information, published by IBM (4th. Ed.,
`ation of the mainframe 16; and communicationsinter-
`2/1987, Order #GC33-0155-3), for IMS in IMS/VS
`face 50 to a modem 52, for the purpose ofelectronically
`Version 2 General Information Manual, published by
`sending data from the mainframe 16 to remote locations
`IBM (2nd. Ed., 6/1987, Order #GC26-4180-1), and for
`over communication line 53, which can be a normal
`CPCS in Check Processing Control System Program Ref-
`telephone line, a dedicated phone line, or other means
`erence and Operations Manual, published by IBM (9th.
`Ed., 6/1988, Order #SH20-1228-8), and for further
`of transmitting data. Althoughit will be understood that
`the central site. typically includes multiple user termi-
`information about VWTAM communications, VTAM
`nals, operator consoles, disk memories,
`tape drives,
`Programming, published by IBM (3rd. Ed., 9/1985,
`communication links, modems, and even multiple main-
`Order #SC27-0611-2 File $370/4300/30XX-50),
`the
`frames tied together as one large, virtual mainframe,
`disclosure each of which is incorporated by reference
`herein.
`many such duplications have not been shown for sim-
`plicity.
`Mainframe 16 running MVS operating system 20,
`The mainframe 16, whenfirst initialized by applying
`uses terminal 42 for interaction with the IDMS program
`running in A Region, and user terminal 43 for interac-
`power or using reset procedures loads a program from
`a predetermined location in disk memory 18 into main
`tion with the CICS program running in B Region. Fur-
`memory 14 that controls the hardware of the mainframe
`ther, MVS hasallocated several files, called datasets,
`and the software being used on the mainframe. This
`two of which are shownat 72 and 74 in disk memory 18,
`for use by CICS. Base dataset 72 contains information
`program is knownas the “operating system”. The oper-
`ating system program Multiple Virtual Storage (MVS)
`vital to the bank, namely data concerning its customer
`20, is loaded into main memory 14 and controls the tape
`accounts. Table 1A showstypical data contained in the
`base dataset 72.
`drive 46 and disk memory 18 as well as the operator
`If the information contained in base dataset 72 were
`console 44 and user terminals 42, 43. MVS 20 also con-
`trols the placement of programs and data into main
`lost or inaccessible, the disruption to the bank’s business
`would be disastrous. Possible causes of such data loss
`memory 14, as well as allocating unused main memory
`are varied. The data might be lost due to hardware or
`for use by the various executing programs loaded into
`software failure, or due to a natural disaster. The data
`main memory. MVSlogically divides main memory 14
`into multiple regions,
`sometimes called “address
`could be temporarily lost as a result of a central site
`power failure, or an operating system stoppage. If the
`spaces” to allow multiple operations or programs to
`execute simultaneously. FIG. 1 depicts three regions: A
`poweris lost, or the operating system stops abnormally,
`the data stored in main-memory 14 is lost, but the data
`region 26, B region 28, and C region 30, but the number
`of regions is not limited to three. MVS keeps track of
`stored in disk memory 18 can usually be accessed nor-
`whatis happening in each of the regions, creating new
`mally once power to the mainframe and the operating
`regions from unused memory and removing regions and
`system is restored. The operating system should not
`normally stop absent an operator command to shut-
`programsafter the programs havefinished their opera-
`tion.
`down. The condition of abnormal operating system
`By way of example, the program, Integrated Data-
`failure is designated ABEND (ABnormal ENDing).
`base Management Software (IDMS), a product of Com-
`To avoid total loss of data in the case of permanent
`failure of the disk memory 18, or erasure of the dataset
`puter Associates International, Inc. (formerly a product
`of Cullinet Software, Inc.), is loaded and running in A
`72, the mainframe operator makes backup copies ofthe
`Region and the program, Customer Information Con-
`data stored in disk memory 18. Typically this is done
`trol System (CICS), a product of IBM, is loaded and
`during a period when no DBMSprogramsare running,
`and on a reguiar daily, weekly, or other basis. As shown
`running in B Region. No programs are running in C
`in the example backup data of Table 1B, the bank backs
`Region. The programsof each region do not generally
`interact with each other, but do interact with their own
`up its data at the beginning of every month. A backupis
`MVS-allocated terminals and disks. However, MVS
`typically done by an operator issuing a command at
`console 44 which starts image backup program 106
`provides inter-region interaction by the use of an extra-
`regional memory area known as the Common System
`copying the data from disk memory 18 to tapes 39. The
`Area (CSA) 40, and subprograms of MVScollectively
`tapes 39 are then sent off-site for secure storage. Typi-
`knownas Cross Memory Services CXMS) 48. MVSalso
`cally,
`this step makes an entire copy of all data. A
`includes subprograms collectively known as Virtual
`backup of an entire database is called an image backup
`Telecommunications Access Method (VTAM) 64 for
`since an image ofall the data is taken. The opposite of
`controlling communications through the modem 52.
`an image backup is an incremental backup where data is
`The programs IDMSand CICS are commonly called
`not backed up at once, but is backed up in increments.
`Database Management Systems (DBMSs). The purpose
`Image backup procedures have drawbacks. Hf the
`of a DBMSis to accept input data from users, store data
`data is saved at the beginning of the month, andadisas-
`in an orderly fashion, and retrieve it for later users.
`ter occurs in the middie of the month, the data restored
`from tape would be a half month out of date. For a
`Although a detailed discussion will not be given of each
`DBMS,a discussion of the detailed operation of one
`bank, this would be unacceptable. Customer balances
`DBMS, CICS, will provide one skilled in the art an
`would not reflect deposits and withdrawals that oc-
`understanding of how to make and use the invention.
`curred during that half month. A backup could be made
`Further
`information will be found for MVS in
`more often, but even a daily backup would be out of
`MVS/Extended Architecture Overview, published by
`date if the disaster occurred in the middle of the day.
`IBM (First Ed., 3/1984, Order #GC28-1348-0 File
`Another drawback to image backupsis that, since an
`$370-34), for [DMS in IDMS/R Concepts and Facilities,
`exact copy of the entire database must be made, in most
`published by Associates Computer International, Inc.
`installations no part of the database can be changed
`
`40
`
`45
`
`60
`
`27
`
`27
`
`

`

`7
`during the backup process. Because no changes can be
`made while the image backup copyis being taken, the
`mainframe’s operations must be suspended while the
`backup is in progress. Since computer downtime is
`costly, backups must be kept to a minimum. Further
`costs accrue if the computer center is operated 24 hours
`a day, since work must stop during the backup.
`Banks and other businesses that process transactional
`data store change logs. A changelogis a dataset show-
`ing all the changes to a base dataset that have occurred
`since the last backup. The DBMSsoftware packages,
`CICS, IDMS,and IMS,include programs which auto-
`matically produce change log datasets. Table 1C lists an
`example of change log dataset 74, and details all the
`changes madeto the base dataset 72, Table 1A. Since a
`record of a change could take up muchless disk space
`than the data which could have been altered by the
`change, the log dataset 74, sometimes called journal
`data, is continuously backed up to tape for safekeeping.
`Whena disaster strikes, the operator loads the backup
`data from the last image backup and, in someinstalla-
`tions, the change log from tapes 39 onto the disk mem-
`ory 18, and issues a command via the operator console
`44 to initiate a roll-forward recovery program (not
`shown in FIG. 1) provided by the supplier of the
`DBMSsoftware. Roll-forward recovery programs are
`well known in the art and read a record of a change
`from the log dataset 72, Table 1C, and use the record to
`determine how the base dataset 72, Table 1B, was
`changed. Knowing how the base dataset 72 changed,
`the recovery program can change the restored backup
`data to reflect the change. After all the change records
`are applied to the backup data in chronological order,
`the backup data will have changed to accurately reflect
`the most current base data.
`Table 2A illustrates an example of the operation of
`the roll-forward recovery process. Table 1B depicts the
`base dataset 72 as it was restored, current as of Jul. 7,
`1989, but out of date as of Jul. 20, 1989, the time of an
`assumed disaster. Table 1B indicates that the customer
`account 01-6056 had a credit limit of $1,000 as of Jul. 1,
`1989. In the time between Jul. 1, 1989 and Jul. 20, 1989,
`several changes were made to the value of the credit
`limit, as shown by Table 1C.
`Changes to base dataset 72, Table 1A, are made by a
`CICS user issuing commands to the DBMS main pro-
`gram 83 in region B (shown in FIG.2) via user terminal
`43. DBMS main program 83 acts upon a.command by
`changing the value stored in the base dataset 72 and
`adding a record of the change to log dataset 74, Table
`1C. The modification of base dataset 72 and the writing
`to log dataset 74 is done by a subroutine of CICS,jour-
`nalling routine 66. The journalling routine 66 ensures
`that every changeto base dataset 72 is logged to the log
`dataset 74. The credit limit for the account 01-6056,
`changed four times between Jul. 1, 1989 and Jul. 20,
`1989, as shown in Table 1C,lines 1, 6, 9, and 16. After
`the assumed Jul. 20, 1989 disaster, the correct value of
`the credit limit can be restored by taking the backup
`value $1,000, in Table 1B, and applying the changes of
`lines 1, 6, 9, and 16, in order, to arrive at the final value
`of $6,000. The same can be done for the balance and the
`interest rate. All other accounts and base data can simi-
`larly be restored using this method.
`Most DBMSstypically hav