FWKCS(TM) Contents_Signature System
` Version 1.22
` 1993 August 10
`
` (C)Copyright Frederick W. Kantor 1988-1993. All rights reserved.
`
`EMCVMW 1004
`
`

`

` All of the programs and documents, comprising the entire contents
` of
` this Authenticity Verification Zipfile FWKCS122.ZIP,
` together with this Zipfile itself,
` are,
` in accordance with their respective dates of creation or revision,
` (C)Copyright Frederick W. Kantor 1988-1993.
` All rights reserved.
`
`

`

`Your use of this program is at solely your own risk: PLEASE HAVE PROPER BACKUPS
`........................ C A V E A T O P E R A T O R ........................
`
` Disclaimer:
` ~~~~~~~~~~~
` Frederick W. Kantor disclaims any and all warranties as to this
` software, whether express or implied, including by way of example
` and without limitation any implied warranties of merchantability,
` fitness for a particular purpose, functionality, data protection,
` or data integrity.
` The information provided in the programs and in the documentation
` may contain error(s). Your use of any of this material is at
` solely your own risk. All programs provided in this package, and
` the format(s) of any file(s) which they generate or use, are
` subject to change without notice.
`
`

`

` Distribution
` ~~~~~~~~~~~~
` If you received this on a disk, the disk contains
` FWKCS122.ZIP the FWKCS(TM) Contents_Signature System, in an
` Authenticity Verification Zipfile.
` If you upload this material to a bulletin board system, the only part
` which you have permission to upload is FWKCS122.ZIP, in its original,
` unmodified form.
` If you upload FWKCS122.ZIP to a bulletin board system which does not
` automatically post the FILE_ID.DIZ description provided in the
` zipfile, please provide the instructions given in the FILE_ID.DIZ
` file.
`
`

`

` Preface
` ~~~~~~~
` In the late 1980's, there was an unanswered problem of duplicate files
` on electronic bulletin board systems ("BBS('s)"). People would change
` the name of a file and upload it to a bulletin board, to get time_credit
` for a "new upload" which they could then use to download other files.
` And they, in turn, were then potentially subject to becoming the victim
` of someone else's misdirected eagerness to get spurious upload credit.
` Meanwhile, the bulletin board system operators ("sysop(s)") were paying
` for hardware to provide capacity for these spurious files, and spending
` many hours trying to find and delete them.
` This problem grew more complex with the widespread adoption of
` compressed library files as the preferred means for moving collections of
` files which needed to be kept together, such as the interdependent parts
` of a program system, or a program with its documentation, etc.. Once
` that came into use, a person could easily change a comment on a file, or
` change the name or date of a file in a library, or change the order in
` which the files appeared in a library, and call the result "new" -- and
` then, even a direct binary comparison of the compressed libraries would
` no longer show that they contained essentially the same material. Also,
` people could take parts of different compressed libraries, put them
` together to make a "new" package, and upload that. Many expensive long
` distance calls were being made by people downloading "new" files, to
` discover later that they already had the material... Not a pretty
` picture.
` The nascent BBS community was beginning to touch on (and some large
` BBS's were already burdened by) a potentially serious problem, via a
` feedback mechanism in which an eagerness for (in this case, spurious)
` short_term reward can harm or destroy a long_term asset -- a feedback
` property called, in economics, "the tragedy of the commons".
` This problem came to my attention via The Invention Factory BBS, in
` New York (sysop: Michael Sussell). Guided by concepts from my work on
` the foundations of physics (see Frederick W. Kantor, "Information
` Mechanics" (Wiley, 1977)), I looked into this problem, treating it as a
` question about how information is represented. It became apparent to me
` early that 32 bits, such as is used in a cyclic residue check ("CRC") to
` support tests of file integrity, would not provide sufficient statistical
` resolution. To provide more resolution, high speed operation, and
` compatibility with existing equipment and software, I designed what I
` called a "contents_signature" ("cs"). As measured experimentally, it
` typically carries a pair_wise statistical error rate of less than one
` part in ten trillion, much better than is mathematically possible for a
` 32_bit CRC. To permit wide use of this cs, I placed the format for the
` resulting cs list file in the public domain.
` Many special functions were required to implement the system, starting
` with those required for generating and manipulating contents_signatures
` efficiently, and including such capabilities as rapidly acquiring new
` information, rapid retrieval, consolidating files and rebuilding the data
` base while the system is up and running, and so on, so that it would be
` able to operate properly in automatically screening incoming files for a
` large BBS. Because the system design is guided by information mechanics
` ("IM") concepts, in order to avoid conflict of assumptions and provide
` high performance, the entire FWKCS.EXE code is written directly in
` assembly language. Many powerful program features, originally developed
`
`

`

` to meet internal needs of FWKCS, have been made externally available so
` that they can be controlled and used by other programs.
` Recent events have included the development of FWKCS procedures for
` dynamic allocation of supporting computer resources on a network, with
` both "split testing" and the new client/host configuration; the use of
` FWKCS in the automatic interpretation of text; and the use of FWKCS
` contents_signatures in blocking commercial programs accidentally
` uploaded to public electronic bulletin boards.
` Among the functions it can now automatically perform, here are some
` ways in which it has helped the BBS community avoid "the tragedy of the
` commons":
` To work properly with Phil Katz's widely used "Zipfile"
` structure, FWKCS has the special ability to make a "zipfile
` contents signature", ("zcs") which is independent of zipfile
` comment, file comments, zipfile name, zipfile date, order in
` which files are included in the zipfile, the names and dates of
` files in the zipfile, zipped path information, and file
` compression ratio. By comparing their zcs's, FWKCS can find,
` with a statistical pair_wise error rate typically less than one
` part in ten trillion, zipfiles which contain essentially the
` same inner file material. This has resulted in major clean_ups
` on many BBS's (in one case, over half a Gigabyte of space was
` thus freed).
` By means of "contents_signatures", FWKCS automatically
` compares all the files in a newly received standard zipfile
` ("standard" means that it is built according to the rules set
` down by Phil Katz, up to and including a full DOS tree of
` subdirectories, with no "gimmicks") with all the files in all
` the standard zipfiles on an entire electronic bulletin board
` system, working in real time, independent of filenames, file
` extensions, dates, compression ratios, comments, zipped paths,
` and order in which the files appear in the zipfile. It
` recognizes as redundant a zipfile made of pieces scattered in
` many different zipfiles on the system. It is used to prevent
` wasteful duplication, and to protect against unwanted files.
` It automatically evaluates uploaded files, and prepares reports
` which are sent to the users. FWKCS is in use on electronic
` bulletin board systems with multiple servers and up to more
` than 80 nodes, and runs also on a single personal computer for
` personal use. It is also used for handling a mix including
` zipfiles, "GIF" files, and plain files.
` FWKCS can run tests on all the files in a non_encrypted
` standard zipfile -- including the case of zipped paths for a
` complete DOS tree of subdirectories -- while using only a single
` subdirectory as a workarea. It preserves the Authenticity
` Verification stamp and the internal structure of the zipfile
` being tested. In this way, it helps serve the need of the
` creative author to protect the integrity of his or her work,
` while helping the bulletin board operator to protect against
` harmful files.
`
`
`

`

` FWKCS can be used remotely, via LOOKUP. This can contribute
` significant savings in time and communication costs for users
` of electronic bulletin boards. Cross_references can also be
` followed up remotely, using RCROSREF.
`
` This software is offered on a test_before_you_decide basis; to run an
` unregistered copy of FWKCS(TM), keep a copy of REGISTER.FRM in the same
` directory with FWKCS.EXE. You can put FWKCS.EXE on your path, and call
` it from anywhere on your system. The suggested test time is two weeks.
` The registered version saves time, because it does not have to verify
` REGISTER.FRM. This software is not free -- if you do not register, then
` you should discontinue using it. In recognition of the reality of "the
` tragedy of the commons", one second is added for each day after day 30
` before proceeding, counting the date of first execution as day zero.
` PLEASE TAKE NOTICE, that the unauthorized use of this software for
` more than thirty (30) days in a public, commercial, school,
` institutional, or governmental environment, without payment and
` registration, is not permitted.
`
` As you might glimpse in the following Table of Contents, FWKCS has
` become a robust platform for supporting contents_signature functions.
` With its speed, flexibility, power, and resolution, it stands in a class
` of its own.
` ----------------------------------------------------------------------------
`
`

`

` page i
` Table of Contents
` Overview and Introduction............................................ 1
` To the reader...................................................... 1
` Note: "Kilobyte", "KB", "Megabyte", "MB", "Gigabyte", and "GB"... 1
` Note to programmers: interfacing to FWKCS.EXE, FWKCSC.COM,....... 1
` on_line help menus................................................. 2
` See Appendix 1 for annotated list of files and programs............ 2
` Overview........................................................... 2
` Purpose:........................................................... 2
` Please be careful................................................ 3
` Some Trademarks.................................................. 3
` Thanks........................................................... 3
` Introduction....................................................... 4
` What is a "contents signature"? ("cs")........................... 6
` The FWKCS(TM) Contents_Signature................................. 7
` Contents_Signature structure and internal format............... 8
` Contents_Signature input format................................ 8
` /f to Find if there is a matching contents_signature......... 8
` /g to Get all matching contents_signatures................... 8
` /u to Update the contents_signature collection............... 8
` Zipfile_Contents_Signature ("zcs")............................. 9
` Evaluating Statistical Errorload................................. 9
` Reports of Experimental Data................................... 9
` From Channel 1.............................................. 10
` From The Invention Factory.................................. 10
` Note........................................................ 10
` Practical consequences........................................ 10
` Outline of data collection and calculation.................... 11
` for on_line help, see \CS\SPECTRUM /* <enter>............... 11
` for on_line help, see \CS\FWKCSS /* <enter>................. 11
` Comparing cs with 32_bit CRC.................................. 12
` for on_line help, see \CS\FWKCST /* <enter>................. 12
` Remarks....................................................... 12
` Some Conceptual Errors to Avoid............................. 13
` References.................................................... 16
` FILES=30 or OS/2 2.0 DOS_FILES setting of 30...................... 16
` network queues.................................................... 16
` node identification on a network or multitask system.............. 16
` reserved names based on node ID................................. 17
` system "housekeeping"............................................. 18
` for complex external activity, short directory paths in DOS....... 19
` option for short path for temporary files and workareas......... 19
` The FWKCS command_line.............................................. 19
` Grammar 1,2,3 : /(r)(tNNNN.NN)(&e,el;)(&n)(xNNN)other_option(s)... 20
` Format 1.......................................................... 21
` Format 2.......................................................... 21
` reading file(s) into command line, use of macros.................. 21
` Format 3.......................................................... 21
` capturing expanded command line................................. 24
` Editing FWKCS/FWKCSC Default Settings (Advanced Users)............ 25
` re editing default settings, see GET_DFLT /*<enter>............. 25
` re bulk deletion of FWKCS settings, see REVERT /*<enter>........ 26
` Error Log......................................................... 26
` &e,d:\csa\fwkcntrl.err.......................................... 26
` e,d:\csa\client.err;............................................ 26
`
`

`

` page ii
` d and d! - Default names, paths, macros........................... 27
` using /dx to suppress an operation.............................. 28
` macros............................................................ 29
` e - Errorlevels + messages........................................ 30
` l - License, disclaimer, and dedication............................. 30
` FWKCS/FWKCSC Environment Variables.................................. 31
` FWKKTS - Keep time_slice.......................................... 31
` FWKNODE=ccc - Node identification................................. 32
` "preface" options (see also Grammar, above)......................... 32
` r - Redirect message or screen to "standard output"............... 32
` tNNNN.NN - Time to keep retrying, 0-9999.99 seconds............... 32
` &e,el; - extra Error log. (See also Error Log, above.)............ 32
` &n - deNy write................................................... 32
` xNNN - eXclude use of upper NNN (decimal) Kilobytes............... 33
` "/" options ("/1" options are listed after these)................... 33
` Single "/" followed by (preface+)option(s) are all contiguous..... 33
` f - Find if matching contents_signature(s)...................... 33
` g - Get all matching contents_signature(s)...................... 33
` f,g,u format.................................................... 33
` re GET.BAT, see GET /*<enter>............................... 34
` f,g suboptions.................................................. 34
` j,junked - keep log of proposed deletion(s)................... 34
` m - collect for deletion if cs Match.......................... 34
` nN,NN(NN)(,NN); - Novelty test (four tests, 3 current)........ 34
` Example: n1,30.............................................. 34
` Example: n2,1 (obsolete; see n4, below)..................... 35
` Example: n3,1............................................... 35
` Example: n3,2............................................... 35
` Example: n4,1,49............................................ 35
` o,oust - deletion list........................................ 35
` p - Purge non_AV from AV zip.................................. 36
` r,refs - cross_Referencing.................................... 36
` w - Wipe...................................................... 36
` re WIPE.BAT, see WIPE /*<enter>............................. 36
` h - Help (or FWKCS <enter> sans option; or FWKCS /* <enter>).... 36
` see also /** , /0 , /1 , /a , /d , /d! , /e , /l, /s*......... 36
` i - make Index CSLIST.NDX for use with CSLIST.SRT............... 36
` m(cdP|S) - Merge unique (Difference) lines from sorted files.... 36
` c - line terminator = Carriage_return line_feed............... 36
` d - Difference, delete third file if named and empty.......... 36
` p - keep Path, tests made according to column 17 flag......... 36
` s - Sans path, tests made according to column 17 flag......... 36
` Examples...................................................... 37
` mn - Make Nullfile.............................................. 37
` r /1r /r/1 - Redirect screen a, d, e, *(h), l, s*, 0, 1*........ 37
` revert - reset all entries on right of "=" on /d and /d!........ 38
` register - set, clear, or report registration status............ 38
` Report registration status:................................... 38
` Clear the registration from a registered copy:................ 38
` s - Sort........................................................ 39
` bNNN - merge_file input Buffer, least size, in "page"(s)...... 39
` d - Delete each temporary file promptly....................... 39
` fNNNN - Fixed field (line) length............................. 40
` (-)NNNN(:NNNN), - (descending)key_position(:key_length)....... 40
` r - Reverse sorting order for all keys........................ 40
` c - line terminator = Carriage_return line_feed............... 40
`
`

`

` page iii
` e - Empty line as block_terminator............................ 40
` mNNNN - Maximum string or block size.......................... 40
` p"l" - 'text Paragraph' terminator_letter flush left.......... 40
` pNNN, - to specify any ASCII character 0...255 as such........ 40
` tNNN(,...); - Terminator for string or block.................. 40
` u,tmp; - Use (d:\path\)tmp as directory for temporary files... 40
` t,nnnn.nn - Time for retrying................................... 41
` u(p|s) - Update................................................. 41
` up - restricts Update with tests as in vp, below.............. 42
` ver - show Version, set exit errorlevel = (version number)*100.. 42
` v(cNNNNNNNNp|s) - reVise default files #1-4..................... 42
` cNNNNNNNN - Concurrent operation.............................. 43
` p - keep Path, add only unique cs(+flag+ref)'s.............. 44
` s - Sans path, add only unique cs(+flag+ref)'s.............. 44
` vp, discussion................................................ 44
` tests made according to column 17 flag........................ 44
` vs, discussion................................................ 44
` re concurrent revision, see \CS\S_REVCSL /*<enter>............ 45
` re where to keep CSLIST.* and CSLIST1.* files................. 45
` From the /d screen, with labels added:...................... 45
` "/1" options ("/" options are listed above)......................... 45
` Usage............................................................. 45
` If preface: single "/", preface, single "/1", all contiguous...... 46
` Single "/1" followed by /1 option(s) are all contiguous........... 46
` /1 default is to do fpx3z....................................... 46
` Option * ? h (/1 help) suppresses acdefjmnpsvwxyz............... 46
` /1r Redirect screen to "standard output"........................ 46
` aNNNN - zipfile Age test........................................ 46
` b(1) - process disk_spanning Backup zipfile..................... 46
` c(0) - expect a Control_list.................................... 46
` d - reDate zipfile to latest file in it......................... 48
` e - Edit input from control_list................................ 48
` f - make cs for file(s) in zip(s)............................... 48
` j - strip/ignore "GIF" file's tail.............................. 49
` m(NNN) - Make MULTCNT.RPT....................................... 49
` mNNN - keep sample if multiplet >= NNN........................ 49
` n - open files "deNy_write"..................................... 50
` p - make cs for Plain file...................................... 51
` r - Redirect messages to standard output........................ 51
` s - Scan a sorted contents_signature file for cs_multiple(s).... 51
` Example....................................................... 51
` v - if any zipfile, set errorlevel = 1.......................... 52
` w - set errorlevel = 1 if apparently nonzip file................ 52
` x(N) - eXpress encryption and/or AV............................. 52
` /1x Output Format............................................. 52
` AVONLY.BAT.................................................. 54
` Example:.................................................... 54
` y = make cs for zipfile as if plain file........................ 55
` z = make a "Zipfile contents signature" for (each) Zipfile...... 55
` Auxiliary Functions................................................. 55
` Swapping most of program out of memory............................ 56
` #; - swap out of memory for all SHELL operations................ 56
` #luvz; - swap if List Unzip scanV Zip........................... 56
` Specifying a subdirectory for temporary files..................... 57
` u,tmp; - Use subdirectory d:\path_tmp for temporary files....... 57
` Auxiliary Function 1, search file list for novel entry(s)......... 57
`
`

`

` page iv
` 1 filenames..................................................... 57
` 2 contents_signatures........................................... 57
` 3 do both tests................................................. 57
` tNNNN.NN - timespan for retrying (on network, length of queue).. 57
` #(luvz);u,tmp; See discussion of Swapping most of program out... 57
` Auxiliary Function 2, local fast find / update.................... 59
` tNNNN - timespan for retrying................................... 59
` #(luvz);u,tmp; - See discussion of Swapping..................... 59
` &e,el; - See discussion of Error Log............................ 59
` b - process disk_spanning Backup zipfile........................ 59
` d - reDate zipfile(s) to the latest date of the files inside.... 59
` &j - strip/ignore "GIF" file's tail............................. 59
` &n open files in "deny_write" mode.............................. 59
` re Y.BAT, see Y /* <enter>...................................... 60
` Terse form YT.BAT............................................... 60
` Auxiliary Function 3 - can rename rejected files.................. 60
` tNNNN - timespan for retrying................................... 61
` #(luvz);u,tmp; - See discussion of Swapping..................... 61
` &e,el; - See discussion of Error Log............................ 61
` aNNNN - Age test in months of latest file in zipfile............ 61
` &bN - bit mapped, to specify which AV form(s)................... 61
` 0,1 PKZIP(R) Ver. 1.10........................................ 61
` 0,2 PKZIP(R) Ver. 2.04g....................................... 61
` b - test for Bad Authenticity Verification stamp................ 61
` c(,zcomb;) - strip (replace) zipfile Comment; else keep......... 62
` d - reDate zipfile to latest file in it......................... 62
` &j - strip/ignore "GIF" file's tail............................. 62
` j,junked; - log deletions....................................... 62
` kN - apparent encryption in zipfiles............................ 62
` N = 0......................................................... 62
` N = 1, 2, 3................................................... 62
` &mNN - max. allowed uncompressed length, 0-99999999 Kilobytes... 62
` m - delete from zipfile if cs_match found....................... 62
` &n - open "deNy_write".......................................... 63
` nN,NN(NN)(,NN); - novelty test (four tests, 3 current).......... 63
` o,oust; - list for deletion..................................... 63
` p - Purge non_AV file(s)........................................ 63
` &r,names; - list of up to 64 reserved names..................... 63
` r,refs; - collect references.................................... 63
` contents_signature cross_reference format..................... 63
` s - keep Same extension, still set errorlevel................... 64
` v(2)(,vdat;) - call SCAN for plain_file Virus................... 65
` vn - (obsolescent format; see v2)........................... 65
` v2 - set (NET)SCAN to UNATTEND mode........................... 65
` v1,telsysop; - d:\path\file for warning about "virus"........... 65
` v3 - suppress SCAN's expiration message......................... 65
` v4,d:\path\scan.exe; - specific, overrides /d8 setting.......... 65
` v5,(d:\path\)scan.exe; - global, overrides /d8 setting.......... 66
` w - Wipe via o if column 17 w flag.............................. 66
` z - ZIPs_only, else can handle both zip and non_zip files....... 66
` z1 - if zipfile, must have AV_stamp............................. 66
` z2 - if zipfile, must have AV_stamp recognized as acceptable.... 66
` To enter a new AV_stamp for recognition,...................... 66
` z3(~3+-)=; - 7 controls "(~3+-)=" to pass to PKZIP 2.04g........ 66
` z43+-); - 4 controls "3+-)" to pass to PKUNZIP 2.04g............ 66
` z5 - suppress special treatment of false zipfile disk label..... 66
`
`

`

` page v
` Renaming files under /a3........................................ 66
` re ACCESION.BAT, see ACCESION /* <enter>........................ 67
` Terse form ACCESN.BAT........................................... 67
` Auxiliary Function 4 (all /a3 options, plus zipfile tests)........ 68
` tNNNN - timespan for retrying................................... 69
` #(luvz);u,tmp; - See discussion of Swapping..................... 69
` &e,el; - See discussion of Error Log............................ 69
` f,Free; - optionally specify Free workarea...................... 69
` default = create and delete separate subdirectories........... 69
` Node ID may be used in DIR name, else DOS unique name....... 69
` kN - apparent encryption in zipfile............................. 70
` &q - when files found with same name in a zipfile............... 70
` &tN specify amount of testing................................... 70
` Discussion of virus testing of zipfiles......................... 71
` re ACCESZIP.BAT, see ACCESZIP /* <enter>........................ 72
` Terse form ACCZIP.BAT........................................... 72
` Auxiliary Function 5, bulletin board + library.................... 72
` auto test + accession files..................................... 72
` direct testing of each uploaded file:......................... 72
` split testing:................................................ 72
` client / host:................................................ 72
` - Format for direct use and for split testing..................... 73
` for on_line help, see SPLITEST /* <enter>................... 73
` for on_line help, see DO_BLOG /* <enter>.................... 73
` With PCBoard.................................................. 74
` Generic use................................................... 74
` for on_line help, see BBSTART /* <enter>.................... 74
` for on_line help, see SCANFILE /* <enter>................... 74
` - Format for BBS operation with FWKCS host........................ 75
` for on_line help, see DO_HOSTQ /* <enter>..................... 75
` - Format for BBS node operation as client......................... 76
` for on_line help, see BBCLIENT /* <enter>..................... 76
` Whether or not UPLOAD or ATTACH................................... 77
` --- sets errorlevel (see /e screens, below, or FWKCS/e)......... 77
` tNNNN.NN - time for retrying.................................... 77
` #; - swap all................................................... 77
` #luvz; - swap if List Unzip scanV Zip........................... 77
` u,tmp; - Use d:\path_tmp for temporary files.................... 77
` &e,el; - extra Error Log........................................ 77
` f,Free; - (optional) specific empty subdirectory to reuse....... 77
` when to add a ##CELLAR........................................ 77
` f=d:\Free; - here Free is a path................................ 79
` re ADD_CELR.BAT, see ADD_CELR /*<enter>. (re f, f=)......... 79
` &n - open files "deNy_write".................................... 79
` x(0) - ascii output (not use @codes)............................ 79
` If UPLOAD or ATTACH............................................... 79
` If ATTACH, contents_signatures are not added to data base..... 79
` Split Testing................................................... 80
` re installation, see SPLITEST /* <enter>...................... 81
` re routine operation, see DO_BLOG /* <enter>.................. 81
` Upload Credit................................................... 81
` excludes (zip)file if cs_match with x_flag...................... 81
` re marking material for exclusion, see EXCLUDE /*<enter>...... 81
` !N - bit_mapped option for deferred dispos