15 Dec 1999. Daniel Hellerstein (danielh@crosslink.net) UpDater: A mirroring addon for SRE-http (beta 1.07) Abstract: UpDater is a "mirroring program" for OS/2 web servers. UpDater is used to update a set of files on a "remote" server (or on a "remote" client) so that they are the same as a set of files on a "central" server. UpDater uses rsync and gzip to minmize bandwidth requirements, and is managed via a www-aware interface (using HTML forms). ---------------------------- Contents: I) Introduction Ia) Feature list II) Installation III) Defining Packages on a Central Server IIIa) Example of a package file IV) Defining a Sitelist on a Remote Server IVa) Example of SITELIST.IN V) Using encryption Appendices) Appendix 1) File list Appendix 2) User-configurable parameters Appendix 3) Installing UpClient as a NetScape helper application. Appendix 4) The UpDater API Appendix 5) Disclaimer and terms of use: ---------------------------- I) Introduction: UpDater is a "mirroring program": it is used to update a set of files on a "remote" server (or on a client's workstation), so that they are the same as a set of files on a "central" server. UpDater has a www-aware user interface -- it is invoked and controlled with HTML forms. After choosing options (such as what files to download, and where they should be copied), UpDater will use TCP/IP socket calls to acquire files from the central server, and save them to it's local hard disk. To minimize bandwidth requirements, UpDater uses GZIP and Rsync. Rsync is a differencing algorithim that is used to transmit only the fraction of a file that has changed, and GZIP is used to compress this fraction. What is especially nice about RSYNC is that the central-server does NOT need to maintain old versions of files. Instead, the remote server sends a synopsis of an old version of a file, and the central server uses this synopsis to compute a difference file. Thus, one trades off processor utilization (the need to compute a synopsis, and to compute a difference given this synopsis) to lessen bandwidth requirements and to remove the need for the central server to maintain extensive archives of old versions. UpDater has three components: two "remote" components, and a "central" component. The central component is installed on a "central" server, a server that maintains "packages" of files. Defnition: A package is a set of files that have some logical relationshiop. Basically, you can think of a package as a set of files that could be distributed on a CD-ROM (or on a set of floppies). The two remote components are for either a "remote server" (running SRE-http), or for a "remote client" (running NetScape under OS/2). Remote server: A pair of programs are used to implement the remote server component of updater. The first part handles communication with a client, using HTML forms to allow the client to select a package of files from one of several central servers. After determining the desired package, control is passed to the second part, which runs as a "daemon" on the remote server -- this daemon does the actual work of obtaining the files from the central server. Remote client The "remote client" communicates directly with a central server. The central server will send the remote client a special "application/x-updater" response; which will then be used by the UpClient Netscape helper application. UpClient will ask a few questions, and then obtain files from the central server. Actually, a remote server (or a remote client) can obtain packages from a number of central servers. Furthermore, a remote server can also be a central server for an unlimited number of other "remote" servers (and remote clients). NEW: A stand-alone UpDater (UpDateMe.CMD) is now included with UpDater (it is a classic OS/2 REXX program). Ia) Feature list UpDater features include: * Can be used as addon for the SRE-http web server; or as a cgi-bin script on most OS/2 web servers that support cgi-bin. * A central server can distribute an unlimited number of "packages" * Each package can have an unlimited number of files, defined using wildcardable "include" and "exclude" file descriptors. * Access to specific packages can be limited to authorized users * Packages can be obtained by "remote servers" or by "remote clients" * A remote server can access any number of central servers, and is managed via the net (using HTML forms). * A remote client (that can be run under NetScape on any OS/2 machine) can access any number of central servers, and is managed via the net (using HTML forms, and via a non-GUI Netscape helper application. * GZIP and RSYNC are used to reduce the size of file transfers (both of these "encodings" can be suppressed on a file specific basis) * A package specific installation program can be run on a "remote client" after all the files have been UpDated. * Files can be encrypted before being sent to the client. ---------------------------- II) Installation UpDater is an OS/2 REXX program designed to work either as an addon for the SRE-http web server (http://www.srehttp.org), or as a more-or-less generic cgi-bin script.The "remote client" program (UpClient) does NOT require SRE-http or cgi, but it does require Netscape running under OS/2. UpDater is installed using an installation program named INSTALL.CMD. a) Unzip UPDATER.ZIP to an empty temporary directory. b) From an OS/2 prompt, i) Change your default directory to this empty temporary directory ii) Run INSTALL For example, assuming you unzipped UPDATER.ZIP to E:\TEMP C:>e: E:>cd temp E:\TEMP>install c) Choose if you want to install UPDATER as an SRE-http addon, or as a cgi-bin script Note: if you are an sre-http user, you can choose either method. However, the "addon" mode is a bit quicker, somewhat more stable,m and has a few extra features. d) Answer the questions that appear on your screen. In particular, you'll be asked to provide the name of a UpDater directory. This should be an empty directory (it will be created if it doesn't currently exist). Also, if you are installing as a cgi-bin script, you'll be asked to provide the "cgi-bin prefix". This is what your server uses to indicate "this URI is a cgi-bin script". For example, in http://foo.bar.net/cgi-bin/test-cgi?name=daniel and in http://mysite.org/cgi-bin/updater? the cgi-bin prefix is "cgi-bin". e) EXTRA step for use as cgi-bin script. UpDater requires the rexxlib.dll dynamic link library. The bad news is that Rexxlib is NOT freeware. The good news is that we have an unlimited distribution license for rexxlib, so long as it is used with programs we distribute. Therefore, if you do not have a copy of rexxlib.dll, you should either a) Buy it, it's a useful resource for REXX programmers. For details, see http://www.quercus-sys.com or b) Download it from (for use with UpDater), You can get it from http://www.srehttp.org/pubfiles/updlls.zip After you've obtained REXXLIB.DLL, copy it to somewhere in your LIBPATH; for example, to x:\OS2\DLL (where x: is your boot drive). Note that both the "central server" and "remote server" components of UpDater will be installed (since a single machine can be both a remote server and a central server). In addition, the files necessary to use UpDater as a remote client will be copied to the UpDater directory. After installing UpDater on a central and a remote server (or, on single machine if you want to test it out), you'll need to create a few files. In particular: a) Central servers use "package files" b) Remote servers use a SITELIST.IN file. If you will be using the "remote client" mode, you will also need to c) Install UPCLIENT.CMD as a NetScape helper applicaiton. Package files and SITELIST.IN are described in the next two sections. Installation of UPCLIENT.CMD is described in the appendix. Having completed the preceeding, you can run UpDater. It's easy to do: * if you are maintaining a remote server: a) from your personal workstation, point your browser at this "remote server", and enter /RUPDATER? (in the location field). Note that your "personal workstation" can be the same machine as the remote server. b) You will then be presented with a list of "sites". Choose one. c) You will then be presented with a list of "packages" on this site (where a site is a nickname for a central server). Choose one. d) A short HTML form will be displayed. Choose your options (in particular, the "destination directory"), and submit your answers. e) Watch it go (using a "view status" link). This monitoring is a bit clumsy, due to the use of a daemon to do the work (this daemon is used for several reasons, a primary one being to avoid timeouts on the remote server). * if you are using a remote client (say, to update files on your personal workstation): a) from a remote client (for example, from your personal workstation) point your browser at the "central server", and enter /UPDATER? (in the location field). b) You will then be presented with a list of "packages" on this central server. Choose one. c) The UpClient helper application will be launched. You will answer a few questions (in particular, the destination directory), and watch it go. Notes: * UpClient's user interface is non-gui. * UpClient works a lot better when a few libraries and programs are also installed on the remote client's machine (see appendix 3 for the details). * The various .CMD files (UPDATER.CMD, RUPDATER.CMD, UPCLIENT.CMD, and UPDAEMON.CMD) contain several user-configurable parameters -- see Appendix 2 and Appendix 3 for the details. * If there are large files in a package, or if a "list of files in a package" is large, you may need to increase your GoServe timeout limits. * It is possible for a remote client to talk to a central server directly -- so as to review what's there, and to get one file at a time. To do this, point your browser at the central server, and enter /UPDATER?PACKAGE=!LIST in the location line. * If you need to encrypt files before sending them, please see section V. * The UpDateME program can be run from anywhere -- you will be prompted for a number of parameters (some of whose defaults you can set by editing the user changable parameters in UPDATEME.CMD) * The various UpDater programs write status messages to a PMPRINTF window. To view a PMPRINTF window, get the free PMPRINTF program at http://www2.hursley.ibm.com/goserve ---------------------------- III) Defining Packages on a Central Server "Packages" are defined as "a set of related files". "Packages" are what UpDater distributes. Basically, you can think of a package as a collection of files that one could distribute on a CD rom (or, for the old timers out there, on a handful of floppies). Instead, the files that comprise a package will sit on your central server's hard disk (or cd-rom drive), and be distributed over the internet. To define a package on a central server, you must create a package file. These are fairly simple, "mime-like" files; they must have an extension of .PKG, and they must be in the PACKAGE subdirectory of the UpDater directory (that you defined when you installed UpDater). For example, E:\GOSERVE\UPDATER\PACKAGE\SAMPLE.PKG. A package file contains several entries, with each entry seperated by a blank line. Entries must begin with a "name:" (without the quotes), where name is one of the following: title: -- the title of this package description: -- a longer description of this package privileges: -- required privileges (possibly *) include: -- files to include exclude: -- files to exclude no_encode: -- files for which content-encoding is suppresed cache: -- days to cache a package's "list of files" run: -- an installation program to run after updating encrypt: -- supported encryption methods The following alphabetical list describes these options in greater detail. The only required options are TITLE and INCLUDE. cache: number of days to retain the "list of files" (in this package). To reduce the amount of time spent generating "list of files" from the definitions found in a package, you can specify the number of days to use a "cached" version of this list. For rapidly changing packages this is not a good idea, but for rarely changing packages that contain lots of files the use of cache can reduce processing time (hence speed up throughput). description: description of this package. A longer, typically multi-line, description of this package. encrypt: a space delimited list of supported encryption methods These are encryption methods that the central server is able to apply to the files in this package. exclude: a list of files to exclude A list of files to exclude, with * wildcards. You can use /S to include subdirectories. Files that match include: entry are checked against exclude: entries, and if a match is found, they are not used. include: a list of files in this package. REQUIRED. This is a list of files that comprise this package. Each item in the list should appear on it's own line, and should have two values: a relative destination directory, and a fully qualified file name. The fully qualified file name can include wildcards (* characters). In addition, you can also include a /S modifier at the end, to indicate "include files in subdirectories". Basically, UpDater will copy the files on the central server (as identified by these filenames) to a relative (to a base-directory) directory on a remote server. no_encode: a list of files to NOT content-encode A list of files, using the same syntax as exclude. Files in no_encode will not be GZIP'ed or RSYNC'ed. This is especially useful when you have .ZIP files, or other compressed files -- since further compression rarely helps, and differences are often difficult to detect. privileges: sre-http privileges required to obtain this package. OPTIONAL. These are user-specific privileges that are set in USERS.IN (or ATTRIBS.SRE). Note that these are in addition to any privileges that SRE-http may require on a selector specific basis. In other words, you might have a general set of privileges that permit access to the UpDater addon, and a variety of specific privileges required for each of several packages. Note: If you do not want to limite access, you should NOT specify a privileges: entry. run: execute an installation program on a remote client You can specify a program to run after UpClient has completed. This will only be run if the client permits it -- which requires BOTH checking a box in a form, and answering Yes to a "do you want to run" question asked by UpClient. The syntax of run: is ... run: [relative_path]program [PM] [arg=a sting] where [relative_path] -- the path (relative to the destination directory on the client's computer) that contains the program to run. For example /cmds/foo.cmd means "the foo.cmd program in the destdir\cmds directory" If you do NOT specify relative_path, the program must be in the remote computer's PATH. [pm] -- If PM is specified, then this is a presentation manager program. If you do NOT specify PM wit presentation manager programs, an error will occur. For .CMD files, if you specify PM, then the child process (launched when the command file is run) will NOT be automatically closed. [arg=] -- If arg= is specified, then you can specify a string (immediatly after the = sign) to be fed to this program. Note if you specify PM amd arg=, the PM must preceed the arg= title: the title of the package. REQUIRED. This should be a fairly short (50 character) name for this package. It will be used by "remote servers" when communicating with clients. Notes: * lines that begin with a semi-colon (;) are comments, and are ignored. * / and \ can be used interchangeably. Leading and trailing \ (and /) are ignored. * entries are case insensitive * You can use decimal fractions when specifying cache. For example, cache: 1.5 means 36 hours. * Not specifying a cache: entry, or using cache: 0, mean "do not cache". * Cache validity (whether it's too old) is determined by looking at the datestamp of the "list of files", and NOT at the datestamp of every file mentioned in this "list of files". * Cached versions of "file lists" are stored in the PACKAGE subdirectory of the central server's updater directory. They will have an extension of .PKC. * Examples of run: run: /install.cmd run: epm PM READ.ME run: /bin/myinstall PM in\params.1 out\out.2 Note that since EPM is NOT preceeded by a /, it will be run from the client's machine PATH. ---------------------- IIIa) Example of a package file The following is an example of a package (.PKG) file (say, SREHTTP.PKG): ;---------------------------------- ; example of a package file title: The sre-http files description: This is the sre-http distribution files, including a set of addons. privileges: * include: \ f:\srehttp\* \lib f:\srehttp\srf\* \addons\updater f:\srehttp\addons\updater\* /S \newstuff e:\work\dec99\* \htmls g:\users\*.htm* /s exclude: f:\srehttp\*.ZIP f:\addons\*.FAQ /S no_encode: f:\newstuf\*.zip /S f:\srehttp\*.GZ cache: 0.5 run: /install.cmd encrypt: blowfish des ; end of example ;---------------------------------- Notes: * leading and trailing \ (or /) in the "relative directory" entries are optional * a * is used as a wildcard. It can be used anywhere in a filename (but not in a directory name). * a /s after a wildcard filename means "include/exclude/no_encode files in subdirectories" Thus, if the remote server is using E:\SRE as the "base directory" for this package, the following subdirectories will be created: E:\SRE -- will contain F:\SREHTTP\* but excluding files that end with .ZIP. E:\SRE\LIB -- will contain F:\SREHTTP\SRF\*, but excluding files that end with .ZIP E:\SRE\ADDONS\UPDATER -- will contain F:\SREHTTP\ADDONS\UPDATER\* and it's subdirectories, but excluding files that end with .FAQ E:\SRE\NEWSTUFF --- will contain E:\WP\DEC99\*.* E:\SRE\HTMLS -- will contain G:\USES\*.HTM* (and files in subdirectories that match *.HTM*) Also, an SREHTTP.PKC "cache" file will be created if it doesn't exist. If it does exist, and is less then 12 hours old, it will be used (the central server will not regenerate a list of files in F:\SREHTTP, etc.) ---------------------------- IV) Defining a Sitelist on a Remote Server A remote server can obtain packages from many different central servers. In order to specify what central servers should be monitored, a list of accessible "sites" is required. Note that this requirement for a "list of accessible sites" is largely for security reasons -- it limits the damage that an unauthorized user of UpDater can cause. This list is specfied a SITELIST.IN file, that is stored in the SITES subdirectory of a remote servers's UpDater directory. For example: E:\GOSERVE\UPDATER\SITES\SITELIST.IN. SITELIST.IN file contains "mime-like" records, with each record containing several single-line entries, and with records seperated by blank lines. Entries have the following structure: name: value where name may be one of address, description, password, root, site, or username. In greater detail: address: Required The IP address (name or numeric) of a server running the central component of updater. Example: address: www.srehttp.org description: Site title. A one line description of the site. Example: Description: This is the base version of SRE-http. key: a "shared-secret" -- only supported when run as an SRE-http addon. Key is used to specify a "shared-secret". This shared-secret is used as a key for decrypting files from this site, assuming the file has been encrypted due to an Encrypt: option in the package file. Key is case-insensitive (it will be capitalized), and should NOT contain embedded spaces (only the first word will be used). Example: key: wa752a99 If key is used, password and username must also be specified -- since the server uses the "user specific secret privilege" to determine what key to use when encrypting. Of course, the key entered in key:, and the key extracted by the server from a secret privilege, MUST be identical! password: Optional -- and only supported when run as a SRE-http addon. A password to send to the site. Use this with Username: Note that the username/password will be sent to the central server (at the address:) -- it is NOT used to control access to this remote server. Example: username: joe password: schalbotnik root: required A space delimited list of fully qualified directories. These are directories on this (remote) server to which files from a package can be copied to. You can specify one, or many, directories -- when it's time to download a package, you'll specify a "destination directory" that must be under one of these "root" directories. This is a security option -- it prevents hackers from writing junk all over your machine (or, worse, causing files to be deleted). We recommend that these "root directories" be dedicated to the storage of these "packages", or that you exercise caution in who you grant UpDater access to. Example: root: G:\STUFF F:\GOODJUNK site: Required. A one word "nickname" for the site that is running the central component of UpDater (as identified by the address:). Example: site: SRE-http_home username: Optional and only supported when run as a SRE-http addon. A username to send to the site The username and password are sent to the site as part of a "basic authorization" header. The username and password can be used by a central server to limit access to selected packages. Note that a client who invokes Rupdater may also have her own username and password which is specific to the "remote" site (that rupdater is running on). This "remote-site access" username and password is NOT sent to the central site. In other words, Rupdater has two levels of access controls: a) For a client to use Rupdater, she must have access privileges defined for her on the remote site b) For the remote site to talk to a central site, it must have access privileges defined for it on the central site, and an appropriate username and password defined in SITELIST.IN. Thus, a "superuser" of both the remote and central site might NOT be able to use Rupdater to instruct the remote site to obtain files from the central site -- it depends on whether an appropriate sitelist.in entry is specified (an entry that contains a working username and password). Example: username: joe password: schalbotnik -------------------- IVa) Example of SITELIST.IN Example of a SITELIST.IN file ;======================================== ; This is a sample sitelist.in file for UpDater site: srehttp address: www.srehttp.org name: srehttp description: The SRE-http home page root: e:\srehttp f:\srehttp site: cool_stuff address: www.invisible.net description: Wonderful stuff,but not for everyone username: mutwo password: xy321 root: g:\wonders\set1 ;======================================== -------------------- V) Using Encryption When used as an SRE-http addon, UpDater can encrypt the contents of the files before sending them over the net. To do this, you need to: a) specify the supported encryption methods in the package file b) specify an ?ENCRYPT secret privilege for appropriate users c) install encryption/decryption software on the central server and on the remote client (or server). Step a is accomplished by using an ENCRYPT: option in a package file (see section III for the details). You must also specify a Privileges: option. Step b is accomplished by modifying user specific privileges (say, by modifying the USERS.IN file). The details of "secret privileges" are discussed in the SRE-http manual (SREHTTP.HTM), and in ENCRYPT.DOC. Basically, for users who are granted access to this UpDater package, you should add "privileges" of the form: ?UPDATER:a_shared_SECRET where a_shared_secret is the "shared-secret". Note that a_shared_secret can be as many characters as you wish, it must NOT contain spaces, and it will be capitalized. Step C can be accomplished in one of two ways: a) using "encryption" and "decryption programs. b) using encryption and decyption procedures a) Using programs. You should install encryption (and decryption) software somewhere in the PATH of the central (and remote) computers-- for example, in the x:\OS2\APPS directory (where x: is the boot drive). To tell UpDater how to use this software you must create an ENTRY. "stem" variable in UPDATER.CMD (the central server proram), and in UPCLIENT.CMD (the remote client program) and/or UPDAEMON.CMD (the remote server program). ENCRYPT. variables should have the following format: ENCRYPT.!METHOD=progname opt1 %in opt2 %out opt3 "%key" ... where: METHOD - the "method" (case-insensitive). progname - the program name opt1 .. - options understood by the program %in - a placeholder for the input file %out - a placeholder for the output file %key - a placeholder for the "key". This will replaced by the xxx in the ?UPDATER:xxx shared-secret. Examples: encrypt.!blowfish=e:\apps\bf -q -e %in %out "%key" could be used by the UPDATER.CMD to "encrypt", and encrypt.!blowfish=g:\bf -q -i %in %out "%key" could be used by UPCLIENT.CMD to "decrypt". b) Using a procedure. If your encryption/decryption is via a rexx-callable procedure (say, in a DLL you load seperately, or in an external file), use: ENCRYPT.!METHOD=proc_name(arg1,%out,arg2,%in) where the ( must IMMEDIATELY follow the proc_name (no intervening spaces allowed). Otherwise, use arguments exactly as you would if you were calling this procedure from your own rexx program. For example: ENCRYPT.!MYWAY=encfast(%in,%out,%key,"q") Notes: * UpDater will replace %in and %out with the approriate filenames. * You can freely mix %in, %out, and the options & arguments. * Be sure to put an ! before the METHOD. * you can use the UPDATER?SEND= option as a means of distributing decryption software. * Due to the lack of a standard method of assigning usernames and passwords, the CGI-BIN version of UpDater does NOT support encryption. * BLOWFISH (available at hobbes.nmsu.edu) is a very effective encryption/decryption tool. * Do NOT use ' characters when defining an ENCRYPT.! stem variable. You CAN use " characters instead. If ' is necessary, use ''. ---------------------------- APPENDICES ---------------------------- Appendix 1) Files in UPDATER.ZIP rxzlib.dll -- A dynamic link library used to gzip (requires EMX) install.cmd -- The installation program. rupdater.cmd -- The "client interface" portion of the remote server component rxrsync.dll -- A dynamic link library containing several useful procedures. sample.pkg -- A sample "package" file sitelist.smp -- A sample SITELIST.IN file updater.doc -- This documentation file updater.cmd -- The "central server" component updaemon.cmd -- The "daemon" portion of the "remote server" component updmncgi.cmd --The "daemon" portion of the "remote server" component, cgi-bin variant. upclient.cmd -- The Netscape helper app (that is used as the remote client). updateme.cmd -- Stand-alone "client-side" updater upd-cgi.cmd -- Base files used to create cgi-bin version of updater.cmd rup-cgi.cmd -- Base files used to create cgi-bin version of rupdater.cmd instcgi.cmd -- Install updater as a cgi-bin script (it is called by install.cmd) ---------------------------- Appendix 2) User configurable parameters The several .CMD files contain similar user configurable parameters. For the details, open up the files and look for the "user configurable options" section at the top of the file. The following describes the more important parametes: UPDATER_DIR: The UpDater directory (RUPDATER.CMD and UPDATER.CMD) This is where UpDater stores various files, and where UpDaemon.CMD is installed. Note that UPDATER.CMD and RUPDATER.CMD are installed to your GoServe addon directory. Examples: updater_dir='F:\UPDATER' updater_dir='' Notes * the second example means "use the UPDATER directory of the GoServe working directory. * UPDATER_DIR (in RUPDATER.CMD and UPDATER.CMD) is set by the UpDater installation program. ADDCRC: Compute an Adler-32 value when constructing a package (UPDATER.CMD) The CRC-32 value is used by UpDater's "only update if contents have changed" option. If you do NOT enable ADDCRC, then this option will not work. Set ADDCRC=1 to enable, ADDCRC=0 to disable. Note that computation of an Adler-32 takes a bit of time, but not much (typically, less then a second per Mb of file). Example: addcrc=1 ALWAYS_USE_CACHE: Always use cache entry if it's current (UPDATE.CMD). In a few places, you can "request the latest version" (this causes a &nocache=1 option to be added to a request). If you do NOT want to allow clients to suppress the use of cached "list of files in a package", set ALWAYS_USE_CACHE=1. If you do want to allow this, set ALWAYS_USE_CACHE=0. BACK1: The "body background" (RUPDATER.CMD and UPDATER.CMD) BACK1 is used to set the background color (or image) used when UpDater communicates with client. Examples: back1='background="/imgs/web2.gif"' back1='bgcolor="#aa9933"' CanDoGZIP: Enable GZIP encoding (UPDATER.CMD, UPDAEMON.CMD, UPCLIENT.CMD and RUPDATER.CMD) Set this to 1 to enable GZIP encoding, 0 to disable. GZIP encoding takes a bit of CPU time, but can substantially reduce the size of a file (hence saving bandwidth). Note that, as currently built, the rxZlib library requires EMX 0.9d (emxrev should report 60 or above). You can get emx from http://hobbes.nmsu.edu (search for emxrt.zip). Example: candogzip=1 CanDoRsync: Enable RSync differencing (UPDATER.CMD, UPDAEMON.CMD, UPCLIENT.CMD, and RUPDATER.CMD) Set this to 1 to enable RSync differencing, 0 to disable. RSync differencing takes a bit of CPU time, but can substantially reduce the size of a file (hence saving bandwidth). Note that some OS/2 servers with minimal implementations of CGI-BIN may not be able to support Rsync. In particular, the server must be able to write non-standard request headers to the environment, and these request headers may be as long as 500 characters. Example: candorsync=1 Timeout: number of seconds to wait between socket calls (UPCLIENT.CMD and UPDAEMON.CMD) If a socket call takes longer then timeout seconds, then UpDater will give up on this file. Large values may be needed over very slow lines. Note that Timeout is NOT the total time required to get a file, it's the max seconds for each 1000 byte "sockrecv" call. timeout=30 ---------------------------- Appendix 3) Installing UpClient.CMD as a Netscape helper application. The UpClient helper application is a REXX program (UPCLIENT.CMD) that will update a package of files to your computer. To use UPCLIENT.CMD, you must be running NetScape under OS/2. Assuming this describes you, after downloading UpClient.CMD you should 1.Copy UPCLIENT.CMD to a directory in your PATH (i.e.; to x:\OS2\APPS, where x: is your boot drive) 2.Tell NetScape to use UPCLIENT.CMD with all application/x-updater responses To do this, you should: a) Open up the Preferences menu in Netscape (in NS 4.6, Edit-Preferences) b) Select Applications c) Select the New Type option d) Fill in the fields of the New Type screen with: i) Description of type: UpDater Client ii) File Extension: UPD iii) MIME Type: application/x-updater iv) Application to use: "cmd.exe /c "x:\os2\apps\upclient.cmd" be sure to include the " characters You can improve UpClient's efficiency by making sure the following programs are installed on the remote client's machine: a) RXRSYNC.DLL -- it should be in the remote client's LIBPATH b) RXZLIB.DLL -- it should be in the remote client's PATH. Note that RXZLIB.DLL requires emx rev 0.9d (emxrev should report 60 or above). EMX can be found at http://hobbes.nmsu.edu (search for EMXRT.ZIP). Note that the INSTALL program will setup your central server with a page that describes how to install UpClient, and which contains links to UPCLIENT.CMD, RXRSYNC.DLL,and RXZLIB.DLL. To see this page, just hit the central server with a UPDATER? request, select any package, and select the "using the UpClient helper app" link. Or, use a request of updater?action=UPCLIENTDOC. Once UpClient is installed, when you update a package from an UpDater central server Netscape will run UpClient in a seperate window. After asking a few questions, UpClient will use tcp/ip socket calls to GET files and write them to your hard drive (and possibly run an installation program after all the files have been retrieved). For details on rxRsync and rxZlib, see http://www.srehttp.org/apps/rxrsync/ and http://www.srehttp.org/apps/rxzlib/ ---------------------------- Appendix 4) The UpDater API UpDater has a built in, web-aware, user interface. Primarily based on html forms, this provides an effective, though not particularly fancy, means of updating files. Ambitious users may wish to tinker with this interface (say, by embedding the commands in a customized page), or may want to use the UpDater capabilities for more refined purposes. To assist in such endeavours, the following lists the "options" understood by UPDATER.CMD, which is the "central server component" of UpDater. The remote components (RUPDATER.CMD, UPCLIENT.CMD,and UPDAEMON.CMD) will not be discussed -- the notion is that you'll be replacing these with your own tools! As a "server side program" (running as a CGI-BIN script or as an SRE-http addon, one communicates with UPDATER.CMD through bu using web requests that contain options. Although there are several ways of defining options (such as through html forms), for illustrative purposes this documentation will assume that options appear after a request string. For example, in the URI http://foo.bar.net/updater?action=display&package=craziness the "request string" is updater?action=display&package=craziness the "resource" is updater and the options follow the ?, with options seperated by & signs. Thus, there are two options, action=display package=craziness Note that options typically have a "varname" and a "value". For example varname=action, value=display 4.a) UPDATER.CMD options The following is an alphabetical list of options recognized by UPDATER.CMD The most important options are: action, file, entry, package, and send. Note that options are used in requests of the form: updater?opt1=val1&opt2=val2&... action: An action to take. Recognized values are: display: Display a page describing the package that is identified by the package option. file1: Get a file from this package, and return it. Also requires that an entry option be defined. The file will be returned using one of the standard mime type (such as text/plain), or using a mime type of application/octet-stream (if the file's extension does not map to a common mimetype). file: Get a file from this package, and return it (possibly after GZIP compression and RSYNC differencing). Also requires that entry option be defined. The file will be returned with a mimetype of application/x-updater. Ax x-updater response header is added, that may contain several values (described below). getl: Return a list of all the files in the package. This list is described below. An x-updater response header is also returned. You can examine this for "GZIP" and "RSYNC" entries -- they signify that the server can handle GZIP and RSYNC. getl2: Return an html form used by a client to specify where/how to update. Upon submitting this form, UpClient will be called and will be passed a list of files in the package. upclientdoc: display a page that describes the UpClient helper app. ashtml: When an ashtml=1 option is specified in combination with a action=GETL, then the "list of files" will be returned as an html page containing "action=file1" links to the various files in the package. In addition, when ashtml=1 is specified with a package=!list option, then the "list of packages" will be returned as an html page containing "action=display" links to the various packages on the server. client: Used with action=getl. When client=1, this request is from a "remote" client. This has two effects: i) updater will check for some options (destination directory and possibly username/password), and return an error message if they have not been specified. ii) if ashtm<>1, some extra information will be added to the "list of files" header (see below for the details) encrypt: only supported in SRE-http addon version. A comma delimited list of encryption methods that the client supports. This should be a subset of the server supported methods (as specified in the ENCRYPT option in the "list of files", described in section 4.b.ii). Note: the capitalized shared-secret used by the central-server to encrypt the file is pulled from an ?UPDATER:shared_secret "secret privilege". It is your responsibility to know what this value is (UpDater does NOT use public-key codes or other SSL-like mechanisms). entry: The "entry number" in the package. The entry number is basically the index into the list of "root directory\ies" specified in the .PKG file of the specified package. It is combined with a file option to define a particular file. Example: entry=1 file: A filename; which may contain subdirectory information. The actual file this refers to will depend on the value of the entry (and package) options -- the filename is relative to the subdirectory identified bye the entry option. Examples: file=foo.bar foo=/programs/math/calc.exe getuser: When getuser=1, and client=1, and action=GETL, then updater will check that a username and pwd options have been specified. gzip If gzip=1, then "use gzip, if possible, to compress the response" md4: A 32 hex character MD4 hash of an old version of a requested file. If the md4 of the requested file matches this md4 value, then a "304 not modified" response will be generated (for the CGI-BIN version of md4, a nocache: If nocache=1, and action=getl, then do NOT return a cached version of the "list of files". This is ignored if ALWAYS_USE_CACHE=1 is set in user configurable parameters section of UPDATER.CMD. package: Specifies the name of a package on the central server. Packages are specified in .PKG files in \PACKAGE subdirectory of the UPDATER directory. See section III (above) for details on package files This is an important option, almost all the other options refer to actions to take using this package. Example: package=sample package=my_package package=!list Note: do NOT included the ,PKG in the value The last example, value=!list, is used to return "list of packages". See Appendix 4b for the details. rsync: If rsync=1, then "use rsync, if possible, to shorten the response". This MUST be combined with an "rsync-signature_updater" request header that contains the "rsync synopsis" of a "prior version" of the requested file. send: Transfer a file in (or under) the updater directory. This is currently used by the UpClient helper app page to allow clients to obtain the upclient.cmd (and other files). The syntax is: UPDATER.DOC, RXZLIB.DLL, GZIP.EXE, RXRSYNC.DLL, UPCLIENT.CMD To add files to this list, you should change the SEND_FILES parameter in UPDATER>CMD. Examples: send=upclient.cmd send=morefile\help.doc Note: SEND could have been specified using "action=SEND&file=filename". However, since these files (in the updater directory) are NOT identified with a "package" or "entry" option, we decided to use a seperate option. text: When text=1, and a send option is specified, then transfer the file as a text/plain mimetype. backupdir dodelete check quiet dorun: These options are used when action=getl and client=1 (and asthml<>1). They control what extra "remote client" specific information is added to the "list of files header". For example, UpClient uses the values: backupdir: the relative location of a backup directory dodelete: 1 (used to signal "delete all orphaned files from destination directory" check: 0,1,2 or 3. The time of "check" to do before requesting the file. 0=none, 1=time, 2=crc32, 3=add an md4 option quiet: 1 (non-verbose reporting in UpClient) dorun: Name of an "installation" program 4.b)The "list of files" in a package. When not used to directly control the process of updating files, UPDATER.CMD returns several types of responses. These are: * list of packages * a "list of files" in a package * a file, that may have been GZIP'ed and RSYNC'ed. The following describes these types of response. 4.b.i) List of packages. When package=!list, then a "list of packages" is returned. You can then use this list to choose a package for use in later requests to UPDATER.CMD. The format of this list is: n yyyy/mm/dd/hr/min Name1 title 1 ... yyyy/mm/dd/hr/min NameNthg title nth where: yyyy/mm/nn/dd/hh = timestamp (last revision of package file) name1 ... nameNth = name of the package title = a short, mutli word, description of the package Example: 4 1999/12/10/02/29 sample This is a sample package file 1999/12/10/00/17 SREHTTP The SRE-http package 1999/11/23/21/14 ZIPFIP The ZIPFIP package Note that the timestamp refers to the .PKG file, and NOT to files that may be referenced by the .PKG file. 4.b.ii) "List of files" in a package. The list of files in a package contains information on what files comprise a package. The format is: header_line yyyy/mm/dd/hr/min entry File1 relDir no_encode crc32 where: yyyy/mm/nn/dd/hh = timestamp (last revision of package file) entry = entry number (index into the .PKG file) file1 ... fileN = name of file in this package relDir = relative directory to copy file to This is typically relative to a root directory that you provide no_encode = Hint to not attempt to encode this file (RSYNC and GZIP should not be attempted) crc32 = Adler-32 checksum (can be used to see if file has changed) Note that an md4 checksum is NOT send as part of this "list of files" The header line has the format: var=value var1=value var2=value That is, each var=value pair is seperated by a space. Recognized variables are: package = required -- the package this file belongs to server = requird -- ip address that this package is on selector = recommended -- the "resource", on this server, that will process UpDater requests. GZIP = optional. If GZIP=1, then the server knows GZIP RSYNC = optional. If RSYNC=1, then the server knows RSYNC CREATED = Timestamp, when this "list of packages" was created. Format is basedate.decimal_fraction_of_a_day PKG_DATE = Timestamp, when this .PKG file was last modified. CURRENT_TIME = Timestamp, when this "list of files" was sent. Note that CURRENT_TIME is ONLY used when this is a cached "list of files". CURRENT_TIME will always be greater then or equal to CREATED. ENCRYPT = A comma delimited list of encryption methods supported by this server (for this package). Example: Package=sample Server=localhost CREATED=730101.92986 PKG_DATE=730097.1035 GZIP=1 RSYNC=1 CURRENT_TIME=730102.02361 ENCRYPT=BLOWFISH,DES SELECTOR=UPDATER 1999/07/20/20/56 1 /ACCESCFG.CMD / 0 652FD5DA 1999/11/11/14/33 1 /ACCESCHK.RXX / 0 A2D10DAA 1999/06/19/10/30 1 /access.cnt / 0 45DD05E8 1999/06/13/16/14 1 /access.in / 0 A69D859E Note that the "header line" should be on one long line (in the above example, it is split for readability sake). 4.b.iii) A file When action=file, then a file will be returned. This file may have been GZIP'ed and RSYNCed, and it may have been encrypted. This return will have the following request headers: Content-type: application/x-updater X-updater: alist It may also have, Content-encoding: alist X-NotMod: 1 X-updater: Xupdater takes a comma delimited list of case-insensitive options. The important options are: DIR=/rel_dir MD4=md4_value ERROR where rel_dir is a "relative directory": it can be used instead of the "relDIR" specified in the list of files. md4_value is a 32 hex character md4 value. It is only included when an RSYNC has been used. As an error check, this value should be compared against the MD4 value of reconstructed file. ERROR takes no value -- it signals that "an error occurred" Examples: X-updater: dir=/sub1 X-updater: dir=/home/particles,md4=3234a67abdf2f456ca90155156789012 X-updater: dir=/secure/stuff1,encryt=blowfish Content-encoding: Content-encoding takes a comma delimited list of case-insensitive "content-encodings". UpDater supports three kinds of content-encodings: GZIP, RSYNC, and ENCRYPT=method. Note that the order of appearance of these content-encodings is the reverse order of their application --- you should do the last one first. For example, Content-encoding:rsync,gzip,encrypt=method1 means ENCRYPT=method1 -- first deencrypt, using the encryption "method1" GZIP -- then inflate (de-compress) it RSYNC -- then apply this "GDIFF" difference file to the old file Examples: Content-encoding: gzip Content-encoding: rsync,gzip Content-encoding: gzip,Encrypt=blowfish appears, a and b might be one of GZIP and RSYNC -- which signifies that the response has been GZIPped, and that it is a GDIFF "difference file" that should be combined with the "old file" (from which the rsync-signature was generated) to create the new file. X-notmod: X-notmod, which will always have a value of 1, is only used when to signal "the file has not changed" (i.e.; an md4 comparison by the server show that the contents of the file have not changed). X-notmod is only used when running as a cgi-bin script. When running as a SRE-http addon, a 304 return code is used to signal that "the file has not changed". ---------------------------- Appendix 5) Disclaimer and terms of use: The various components of UpDater are freeware that is to be used at your own risk -- the author and any potentially affiliated institutions disclaim all responsibilties for any consequence arising from the use, misuse, or abuse of this software (or pieces of this software). You may use this (or subsets of this) program as you see fit, including for commercial purposes; so long as proper attribution is made, and so long as such use does not in any way preclude others from making use of this code.