5 April 1999: The PRELOAD addon for SRE-http


The PRELOAD addon for SRE-http is used to send multiple requests
to HTTP servers. It's major purpose is to "preload" requests to
your server, thereby generating a useful GoServe/SRE-http cache.
PRELOAD. Best of all, this "preloading" is sensitive to server load --
it will suspend activity when there are active clients.

PRELOAD is designed to work as an "custom initialization procedure",
hence it requires SRE-http ver 1.2m  or above.
Given that you have (or will now obtain) an up-to-date 
version of SRE-http, installation is simple:

1) Copy PRELOAD.CMD and PRELOAD.RXX to your GoServe working directory.
   Note that you should NOT copy these files to the ADDON directory --
   copy them to the "main" GoServe working directory.
2) Using your favorite text editor, or using the EDITSREF.CMD utility,
   modify the CUSTOM_INITS parameter in INIT_STA.80 (or INITFILT.80,
   if you are using a version of SRE-http prior to 1.3d)
3) Using your favorite text editor, modify the PRELOAD.n "stem variables"
   in PRELOAD.RXX.
4) Restart GoServe/SRE-http.  
5) Optional: copy RELOAD.CMD to the SRE-http ADDON directory.

------------------------------------------------------------------------
The following sections describe PRELOAD in greater detail.

        * Modifying CUSTOM_INITS
        * Modifying PRELOAD.n stem variables
        * PRELOAD.n Special Features
            i) USERNAME and PASSWORD
           ii) Wildcards
          iii) Search and load URLS found in a document
        * Optimization Hint: Suppressing SRE-http auditing 
        * Running PRELOAD against a selected IP address
        * RELOADing the PRELOAD requests.
        * PRELOAD runs as a thread
        * Monitoring PRELOAD
        * An Alternative

                -----------------------------------------------


******* Modifying CUSTOM_INITS

  All you need to do is add ' PRELOAD ' to the CUSTOM_INITS variable.
  For example, you can set CUSTOM_INITS='PRELOAD '. 
  For further discussion of CUSTOM_INITS, see INITFILT.DOC


******** Modifying the PRELOAD.n stem variables

   The PRELOAD.n stem variables are set at the top of the PRELOAD.RXX
   file.

   Each of these variables should contain an http: style of request.  
   For example:
             preload.1='http://foobar.net/dir1/stuff.html'
   If you do not include the http://x.y.z/ portion of the address,
   then the "default server" will be used. For example, if goserve/srehttp
   is running on mywww.banjo.org, then
            preload.2='songs/wva.html'
   would be equivalent to
            preload.2='http://mywww.banjo.org/songs/wva.html'
 
   You must also specify the number of "entries" in a preload.0 statement.

Example:
   preload.0=5
   preload.1='index.html'
   preload.2='imgs/'
   preload.3='imgs/bigpic.gif'
   preload.4='docs/describe.txt'
   preload.5='docs/client1.html , master  as43 '

***** PRELOAD.n Special Features

 i) USERNAME and PASSWORD

    The full form of a preload.n statement is:
        preload.n='http://x.y.z/selector , username password '
    Example:
        preload.5='docs/client1.html , master  as43 '
    would add a "basic authorization" username=master and password=as43
    to the request for docs/client.html.
    Note that the username password must be preceded by a
     " , "  -- a comma with a least 1 trailing and leading space

  ii)Wildcards

    You can use * wildcards to create multiple requests (one for
    each file that matches a *'ed file-pattern.  To do this,
    instead of the URL, you must specify a "file-pattern" and a 
    "selector-pattern".
    The file pattern should be a fully qualified filename that must contain
    one * (wildcard character), it is used to generate a list of matching
    files. These matching files are then converted into request
    selectors, using the selector-pattern.
    Example:
      If:
         D:\WWW\SAMPLES contains FILE1.HTM and FILE5.HTM
      Then
         PRELOAD.6='D:\WWW\SAMPLES\*.HTM  SAMPLES/*.HTM '
      Would generate the following requests
         SAMPLES/FILE1.HTM
         SAMPLES/FILE5.HTM


 iii)Search a document for links

     In addition to requesting a document, you can also request all "local"
     <A HREF= and  <IMG SRC= URLS referenced in the document.

     To do this, preface a PRELOAD.n entry with #LINKS.  For example:
          preload.1=' #LINKS /index.html '

     Notes:
          * be sure that you do NOT use #LINKS with non-html documents.
          * #LINKS will NOT work with  wildcard (*) entries.
          * You MAY include a " , username password "
          * #LINKS is NOT recursive -- it doesn't "search links for links"
          * To load "non-local" urls (urls with an http:// in
            them), use #LINKS_ALL instead of #LINKS.



***** An Optimization Hint: Suppresion of SRE-http auditing

   To suppress SRE-http auditing, and thereby speed up preloading,
   you should set NO_RECORD=1 (at the top of PRELOAD.RXX). 
   When NO_RECORD=1, a !NORECORD will be appended to all "local" URLS.

   Alternatively, you can explicitly include a !NORECORD in your 
   PRELOAD.n entries. For example:
         PRELOAD.2=' !NORECORD/IMGS/ '
   or
         PRELOAD.6='D:\WWW\SAMPLES\*.HTM  !NORECORD/SAMPLES/*.HTM '

   Note that you can NOT include a !NORECORD in #LINKS entries. If you have
   #LINKS entries, you should set NO_RECORD=1


****  Running PRELOAD against a selected IP address

   If you support multiple hosts, you can select which IP address to use
   as the default (useful if you intend to use *'ed entries).  To do this,
   just enter the IP address after the PRELOAD in CUSTOM_INITS. For example:
        custom_inits='PRELOAD FOO2.BAR.ORG '


***** RELOADing the PRELOAD requests.
 
   Since entries in the various SRE-http caches expire after a few days, 
   it is occasionally useful to "reload" the entries in the PRELOAD.n
   stem variables.  To do this, the RELOAD.CMD SRE-http
   addon can be used -- just invoke it as an "addon". 

   For example, you can issue the following from a web browser:
          http://www.mine.org/reload?
   (where www.mine.org is your servername).

   You'll get a simple confirmation message back, and PRELOAD.RXX
   will be invoked (as with PRELOAD.CMD, intermediate results are
   written to the PMPRINTF window).

   If you want to "reload against a specified default server", just
   include this servername after the ?.  
    For example:
       http://www.mine.org/reload?www2.mine.org

      
***** PRELOAD runs as a daemon

   PRELOAD works by having the "custom initialization procedure" PRELOAD.CMD
   launch PRELOAD.RXX as a thread.  PRELOAD.RXX then monitors the state
   of the server.  When it sees that there are no active clients, it will
   process one of the "selectors" specified by the next PRELOAD.n variables (or,
   one of the "selectors" implicitily specified by a wildcarded
   PRELOAD.n variable).  This monitoring occurs before each "request" -- so
   if a client should heavily hit your server whilst PRELOAD is working, 
   PRELOAD will finish the current request, and then wait.

*****  Monitoring PRELOAD

   The PMPRINTF window will monitor PRELOAD's actions. You might want to
   check it, with an eye towards errors (i.e.; 404  responses that
   indicate you misspecified a URL).