Outline of SRE-http, ver 1.3h

Since SRE-http has numerous options, many of which have similar effects, at times it can get confusing!

This document outlines the steps followed by SRE-http when a new request is received. The goal is to provide a general outline, without burdening the reader with the details of how it actually works. That is, some familiarity with the various SRE-http parameters is assumed, or a willingness to check the main documentation or the description of the initialization parameters !

Notes:

• The SREPROXY variant modifies this logic slightly (see SREPROXY.DOC for details on how SREPROXY works).
• Note that VARIABLE_NAMES refer to variables contained in INITFILT.80 or INIT_STA.80.
• For simplicity sake, we abstract from the intialization of SRE-http (which occurs on the first call to SRE-http).

  Note that you can use CUSTOM_INITS to execute custom initialization procedures

• SRE-http will write errors and warnings to its error log file (it's usually named \GOSERVE\DATA\ERROR.LOG).
• The SRE-http event monitor can be used to check for flags set by external processes (or by the administrator), and respond accordingly.

The outline

Multiple hosts

1. A HOST: request header is read (if it exists)
2. The IP address of the client is extracted.
3. Using the HOST: information and the IP address, the HOSTS. stem variables are searched for a match.
4. If one is found, the appropriate host nickname is returned (and used throughout SRE-http). In general, host-specific values for the various SRE-http parameters are used if available, otherwise default (non-host-specific) values are used.

Preliminary checks

1. The LOAD_THRESHOLD and BACUPSERVERLIST variables are examined to see if this request should be forwarded to another server. Note that the simple load balancer shipped with SRE-http is easily upgraded (typically, all you need to do is replace the BACKSRVR.SRF file).
2. If PRE_FILTER="YES", call the (several) pre-filter(s) (listed in the PREFILTER_NAME variable).
3. Since to that GoServe's caching, or SREPROXY, or a pre-filter did not resolve this request. If one of these did respond, then goto the "post -filter step" (see below).
4. Check the list PUBLIC_URLS (as read from the PUBURLS.IN file and the realm definitions file). If the requested selector appears on this list (perhaps by matching a wildcarded entry) then:
   • Check if it's a literal public_url.
     If so, then return a file in the data directory, or use an exact name contained in the public_url entry.
     If it's also a literal_norecord public_url, then exit.
     Otherwise, goto the "post-filter step".
   • If it is a public_url, but not a literal public_url;
     lookup dynamic client privileges,
     skipto the "alias check" step (skip the logon checks).

Logon and access controls

Logon

1. If the client is an OWNER or has an IP address that matches an INHOUSEIPS., skip the logons (grant her entry to the site).
2. If the client is an UNALLOWEDIPS, deny entry.
3. If DNS_CHECK='YES', check for the domain name server for the existence of a matching IP "name".
4. Otherwise, if CHECKLOG is on, look up (or first ask for) the username/password in the USER_FILE.

   ..but, first check the LOGON_LIMIT

5. If LOGON_LIMIT is violated, or an incorrect username/password is entered, SRE-http can either,
   Re-request username/password, (site logon authorization responses use THE_REALM as the "realm name").
   Send the LOGON_FAIL_FILE

Determining client-privileges

1. If the client is an OWNER, grant her SUPERUSER "client privileges"
2. If the client is an INHOUSEIPS., grant her any privilege mentioned on her INHOUSEIPS. entry, and the privileges contained in INHOUSE_PRIVS.
3. Based on the client's username/password (if provided), client-privileges are read from the USER_FILE
4. Everyone gets PUBLIC_PRIVS;
5. If CHECK_ADD_PRIVS is on, the "dynamic" privileges cache is checked for client-specific transient privileges.
6. If ADD_USER_NAME is on, add the "username" to the client-privileges list.

Determining Access Rights using Resource-Privileges

1. Lookup SEL-specific resource privileges, permissions, etc. in the ACCESS_FILE (or the realm definitions file). Add the "action" portion of the URL if ADD_RESOURCE_NAME is on.
2. If ALLOW_ACCESS is on, the list of resource privileges is compared against the client privileges.
   • All the "MUST_HAVE" resource privileges must be satisfied,
   • at least on of the "ONE_OF" resource privileges must be satisfied.
3. If the resource privileges are not "satisfied":
   • If ACCESS_FAIL_FILE=0, re-request a username/password (possibly using the "SEL-specific" realm name, or the 1st resource privilege).
   • Otherwise,
     If a "SEL-specific" access failure file is available, send it to the client
     If there is no "SEL-specific" access failure file, send the ACCESS_FAIL_FILE
4. In addition to the "resource privileges list" and the SEL-specific realm, several permissions are read from the ACCESS_FILE and the realm definitions file.

   Any of the following permissions might appear: NO_SSI, YES_SSI, NO_SSP, CACHE, NO_HTACCESS, NO_ALIAS, NO_VIRTUAL, NO_POSTFILTER, DELETE, and PUT.

5. If CHECKLOG< >"NO" or ALLOW_ACCESS< >"YES", then caching is suppressed. This suppression is withdrawn when the CACHE permission appears.

Request selector modifications and URL redirections

1. But first, if PRE_FILTER equals YES (rather then FIRST), call the pre-filter (and check for an "all done" return status).
2. If it is an empty selector, then set the selector to be the DEFAULT
3. NOTE: Non-literal public_urls jump to here !
4. If CHECK_ALIAS is on, and the NO_ALIAS permission is not present: Compare the request selector to the aliases contained in the ALIAS_FILE and the realm-definitions file.
   Matches may indicate "local redirection", "remote" redirection, or "immediate file transfer".
   If a local redirection, the request selector is modified.
   If a remote URL redirection (or a file transfer); do the"redirection" (or the "transfer"), and goto the post-filter step.
   If a negotiable resource is indicated, then either server-side negotiation is used to generate a new "selector" (similar to "local redirection"), or a "variants list" is immediately sent to the client.
   
5. If the http method is GET, then check "auto naming"
   If NOEXT_TYPE is on, check the (now alias modified) selector for an extension.
   If no extension found, then (depending on the value of NOEXT_TYPE) either add an extension, or search a directory for a file (using the entries in AUTO_NAME), or display a directory of files in the requested directory (using the !DIR facility).
6. Perform the HOME_DIR for ~ replacement (with possible use of the $ to signifiy "www specific subdirectories").
7. If the selector contains a *, and WILD_SELS_OKAY is not equal to 0, then search for a file that mathes this wildcarded. Depending on the value of WILD_SELS_OKAY, this might yield a !DIR style listing of multiple matches.

Processing the Request

Note that in all cases,

• If the request selector can not be matched to a file, a response that uses HOME_NAME and NOT_FOUND_URL is returned.
  Note that NOT_FOUND_URL may invoke a custom-written "not found" response file.
• If a NO_VIRTUAL permission appears, then virtual directories will not be checked
• If a "pre-reply procedure" was specified (using the PRE_REPLY_PROCEDURE parameter in INIT_STA.80), then it (an external rexx procedure) will be called just prior to sending contents to the client.
• If encryption is enabled, and the selector is to be encrypted (as specified with a !ENCRYPT modifier, or an entry in an advanced options file), then encrypt the response (that may have been modified by the "pre-reply" procedure) using the appropriate method. This may entail changing the mimetype used in the response.
• Post-filter processing is done before exiting, but after closing the connection to the client.
• If DO_HTACCESS is on, then examine the directory (and parent directories) of the requested file for HTACCESS files, and use information therein for access control (and possibly for redirection). Note that for CGI-BIN programs this directory will be the location of the CGI-BIN program (similar rules hold for other requests for server side processing).
• If an advanced options file was specified (in the ACCESS_CONTROL_FILE), EXEC directives will be executed before "processing the verb".

• Image map requests

  The request string is compared against the NCSA_ISMAP and CERN_ISMAP variables. If a match is found, the image map routine is called, which invokes a redirection to the chosen URL.
  Note that the MAX_POINTDIST variable is used by the image map routines.
  Note that HTACCESS is never checked for image map requests (but HTML documents that contain clickable images are subject to HTACCESS controls!)

• CGI-BIN requests.

  If a GET or POST request starts with a CGI-BIN/, and the CGI_BIN_DIR variable is set, then the request is assumed to be a CGI-BIN requests.

  1. The alias file is checked again, this time for a match for a matching CGI-BIN script name. If found, the alias entry should contain the fully qualified directory containing the CGI-BIN script. If not found, the CGI-BIN script is assumed to be in the CGI_BIN_DIR directory. Exception: If the CGI-BIN portion of the request selector is preceeded by a directory entry, then this subdirectory (of CGI_BIN_DIR), or this virtual directory, will be used.
  2. The CGI-BIN processor is called, and the results from the CGI-BIN script are returned (or an error message if no such script exists).
     Notes:
     • There are 2 variants of the CGI-BIN processor -- a standard CGI-BIN script processor, and a (somewhat experimental) "send output as it is generated" script processor. The CGI_BIN_PIECES parameter, and the !CGI-BIN_n special directives control which one is used.

  3. Jump to the post-filter step.
• Special Requests If the request starts with a !, then one of the several special request might be processed.
  1. The valid special requests are: ping, statistics, host, reset, save, and variable.

     Save,reset, and variable require SUPERUSER or CONTROL privileges. Otherwise,an error response is returned.

  2. The !DIR special request will create a directory listing. It looks at the DIR_OPTIONS and DIR_EXCLUSION parameters for options. It also will check the "directory cache" for a match to the requested directory, and use it if possible.
     
  3. Several special directives are also recognized:
     • !SENDAS_type_subtype/sel is used to force SRE-http to return the sel with a content-type:type/subtype response header (that is, the normal SRE-http methods for determining MIME type are suppressed).
     • !DELSEND/filename.ext will transfer a file from the TEMPFILE_DIR, and then delete it.
     • If a !RANGE special directive is found, then only the indicated range of the resource will be sent (this can be partially suppressed with the RANGE_AT_END variable).
     • !FORCE/asel will force a logon, and will suppress use of the SSI-Cache.
  4. For special requests: Jump to the post-filter step.
     For special directives: preceed to GET step
     
• GET and HEAD requests
  HEAD requests are treated the same as GET requests -- except for the "minor" detail that the actual contents of the response is not sent!
  There are 4 classes of GET (and HEAD) requests.
  1. Request for non-html files.

     The request selector is matched to a file, using the data directory or a local or remote virtual directory. If DO_HTACCESS is on, then check appropriate HTACCESS file(s) for access privileges.

  2. Requests for Html files.

     The request selector is matched to a file, using the data directory or a local or remote virtual directory. If DO_HTACCESS is on, then check the appropriate HTACCESS file(s) for access privileges.

     • HTML files are recognized by extensions of .HTM, .HTML, .SHT, .SHTML, or .HTML-SSI. You may add to the list of HTML file "extensions" by modifying MEDIATYP.RXX (MEDIATYP.RXX can also be used to add new MIME type definitions). You can also change the "server side includes allowed" list by modifying the SSI_EXTENSIONS variable. Lastly, you can use MIME advanced option to explicitily specify the mime type.
     • On GET requests: If AUTO_HEADER="ALWAYS", then LINK, NAME and META-EQUIV elements in the HEAD are used to create extra response headers (for HEAD method requests, this header extraction will occur if AUTO_HEADER='ALWAYS' or 'HEAD')
     • If SSI_SHTML_ONLY is on, then check the SSI_EXTENSIONS (the "server side includes allowed" list) to see if this file can have server side includes added. Otherwise, all HTML files are checked for server side includes. In both cases, if NO_INCLUDE is on, or a NO_SSI permission applies to the selector, server side processing will not occur. However, a YES_SSI SEL-specific permission forces SRE-http to check an HTML document for server side includes (YES_SSI overrides NO_SSI, NO_INCLUDE, and SSI_SHTML_ONLY).
     • If SSI_CACHE_ON is on, then check to see if the SSI-Cache contains an entry for this file. If it does, either send it "as is", or use it in the following steps.
     • If the advanced options are enabled, check for SSI_suppression directives.
     • If the request selector includes a ? followed by a text string, the text string is assumed to contain OPTIONS, which can be used in server side includes.
     • Searchable indices, which are often specified as HTML files followed by ?string, should be interpreted through the use of an alias (the alias maps the request for an HTML file to a call to an appropriate external program, such as DOSEARCH).
     • Note that server side includes are processed recursively
     • REPLACE_RULES, specified in INITFILT.80 and in the advanced-options file, can be used to replace strings anywhere in an HTML file.
     • The DELIM_1. and DELIM_2 variables are used to define keyphrases
     • The following variables may be used for various server side includes: HEADERS., FOOTERS., INHOUSE., SUPERUSER., WEBMASTER, OPTION_HIT_LINE . In addition, the REPSTRGS_FILE may be checked for custom written "static REPLACEment strings".
     • The COUNTER_FILE may be used to record the "number of hits" -- after checking the repetitive hits cache (the repetitive hits cache is configured by the HIT_CACHE_LEN, HIT_CACHE_DURATION, and HIT_OWNER_SUPPRESS variables).
     • If NO_INTERPRET_CODE is on, or a NO_CODE permission applies to this request, then SELECT and INTERPRET CODE keyphrases are ignored.
     • Dynamic privileges can be added by using ADDPRIVS.RXX in an INTERPRET FILE keyphrase. Note that the added client privilege is modified by the ADD_PRIVS_PREFIX variable.
     • Secret privileges (that start with a ?) will be extracted (and reported to addons along with "regular" privileges)
     • If advanced options are enabled, HEADER and RESPONSE directives are processed. In addition, see if a "selector specific" mimetype was specified.
     • Lastly, the HTML document (perhaps with server side includes) is returned to the client.
  3. Server side processing requests
     • If NO_PROCESSING is on, or a NO_SSP permission applies to this selector, then no server side processing will be permitted.
     • If the request is to one of the several "built in" routines (SENDFILE, ASKMESSBOX, VIEWMESS, MESSAGE, GET_URL, !DIR, PUT_FILE), then SRE-http handles it directly.
     • Requests to SRE-http addons (such as DOSEARCH, GETAFILE, STATUS) are processed by calling the appropriate external procedure, which is located either in:
       macrospace (i.e.; as loaded there by a KEEP_ADDONS parameter),
       in the addon directory (as determined by the ADDON_DIR parameter), with typical value of x\GOSERVE\ADDON), or
       in a local virtual directory.

       These programs should return results directly to GoServe (i.e.; by using 'FILE xxx' types of commands). Note that failures in these "external" programs will result in error messages being returned to the client -- SRE-http will not crash.

     • If DO_HTACCESS is on, then check any appropriate HTACCESS file(s) (in the directory-tree that contains the program). For "built in" and "included" programs, the GoServe directory is checked for an HTACCESS file.
  For GET, HEAD, and (depending on how the addon is written) POST requests, SRE-http will check several conditions before returning the response. These include:
  • If-modified-since, If-unmodified-since, If-Match, and If-none-match "conditional get" request headers are checked. If they should be relevant (say, a resource is older then the if-modified-since date), then the contents of the resource may not be returned (for example, a 304 Unmodified response may be returned).
  • If specified, delta-encoding may be attempted. Furthermore, the current "instance" of the resource (the actual response body) may be saved (in the TEMP\DELTAS) directory for possible use in a future delta-encoded response.
  • Appropriate cache and expiration header will be added (whose values wil depend on the temporariness, and the accessiblity, of the resource). In particular, for "temporary files", an offset (set by the FIX_EXPIRE variable) may be added to the current time, and then used in the Expires header.
  • A Content-MD5 digest may be added (if CONTENT_MD5>0)
  • If the CE_GZIP parameter is enabled, or a CE_GZIP advanced option applies to this selector, then the response is GZIPped and a GZIP content-encoding header is added. This is not attempted if a GZIP content-encoding header is already present (say, due to content-negotiation).
  • If the acceptable transfer encoding request header indicates that GZIP compression should occur (TE: gzip), and there is no GZIP content-encoding, then the contents will be GZIPped, and the appropriate response header (Transfer-encoding: gzip,chunked) will be added.
  • If SREPROXY is being used, then possibly record some request information.
  cache
• POST requests POST requests are handled just like GET requests for server side processing. The only difference is that the argument list is pulled from the "body" of the request, rather then from the "request selector".

  If a "file upload" request occurs, the UPLOAD_MAXSIZE, UPLOAD_MINFREE and UPLOAD_DIR variables are used. (note that this applies to GET method transactions also).

  If DO_HTACCESS is on, then check any appropriate HTACCESS file (in the directory tree that contains the program). For "built in" and "included" programs, the GoServe directory is checked for an HTACCESS file.

• PUT requests PUT request (for transfering files to the server) are honored only if a PUT permission applies to the selector.

  If DO_HTACCESS is on, then check any appropriate HTACCESS file(s) (in the directory-tree that is the target of the PUT).

• DELETE requests DELETE request (for deleting files on the server) are honored only if a DELETE permission applies to the selector.

  If DO_HTACCESS is on, then check any appropriate HTACCESS file(s) (in the directory tree that contains the file to be deleted).

Post-filter processing

1. If the WRITE_LOGS parameter is enabled, then (perhaps) write the common-log audit file, the browser log, and the referer log. The SREFLOGS.INI will dictate exactly where and how to write these log files. SREFLOGS.INI, and SCHEDULE.CFG, are also used to dictate when to run "scheduled programs".
2. If the RECORD_OPTION is on and the RECORD_ALL_FILE is set, then record this selector.
   ... but first check the repetitive hits cache to see if it has been recently recorded.
   Note: recording may be stored in an internal cache for several minutes.
3. If the POST_FILTER variable is on, then call a (set of) post-filter(s) with listed in POSTFILTER_NAME (note that the SMTP_GATEWAY can be used by the POST_FILTER).