1 November 1999 Abstract: SRE-http's SEL-specific "advanced options" are used for a variety of purposes, including setting parameters, enabling less-frequently used options, and executing mid-filter procedures. 1. Introdution 2. Specifying advanced options 3. The advanced options 3a) EXEC Commands 3b) HEADER and RESPONSE directives. -------------------------------- 1) Introduction SRE-http's selector specific advanced-options provides a set of less-frequently needed, "advanced-options" to the ambitious webmaster. These advanced-options include: * execution of "mid-filter" procedures, * customization of response headers * suppression of specific server side includes. * selector specific mime type * selector specific replacement-rules * set the FIX_EXPIRE, CONTENT_MD5, PRE_REPLY, and other parameters * enabling simple hit-metering * enable or disable encryption * specify the maximum number of simultaneous users of a resource * in the future, additional features may be added. -------------------------------- 2) Specifying advanced-options There are two ways of specifying selector-specific advanced options. a) Using an advanced options file. You can specify a selector-specific advanced option file either in ACCESS.IN or in ATTRIBS.CFG -- you can use the intermediate configurator to do this (or see IN_FILES.DOC for the details). By default, these advanced options files should be located in, or under, the DATA\ subdirectory of your GoServe working directory (technical note: it will be in or under SRE-http's WORKDATA_DIR directory). Note:there is NO global (default) advanced-options file. b) Specifying options in ATTRIBS.CFG In the usual case where one needs to set a few advanced options, the use of the Option: lines in ATTRIBS.CFG is easier then specification of a seperate advanced options file. Note that for any REALM specified in ATTRIBS.CFG you can use both an advanced options file line (ADVOPT_FILE) and one (or several) Options: lines. Some examples:: * Assuming that your SRE-http "WORKDATA_DIR" directory is D:\GOSERVE\DATA, the following entry in ACCESS.IN: /SAMPLES/* * , , MYREALM , , ADV_OPT.CTL tells SRE-http to use the D:\GOSERVE\DATA\ADV_OPT.CTL advanced options file for all selectors that match /SAMPLES/* * The following entry in ATTRIBS.CGG could also be used to set some advanced options (for all selectors that match /SAMPLES/*): REALM: Myrealm RULE: /SAMPLES/* Option: header add X-Grader: this is mediocre resource Option: set fix_expire 0.1 -------------------------------- 3) The advanced options There are several types of selector-specific advanced options: 1) EXEC commands -- execute an external REXX procedure. 2) HEADER and RESPONSE directives -- specify a custom response header 3) SSI_suppression directives -- suppress specific SSI actions. 4) Set MIME type 5) Replacement rules 6) Set a few variables 7) Enabling hit-metering 8) Controlling the maximum number of simultaneous users of a resource The following documents how to use these options. We also recommend perusing the ADV_OPTS.CTL and ADV_OPTS.CMD sample files. Notes: * each of these options should appear: i) on a single line in your advanced option file, or ii) in ATTRIBS.CFG * lines in an advanced option file that begin with ; are comments 3a) EXEC Commands EXEC entries are used to execute external REXX procedures. This execution occurs just before the "verb is processed", and after all "selector modifications/redirections" and "access control checks." In a sense, the EXEC commands can be thought of as "mid-filter" hook, occuring after the (optional) pre-filter, and before the (optional) "pre-reply" procedure. Example: EXEC ADV_OPTS.CMD Hello from job 1. In this example, ADV_OPTS.CMD should be in the GoServe working directory. Note that ADV_OPTS will be sent a set of variables: ARGUMENT: The "argument" following the procedure-file name. This may contain commas (a comma containing ARGUMENT does NOT result in multiple arguments being sent to the called procedure). In the above example, ARGUMENT would be "Hello from job 1." (without the quotes). SOURCE: The GoServe "source" line REQUEST: The GoServe request line SEL: The (SRE-http) modified selector VERBOSE: The SRE-http VERBOSE parameter SERVERNAME: The name of the server processing this request (might be an alias) HOST_NICKNAME: SRE-http "host nickname" for the server processing this request DATA_DIRECTORY: The default data directory (might be servername dependent) HOME_DIRECTORY: The ~ replacement directory (the "home" directory). TEMP_DIRECTORY: The SRE-http TEMPDATA_DIR "temporary files" directory These external procedures MAY issue GoServe completion codes. If they do, they should return status_code bytes_sent (i.e.; 200 15122) If they do not, they should return a 0. All other returns are ignored. Note that ADV_OPTS.CMD contains a simple example of such an external procedure. Hint: one use of EXEC is to support HTTP M-extensions, as described in the "HTTP Extensions" Internet-Draft. Otherwise, these M-extensions will result in a 510 error return. 3b) HEADER and RESPONSE directives. You can include HEADER and RESPONSE entries -- they will be sent almost verbatim to GoServe. This response header customization is only available for GET and HEAD requests that invoke a file transferal (including HTML documents with server side includes). That is, for "server side processing" requests (such as CGI-BIN scripts, SRE-http addons, PUT and DELETE requests, and imagemaps) HEADER and RESPONSE entries will NOT be used. As a convenience, SRE-http will perform a few substitutions on HEADER and RESPONSE entries: $GMT : The current GMT day, date and time. $SERVERNAME: The "server name" of the server processing this request $SIZE: The number of bytes to be sent $CODE: The status code (almost always 200) Examples: response HTTP/1.1 406 Variant not available header add X-Webmaster-Name: Daniel Platypus header drop expires header add X-comment: X- headers are created by SRE-http header add X-Our-server: $servername header add X-Size-1: this is $size size header add X-Code-1: this is $code code Notes: * This customization is akin to ICS's "meta information". * Headers specified using advanced options override automatically generated headers. For example, if a cache-control header is specified as an advanced option, then the automatically generated cache-control header will not be used. * The HEADER DROP will drop a header (if one exists). This is cumulative, so that... HEADER DROP EXPIRES HEADER ADD Expires: not-now yields an Expires: not-now response header. * HEADER ADD is cumulative -- if the named header exists, the new value is appended. Thus, to replace a header, you should use HEADER DROP followed by HEADER ADD. * See GOSERVE.DOC for a description of the HEADER and RESPONSE GoServe directives. * When adding a header, be sure to include an ADD after the HEADER * CACHE-CONTROL is special -- it is not cumulative. To drop it, specify HEADER ADD Cache-Control: 0 To change it, specify HEADER ADD Cache-Control: my_cache_control 3c) Server Side Include Suppression Although you can suppress all server side includes (either for all documents, or on a SEL-specific basis), you may wish to only suppress a select set of these includes. In particular, you may wish to supress HEADERS and FOOTERS for a portion of your HTML documents. This can be easily done with the various server side include suppression options. The available options are: ssi_no_cache : suppress caching (similar to a keyphrase) ssi_no_header : Do NOT include HEADERS lines ssi_no_footer : Do NOT include FOOTERS lines ssi_no_replace : Do NOT process keyphrases ssi_no_include : Do NOT process keyphrases ssi_no_option : Do NOT process keyphrases ssi_no_interpret : Do NOT process or keyphrases. ssi_no_select : Do NOT process ... keyphrases ssi_no_#filestat : Do NOT process or keyphrases ssi_no_#config : Do NOT process keyphrases ssi_no_#echo : Do NOT process keyphrases ssi_no_#include : Do NOT process keyphrases ssi_no_#exec : Do NOT process keyphrases Note that these options should appear one-per-line. Examples: ssi_no_cache ssi_no_header ssi_no_footer ssi_no_select ssi_no_#echo ssi_no_#exec 3d) Mime types You can specify the mime type for this (these) selector(s). Just enter MIME type/subtype For example: MIME text/plain MIME image/jpg You should have, at most, one MIME line per selector (latter entries are ignored). 3e) Replacement rules You can specify selector specific "replacement rules". These will be used instead of the generic REPLACE_RULES. replacement rules (set in INITFILT.80). The syntax is REPLACE_RULES=old_string==new_string Notes: * Spaces will NOT be stripped from either old_string or new_string -- thus "invisible spaces" at the end of new_string will be retained. * Note the use of == as a seperator between old_string and new_string. * Unlike REPLACE_RULES.n (in INITFILT.80), do NOT include a "stem"; the order of appearance is the order of processing * If you do NOT specify any replacement rules here, generic ones (specified in INITFILT.80) will be used. Example: REPLACE_RULES=$(== Hint: Using Replace_rules and Load Balancing. Replace rules can come in handy when you are load balancing to subsidiary- sites. In order to guarantee that all requests pass through a "main-site", you may have to include a element in the section of HTML documents on subsidiary-sites. Since adding these elements may be tedious, the following REPLACE_RULES can be used instead: REPLACE_RULES=== where main.site.url should be the IP address of the main-site. 3f) Setting variables You can set several SRE-http variables on a request specific basis. These are: CG_GZIP (enable/disable on-the-fly GZIP content encoding) CONTENT_MD5 (add an Content-MD5 response header) FIX_EXPIRE (add an offset to an otherwise immediate expiration). PRE_REPLY (suppress or enable the use of a "pre-reply" procedure). ENCRYPT (enable or disable encryption) DELTA_ENCODING (enable delta encoding) DTemplate (specify a delta encoding "DTemplate") DCluster (specify a delta encoding "DCluster") The syntax to use is: SET varname value where varname is either CONTENT_MD5, FIX_EXPIRE, PRE_REPLY, or ENCRYPT For example: SET CONTENT_MD5 0 (suppress generation of Content-MD5 header) SET CONTENT_MD5 1 SET CONTENT_MD5 2 (use MD5.EXE, assuming EMX is avaiable) SET FIX_EXPIRE 0 SET FIX_EXPIRE 0.66 SET PRE_REPLY 0 SET ENCRYPT SRE_A SET DELTA_ENCODING 1 (1 means enable, 0 means disable) SET DTEMPLATE /quotes/? SET DCLUSTER /quotes/basequote.1 SET CE_GZIP 1 (enable on-the-fly GZIP content encoding) SET CE_GZIP 0 (disable on-the-fly GZIP content encoding) Notes: * PRE_REPLY can take the following values: 0 - suppress the pre-reply procedure This is not necessary if the PRE_REPLY_PROCEDURE='!filename' format is used. 1 - use a "selector specific" pre-reply procedure. This only has an effect when the "!filename" (selector specific) format of PRE_REPLY_PROCEDURE is used; it is not necessary when the PRE_REPLY_PROCEDURE='filename' format is used. * If PRE_REPLY_PROCEDURE=0, then PRE_REPLY has no effect. * PRE_REPLY_PROCECURE is set in INIT_STA.80 (see INITFILT.DOC, or PREREPLY.DOC, for more details). * If encryption is not enabled (ENABLE_ENCRYPTION=0) then SET ENCRYPT has no effect. * SET ENCRYPT 0 means "never encrypt" -- this will override a !ENCRYPT "special directive" (that is prepended to the selector) * SET ENCRYPT (without any other argument) is a shorthand for SET ENCRYPT SRE_A * DELTA_ENCODING is used when the default DELTA_ENCODING (set in INIT_STA.80) is equal to 1 or 2 (if the default DELTA_ENCODING=0, then DELTA_ENCODING is never attempted) * DCLUSTER and DTEMPLATE are used to specify a Dcluster or Dtemplate request header (they will be included verbatim). These response headers are used to improve delta-encoding (as described in DELTA.DOC). * CE_GZIP overrides the CE_GZIP parameter set in INIT_STA.80 3g) Enabling hit-metering. In order to get some measure of the number of times proxy servers are resolving requests for resources from your site, SRE-http supports a form of simple-hit-metering. This algorithim (based on RFC 2227) depends on proxies reporting their activity to your server. To enable this, you must specify a METER advanced option. When METER is specified (and when a proxy says that it is willing to support hit metering), SRE-http will: * include a METER: response header, and * will watch for special METER: request headers. The information in these METER: request headers is written to HITMETER.CNT. The syntax of this option is: METER=[options] where: options are a set of optional modifiers, that are described in RFC2227. These options include directives on cache lifetime, and on when and how to report hits. Alternatively, you can not specify options, which will enable hit-metering, but not effect cache behavior. Examples: METER= METER= max-uses=3, dont-report Note that if you do not specify a METER= option, then hit-metering will not be attempted (even if a proxy reports that it is willing to provide hit-metering). For further details, please HITMETER.DOC, and see the description of the PROXY_CACHE variable in INITFILT.DOC. 3h) Controlling the maximum number of simultaneous users of a resource In order to selectively control the intensity of use of your server's resources (such as CPU intenstive scripts and addons), you can specify an 'intensity of use' option". Basically, if more then this number of requests for this resources are currently active, the server will respond with a "Server busy, try again in a few minutes". The syntax is: MAXUSER class = n ; failure=maxuser_fail_file retryafter where: class : identifies a "class" of resources. This class can be unique to the selector, or can be shared by several selectors (say, due to several MAXUSER options in several advanced options files). These "classes" are similar to "realms", but are only used by this MAXUSER option If not specified, a "default" class is used. n : n is the maximum number of simultaneous users allowed. maxuser_fail_file: Optional. If specified, maxuser_fail_file is used when maxuser is not satisfied. Maxuser_fail_file should be a fully qualified file name, or a file name relative to your GoServe working directory. retryafter: Optional. Number of seconds to use in a Retry-After header. If specified, a Retry-After reponse header is added to the maxuser_fail_file response (http/1.1 clients can use Retry-After response headers to automatically re-request the resource). Example (as may appear in attribs.cfg): realm: foo1 rule: /addon/gif_text* option: MAXUSER gifProgs = 3 ; failure=nogif.htm 60 realm: foo2 rule: /addon/blendgif* option: MAXUSER gifProgs = 1 Notes: * "class" is case insensitive. * a semi-colon (:) MUST appear before a failure=maxuser_fail_file * if a MAXUSER_FAIL_FILE is not specified ,a generic "server is busy" response will be returned (when the MAXUSER is not satisfied). * As with the LOGON_FAIL_FILE, when the MAXUSER_FAIL_FILE is an HTML document, server side includes are NOT attempted. However, a few special substitutions will occur (when the MAXUSER_FAIL_FILE is an HTML document): phrases are substituted with the request selector. phrases are substituted with the servername. phrases are substituted with the WEBMASTER variable. phrases are substituted with a short description of the reason for failure phrases are substituted with a URL which points back to this resource * MAXUSER is defined on a selector specific basis, but the maximum users are defined across a "class" of selectors. Thus, one could limit users of a set of resources (say, only 2 simultaneous users of any one of several helper apps). * The fact that n can vary for each of several selectors assigned to a "class" lends some flexibilty. * In the above example, i) blendgif will not be started if any instance of blendgif or gif_text is currently active ii) up to 3 instances of gif_text are alllowed iii) or 1 instance of blendgif and 2 instances of gif_text (assuming that blendgif was started first) * CAUTION: SRE-http uses mutex semaphores to implement this MAXUSER feature. Unfortunately, REXXLIB's implementation of mutex semaphores is somewhat buggy; and there may be cases where "ghost semaphores" will interfere with program flow. SRE-http attempts to deal with this (by using alternative semaphores when required). However, on very dynamic and unstable sites (where threads are being opened and closed rapidly), this workaround may not be effective. If problems arise that can be traced to the use of MAXUSERS, you can * enable more thorough bug reporting by setting VERBOSE=3 (in INITFILT.80) * view these bug reports by running PMPRINTF * contact the author (Daniel Hellerstein) for advice (danielh@crosslink.net) 4) A sample "advanced options file" The following is a sample of an advanced options file. You might also want to examine ADV_OPTS.CTL. ;------------- Begin Example ------------ ; call the sample external procedure exec adv_opts.cmd This is an argument , with commas embedded. ; ; add a custom response header header add X-Webmaster-Name: Timothy Platypus ; ; Suppress headers, footers, and server side "excludes" ssi_no_header ssi_no_footer ssi_no_select replace_rules= C.== $ Customiz ; suppress content-md5 header set content_md5 0 ;------------- End Example ------------ ---- End of document