23 February 2002

SRE2002: An Internet Server for OS/2

Abstract SRE2002 is an Internet server for OS/2. More precisely, SRE2002 provides a set of functions and capabilties that allow you to implement a web server through a "filter" program that is written in REXX. In addition to a fairly simple filter included with SRE2002, several filter programs are publically available (such as sreLite2 and SREhttp/2) - or you can write your own!

1) Introduction

SRE2002 is an Internet server for OS/2. More precisely, SRE2002 provides a set of functions and capabilties that allow you to implement a web server through a filter program that is written in REXX. In addition to a fairly simple filter that comes with SRE2002, there are several publically available filters, such as SREhttp/2 (http://www.srehttp.org). Or, or you can write your own filter!

SRE2002 is designed for small to intermediate sized sites. It's primary strengths are flexibility and ease of customization. Speed, resource use, security, and scalability are not the primary goals. That said, these are operational concerns; and quite a bit can be done to enhance performance along any of these dimensions.

 In many ways, SRE2002 is an outgrowth of GoServe, which was designed and created by Mike Cowlishaw. In fact, it is the end of support for GoServe that motivated the creation of SRE2002. In comparison with GoServe, the primary advantage of SRE2002 is increased support for http/1.1 functions, including a greatly improved cache. Just as important, SRE2002 can be readily modified (it's written in REXX) -- we are always interested in suggestions and ideas! Are you are upgrading from GoServe/SRE-http? You can find a number of hints and suggestions in UPGRADER.TXT.

Of course, there is a disclaimer ..

Before choosing & using SRE2002, you must read the disclaimer (in appendix 1). Basically, SRE2002 is gnu style freeware; and is to be used at your own risk. We do try to create a fairly secure product, and we try to quickly fix any problems (or potential problems). However, we can NOT guarantee that SRE2002 has no security holes or is free from other potentials for failure.

More help can be found in ...

 SRE2PRC.HTM contains detailed descriptions of SRE2002 functions. SREAPI.TXT contains further details on working with SRE2002 SRE_DMN.TXT discusses how you to user SRE's functions that create and manipulate daemons, flags, and queues. SRE_TASK.HTM discusses SRE2002's built in task manager. SRE_DCSH.TXT discusses SRE2002's built in database facility. IM_USE.HTM discusses how to use and create SRE2002 Instance Manipulation modules .

2) Installation

 Before starting... if you are less familiar with the http/1.1 protocol, or are new to http servers, you may want to first review some basic terminology.

The easiest way to install SRE2002 is to use the installation program.
1. UnZIP SRE2002.zip to an empty temporary directory
2. open up an OS/2 prompt, and CD to this directory
3. run INSTALL.CMD
For example, assuming you unzipped SRE2002.zip to E:\TEMP:
E:\TEMP>install
4. You will be asked to enter a working directory -- this is where SRE2002 will be installed.
Once you've installed SRE2002, you'll probably want to install one of the more advanced filters (such as sreLite2). However, for very simple sites (or just to see how it works), the SIMPLE filter that comes with SRE2002 is quite adequate.

Notes

• If you are using Object Rexx (under OS/2 4.0), you will not be able to use SRE2002.

• If you are using Warp 4.0 (or Warp Server 4.0), you should be using the latest fix packs for TCP/IP (INETVER should report 4.02t or above).
You can get the latest TCP/IP fix-packs from:
ftp://ftp.software.ibm.com/ps/products/mpts/fixes/english-us/
or you can try
http://duanec.indelible-blue.com/fixes/latestwarp4.html

• SRE2002 does NOT modify any system files. To remove SRE2002, just delete the SRE2002 working directory (that you chose in step 4 above).

• SRE2002 uses several DLLS. If you are running OS/2 3.0, or you plan to use a multitude of CGI-BIN scripts, you may want to move these DLL files into your LIBPATH; say, to x:\OS2\DLL, where x: is the system drive. Alternatively, you can add x:\sre2002\DLL to the LIBPATH line in your CONFIG.SYS file (where x:\sre2002 is your SRE2002 directory).

• By default, the SIMPLE.RXX filter (that comes with SRE2002) is used.

• The installation process creates several subdirectories (under the working directory).
• BAK : Backup of prior SRE2002 versions (used by INSTALL program)
• BIN : Programs and modules used by SRE2002
• CFG : Configuration files
• DLL : Dynamic link libraries
• DOC : Documentation files and manuals
• LOG : Log files
• TEMP : Temporary files created by SRE2002
• SIMPLE : The "SIMPLE" filter and related files
• It is possible to simultaneously run more then one copy of SRE2002, with each copy handling a different port (say, port 80 and port 8080). To do this, we recommend having totally seperate subdirectories for each version (for each port).

To use a different port, all you need to change is the PORT parameter in SRE2002.CFG.

3) Starting SRE2002

To start SRE2002,
1. Open an OS/2 box
2. CD to the working directory
3. Run SRE2002
For example, assuming the working directory is D:\SERVER:
D:\SERVER>SRE2002
4. SRE2002 will start, and after some initializations a text-based status screen will be displayed.

Notes

• From the SRE2002 status screen, you can enter several options:  Esc Shutdown SRE2002 After hitting Esc, it may take a few seconds to shutdown. In some cases, hitting Ctrl-C may be faster. However, it's cleaner to shut down using ESC (various resources will be properly freed up). R Refresh the screen. This is useful if error messages, or other unusual conditions (such as your filter using SAY) causes information to be written to the screen in an uncontrolled fashion. M Modify parameters. This will let you modify the dynamically-modifiable parameters. You can suppress this option by setting the NO_MODIFY_ONTHEFLY option. S Snipe(kill) connections. Use this to list, and selectively close, currently active connections. When in snipe mode, no more connections will be accepted (though currently open connections will continue to be active).

• After shutting down SRE2002, you can immediately restart it (by re-running SRE2002.CMD). However, we recommend restarting SRE2002 from a freshly opened OS/2 session (and closing the session SRE2002 had been running in).
• If you do not like the colors used in the status screen, you can modify them -- see the user configurable parameters section at the top of SRE_STAT.RXX (in the BIN subdirectory).
• To see SRE2002 start up options, run SRE2002 ?
• SRE2002 uses PMPRINTF to display status and error messages. To view PMPRINTF output, you can use PMPRINTF.EXE, which can be found at http://www2.hursley.ibm.com/goserve/
• SRE2002 writes error messages to an ERROR.LOG file (in the LOG subdirectory).

3.a) Using the SRE Watch monitor

The SRE Watch program offers another means of running SRE2002.

SREWATCH will start SRE2002, and will monitor its status. If SRE Watch detects that SRE2002 has crashed (or is otherwise hopelessly hung), it will attempt to kill, and then restart, SRE2002. In fact, you can even tell SRE Watch to reboot the machine when things get bad enough.

To run SRE Watch, run the SREWATCH.CMD program (located in the SRE2002 working directory) instead of SRE2002.CMD. You might need to set a few parameters (in SREWATCH.CMD) before the first time you run it, but most people can use it as delivered.

• For details on using and configuring SREWATCH, read SREWATCH.CMD with any text viewer (or editor). Or, from an OS/2 prompt, run SREWATCH.CMD with a ?
For example: D:\SRE2002>srewatch ?

• The use of SREWATCH is highly recommended for those running unattended servers.
For example: run SREWATCH.CMD from your STARTUP.CMD file, and enable the SRE Watch "reboot if necessary" option,

4) Configuring SRE2002

SRE2002 uses 3 configuration files:
1. SRE2002.CFG: Server parameters
2. HOSTINFO.CFG: Host definitions
All of these are ascii (text) files that can be modified using any text editor (such as EPM). They are located in the CFG subdirectory of SRE2002.

SRE2002.CFG is a text file that you can modify using any text editor (such as EPM). You can also modify SRE2002 by entering, from an OS/2 prompt:
SRE2002 -m
You will be presented, in a sequential fashion, with a list of the most important SRE2002 parameters, and their current values. You can change, or retain these values; you can also dislay some rudimentary on-line help.
Alternatively, if you enter
SRE2002 -mall
you will be presented with a list of all the parameters contained in SRE2002.CFG, many of which you'll never need to change.

Note that several SRE2002 filters (such as sreLite2 and SREhttp/2) come with configuration utilties that provide an on-line method of modifying SRE2002.CFG (the server parameters file), HOSTINFO.CFG (the host definition file), and BADIPS.IN (the unallowed IP addresses file).

4) Description of SRE2002 configuation parameters.

The variables set in SRE2002.CFG are:
The most important of these variables are:
 AUDIT_ITEMS DATADIR INIT_PROC FILTER LIMITCLIENTS PORT REQ_CACHE_ENABLE REQ_CACHE_RECORD STATUS_ITEMS
(almost all of these "most important" variables can be modified by running SRE2002 -m).

Notes

• The following parameters can be reset on the fly (from the SRE2002 status screen, hit M).  DATADIR LIMITBODY LIMITCLIENTS LIMITHEADER LIMITTIMETOTAL LIMITTIMEWAIT LIMITTIMEINACTIVE SERVER_SOFTWARE SERVER_SOFTWARE_SHORT

4a) Detailed description of SRE2002 parameters

The following provides a more detailed description of the SRE2002.CFG parameters.
AUDITFILE
Where to write audit items (as specified by AUDIT_ITEMS) to. Should be a relative filename; relative to the SRE2002 working directory. By default, it's LOG\AUDIT.LOG.

AUDIT_DELAY
How many minutes to wait when writing to audit file. This is a "lazy write" parameter -- SRE2002 will store this many minutes worth of AUDIT output, and write it to the audit file all at once.

You can use fractional values. For example:
AUDIT_DELAY=0.25
means "write to audit file every 15 seconds"

Notes:

• this delay is approximate -- the actual delay will depend on system activity
• REQ_CACHE_RECORD is used to set the delay when writing to the "cache hit" file

AUDIT_ITEMS
Specifiy what additional events to write to audit file. Should contain a space delimited list of options. Supported options are: CLIENT, MESSAGE, REQUEST, SENT, RESET, DIAG, and DIAG2.
Example: audit_items=CLIENT SENT

These options yields the following information.

CLIENT
Client and server info. For example:
    C : 1A 00:25:07: 131.10.51.21 80 1 15.12.112.3 1104
where:
C :    -- this is the "client audit item"
1A    -- transaction and request number
00:25:07    -- time of connection
31.10.51.21    -- ip address of server
80    -- port of server
15.12.112.3    -- ip address of client
1104    -- port of the client
MESSAGE
Messages written using the SRE_WRITE_MESSAGE function. SRE_WRITE_MESSAGE is used to write text to the message area of the SRE2002 status screen.
DIAG
Diagnostic messages. A variety of diagnostic messages. For example:
   D : 3A 00:33:50: REQFIELD = If-match If-Unmodified-Since If-None-Match
D : 3A 00:33:50: FILE = CHUNK NOWAIT ERASE type text/plain NOCACHE Name G:\SRENEW\TEMP\_17111__.TGZ

DIAG2
A superset of DIAG -- you should not use both DIAG and DIAG2. The following are examples of information produced by DIAG2, but not DIAG.
  D : 3A 00:33:50: Header = HEADER NOAUTO
D : 1A 00:25:06: ReqHdr = Referer:do_get@localhost
D : 1A 00:25:06: ReqHdr = If-none-match: "dir.d","dir.doc"
REQUEST
The request string. For example:       R : 1A 00:25:06: GET /samples/daemons.doc HTTP/1.1
RESET
A reset has occurred. For example:
RESET : 21:34:52 5 Jul 2001
SENT
Summary of bytes sent & recieved. For example:
S : 1A 00:25:07: 158, 200 8992
Where:
158 : Bytes recieved
200 : response code
8992 : Bytes sent

Note that you can use the SRE_AUDIT function to write custom messages to the AUDIT file.

The default "data" directory. This is the "root" directory of you web site -- it's where resources corresponding to selectors of the form "/afile.ext" are located.

Note that the filter does NOT have to use this directory!

Notes:

• DATADIR() can be used, by filters, to obtain this value
• the DATADIR parameter directory applies to "default" (non host-specific) requests. The data directory for host specific are defined with the SRE_HOSTINFO procedure (or with the INITHOST.RXX initialization procedure).
• For security reasons, we STRONGLY recommend that the SRE2002 working directory NOT be in or under the datadir directory.
• The data directory can be modified on per request basis -- see the description of SRE_COMMAND for the details.

DEFAULT_HOSTS
A space delimited list of host names that are implicitily assigned to the default host. For example, they will have no host nickname, and they will use the default data directory.

Furthermore:

If a host is not:
• the canonical host, or
• listed in DEFAULT_HOSTS, or
• defined in HOSTINFO.CFG,

then

• a 400 Bad Request response is returned to the client.

Notes:

• The canonical host is defined by TCP/IP calls, or by the SERVERNAME_USE parameter. It is always accepted. Thus, DEFAULT_HOST augments the canonical host name, and complements HOSTINFO.CFG
• In this context, host names is synonymous with domain name
• The host (to whom this request is directed) is determined by (in order of precedence):
• A host in the selector. For example, a request line of:
GET http://foo.bar.net/hello.html HTTP/1.1
• A Host: request header. For example:
Host: foo.bar.net
• If niether of the above, then use the canonical host
• You can use wildcarded host names. In particular, the installation default, *, means any host will be accepted.
• Examples. If the host is not the canonical host, and it's not defined in HOSTINFO.CFG...
If DEFAULT_HOSTS is then the following hosts will be served (using default parameters)
* All hosts wil be served
0 No additional hosts will be served
foo.bar.net foo.bar.net will be served.
foo.bar.* goo.com goo.com, and any host name that starts with foo.bar., will be served

DEFAULT_SOCKET_TIMEOUT
Time to wait on a socket, in seconds. If a read or write action on a TCP/IP socket takes longer then DEFAULT_SOCKET_TIMEOUT to do anything, then a timeout will result (the connection will be closed).

Notes:

• The LIMITTIMEINACTIVE and LIMITTIMETOTAL timeouts refer to total connection time, assuming the connection is still healthy. DEFAULT_SOCKET_TIMEOUT refers to how long one should wait before concluding that the "connection is dead".

This option controls which entity-headers SRE2002 should not consider when auto-creating an ETAG (via the AUTO_ETAG option of SRE_COMMAND). It should either be empty, or contain a case insensitive, space delimited list of entity-headers.

Basically, the rules for etags are (as defined in RFC2616):

• An etag value should be unique for all entities derived from a request-uri.
• Entities are defined as having an entity-body and entity-headers
• Thus, if the entity-body, or an entity-header, changes its value (again, for a given request-uri), then the etag should change.
• Unfortunately, this can mean that essentially inconsequential changes in entity headers require a new etag (such as changes in file creation dates, or dropping a Content-Length header if chunked transfer encoding is used). The required changes in the etag can diminish the effectiveness of caches and proxy servers, as well as diminish the ability of SRE2002 to use conditional (304) responses to requesst that contain If-None-Match request headers.

Since RFC2616 is silent on this issue, SRE2002 uses the ENTITY_HEADERS_IGNORE parameter to select entity-headers to not consider when automatically creating an etag. When you select what to include in this list, you should consider whether it matters that a client may recieve an entity with headers that may not be completely up-to-date, versus the probability that an inconsequential change in an entity-header will unnecessarily suppress caching and conditional responses.

In general, we recommend including the following entity-headers: CONTENT-LENGTH, CONTENT-MD5, EXPIRES, and LAST-MODIFIED.

However, if you have resources subject to unexpected changes in freshness (that is, resources whose expiration times may depend on circumstances), then EXPIRES and LAST-MODIFIED probably should not be included (at the cost of reducing SRE2002's ability to use 304 responses).

Notes:

• In no case will general-headers and response-headers (as defined in RFC2616) be considered when automatically creating an etag.
• As defined by RFC2616 ...
• Response-Headers: Accept-Ranges Age Etag Location Proxy-authenticate Retry-After Server Vary WWW-Authenticate
• all other headers in the response are considered to be entity-headers.
Some examples of entity-headers include: Allow Content-Encoding Content-Language Content-Length Content-Location Content-MD5 Content-Range Content-Type Expires Last-Modified

ERRORFILE
The file where errors are recorded to. By default, this is log/errors.log

EXIT_PROC
File, or a comma delimited list of files, containing exit procedures. These exit procedures are called just before SRE2002 shuts down.
• If you do not specify any full path, a file is assumed to be relative to the SRE2002 \BIN directory.
• If you specify a relative path, a file is assumed to be relative to the main SRE2002 directory (the directory containing SRE2002.CMD).
• Or, specify a 0 if you do not have an exit procedures.

Examples:

1.     EXIT_PROC= PEND1.RXX
The PEND1.RXX procedure (say, in x:\SRE2002\BIN\PEND1.RXX) will be called.
2.     EXIT_PROC= EXIT1.RXX , SRELITE2\LITEEXIT.RXX
First the EXIT1.RXX procedure (say, in x:\SRE2002\BIN\EXIT1.RXX) will be called. Then, LITEEXIT.RXX will be called (say, in x:\SRE2002\SRELITE2\LITEEXIT.RXX).
3.     EXIT_PROC=0
No exit procedure

Notes:

• Each EXIT_PROC procedure is called with the http port, the servername, and the directory containing the filter. Thus, exit procedures should start with something like:

parse arg PORT,SERVERNAME,FILTER_DIR

• EXIT_PROC procedures can return anything (the returned value is ignored).
• Possible uses for exit procedures include cleaning up daemons and flags that are created by a filter.

FILTER
Name of the filter program. Should be a filename relative to the SRE2002 working directory. Example: FILTER = SREhttp2.RXX

FILTER_VERSION
Descriptive name of the filter program. Should be a short phrase that describes this filter.
• Example: FILTER_Version= Simple Filter version 1.10a
• The SRE_SERVER('F') function returns the value of the FILTER_VERSION parameter.
• FAST_FILTER
Name of "fast-filter" program. If specified, the FAST_FILTER is called first. If the fast-filter does not return a response to the client, the "normal" filter is then called. For more details on use of a "fast filter", see Appendix 4.
Examples:
FAST_FILTER = FASTFILT.RXX
FAST_FILTER = 0 -- do not use a fast filter.

IM_TYPES
Define the available Instance Manipulation modules. These modules may be called by SRE2002 after recieving a FILE to send to the client, but before any final encodings. Examples of instance manipulation include range extraction, and delta encoding.

Syntax: IM_TYPES= im_name1 im_file1 , im_name2 im_file2 , ....

where each (case insensitive) im_name1 & im_file1 pair are seperated by a comma:

• im_nameN = a short name identifying this module. This name can be used in the IM_DEFAULT configuration parameter, or with the IM option of SRE_COMMAND('FILE ...')
• im_fileN = a rexx program that implements this module. This should either be a fully qualified filename, or a filename relative to the SRE2002 directory.
• Example:
IM_TYPES=XRange bin\xrange.rxx , deltas D:\TEST\DELTA\delta.rxx

IM_DEFAULT
The default Instance Manipulation module. This is used if no IM option is specified [in a SRE_COMMAND('FILE ...')].
IM_DEFAULT can be set to:
• 0 = By default, do not attempt an Instance Manipulation.
• RANGE = Extract a range (if the client requested one). This uses SRE2002's built-in range extraction procedures.
• or, one of the id_names specifed in IM_TYPES.
Examples:
IM_DEFAULT=DELTAS
IM_DEFAULT=RANGE

INIT_PROC
File, or a comma delimited list of files, containing initialization procedures.
These initialization procedures are called just before SRE2002 starts accepting requests.
• If you do not specify any full path, a file is assumed to be relative to the SRE2002 \BIN directory.
• If you specify a relative path, a file is assumed to be relative to the main SRE2002 directory (the directory containing SRE2002.CMD).
• Or, specify a 0 if you do not have an initialization procedures.

Examples:

1.     INIT_PROC= PSTART1.RXX
The PSTART1.RXX procedure (say, in x:\SRE2002\BIN\PSTART1.RXX) will be called.
2.     INIT_PROC= INITHOST.RXX , SRELITE2\LITEINIT.RXX
First the INITHOST.RXX procedure (say, in x:\SRE2002\BIN\INITHOST.RXX) will be called. Then, LITEINIT.RXX will be called (say, in x:\SRE2002\SRELITE2\LITEINIT.RXX).
3.     INIT_PROC=0
No initialization procedure

Notes:

• Each INIT_PROC procedure is called with the http port, the servername, and the directory containing the filter. Thus, initialization procedures should start with something like:

parse arg PORT,SERVERNAME,FILTER_DIR

• Each INIT_PROC procedure must return either a
• 1 for success, or a
• 0 for failure
• If 0 is returned, SRE2002 will immediately shutdown.

• Possible uses for initialization procedures include launching a daemon manager and reading in host definitions.

LIMITBODY
The maximum size of a request body (as may be included in a POST or PUT request). If the request body is greater then this size, a 413 response is returned.

Example: LIMITBODY=150

Notes:

• LIMITBODY can be set on a connection specific basis -- see the description of SRE_COMMAND for the details.

The maximum size of request headers, in Kbytes. If the request has headers greater then this size, a 413 response is returned.

Note that there are no size restrictions on a single request header -- so long as all request headers (including the request line) are less then LIMITHEADER kbytes.

LIMITCLIENTS
Maximum simultaneous connections. Each connection handles a single request at a time (though a "maintained connection" can handle a sequence of requests).

Example: LIMITCLIENTS=15

LIMITTIMETOTAL
Maximum time allowed per request (in seconds). If the total time required to resolve AND transmit a response exceeds LIMITTIMETOTAL, the connection will be closed. No error message is sent to the client -- the connection is just closed.

Example: LIMITTIMETOTAL=160

Notes:

• LIMITTIMETOTAL must be greater then LIMITTIMEINACTIVE.
• LIMITTIMETOTAL can be set on a connection specific basis --
• see the description of SRE_COMMAND for the details.
• LIMITTIMETOTAL refers to a particular request, not to all the requests that may be handled over a single maintained connection.

LIMITTIMEWAIT
Maximum time to wait on a maintained connection (in seconds). This is the number of seconds that SRE2002 will wait for a new request to arrive. This wait occurs after a prior request has been resolved.

Example: LIMITTIMEWAIT=10

Notes:

• LIMITTIMEWAIT can be set on a connection specific basis -- see the description of SRE_COMMAND for the details.

LIMITTIMEINACTIVE
Maximum intra-request idle time (in seconds). If LIMITTIMEINACTIVE seconds have passed without any bytes being transmitted (or recieved), SRE2002 will close the connection.

Example: LIMITTIMEINACTIVE=20

Notes:

• LIMITTIMEINACTIVE must be less then LIMITTIMETOTAL
• You can set LIMITTIMEINACTIVE on a request specific basis -- see the description of SRE_COMMAND for the details.
• LIMITTIMEINACTIVE should be greater then DEFAULT_SOCKET_TIMEOUT. Note that DEFAULT_SOCKET_TIMEOUT is typically used to detect a bad connection (it's the maximum wait time allowed within a sequence of socket calls). LIMITTIMEINACTIVE is the maximum time SRE2002 will wait for the filter to "do something".

MAXCOUNTERS
Maximum number of "counter" variables. Counter variables are used to maintain counts. Currently, SRE2002 maintains the following counts: BYTESREADTOTAL, BYTESSENTTOTAL, TRANSACTIONS, REQUESTS, LIMITS, ERRORS, BROKEN, and CACHEHITS.

Thus, MAXCOUNTERS should be at least 10.

Examples: MAXCOUNTERS=30

Notes:

• Each counter variable requires about 30 bytes of storage
• You can set, augment, and read counter variables with the SRE_COUNTVARS function.

MAXCLIENTNAMES
Size (number of entries) in "client name" cache. The client name cache is used to lookup an IP name given an IP numeric address (IP requests contain the IP address of the client, but not the IP name). Since this may require a DNS lookup, SRE2002 can maintain a cache of the most recently used names (thereby avoiding a potentially costly call to your DNS server). This is most useful when clients tend to revisit your site several times in a short time periods (say, to fetch embedded images).

Example: MAXCLIENTNAMES=250

Notes:

• each entry in the clientname cache requires about 100 bytes
• the clientname cache is dynamic. Once it is filled, a LRU algorithim is used to remove old entries (so that new entries can be added).
• To disable the "client name cache", set MAXCLIENTNAMES=0
• the SRE_CACHE function can be used to set and view entries in the clientnames cache.

MAXSTATVALS
Size (number of entries) to use to compute statistics SRE2002 can compute several sets of statistics, including average, minimum, variance, and maximum. To do this, a set of the "most recent" values is required. MAXSTATVALS sets how large this set should be.

Currently, SRE2002 maintains statistics on the following measures:
TOTAL_TIME,START_TRANSMIT_TIME, BYTES.
SRE2002 also retains the most recent values of CLIENTADDR (dotted numeric address) and REQUEST (the request line).

In other words, MAXSTATVALS sets the number of "measures of the most recent responses" to retain.

Example: MAXSTATVALS=100

Notes:

• To set and view "statistics", you can use the SRE_LIST function.

MAXMD5S
Size (number of entries) of MD5 cache. Http/1.1 allows the server to attach a Content-MD5 response header, which contains the md5 hash of the response body. Clients can use this value to detect transmission problems, tampering, or other errors.

Since computation of an MD5 is mildly CPU intensive (especially for long files), SRE2002 maintains an "MD5" cache. The MD5 cache is used for "permanent" files -- when SRE2002 is asked to compute an MD5 for such a file, SRE2002 can first check it's cache to see if such a value has already been computed.

Examples:
MAXMD5S= 200
MAXMD5S=0 -- do NOT maintain an MD5 cache.

Notes:

• If you do NOT intend to include a Content-MD5 response header, you might as well set MAXMD5S=0.
• Each entry in the MD5 cache requires about 125 bytes.
• The SRE_MD5 function will automatically check the SRE_MD5 cache (if it is enabled) before computing an MD5 of a file.
If you use the SRE_MD5 function with the MD5 cache enabled, all MD5 computations are serialized. This seems to to help system stability. Thus, if you set MAXMD5S=0, then there will be no MD5 cache, and MD5 computation will not be serialized.
• The SRE_CACHE function can be used to view the current contents of the MD5 cache.
• The MD5 cache is dynamic. Once it is filled, a LRU algorithim is used to remove old entries (so that new entries can be added).

NO_MODIFY_ONTHEFLY
Flag to suppress the M "modify a few parameters on-the-fly" feature (parameters in the SRE2002.CFG file)

Examples:
NO_MODIFY_ONTHEFLY=1 suppress on the fly modification       NO_MODIFY_ONTHEFLY=0 allow on the fly modification

Notes:

• you can still modify SRE2002.CFG using SRE2002 -m , or edit it directly with a text editor.

The NO_TASKS parameter can be used to limit who can launch tasks. The following values of NO_TASKS are recognized.
• 0 = disable the task manager.
• 1 = enable the task manager
• 2 = the task manager is enabled when SRE2002 starts, but is disabled before the first request is received. This is useful if you want to use an initialization procedure to launch a few (perhaps recurring) tasks.
In particular, the use of NO_TASKS=2 may reduce some security concerns due to scripts launching unwanted/resource-wasting tasks (though it doesn't allay it much, since a well-written script can do just about anything).

PORT
Port the server is running on. The standard http port is 80.

Example: PORT=80

REQ_CACHE_ENABLE
Enable the request cache.
 REQ_CACHE_ENABLE=nn enable, with nn the time-to-live of each entry, in days. Thus, REQ_CACHE_ENABLE=1 means "enable the cache, each item has 24 hour lifespan" REQ_CACHE_ENABLE=0 do NOT attempt to resolve requests from the request cache

PRE_REQUEST
File, or a comma delimited list of files, containing pre-request procedures.
These procedures are called before each request is processed (it is called after the client's request had been read, and before checking the request cache). Note that PRE_TRANSACTION procedures are called well before the PRE_REQUEST procedures.
• If you do not specify any path, a file is assumed to be relative to the SRE2002 \BIN directory.
• If you specify a relative path, a file is assumed to be relative to the main SRE2002 directory (the directory containing SRE2002.CMD).
• Or, specify a 0 if you do not have any pre-transaction procedures.

Examples:

1.     PRE_REQUEST= BALANCER.RXX
The BADIPS.RXX procedure (say, in x:\SRE2002\BIN\BALANCER.RXX) will be called.
2.     PRE_REQUEST= BALANCER.RXX , SRELITE2\PREAUDIT.RXX
First the BALANCER.RXX procedure (say, in x:\SRE2002\BIN\BALANCER.RXX) will be called. Then, PREAUDIT.RXX will be called (say, in x:\SRE2002\SRELITE2\PREAUDIT.RXX).
3.     PRE_REQUEST=0
No pre-request procedures are called.

Notes:

• Each PRE_REQUEST procedure is called with the following arguments (these are a subset of the arguments sent to the filter).  source port and ip address info:    serveraddr serverport transaction clientaddr clientport request_line The request line (sent by the client):    verb selector protocol hostinfo The host-nickname to whom this request is directed:    host host_nickname data_directory socket The socket used for this request.

• Each PRE_REQUEST procedure must return either a
• 1 for success -- request processing will continue
• 0 for failure -- don't process this request, but wait for further requests (over a possibly maintained connection)
• An http response, structured as:
response_line '0d0a'x resp_headers '0d0ax' blank_line '0d0a'x a_message
Where ...
• response_line is the response line. For example:
HTTP/1.1 200 ok
• a_message is the content body (which can be text or binary).

Note that blank_line means a blank line. That is, the resp_headers should be a '0d0a'x delimited list of response headers, that end with an empty line.

• Possible uses for pre-request procedures include load balancing, and status monitoring.
• You can not modify PRE_REQUEST procedures on-the-fly (if you modify a pre-request procedure, it's changes will take place when you restart SRE2002).

• In addition to the information provided in the arguments to the PRE_REQUEST procedures, you can use many of the SRE2002 procedures.For example, you can use...
• SRE_VALUE to obtain system and global variables.
• The string manipulation procedure (such as SRE_REPLACEWILD and SRE_PACK64)
• The time procedures (such as SRE_GMT and SRE_DATESTAMP)

PRE_TRANSACTION
File, or a comma delimited list of files, containing pre-transaction procedures.
These procedures are called before each transaction is processed (it is the first thing that the transaction daemon does).
• If you do not specify any path, a file is assumed to be relative to the SRE2002 \BIN directory.
• If you specify a relative path, a file is assumed to be relative to the main SRE2002 directory (the directory containing SRE2002.CMD).
• Or, specify a 0 if you do not have any pre-transaction procedures.

Examples:

First the BADIPS.RXX procedure (say, in x:\SRE2002\BIN\BADIPS.RXX) will be called. Then, TOO_BUSY.RXX will be called (say, in x:\SRE2002\SRELITE2\TOO_BUSY.RXX).
3.     PRE_TRANSACTION=0
No pre-transaction procedures are called.

Notes:

• Each PRE_TRANSACTION procedure is called with the client's (numeric) IP Address, and a 1 if VERBOSE is a status-item.

• Each PRE_TRANSACTION procedure must return either a
• 1 for success -- transaction processing will continue
• 0 for failure -- immediately close connection
• 0 a_message for failure -- return a 503 response, using a_message in a X-Message response header.
• If 0 or 0 a_message is returned, no other PRE_TRANSACTION procedures will be called for this transaction.

• Possible uses for pre-transaction procedures include checking for "wildcarded" BAD ip addresses (i.e.; using the BADIPS.RXX procedure), or checking to see if the server is currently too busy.
• In addition to the information provided in the arguments to the PRE_REQUEST procedures, you can use several of the SRE2002 procedures. For example, you can use...
• SRE_VALUE to obtain system and global variables.
• The string manipulation procedure (such as SRE_REPLACEWILD and SRE_PACK64)
• The time procedures (such as SRE_GMT and SRE_DATESTAMP)
• You can not modify PRE_TRANSACTION procedures on-the-fly (if you modify a pre-transaction procedure, it's changes will take place when you restart SRE2002).

REQ_CACHE_CALL_ANYWAYS
For auditing and other purposes, SRE2002 can call the filter even when the request has been resolved through use of the request cache.

REQ_CACHECALL_ANYWAYS can take the following values:
 REQ_CACHECALL_ANYWAYS 0 Do not call the filter after use of the request cache REQ_CACHECALL_ANYWAYS * Always call the filter REQ_CACHECALL_ANYWAYS 1 Always call the filter (same as *) REQ_CACHECALL_ANYWAYS A_LIST Call the filter if any of the "abbreviations" in the spaced delimited A_list matches the selector
Example: REQ_CACHECALL_ANYWAYS=/DOCS/ /HTMLS/

Notes:

• See Appendix 2 for a discussion of the SRE2002 request cache.
• See the REQ_CACHE_RECORD parameter for a discussion of alternate means of auditing cache resolved requests.
• See the description of the arguments sent to the filter, and the description of the STILL_ALIVE function, for a discussion of how to detect whether the request cache was used.

REQ_CACHE_HITFILE
File containing record of cache hits. This file is used when REQ_CACHE_RECORD is enabled. It should be a relative filename (relative to the SRE2002 installation directory). You might want to place it in the LOG subdirectory of the working directory.

Example: REQ_CACHE_HITFILE=LOG\CACHEHIT.LOG

REQ_CACHE_RECORD
Record cache hits to the REQ_CACHE_HITFILE file.

As an alternative to "calling the filter anyways", SRE2002 can maintain a simple log of all requests for which a "request cache" response was used. This log file (specified in REQ_CACHE_HITFILE) contains a running sum, by selector, of the 200 and 30x (unmodified) responses).

REQ_CACHE_RECORD can take the following values:
 0 Do NOT use the cache-hit file mmm Use the cache hit file, and save results very mmm minutes
Example: REQ_CACHE_RECORD = 5

Notes:

• the cache hit file is NOT augmented when you "call the filter anyways"

REQ_CACHE_SIZE
The size of the request cache, in number of entries. Each entry requires about 300 bytes.

Example: REQ_CACHE_SIZE=200

REQ_CACHE_VERIFY
Enable http/1.1 style verification prior to use of the request cache.

REQ_CACHE_VERIFY can take the following values:
 0 never verify (always use cache entry if it exists) 1 verify, check Pragma, Cache-Control, If-None-Match, and If-modified request headers. If these are binding (i.e.; if Cache-control: no-cache is specified), then do NOT use the request cache. 2 verify, but ignore Pragma and Cache-Control:no-cache request headers (that is, just check etag and date)

SECURITY_LEVEL
A value that is used by some filters (such as the SIMPLE filter, sreLite2, and SREhttp/2) to determine who is allowed to remotely modify server parameters.

The following values of SECURITY_LEVEL are used by the SIMPLE filter (the simple filter that comes with SRE2002) and the sreLite2 filter.
value of
SECURITY_LEVEL
Interpretation
0 Remote configuration of server parameters is not permitted. Instead, you'll have to hand-edit (using your favorite text editor) the parameter files. In other words, the on-line configuration utilities are totally disabled (at least those that come with SRE2002 or with SRELITE2)
1 Remote configuration is only allowed from the server's own machine. In other words, it's not really remote -- you have to be running a browser on the same machine that SRE2002 is running (actually, the client ip address must match the server ip address, so a careful spoofer could cause damage). However, this does permit an "on-site webmaster" to use the on-line (html form based) configuration utilities.
otherwise
• SIMPLE: Anyone is allowed access
• sreLite2: By default (if you use the version of SRELITE2.CFG file that comes with sreLite2) remote configuration is granted to users with a SUPERUSER privilege. Note that there is no encryption of the http stream, so this is not a very high level of protection.
If you have removed the /SYSTEM/* SUPERUSER SEL_REQUIRES entry, then anyone will be able to use the configuration utilities!
For other filters, the filter's documentation should describe how SECURITY_LEVEL is used.

The dotted numeric IP address that SRE2002 will report to the world. This is not used by the TCP/IP layer, it only effects SRE2002 functions (such as by SRE_EXTRACT("SERVERADDR")).

• The dotted (numeric) IP address of this site.
• A 0, which means use TCP/IP calls to determine this site's dotted IP address

In general, it is safest to use a value of 0. However, there are cases (such as when your server is behind a NAT-capable router, and has a 192.168.0.nn address) for which the numeric ip address is meaningless. In such cases, set SERVERADDR_USE to the IP address of this site.

Notes:

• To find what TCP/IP reports as this site's IP address, check (with the SRE_VALUE function) the SRESYS variable SERVERADDR_CANONICAL

Examples:

• SERVERNAME_USE
The domain name that SRE2002 will report to the world. This is not used by the TCP/IP layer, it only effects SRE2002 functions (such as by SRE_SERVERNAME('default')).

SERVERNAME_USE should be one of:

• A dotted domain name
• A 0, which means use TCP/IP calls to determine this site's domain name

In general, it is safest to use a value of 0. However, there are cases (such as when your server is behind a NAT-capable router, and has a local name) for which the TCP/IP derived domain name is meaningless. In such cases, set SERVERNAME_USE to the "canonical" domain name of this site.

Notes:

• SERVERNAME_USE is only used for the "default" (the canonical) host.
In particular, it is not used when a Host: header matches one of your host definitions.
• To find what TCP/IP reports as this site's domain name, check (with the SRE_VALUE function) the SRESYS variable SERVERNAME_CANONICAL

Examples:

• SERVERNAME_USE=www.foobar.net
• SERVERNAME_USE=0
• SERVER_SOFTWARE
Reported name of this server software. This variable can be used when reporting the name of the server software; say, in a Server: response header.

Set SERVER_SOFTWARE=0 to use the default name (that is hard coded into SRE2002.CMD).

Example: SERVER_SOFTWARE = SRE2002 for OS/2, ver 1.10

Notes:

• The SRE_SERVER('N') function returns the value of the SERVER_SOFTWARE parameter.

SERVER_SOFTWARE_SHORT
Short reported name of this server software. This is an alternative to SERVER_SOFTWARE.

Example: SERVER_SOFTWARE_SHORT= SRE2002

Notes:

• The SRE_SERVER('H') function returns the value of the SERVER_SOFTWARE_SHORT parameter.

STATUS_ITEMS
Specify what to display on the status screen. SRE2002 can display several types of dynamic information on it's status screen (the status screen is what SRE2002 displays while it is running).

STATUS_ITEMS should be a space delimited list containing one or more of the following tokens:

ERRORS, CONNECTION, CLIENTS, CLOCK, LASTACCEPT, LIMITS, PEAK, STATS, VERBOSE
where:
 CLIENTS report number of currently active connections CONNECTION report each connection CLOCK display current time ERRORS report cumulative number of errors. LASTACCEPT display time of most recent connection LIMITS report cumulative number of timeouts PEAK report peak number of connections STATS report summary statistics VERBOSE report various other items

STATUS_MESSAGE
A message displayed on the side of the status screen. Only the first 12 characters are used.

TRACKING_ON
Enable SRE2002's "client tracking" facility.
TRACKING_ON = 0 suppress
TRACKING_ON = 1 enable

The tracking facility instructs SRE2002 to store "status" and other "tracking" information. You can the retrieve this tracking information, on a client specific basis, by using SRE_TRACKIT.

For more details on tracking, see Appendix 5.

UNALLOWED_FILE
A file name (relative to the SRE2002 working directory). This file can contain a list of UnAllowed numeric IP addresses. All requests from clients with these IP addresses are immediately closed -- no response is given, the socket is shutdown.

If you do NOT need this capability, set UNALLOWED_FILE=0

Examples:
UNALLOWED_FILE=0

The UNALLOWED_FILE should contain one IP address per line. Lines that start with a semi-colon are comments (and are ignored), as are blank lines. Note that IP addresses should be the dotted numeric address (do not use domain names).

Alternatively, you can enter IP addresses that contain a * wildcard character. These entries are used by the BADIPS.RXX PRE_TRANSACTION procedure.
For example:
125.22.*.1 matches 125.22.0.1 to 125.22.255.1

Examples of UNALLOWED_FILE entries

      98.13.61.22
98.13.61.23
;clients from 98.14.* are not granted access
98.14.*
Note that you can add and remove entries from the UNALLOWED_FILE at any time -- SRE2002 will check the UNALLOWED_FILE about once a minute.

5) Using SRE2002

This section is meant for those interested in writing custom filters for SRE2002. It may also be of interest to those interested in writing utilties (such as CGI-scripts, or addons) for use with SREhttp/2 or other SRE2002 filters.

Before proceeding, consider this short introduction to how SRE2002 works. Abstracting from configuration and startup issues, and ignoring some fancy tricks one can use, the sequence of events leading to resolution of a request is:

1. SRE2002 (the main thread) recieves a TCP/IP request from a client.
If the client's IP address is listed in the UNALLOWED_FILE, the connection will be immediately closed.
2. SRE2002 launches a daemon (the "transaction daemon"), which reads the request line, request headers, and (possibly) the request body.
Before doing anything (that is, before reading any bytes from the socket), the transaction-daemon will call PRE_TRANSACTION procedures (if any have been specified).
3. The transaction daemon launches another daemon (the "request daemon"). This request daemon calls the "filter", and provides several arguments that give information about the request -- such as the "selector" and the client IP address.
Before launching the request daemon ...
1. the PRE_REQUEST procedures are called (if any have been specified). Depending on the action of the PRE_REQUEST procedure(s), the request daemon may not be invoked.
2. if the request cache has been enabled, the request-uri is looked for in the request-cache. If found, the file it points to returned to the client (and the request daemon is not invoked).

If a fast_filter was specified, the request daemon will call it first; and call the regular filter only if the fast-filter was unable to resolve this request.

4. The filter then uses these arguments, along with additional information, to fashion a response. Typically, the response consists of the contents of a file stored on the server's hard disk. However, if may be a 3xx (redirection) or a 4xx (access denied) response.

5. In addition to the arguments sent from SRE2002, the filter can use function calls to request additional information (such as the value of specific request headers) from SRE2002. Operationally, these function calls commumicate with the "transaction daemon".
In some sense, one can think of SRE2002 as providing an interface to the Internet; an interface with an API that a REXX program (a "filter") can use to recieve http requests and send http responses.
6. When the filter is ready to send a response (to the client), it uses function calls to tell the transaction what the contents of the response should be. The transaction daemon takes care of all the tcp/ip details, and upon completion of the response informs the filter.
Prior to sending a response to the client, the transaction daemon may also apply instance manipulations to the contents. For example, range extraction is built into SRE2002; and other manipulations (such as delta encoding) can be supported via special IM modules for SRE2002.
7. The filter may then do some auditing or other such tasks.
8. The filter informs the transaction daemon that it is done, and exits (the request daemon immediately ends).
9. The transaction daemon waits to see if another request has been sent (that is, the connection may be a "maintained connection"). If a new request is available, go back to step 3. Otherwise, the daemon records some transaction information, and exits.
Thus, there are two important sets of information provided by SRE2002 -- the arguments, and the functions.

 Note: "daemons" are independent threads. Actually, in order to avoid operating system overhead, daemons are "recycled", so that rather then ending, they go into a dormant state and await re-invocation.

5a) Arguments sent to the filter

When the filter is called (in step 3 above), it is provied with the following arguments:
source,request,sel,host_stuff,id_info,reqnum,authh,ISALIVE
where ...
 source where, and to whom, the request is sent the request line the "selector" portion of the request line the host (and host-nickname) to whom the request is sent used to expedite processing of some SRE2002 functions the request number the value of a Authorization: request header flag indicating whether the request is still alive

In greater detail ...

SOURCE
Contains the following information in a space delimited list
 servaddr ip address of destination server (may vary if your server is handling multiple ip addresses) port port that recieved the request transaction the transaction number who numeric IP address of the client whoport port used by the client
You can parse "source" using:
parse var source myaddr port transaction who whoport .
REQUEST
The request line.
For example:     GET /samples/foo.bar HTTP/1.1
SELECTOR
The "selector" portion of the request line, with leading / removed.

From the above example, the selector is samples/foo.bar

HOSTINFO
Information about the host this request was sent to. This will be three words, seperated by commas:
where:
 host either the ip name of your site, or the value of a HOST: request header. host_nickname the "host-nickname" assigned to this host, or ' ' (if a host-nickname was not defined for this host) datadir the "default data directory". If no host_nickname is available, this will be the value specified by the DATADIR parameter (in SRE2002.CFG). Otherwise, it's a host-specific data directory (which may be the same as the "default" data directory).
Examples:
jones.gonzo.net,,f:\www
bigtime.circus.org,circus1,g:\webs1\circwww

The first example could be to a site for which no hosts have been defined.

Notes:

• Specification of host-nicknames is done with the SRE_HOSTINFO procedure, (or with the INITHOST.RXX initialization procedure).
ID_INFO
Request and transaction daemon identifier information. Several SRE2002 functions (such as SRE_REQUEST_INFO) will run just-a-bit-faster when you supply them with this parameter.
REQUEST_NUMBER
The request number. This has two components: a numeric "transaction number", followed by an alphabetic "request number". The request number, which starts from A, is the "request within this transaction", it is NOT the total number of requests recieved.

Note that the sequence of request numbers is: A,B,..,J,AA,AB,..,AJ,BA,...

Example: 4A

Since the Authorization: request header is often used, it is provided as an argument (it can also be read using the REQFIELD function).

Example: Basic ZGFuOmRhbg==

IS_ALIVE
Flag indicating whether the request is still alive. The request may already have been satisfied (typically due to a request-cache hit), and the filter may have been "called anyways" -- say, so that the filter can audit the request.

If IS_ALIVE=1, the request has NOT been satisfied.
Otherwise, it has.

Notes:
• you can obtain the values of these arguments using:

parse arg source,request,sel,MOREINFO,idInfo,reqnum,authh,isalive

• Upgraders note: the first three arguments are identical to what GoServe sends.

5b) The SRE2002 functions

SRE2002 includes a library of procedures that can be used when writing filters, or when writing addons for preexisting filters. Note that filters (such as sreLite2 or SREhttp/2) may have their own set of functions -- in many cases these provide additional capabilities that addon/script creators may find useful.

The following briefly describes a few of the more important SRE2002 functions. You can find more complete descriptions in SRE2PRC.HTM.

 SRE_AUDIT write lines to the AUDIT.LOG file Syntax: Call sre_audit(source_name,amessage) Example: call sre_audit('ADDON_1','Addon #1 completed at 10:30 with no errors') SRE_AUTH_RESPONSE return an authorization required response Syntax: rcode=sre_auth_response(realm_name) Example: rcode=sre_auth_response('The Protected Zone') SRE_COMMAND recieve and send information to the client, etc. Syntax: STATUS=SRE_COMMAND('ACOMMAND OPTION',amessage) ACOMMAND can be one of: ALIVE AUDIT CONTROL FILE HEADER NODATA READ REPSONSE SEND SET VAR OPTION is an option list that depends on ACOMMAND. amessage is an argument that is required for the VAR and other options Examples: rcode=sre_command('FILE type text/html name D:\WWW\HELLO.HTM') rcode=sre_command('VAR type text/plain',a_text_string) rcode=sre_command('STRING This is a simple string! ') rcode=sre_command('HEADER X-Color: Neon Green ') SRE_DATADIR return the default (or host specific) data directory Syntax: datadir=SRE_datadir() SRE_ERROR_RESPONSE return a server error response Syntax: rcode=sre_error_response(type) type can be one of the following http/1.1 error response codes:  403 404 410 413 414 500 501 503 (which respectively mean Forbidden, NotFound, Gone, LargeBody,LargeURI, ServerError, NotImplemented, and Busy) Example: rcode=sre_error_respons(403) SRE_REQFIELD returns the value of a request header Syntax: avalue=SRE_reqfield(header_name) Example: avalue=SRE_reqfield('User-Agent') SRE_SERVER return descriptive name of the server software Syntax: sname=SRE_server() SRE_SERVERNAME return the domain name of the server Syntax: sname=SRE_servername() SRE_WRITE_ERROR write a message to the error log and to the status screen Syntax: call sre_write_error(error_message) Example: call sre_write_error('Foo1: got a BAR error') SRE_WRITE_MESSAGE write a message to the status screen Syntax: call sre_write_message(short_message) Example: call sre_write_message(' Hello cruel world? ')

Appendix 1: Basic copyright and it's never our fault disclaimer

  Copyright 2000,2001,2002 by Daniel Hellerstein.

Permission to use this program for any purpose is hereby granted
without fee, provided that the author's name not be used in
advertising or publicity pertaining to distribution of the software
without specific written prior permision.

Use of this product, or portions of this product, is subject to the
following:
1)  Portions of the code are adapted from other authors' work
(these are noted where appropriate); you'll need to contact these
other authors for appropriate permissions.
2)  SRE-http uses several 3rd party dynamic libraries and executables:
i) Quercus System's REXXLIB procedure library.  The
license for REXXLIB gives the author the right to distribute
REXXLIB without charge.  This right may NOT extend to
redistributors (though as of April 2000 it appears that
REXXLIB has been released to the public domain).
ii) MKEY is freeware, and is part of a larger menuing system.
iii) FILEREXX is freeware. The author (Jeff Glatt) can be found
on comp.os.os2.rexx. However, he no longer works with os/2.

3)  We, the authors of SRE2002 and any potentially affiliated
institutions, disclaim any and all liability for damages due
to the use, misuse, or failure of the product or subsets of
the product.

*   In particular, SRE2002 and related product are NOT     *
*   guaranteed to be secure.                               *

We do design and code our product with careful attention to potential
security holes, and and we try to quickly fix any problems (or
potential problems) that may be discovered. However, SRE-http's
fundamental design precept is "open-source", with security an
important secondary consideration.

If you REQUIRE a highly secure web-server, you should
carefully review and test SRE2002. In other words,
you may need to choose a different server.

Furthermore you may also charge a reasonable re-distribution fee for
SRE2002; with the understanding that this does not remove the
work from the public domain and that the above provisos remain in effect.

THIS SOFTWARE PACKAGE IS PROVIDED "AS IS" WITHOUT EXPRESS
OR IMPLIED WARRANTY.
THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE PACKAGE,
INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS.
IN NO EVENT SHALL THE AUTHOR (Daniel Hellerstein) OR ANY PERSON OR
INSTITUTION ASSOCIATED WITH THIS PRODUCT BE LIABLE FOR ANY
SPECIAL,INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE PACKAGE.

authors for the source code (Daniel Hellerstein, danielh@crosslink.net).


Appendix 2: The SRE2002 request cache

The SRE2002 "request cache" is used as a "proxy like" mechanism for quickly returning a response. It is used for static resources that are associated with non-temporary files on your hard drive.

Basically, the request cache matches a selector to a file. If such a match exists (if a cache hit occurs), the SRE2002 will return the file to the client, without "calling the filter". By not calling the filter, hence avoiding request resolution overhead, responses can be sent substantially quicker -- 50% time savings are not uncommon.

Actually, even though the filter is not called, some processing will take place for "request cache hits". In particular:
• If- conditions will be checked
• Range, or other Instance Manipulation modifications can occur
• A few new headers may be computed (such as current date)
The point is that the request cache is not a cache of contents -- it is a cache of what file a selector points to.

Entries are automatically created in the request cache when:

1. SRE2002 used a 'FILE' mode to respond to a prior request for this selector,
2. caching is not explicitily suppressed (on a selector specific basis),
3. the file is not temporary
4. and, of course, request caching is enabled.

For more details on 'FILE' mode, see the description of the FILE option in SRE_COMMAND.

Use of the request cache does have a few drawbacks :

• The request cache requires memory resources
• The request cache should not be used with dynamic resources
• Auditing of responses resolved "from the request cache" may be incomplete.

These concerns are addressed via the use of the several REQ_CACHE* variables. With these REQ_CACHE* parameters, you can ..
• set the maximum size of the cache (a Least Recently Used algoritihim is used to remove old entries)
• set the level of auditing -- either by calling the filter anyways, or by using a "cache hit log"
• enable checking of request headers to determine whether or not the cache should be checked.
• Please see the description of these REQ_CACHE* parameters (in section 4) for the details.

Appendix 3: Running SRE2002 in loopback mode

The following description is taken from the GoServe documentation, with a few minor changes.
SRE2002 and your favorite browser can be run on a stand-alone machine that is not connected to a network, provided that TCP/IP is installed and the loopback driver is started. This is especially useful for developing Web pages offline, or for demonstrations. To do this, two additions are needed to a standard TCP/IP installation.
1. To the file called 'HOSTS' (no extension) in your \TCPIP\ETC, or in \MPTN\ETC, directory add the line:
127.0.0.1 loopy
where loopy is the name by which you want your machine to be known when using the loopback connection. This name can be a single word (e.g., loopy), or an internet-style name (e.g., loopy.my.org).

It's a good idea to have both formats, on two lines:

            127.0.0.1  loopy.my.org
127.0.0.1  loopy  

Notes:

2. In general, check the value of the ETC environment variable to find out where the HOST file should be placed.

If there is no \xxx\ETC\HOSTS file, create one.

3. If you are running OS/2 4.5 or above ..
Instead of edting HOSTS., you can use the TCPCFG configuator --
1. Run TCPCFG2 at an OS/2 command prompt
2. Select the Host Names tab
3. Click on the Hosts tab (at the bottom of the screen)
4. Add a row. For example:
           IP address: 127.0.0.1
Host Name: loopy.my.org
Alias: loopy
Comments: this is my loopback entry 
4. Before running SRE2002, execute the command:
ifconfig lo 127.0.0.1
This only needs to be run once, so can be run from STARTUP.CMD or from any command referenced in your Startup folder. Note that the second word is the lowercase of the two letters 'L' and 'O'.

If you are running OS/2 4.5 or above ..
Instead of the above, you can use the TCP/IP configurator (TCPCFG2):

5. On the 'Network' page, click on 'loopback interface',
6. then check 'Enable interface' and 'Manually, using',
7. then enter '127.0.0.1' as the 'IP address'.

Once set up, you can then connect to SRE2002 on your own machine using (for example) the URI:
http://loopy
The loopback address will be active even when connected to a network, so you can always connect to SRE2002 running on the same machine using the loopback name that you chose, provided that your browser does not have a proxy or SOCKS server enabled (the proxy won't be able to find your local loopback address).

Even if you are not connected to a network, your browser should not have a proxy or SOCKS server enabled (or it will try and use the network to find it before checking the HOSTS file).

Appendix 4: Using a fast filter

In many cases, one can divide a site's resources into "simple resources" (such as static images), and more complicated resources (such as dynamic documents, and access controlled documents). In general, simple resources can be handled by short (hence, faster) filter. Thus, throughput could be greatly improved if one could use a "simple and fast" filter for requests for these "simple" resources, and use a longer (and slower) filter for more complicated requests.

The FAST_FILTER option allows you to do this. If you specify FAST_FILTER, then SRE2002 will:

1. first call the FAST_FILTER.
2. if the fast filter is able to handle the request, SRE2002 exits (or waits for the next request on a maintained connection)
3. otherwise, SRE2002 calls the "normal" filter
This does involve a tradeoff -- slower response time for "complicated" requests (since the fast-filter has to be called first), but quicker response time for "simple" requests.

Fast-filters are essentially the same as normal filters, they are sent the same arguments, and can access the same parameters and functions. There is one difference -- the fast-filter needs to inform SRE2002 as to whether it was successful or not.

To do this, the fast filter should return either:

• 0 : not successful -- the normal filter should be called
• 1 : success. SRE2002 can close the connection, or wait for next request on a maintained connection
An example of a fast-filter, FASTFILT.RXX, is included with SRE2002.

Appendix 5: Tracking client status

When enabled (with the TRACKING_ON parameter), SRE2002 will maintain
information on the current status of each client. This information is
can be read (or modified) by using the SRE_TRACKIT procedure.

Note that a shared daemon is used to store this information. Therefore, it
may not be completely up to date -- when reading parameters, the actual
parameters are not read; rather, their values as of their most recent
update is used.

Many of the variables that are "tracked" are also available via SRE2002
"EXTRACT" procedure. However, EXTRACT is designed to be used to obtain
values of a client (where a client is synonymous with a connection) "own"
variables.

Basically, SRE_TRACKIT makes it easy to find status information for other
clients.

Currently, the following client-specific parameters are set by SRE2002.

BYTESSENT_REQ  bytes sent for this request (possibly reset if multiple
requests per connection)

requests per connection)

RECIEVE        RECIEVE can take the following values
0 - not currently sending
>0 - amount to be received
values:

REQUEST        the request (within a transaction, starting with 1)
SELECTOR       the full request selector (as recieved from the client)

SEND           Currently "sending" information to client. SEND can take
the following values:
0   not currently sending
>0   sending this many bytes
timeout  timeout occurred on most recent send
broken  client broke the connection
error   error when sending
ok    piece succesfully sent

SENDPIECES     # of pieces sent
0     -- send mode not activated
START -- send mode activated
END   -- send mode completed
number -- number of pieces sent

STATUS         Current action being undertaken.

STATUS may take the following values:
START
ERROR_CONN_CLOSE
ERROR_LIMIT
WAITING
ERROR_417 (expectation failed)
ERROR_TIMEOUT
ERROR_NO_HOST (Host: missing from http/1.1 query)
REQ_CACHE_USED (request-cache used)
PROCESS_xxx (xxx is one of the SRE_COMMAND options)
SNIPE   (S "sniping" is currently active)

TRANSACTION    the transaction

In addition, you can set your own "connection specific" values by using
SRE_TRACKIT.


Appendix 6: The HOSTINFO.CFG file

The HOSTINFO.CFG is used to define HOSTS. HOSTS are alternate domain names served by your IP address. In order to specify these hosts, SRE2002 uses the HOSTINFO.CFG file (actually, the INITHOST.RXX initialization procedure uses HOSTINFO.CFG). The HOSTINFO.CFG file has the following syntax:
   ; comment lines start with semi-colon
; Entries should be of the form: ip_address , host_nickname , host_datadir
; for example:
; 127,0.0.1 , local1 , f:\www\h1
; localhost , local1 , f:\www\h1
; www2.mysite.org , altsite ,  e:\web\altfiles

The IP_ADDRESS can be a dotted numeric ip address, a domain name, or a local name. SRE2002 will detect what "host" the client is sending the request to and compare this client-provided information to the entries in the HOSTINFO.CFG file.

This comparision almost always uses the Host: request header provided by all modern (post 2.0) browsers. Sometimes, as when you are serving multiple numeric IP addresses, the numeric IP can be used.

If the client-provided host information does NOT match any entry in the HOSTINFO.CFG file (for example, if the HOSTINFO.CFG has no entries), then the request is assumed to be to the "generic" host.

The host_nickname is a shorthand that is often used by SRE2002 filters (such as sreLite2). For example, sreLite2 can place access restrictions on selector, using both the resource name, and the host_nickname.

The host_datadir is the default data directory. Most filter will use this as the "root" of the web tree. That is, the host_datadir is used instead of the DATADIR parameter set in SRE2002.CFG. In other words, the DATADIR parameter is only used for requests to the "generic" host.

Notes:

• For more info on working with host definitions, see the manual for the SRE2PRC library.
• The sreLite2 and SREhttp/2 filters both come with on-line configuration utilities that can be used to add entries to HOSTINFO.CFG.

• Appendix 7: A short glossary

The following lists a few of the terms used in the SRE2002 documentation.
Hosts
It is possible for one server machine to handle requests to multiple IP addresses. This "multiple hosting" (also refered to as multi-homing) can occur either by having the server handle several numeric IP addresses, or by the existence of multiple aliases for a single numeric IP address.
Aliases:
A given IP address can be used by several hosts: including the host associated with the "canonical" name, and hosts associated with one of several aliases. Unfortunately, TCP/IP requests arrive with only the numeric IP address, making it difficult for the server to determine to which host the request is directed to. However, almost all browsers (such as Netscape 2.0 and above) include a HOST: request header containing the host to whom this request is directed. SRE2002 (as an http/1.1 compliant server) can use this request header to determine the proper host.
Host nickname
SRE2002 uses host nicknames as a shorthand for identifying when identifying hosts. For example, in the sreLite2 filter you can define parameters to apply to requests to selected hosts. Furthermore, the use of host nicknames allows several hosts to point to identical resources (given that each host has been assigned the same host nickname).

Request string
When a client asks for a URI (say, by clicking on a link), a request string (along with request headers and a request body) is sent to the server (at the domain name and port listed in the URI). The request string consists of three tokens: the http method, the location of the resource on the server, and the http protocol.
 Examples: GET /dir1/sample.htm HTTP/1.0 GET /prices?type=mammal&class=retail HTTP/1.0

Selector (also referred to as the request selector )
The selector is the location of the resource on the server. Specifically, it's the slightly modified second token in the request string.

 Examples (assuming the above request strings): dir1/sample.htm prices?type=mammal&class=retail

Note that the "slight modifications" consists of decoding URL-encoded characters (note that the above examples do not include any URL-encoded characters).

In many cases, the connection between the selector and server resource it refers to is simple (such as when the selector is a filename relative to the SRE2002 data directory). In other cases, it is completely virtual (as when the resource consists of the output of a program that's run using information provided by the client).

 Technical Note: RFC2616 (the http/1.1 spec) defines an HTTP request URL as:   http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]] . Basically, the selector is the same thing as the abs_path [ "?" query ]

Server Response
When a server returns an http response to a client's http request, it can consist of three components:
1. A response line
2. A set of response headers
3. The message body

The response line (which is always included) contains status information (such as whether the request could not be satisfied, or if authorization is required. The typical response line, for a satisfied request, is:
HTTP/1.1 200 Ok

The response headers contain meta-information on the response. These includes time and date information, caching information, server identifiers, and cookie definitions.
 Response headers are divided into three categories:

The message body is the html-file, image, or other resource the client requested.

URI and URL
Universal Resource Indicators (URI), which is the technical term for the more commonly used Universal Resource Locators (URL), is a scheme for specifying Internet resources using a single line of printable ASCII characters. A URI should contain a protocol, domain name, port number (optional), the location of the resource, and an (optional) option list (following a ?).
 Examples: http://your.server.net/dir1/sample.htm http://pet.store.com/prices?type=mammals&class=retail

Appendix 8: SRE2002 Features

SRE2002 provides a suite of capabilities that facilitate the creation of web servers. The following lists some of SRE2002's more important features:
• Supports multiple hosts
• Supports maintained connections (multiple requests over a single connection)
• Supports chunking (as a transaction encoding)
• Supports on-the-fly GZIP compresssion (as a transfer encoding)
• Supports automatic generation of ETAG and Content-MD5 response headers.
• Automatic generation of conditional (3xx) returns (which can be suppressed).
• Automatic range extraction (which can be suppressed).
• A variety of optional procedures can be invoked while processing a request:
1. Pre-transaction procedures (before any processing is done)
2. Pre-request procedures (after reading the client's message, but before resolving the request)
3. instance manipulation (after the contents have been determined)
• A task manager, with scheduling options, is built into SRE2002
• A simple database system, with caching features, is built into SRE2002
• Basic statistics include average response time, total number of transactions, and total bytes of throughput.
• Optional tracking of the current state of all requests
• Snipe individual connections without shutting the server down
• Optional auditing of requests, responses, errors, and other events.
• Time and size limits can be set for all requests, and can be changed on a request specific basis
• A proxy-like request cache is built into SRE2002.
• A list of unallowed IP addresses (that are immediately disconnected) can be specified
• Supports multi-part, and multi-piece, responses
• SRE2002 contains a number of functions, including functions to:
• Extract request headers by name
• Easily generate authorization and error responses
• Support basic and digest authentication
• Return a file, or the contents of a variable, to the client
• Wildcard match one string to one (or several) other strings
• Simplify the creation of, and communication between, semi-permanent daemons