25 Feb 1999 Daniel Hellerstein ((danielh@econ.ag.gov) CHECK_404 -- An addon for the SRE-http web server. CHECK_404 is an addon for the SRE-http web server. It's function is to detect "repetitive" requests for non-existent URLs from a client (that is, requests that yield a 404 response code) . Such events may be due to a probing attack by a malfeasant hacker attempting to find password and other such files. CHECK_404 provides a means for detecting, and appropriately dealing with, this situaion. CHECK_404 consists of 3 components: a) A "daemon" that keeps track of the number of "404" responses issued to clients. b) A "pre-filter" that uses this daemon to identify suspicious clients (defined as clients who have recently requested many non-existent resources), and which will deny service to such clients. c) A "post-filter" that tells the daemon when a "404" response has been issued. II. Installation First, CHECK_404 requires versions 1.3b.0199.c and above of SRE-http. If you have an earlier version of SRE-http, CHECK_404 will not work properly. Assuming that you are up to date with SRE-http, installation of CHECK_404 is simple: a) Unzip CHECK_404 to your GoServe directory. The following files will be created: CHK404.RXX -- the "daemon" CHK_INIT.CMD -- the "daemon launcher" CHK_PRE.CMD -- The "pre-filter" CHK_POST.CMD -- the "post-filter" CHK_SAMP.CMD -- a sample "user specified" procedure CHK_PENL.CMD -- the "penalty" addon. In addition, this file (CHK404.DOC) will be created. You should copy CHK_PENL.CMD to the goserve\addon\ directory (see section V for details). b) Edit INIT_STA.80, and include CHK_INIT.CMD as one of the CUSTOM_INITS procedures. For example, set: CUSTOM_INITS='CHK_INIT.CMD ' Actually, you can (optionally) specify several parameters in CUSTOM_INITS which are used to set thresholds used by CHECK_404 -- see section III for details. c) Edit INITFILT.80, and include CHK_PRE.CMD as one of the pre-filters. For example, set: PRE_FILTER='YES' PREFILTER_NAME='CHK_PRE.CMD' You should also set a few parameters in CHK_PRE.CMD (an optional message string, and the name of a custom procedure to call when a request is denied) -- see section IV for details. d) Edit INITFILT.80 and include CHK_POST.CMD as one of the post-filters For example, set: POST_FILTER='YES' POSTFILTER_NAME='CHK_POST.CMD' That's it -- restart GoServe/SRE, and CHECK_404 will be on the job. II. How does CHECK_404 work. The basic algorithim used by CHECK_404 is: a) When SRE is initialized, the CHECK_404 daemon is initialized (that's what CHK_INIT.CMD does). This step can be customized -- you can set the threshold number of "404" responses, and the timespan to consider when counting these responses, and a few other variables (see section III for details). b) On all requests, SRE call this daemon and asks "has this client been sent too many 404 responses in the last several minutes". If this threshold has been exceeded, the daemon informs SRE of this sad fact, and SRE will immediately deny service to this client. This denial of service consists of (see section IV for details): a) a simple 404 response is sent to the client (that uses the "optional message string") b) the "custom procedure" is called (if one has been specified) c) Assuming b is not binding, if the request is a "404" request (it can not be resolved), SRE will do it's usual thing (such as use MTCHFILE to find possible alternatives). Then, the post-filter will inform the daemon that "this client has just been sent a 404 response". If the response was NOT a 404 response, the post-filter does nothing. In short, CHECK_404 monitors every request, keeping track of the "404 response" status on a client specific basis. This does add extra overhead, and may slow down throughput. III. Launching the daemon The simplest way of invoking the "initialize the daemon" procedure is to specify: CUSTOM_INITS='CHK_INIT ' You can also set several parameters when launching CHK_INIT. To do this, just enter the parameters immediately after the CHK_INIT. The full syntax is: CUSTOM_INITS='CHK_INIT port max404 duration keepout verbose' Where: port == the http port you are using (default is 80). max404 == threshold number of 404 responses (default=8) duration == duration (in minutes) this threshold is computed over (default=5) keepout == duration (in minutes) to bar a client if a max404 violation occurs (default=15) verbose == status message verbosity; 1=some, 4=too much (Default= value of the sref_verbose environment variable) Note that to specify a latter parameter (such as max404) you must also specify the port. If a parameters is unspecified, the default values (as noted below) will be used. Basically, if more then max404 "404" (resource not found) responses have been sent to an ip address in the last duration minutes, then this client is assumed to be an attacker. This condition will be assumed for the next keepout minutes; after which the slate is wiped clean (requests from the client will be treated as innocent). Examples: a) Use the default: CUSTOM_INITS='CHK_INIT ' b) The port is 8080, and use the defaults CUSTOM_INITS='CHK_INIT 8080 ' c) The port is the default (80), but set max404=10 and duration=20 (and use the default value of keepout) CUSTOM_INITS='CHK_INIT 80 10 20 ' d) Similar to c, but specify a 40 minute "keepout", and be verbose with the status messages CUSTOM_INITS='CHK_INIT 80 10 20 40 3' IV. Specifying response options CHK_PRE.CMD contains two parameters that control CHECK_404's response to a "repetitive 404" attack. CUSTOM_PROC: Name of custom procedure to call -- it must be in macrospace, or a file in the goserve directory. If you do NOT want to call a custom procedure, set custom_proc='' Note that this custom_proc will be called with 3 arguments: the client's ip address, the request string, and the servername. Example: custom_proc='CHK_SAMP.cmd' Note that CHK_SAMP.CMD is a simple example. MESSAGE_404: Message to use when responding to a request from an attacker (this will be included in a 404 response). You can leave this blank if you don't want to say anything. Example: message_404='Resource not available' V. The penalty addon Some resources are obviously off-limits, and clients who ask for it are quite likely to be ill-intentioned. You can use the "penalty addon" (CHK_PENL.CMD) to "penalize" such clients. In particular, one would create aliases (either by modifying ALIASES.IN, or by using the configurator) that identify resources that are liable to be asked for by attackers, and assign such requests a "404 request penalty". That is, each such request can be treated as a fixed number of 404 responses. One could be stringent, and set this penalty to be greater then the MAX404 threshold, or one could be more tolerant and set the penalty to one or two. Assuming you can use your favorite text editor to modify ALIASES.IN, you should create entries of the form: sel CHK_PENL?penalty&amessage where sel : a "suspicious" selector penalty: an integer number amessage: an optional, url encoded message For example: password.lst CHK_PENL?1000 config/* CHK_PENL?2 users.lst CHK_PENL?5&Unknown+resource In the first example, a request for PASSWORD.LST is treated harshly (since 1000 is likely to be greater then any threshold you might set). In the second example, request for anything in the CONFIG directory (of your GoServe data directory) are not permitted, and cause a 2 "hit" penalty. The message parameter (as used in the 3rd example) is strictly optional -- it is added to the 404 response sent to the client. Note that this must be url encoded (i.e.; use + for spaces).