20 January 1999.
Working with SRE-http daemons.
I. Introduction.
When creating web-aware applications for SRE-http, the use of daemons can
greatly improve performance. Daemons, defined as threads that
are always resident in memory, are used extensively in SRE-http.
In general, given the "seperate transient thread per request" nature of
GoServe/SRE-http, whenever the server needs to access "semi-permanent"
information a daemon is usually there to help. For example; looking up usernames,
taking care of virtual directory resolution, and recording to the common log audit
files are all handled by seperate daemons.
Although these functions could be performed using external REXX procedures, and
the OS/2 environment, daemons have certain advantages. A minor advantage is that
load time is avoided (a small concern when external procedures are stored in REXX
macrospace). A more important feature is that daemons can retain information in
memory, avoiding the need to reload configuration information and other data
files. Furthermore, this information can be retained in useful formats; in
particular, as stem variables. By combining this data storage capability with
search/sort/etc. code; it becomes relatively easy to create dynamic storage
"objects" with auto-refresh, lookup, and other database features.
Although potentially useful, daemons are somewhat difficult to work with. OS/2
does not offer easy ways of transferring information between daemons; one
doesn't call a daemon the same way one would call a procedure stored in a
library. Although it's not conceptually difficult to devise ways of overcoming
this drawback, the actual implementation of robust mechanisms can become
tedious.
In order to overcome this drawback, SRE-http comes with SREF_DMN_, a set of
daemon manipulation routines. By using these procedures in both the
daemon, and in the programs that rely upon the daemon, much of the hassle of
using daemons can be avoided. The remainder of this document discusses these
procedures.
II. The SREF_DMN_ procedures.
To create and use daemons, four actions are required:
1) Launching the daemon
2) Sending information to the daemon, and recieving a reply
3) Reading information into the daemon, and sending replies
4) Closing the daemon
Note that #1 and #4 are one time affairs; with #4 often never implemented.
Action #2 is associated with a program (i.e.; an addon) requiring some
information. Action #3 is associated with a daemon waiting till it's needed.
SRE-http (as of version 1.2N) includes a family of "SREF_DMN_"
procedures that help implement these various actions. The following lists
these procedures, with more detailed descriptions below.
status=SREF_DMN_LAUNCH(daemon_name,file_name,optional_arg,logfile,errpre,verbose)
status=SREF_DMN_EXIST(daemon_name)
value=SREF_DMN_ASK(daemon_name,an_argument,max_wait)
request_info=SREF_DMN_WAIT(daemon_name,max_wait)
an_argument=SREF_DMN_VALUE(request_info)
status=SREF_DMN_REPLY(request_info,reply_stuff)
status=SREF_DMN_KILL(daemon_name)
Basically ...
SREF_DMN_LAUNCH is called (often by a stand-alone program)
to "launch" the daemon. SREF_DMN_ASK is used to read & supply
information to the daemon, and is often called from an
SRE-http addon. SREF_DMN_WAIT, SREF_DMN_VALUE, and SREF_DMN_REPLY
are used by the daemon to return information (typically to
an addon).
-------------------
SREF_DMN_LAUNCH(daemon_name,file_name,optional_arg,logfile,errpre,verbose)
where:
daemon_name: A name that will be used to label this daemon. It is only used by
the SREF_DMN_ procedures
file_name: The file containing the program to launch as the daemon_name
daemon. If not a fully qualified file name, it is
assumed to be under SRE-http's addon directory.
optional_arg: An optional argument that will be sent to the daemon upon
launching.
logfile: An optional log file that will be used for recording errors (errors
will also be written to the PMPRINTF window). This should be
a fully qualified file name.
errpre: An optional "error prefix". If this is specified, when
SREF_DMN_ASK encounters an error, it will prepend this
string to an error message, and return it. If this is
not specified, SREF_DMN_ASK will return a null string.
verbose: Verbosity of status messages: 1 to 4, 1 is a few, 4 is too many.
If verbose not specified, the value of the SRE_VERBOSE
variable is used (SRE_VERBOSE is set by versions 1.3b.0199.c and
above of SRE-http).
Returns:
n = if n > 0, then success; with n the "thread" the daemon
was launched on
n message = n<0 is an error code, followed by an error message
SREF_DMN_LAUNCH will take the program (typically, a REXX program) in file_name
and launch it in its own thread. It will also create some necessary "queues
and semaphores" that will be used by the SREF_DMN_ procedures.
Note that the daemon will remain active so long as the process that called
SREF_DMN_LAUNCH is still alive. If you use SRE-http to do this launching (say,
with a CUSTOM_INITS, or by requesting your own "launch my daemon" addon), the
"process" is GoServe; hence the daemon will stay live as long as your server is
up and running.
Although launching the daemon through SRE-http is convenient, if you intend to
"kill the daemon" without killing GoServe/SRE-http, it is often safer to run
your "launch my daemon" program in a seperate process that you keep open as long
as needed.
When launching a daemon (in its own thread), SREF_DMN_LAUNCH will supply two
arguments: the daemon_name and the value of optional_args. The program can use
these as it sees fit.
CAUTION:
Some users report that if you launch several daemons in rapid succession,
failures may occur (that is, the daemon might "disappear"). This seems
to be cured by inserting a few seconds delay (i.e.; using call syssleep(3))
right after each call do SREF_DMN_LAUNCH.
-------------------
SREF_DMN_EXIST(daemon_name)
where:
daemon_name: As above
returns:
1 = Semaphore and queue exist
Otherwise, an error code. Common error codes are:
0 = Queue does not exist
-187 = Semaphore does not exist
SREF_DMN_EXIST checks whether the deamon's queue and semaphore are open.
If not, either the deamon was never launched, or something caused its
semaphore or queue to be closed. In either case, you can't talk to it!
-------------------
SREF_DMN_ASK(daemon_name,an_argument,max_wait,semqueue)
where:
daemon_name : The daemon_name used in SREF_DMN_LAUNCH
an_argument: An argument to send to this daemon. It can be
any length, and can be text or binary.
max_wait: Optional. Maximum number of seconds to wait. If not
specified, SREF_DMN_ASK will wait forever.
semqueue: Optional. Semaphore/queue information
returns:
null string: A null string is returned if a time out occured, or if some
kind of error occured. You can examine the
use the logfile (described below) to view
additional information on the error.
Or -- if you've specified an "error_prefix" in
DMN_LAUNCH -- an error message will be returned
instead of the nulll string.
For example, if errpre="ERROR:" then an error
return could look like:
ERROR: SREF_DMN_ASK: daemon semaphore missing:
otherwise: The "reply' returned by the daemon (see SREF_DMN_REPLY)
SREF_DMN_ASK will "send an_argument" to the daemon_name daemon,
and wait for a reply. You can think of SREF_DMN_ASK as a way of treating
daemons as if they were procedures that can take one argument. If you
want to send mulitple arguments to your daemon, you'll have to concatenate
them into an_argument, and have the daemon parse them out.
Note that the daemon might return a null string; which is
indistinguishable from the default "error return". If this might
be a problem, you should specify an the errpre option SREF_DMN_LAUNCH,
and then check for this "error_prefix" upon returning from SREF_DMH_ASK.
Semqueue is strictly optional -- if specified, it should be the value
of the semqueue variable that is supplied to SRE-http addons.
If supplied, semqueue will be used to specify a semaphore and queue
to use to talk to the daemon. If not specified SREF_DMN_ASK will create
a thread specific semaphore and queue. The only advantage gained by
providing semqueue is a slight improvement in throughput
(since the overhead of creating a queue and semaphore is avoided).
Special Feature:
Setting MAX_WAIT='NOWAIT' will cause the an_argument
to be added to the queue, and nothing else (a '' is
returned immediately)
Setting MAX_WAIT='CLEAR' will also add an_argument to the daemon's
queue and not wait for a return. It will ALSO clear this
queue BEFORE adding an_argument.
Note: if an error is encountred, SREF_DMN_ASK will write a note to
the PMPRINTF window. In addition, SREF_DMN_ASK will write a
note to a logfile. If you specified a logfile in SREF_DMN_LAUNCH,
that's the logfile it will used. Otherwise, it will use
DMN_daemon_name.LOG (in the current directory, wherever that
may be).
In addition, if SREF_DMN_ASK is running under a GoServe/SRE-http process,
then an error message is recorded using the SREF_ERROR procedure (see
SREFPRC.DOC for details).
-------------------
SREF_DMN_WAIT(daemon_name,max_wait)
where:
daemon_name: As above
max_wait: Optional. Maximum number of seconds to wait.
If not specified, wait forever.
returns one of:
i) null string : If no request was sent to the daemon within max_wait seconds.
ii) a string starting with ERROR, followd by an error message. For example:
ERROR Could not find daemon_name.
iii) request_info: A package of request information.
SREF_DMN_WAIT will wait for a request to the daemon_name daemon;
typically one that's sent by an SRE-http addon that calls
SREF_DMN_ASK. If a request is recieved, then a bundle of information is
returned.
** This bundle is NOT meant to be directly used.
** Instead, you should use SREF_DMN_VALUE to extract the an_argument
** value from it.
Example: request_info=SREF_DMN_WAIT('MyDaemon',100)
Special Feature:
Setting MAX_WAIT='CLEAR' will cause the daemon's queue
to be cleared; with the last written entry returned.
-------------------
SREF_DMN_VALUE(request_info)
where:
request_info: A bundle of request information, as returned by SREF_DMN_WAIT
returns:
an_argument: The "argument" being sent to the daemon. This will be
the value of an_argument used in a call to SREF_DMN_ASK.
SREF_DMN_VALUE is used to extract information from the request sent to this
daemon. You MUST provide it with a value returned by SREF_DMN_WAIT --
if some other string is provided, a null string will be returned. Furthermore,
if an ERROR response was returned to SREF_DMN_WAIT, then SREF_DMN_VALUE
will return a null string.
-------------------
SREF_DMN_REPLY(request_info,reply_stuff)
where:
request_info: the request_info returned by SREF_DMN_WAIT
reply_stuff: The information you want to return to the request. This is the value
that SREF_DMN_ASK will return -- it can be a text or binary
string of any size.
returns one of:
i) 1 (success)
ii) an error message
SREF_DMN_REPLY is the counterpart to SREF_DMN_ASK -- it returns information
to the requesting thread. As with SREF_DMN_VALUE, the request_info must be
the value returned by SREF_DMN_WAIT -- request_info contains the "name"
of the requesting thread, without which the daemon has no idea of who
to respond to.
-------------------
SREF_DMN_KILL(daemon_name)
where:
daemon_name : as above
returns:
1= success
otherwise = an error message
Kills the daemon launched by SREF_DMN_LAUNCH. This is an optional
procedure, and is included to help clean up resources.
Unfortunately, REXX's ability to remove the queues and semaphores created
by SREF_DMN_LAUNCH is somewhat flakey. It is generally safer
to simply kill the process that launched the daemon. If you launched the
daemon through GoServe/SRE-http, that can be a problem; since you'ld
have to restart your server.
Hence, in cases where you expect to be killing (and perhaps relaunching) the
daemon frequently, we recommend issuing SREF_DMN_LAUNCH from a seperate
process, and stopping/restarting the process when you want to kill/re-launch
the daemon. That is, don't bother using SREF_DMN_KILL -- stop the
process instead.
-------------------
III. Example.
Let's assume you want to create a daemon called RUNNING_SUM. It will take
a number and a name, add it to a running sum specific to that name, and return
the new value of this running sum. Such a procedure might be useful when
keeping track of resource utilization; say of some fancy addon that you
wish to make available on a limited basis.
The following code can be used to implement such a daemon. To try it out,
you'll need to create the following files:
RUNSUM0.CMD -- A simple rexx program to launch the daemon. Can be
placed anywhere, such as in the GoServe working directory.
RUNSUM1.RXX -- The daemon itself -- put it in the Goserve working directory.
RUNSUM.CMD -- An addon that ust counts how many times you've called RUNSUM.
Put it in the addon directory.
RUNSUM.HTM -- Requests RUNSUM.CMD. Put it in your GoServe data directory
(the root of your web tree).
You can create these files by cutting and pasting the following code -- check
out the embedded comments for further details. And, when creating
the .RXX and .CMD files, be sure that the FIRST LINE IS A COMMENT
(that is, the first 2 characters should be /*)
------------------------------------------------------
------ RUNSUM0.CMD. Put in \GOSERVE directory ----------
------------------------------------------------------
/* RUNSUM0.CMD: Launch the running-sum daemon.
This program should be run
from a seperate OS/2 box (or, you could start it by using SRE-http's
CUSTOM_INITS parameter)
*/
call initstuf /* check to see if sref_dmn_ procedures are in macrospace */
say "Note: you can use PMPRINTF to see daemon status messages"
say
/* RUNSUM.STM will be used to store running sums,
the 1 signals "verbose" mode */
astat=SREF_DMN_LAUNCH('RUNNING_SUM','RUNSUM.RXX','RUNSUM.STM 1',,'ErRor:')
if word(astat,1)>0 then do
say " Running Sum daemon successfully launched on thread "astat
say " To kill the daemon, close this OS/2 box. "
say " To let it run, do NOT close this OS/2 box."
end
else do
say " Could not launch daemon. "
say " Error message: "astat
end
exit
/* check initialization stuff */
initstuf:procedure
/* Load REXX libraries */
/* Load up advanced REXX functions */
foo=rxfuncquery('sysloadfuncs')
if foo=1 then do
call RxFuncAdd 'SysLoadFuncs', 'RexxUtil', 'SysLoadFuncs'
call SysLoadFuncs
end
foo=rxfuncquery('rexxlibregister')
if foo=1 then do
call rxfuncadd 'rexxlibregister','rexxlib', 'rexxlibregister'
call rexxlibregister
end
foo=macroquery('SREF_VERSION')
if foo="" then do
say " SRE-http has not been initialized. Perhaps you need to "
say " hit your server with its first request? "
exit
end
return 1
/* END OF RUNSUM0.CMD */
------------------------------------------------------
------ RUNSUM.RXX. Put in \GOSERVE directory ----------
------------------------------------------------------
/*RUNSUM.RXX : The running sum daemon */
parse arg da_name,old_file
parse var old_file oldfile verbose
signal on syntax name goterr ; signal on error name goterr
call pmprintf('RUNSUM.RXX launched as daemon: 'da_name)
/* read old information that's been stored in old_file (if specified) */
if anarg<>' ' then do
if stream(oldfile,'c','query exists')="" then do
call pmprintf('RUNSUM.RXX: Warning: 'oldfile' not found.')
end
else do
foo=cvread(oldfile,'sums')
if foo=0 then do
call pmprintf('RUNSUM.RXX: Warning: could not read 'oldfile'. Result will NOT be saved.')
oldfile="" /* suppress attempts at writing to oldfile */
end
else do
oof=cvtails(sums,'tsums')
call pmprintf(' RUNSUM.RXX : Using 'oof' entries stored in: 'oldfile)
end
end
end
didchange=0 /* flag indicating that A query was answered */
do forever
req_info=SREF_DMN_WAIT('RUNNING_SUM',30) /*wait up to 30 seconds for a request */
/* 1) check for error, and pmprintf it if found. Note that SAY does not work from
within daemons-- hence the use of pmprintf */
if abbrev(req_info,'ERROR')=1 then do /* errror occurred */
parse var req_info . errmess
call pmprintf(' RunningSum Daemon error: '||errmess)
iterate /* assume it was fluke, and try again */
end
if verbose>0 then do
call pmprintf(' RUNNING_SUM Daemon: request length='length(req_info))
end
/* 2) timed out? Take this opportunity to write SUMS. to anarg (if
there have been any changes*/
if req_info="" then do
if didchange>0 & oldfile<>"" then do
didchange=0
foo=cvwrite(oldfile,'SUMS')
if verbose>0 then call pmprintf(' RUNSUM.RXX :Wrote running sums to 'oldfile)
end
iterate
end
/* 3) otherwise, we got legit request. Extract the useful information */
stuff=SREF_DMN_VALUE(req_info)
/* check for error again -- this shouldn't happen, but ... */
if stuff="" then do
call pmprintf(' Running sum error. No information provided ' )
iterate
end
parse upper var stuff aname addvalue /* parse out name and number */
/* explicitily save? Might be needed on busy sites (where 30 second timeout
never happens */
select
when strip(aname)='!SAVE' then do /* save current values */
foo=cvwrite(oldfile,'SUMS')
writeme=' Saved running sums to 'oldfile
end
when strip(aname)='*' then do /* display all current sums */
foo=cvtails('SUMS','TSUMS')
nms=''
do mm=1 to foo
aa=tsums.mm
aa2=strip(aa,'l','!')
aa3=sums.aa
nms=nms||aa2'='aa3'0d0a'x
end
writeme=nms
end
otherwise do /* it's a value, add and save it */
if datatype(addvalue)<>'NUM' then addvalue=0 /* bad input, assume number=0 */
/* using stem variables, get the current running sum for aname. If this is first
value for aname, then initialize a stem variable */
tmp='!'||aname
if symbol('SUMS.'||tmp)<>'VAR' then do
rsum=0
end
else do
rsum=sums.tmp
end
sums.tmp=rsum+addvalue /* add to the running sum */
didchange=1+didchange
writeme=sums.tmp
end
end
astat=SREF_DMN_REPLY(req_info,writeme) /* return the reply */
/* check for problems */
if astat<>1 then
call pmprintf('RunningSum error. Could not reply: 'astat) /* note that an error occured */
end /* wait for next request */
goterr: /* jump here on error */
call pmprintf(' Running Sum Daemon. Error at line ' sigl', rc= 'rc)
exit
/* End of RUNSUM.RXX */
------------------------------------------------------
------ RUNSUM.CMD. Put in \GOSERVE\ADDON directory ----------
------------------------------------------------------
/* RUNSUM.CMD : A simple SRE-http addon to demonstrate the use of the
RUNNING SUM daemon.
To use this, you must first run RUNSUM1.CMD (either as a CUSTOM_INITS, or
preferably in its own process).
Requests for RUNSUM.CMD should look like:
RUNSUM?name=name&value=value
For an example of how to use this, see RUNSUM.HTM
*/
/* get request info supplied by SRE-http. For this simple example, we don't
bother parse arg'ing all the arguments. */
parse arg ddir, tempfile, reqstrg,list
if list="" then do
say " This SRE-http addon is NOT meant to be run from a command prompt."
exit
end /* do */
/* start writing response to client */
call lineout tempfile,' Running Sum '
/* extract NAME and VALUE from LIST (list is the stuff following a ? in a GET request) */
aname="" ; addme=0 /* defaults */
do until list=""
parse var list a1 '&' list
parse upper var a1 avar '=' aval
aval=strip(packur(translate(aval,' ','+')))
select
when avar='NAME' then aname=aval
when avar='VALUE' then addme=aval
otherwise nop
end /* select */
end
/* no name given? Send an error response to the client */
if aname="" then do
call lineout tempfile,' Sorry, you did not specify a name.'
call lineout tempfile,' '
call lineout tempfile /* make sure you close the response file! */
'FILE erase type text/html name ' tempfile
return 'No name specified '
end
/* call the daemon and wait for up to 30 seconds */
tmpval=aname' 'addme
stuff=SREF_DMN_ASK('RUNNING_SUM',tmpval,30)
if stuff="" then do /* possible error message */
if errmess="" then errmess="Timed out. "
call lineout tempfile,' Error in RUNNING_SUM daemon: 'errmess
call lineout tempfile,'
Did you remember to first run RUNSUM0?'
end
else do /* otherwise, return the current running sum */
select
when aname='*' then do
call lineout tempfile,'List of current values
'
call lineout tempfile,stuff
call lineout tempfile,''
end /* do */
when aname='!SAVE' then do
call lineout tempfile,''stuff
end /* do */
otherwise do
call lineout tempfile,'
Running sum value
'
call lineout tempfile,' The current running sum for 'aname' = ' stuff ' '
end
end
end
/*return the response to the client */
call lineout tempfile,'