SRE-DATA: A Database Addon for SRE-http

Features of SRE-DATA

• Runs as a thread, or several seperate threads for several seperate views of several different databases.
• Data retrieval on moderate (say, 100k) databases takes a fraction of a second.
• Wildcard searches can be taken against selected variables (that you define on your data).
• Works with ASCII text, or fixed-record binary, data.
• Text data can have:
  variables delimited with any (database specific) delimiters,
  multiple lines per record,
  fixed-position variables, and
  freely interspersed comment lines.
• Searches can be numeric, text based (with or without case sensitivity), or exact (binary match).
• Can be automatically started when you start GoServe/SRE-http.
• A straightforward "macrospace" procedure is provided to facilitate programmer's access to SRE-DATA (you can even access SRE-Data independently of GoServe/SRE-http).

Table of Contents

• Installation Notes
• Launching SRE-Data
• Outline of SRE-Data
• Creating PROGram files
• Creating KEY and STEM files
• Calling SRE-Data from your own REXX scripts
• Launch an SRE-Data Daemon

1. UNZIP SREDATA.ZIP to an empty, temporary directory
2. Run the INSTALL.CMD program (from an OS/2 command prompt).

1. Copy SREDATAX.CMD, SREDATA.CMD, SREDATA.RXX,QSORT.EXE, and SREDATA.SRF to your GoServe working directory (i.e.; D:\GOSERVE)
2. Copy SREDATA.HTM (this file!) and DATADEMO.HTM to one of your WWW directories.
3. If you want to play with the demo files:
   • Copy DATADEMO.CMD to your SRE-http ADDON directory.
   • Create a SREDATA directory under your GoServe working directory,
   • Copy INITFILT.IN, FIPSNAME.IN, and FIPSNAME.DAT to this SREDATA directory.
   • Copy DEM1SRED.CMD and DEM2SRED.CMD to your GoServe working directory.

Launching SRE-Data

New user? You might want to first read the Outline of SRE-Data.

1. CUSTOM_INITS: You can use CUSTOM_INITS, a parameter in the SREFILTR.80 file, to launch an SRE-Data daemon when GoServe/SRE-http starts up (you'll need to edit SREFILTR.80 to change the value of CUSTOM_INITS).
   Advantages: Automated
   Disadvantages: Somewhat more difficult to manage; and to start new SRE-Data daemons with CUSTOM_INITS, you have to restart GoServe.

   Examples:
   CUSTOM_INITS='SREDATA NAME=FIP PROG=SREDATA\FIPNAME.IN ;  SREDATA NAME=FIP key=DAT1\FIP.Key"
   CUSTOM_INITS=' PRELOAD ; SREDATA Name=INIT1 STEM=SMP\INIT1.stm ; " 

2. Seperate session: (recommended) Run SREDATA.CMD from a seperate OS/2 session.
   Advantages It's easy to start and stop SRE-Data daemons on an as-needed basis; you can also establish a very simple daemon monitor.
   Disadvantages: Each daemon requires a seperate process, which must be kept open (closing the process kills the daemon)

   Example: D:\GOSERVE>sredata name=fip prog=sredata\fipaname.in monitor=1

   
   
3. External procedure: You can launch an SRE-Data by calling SREDATA as an external procedure (the DATADEMO.CMD addon, which is used by the launcher facility, contains an example of how to do this).
   Advantages: Fairly easy to start and stop SRE-Data daemons
   Disadvantages: Monitoring can not be done. Although you can use DATADEMO, you might want to write your own "launcher".

   Example:status=sredata('FIP','PROG','SREDATA\FIPNAME.IN')

   When called as an external procedure, SREDATA will return a 1 if the daemon was launched, and a 0 if an error was detected (you can use PMPRINTF to see what errors occurred). Note that a status of 1 does not guarantee a successful launch, since other errors may have occured in the daemon code (in SREDATA.RXX). Again, keep an eye on PMPRINTF.

An Implementation Note

Description of SREDATA options

When called with CUSTOM_INITS, the syntax is:
     SREDATA  NAME=some_name  type=filename
When called  from the command line, the syntax is:
     SREDATA  NAME=some_name  type=filename MONITOR=0/1
When called as a function, the syntax is:
     status=SREDATA('some_name',type,filename)
                or
    status=SREDATA('some_name,'STRING',astring)
where:
    some_name : An arbitrary name (chosen by you)
          type: One of PROG,DATA, STEM, KEY, LOADKEY, STRING
      filename: A file name containing data or instructions
       astring: A string containing instructions 

• type: type is one of: PROG, DATA, STEM, KEY, LOADKEY, or STRING.

  PROG	"program" for reading a data file. PROG files will contain various options, and must contain a DATA statement (to identify the data file)
  DATA	a data file (the file is read using defaults that a PROG can override)
  STEM	a "stem" file
  KEY	a "key" file
  LOADKEY	a "key" file, to be loaded into memory
  STRING	a character string, that contains a PROGram.
  Notes	STRING can only be used when SREDATA is called as a procedure.
  	The STEM, KEY, and LOADKEY types require files that are produced by the SREDATAX program.

• some_name: some_name is a "handle" you'll use to query the daemon
  If NAME= is not specified, then a "name" of INDEX is assumed.
• filename:filename is the name of a datafile (if type is KEY, STEM, or DATA) or a PROGram file (if type is PROG). Note that relative file names are assumed to be relative to the GoServe working directory (i.e.; d:\goserve)
  
• astring: string should be a string that contains a list of PROGram instructions. That is, it should have the exact same format as PROGram file; with "CRLFs" seperating each option. Note that the string option is only used when SREDATA is called as a procedure.
  
• monitor: monitor, when set to 1, tells SREDATA to start a simple daemon monitor. When set to 0 (or when not specified), SREDATA will die after launching the daemon. The major purpose of the monitor is to provide visual evidence that things are happening (since closing the process that launched the daemon will also close the daemon).

  monitor=1 should NEVER BE USED WITH CUSTOM_INITS

Technical Note

1. If SREF_FIND_INDEX does not exist in macrospace, it will load it (using the SREDATA.SRF file). Note that SRE-http version 1.2L and above comes with SREF_FIND_INDEX preloaded into macrospace; for earlier verions, SRE-Data will load a local copy (contained in SREDATA.SRF).
2. It will then create a queue and a semaphore using the NAME argument (or using a value of INDEX, if no NAME was specified).
3. It will launch SREDATA.RXX as a thread, and give it several arguments.
4. If MONITOR=1 is not specified, it will quit. Otherwise, a simple "monitoring" routine will be run. In either case, SREDATA.RXX will continue (as a daemon).

The real work is done by SREDATA.RXX -- the job of SREDATA.CMD is to launch the SREDATA.RXX as a daemon (in it's own thread), and to make sure that SREF_FIND_INDEX is in REXX macrospace.

SRE-Data Reads Flat Databases

Each record (in a flat database) is expected to contain two types of information:

1. An identifier variable
2. Other information variables

The optional information variables contain any information; text, numbers, or binary representations. Your database can contain any number of information variables, though in practice SRE-Data is designed to work with a relatively small number of information variables.

In some cases you may not have any information variables. For example, you may have a large set of data with a single index file; and the position of an identifier in the index is used to access these other data. In such a case, it can be a great efficiency to use SRE-Data to quickly find this position, and then let some other set of procedures extract and otherwise manipulate the data associated with this position .

SRE-Data has parsing ability

In order to implement this wide range of parsing options, SRE-Data uses PROGram files. These files, which are specified when you launch SRE-Data, are simple text files that contain special instruction codes. Therefore, effective use of SRE-Data requires that you learn how to create these PROGram files.

SRE-Data uses daemons for fast access

This strategy has two drawbacks:

1. You must launch a specific daemon for each database you wish to access. As described above, to do this you'll need to create an SRE-DATA PROG file, and run SREDATA.CMD.
2. For large databases, memory requirements can become daunting. Similarly, if you have dozens of seperate databases, each one will require it's own daemon.

   This cost, in terms of RAM and system-threads, is one reason why you might want to use SRE-Data as an "index searcher", and let some other set of procedures retrieve more substantative data based on the position (in the index) of the desired identifier. In recognition of this style of operation, SRE-Data can be used with special KEY files -- quickly searched files that are not loaded into RAM.

SRE-Data is easy to access

SRE-Data is self refreshing

An alternative

Back to top

In some cases, you may be able to run SRE-Data without specifying a PROG file. However, for any kind of special formatting, PROG files are required!

DATA=filename.ext (required)

Name of the file that contains your data.
File is relative to the goserve working directory (i.e;D:\GOSERVE)

DELIMS=char_list optional

A list of "word" delimiters. For space, use %20 (i.e.; use URL encoding for odd characters).

Default value: DELIMS=%20, (space, and comma)

IN_OFFSET=nlines (or nrecords)

# lines (if FIXED=0) or # of records (if FIXED>0) to offset when reading the input file. Lines (or records) before this output will be ignored.

Default value: IN_OFFSET=0

LINES=nlines

# lines per record (default=1)
If you specify FIXED (see below), this is ignored.

Default value: LINES=1

FIXED=nchars

# characters per record (fixed length records).
If non-zero, LINES is ignored (FIXED overrides LINES)

Default value:FIXED=0 -- which means "use one record per line"

INDEX_FORMAT: arg1 [arg2]

Format for reading identifier variable. Should be specified using either:

INDEX_FORMAT=nth

INDEX_FORMAT=start length

where "nth" means "nth word" (using DELIMS to define words)
and "start length" means "length characters, starting at start character"

Default value: INDEX_FORMAT=1 -- use first word.

TYPE =atype

The type of matching to be done. atype can consist of any combination of the following.

• atype='' (or not specified):verbatim.
• atype='I' : case insensitive
• atype='S' : strip spaces from both ends
• atype='N' : numeric match -- should be used by itself

Default value: ATYPE= -- verbatim search (i.e.; using a==b)

COMMENT_CHARS=char_list

Comment line signifiers. If a line starts with any one of the COMMENT_CHARS, it is ignored (this also applies to comments embedded in multi-line records). Leave this blank to have no comment characters.

Default value: COMMENT_CHARS=;

HEADER=header comment

A header comment that desribes the data. Can be retrieved with the !COMMENT special request

VARIABLES=varname1 arg1 [arg2] ,

A comma delimited list of variables and locations. Used to partition the record into "information" fields, with a varname assigned to each of these partitions. arg1 and arg2 are as defined for the INDEX_FORMAT option: arg1 by itself is "nth-word", arg1 with arg2 is "start - length". A special code, * by itself, means "use the entire record".

Note that you can mix formats (i.e.; use both nth-word and start-length formats), and you can use the same characters (or words) in multiple variables.
Example: INDEX_FORMAT=* , COUNTY_NAME 1 , FIPS_CODE 20 5

VARIABLES can be searched over (slowly, since the search uses a one-by-one comparision)

Default value: VARIABLES= * -- create a variable, with the name of "*", that contains the entire record (including the value of the identifier variable)

Sample PROGram file

; desciptive header
header=This is the FIPSNAME demo database for SRE-Data
;
data=sredata\fipsname.dat
; 20 characters per record
fixed=20
; first "word" is the identifier
index_format=1
; matches can be case insensitive, and will ignore leading & trailing spaces
; * means "allow wildcard searches on variables"
type=IS*
; define a name variable (and a "whole record" variable)
variable= * , name 2
; for other items (such as comment characters), use defaults

Not Using PROGram files

for text files, with spaces and commas uses as delimiters, and where the first work is the identifier

Example:
    crlf='0d0a'x
    astring="header=This is the FIPSNAME demo database for SRE-Data"crlf
    astring=astring||'data=sredata\fipsname.dat'crlf
    astring=astring||'fixed=20'crlf
    astring=astring||'delim=,'crlf
    astring=astring||'index_format=1'crlf
    astring=astring||'type=IS*'crlf
    astring=astring||'variable= * , name 2'
    status=sredata('FIP','STRING',astring)
    Say "Status of SRE-Data: " status

Back to top

STEM

STEM is used to quickly load data, but is otherwise the same as using PROG

KEY

KEY (and LOADKEY) are used to instruct SRE-Data to use special KEY files -- index files whose use requires minimal RAM. Although KEY files are somewhat slower to search, their size can be unlimited. Their primary disadvantage is that they contain no additional information -- they only contain the identifier variable.

Experience has shown that for moderately large datasests (say, 30,000 records with 5 byte identifiers), the standard SRE-Data strategy (of loading the database into RAM) can bog down the machine. In these cases, use of the KEY option is highly recommended.

For moderately large datasets, some speed improvement can be obtained by using the LOADKEY option -- it loads the key file into ram for slightly faster searches. Note: for the same amount of information (i.e.; just an identifier), KEY files are much smaller then STEM files, hence the memory penalty is less.

There is one feature of SREDATAX that may require some caution: the use of the QSORT.EXE external program. Since sorting large datasets (say, 70,000 records) can be excruciatingly slow under REXX, SREDATAX can optionally use a fast sort program: QSORT.EXE. Although QSORT.EXE is designed to work with SREDATAX, there are some cases where it's use is inappropriate. In particular:

Hint: Running SREDATAX as a scheduled program

To run SREDATAX as a scheduled program, you should specify the the following parameters:

     SREDATAX P=program_file t=output_file 

You might also want to examine the SREFLOGS.DOC file for hints on using the SRE-http "scheduling" option in conjuntion with SREDATAX.

Back to top

SREF_FIND_INDEX makes it easy to search for matching records. The syntax is

     results=sref_find_index(target,qname,[varname],[nthmatch],[location_only],[timeout])

results

a 2 part response: recnum,value (note the use of the comma as as seperator).
Example: 315,9001,FAIRFIELD
The first part is the "record # of the matching record", or a 0 if no match, or less then 0 if an error. The second part is a value of what was matched. If a * variable was specified, this will be the value of the record; otherwise, it will be the value of the identifier (which should equal the value of the target).

When no value can be found, or when some type of error occurs while looking for a value this second part will be empty.

Note: When KEY (or LOADKEY) is being used, SRE-Data will return not return the "value of the record". However, you can look up a record number in the original dataset to find the "value" of the record.

target

the string to search for

qname

the "name" used when sredata was invoked (by default, INDEX)

varname

[optional] if not specified, look at the identifier variable. If specified, a "one-by-one" search will be conducted. This can be very slow (in contrast, binary search is used when looking for identifier variables). However, you can include a wildcard character (the *) in the target.

You can also use varname to obtain internal information on the dataset, and to get information on a particular record

nthmatch

[optional] if the identifer variable does not have uniques values, or if you are searching against a VARIABLE, multiple matches may occur (an especially likely occurrence when wildcards are used in VARIABLE searches). In such cases, you can tell SRE-Data to return the "nth" match. For example, setting nthmatch=3 means return the third match.
Notes on nthmatch:

• In addition to numeric values, nthmatch understands a few special codes:
  FIRST : return first match (same as nthmatch=1)
  LAST: return last match
  COUNT : return # of matches (but no value).
• The default (nthmatch=0 or nthmatch="") is to return the first found match. In VARNAME searches, this will be the first match in the file; for identifier searches, this may occur anywhere in the "set" of matches.
• if nthmatch is greater then the number of matching values, a 0 (no match) is returned.
• multiple matches on identifiers
  When multiple matches occur in a "sorted" identifier variable, the ordering of these matches can be quite arbitrary, and need not reflect the order in which these occur in the original data file. Hence, to maintain compatability with VARNAME searches, SRE-Data will first sort these matching identifiers by their record number, and then extract the "nth" match.
  Since this secondary sort takes a little extra time, you can suppress it by entering a negative value for nthmatch.

  For example, using nthmatch=-4 means get the 4th match, as it occurs in the sorted file,
  while nthmatch=4 means get the 4th match, as it appears in the original file.

  To improve throughput, KEY files are pre-sorted by record number (that is, a secondary sort is performed on record number); hence the -nthmatch trick has no effect.

location_only

[optional] do not return "value" (i.e.; just return the record #)

timeout

[optional] seconds to wait for answer from daemon. The default value is 30.

        ?: Variable Info -- number of variables, list of names
     !OBS: # observations             
   !TYPE: matching type 
 !COMMENT: Header comment (empty string if none was specified)
    !DATE: creation date (useful if you are using a STEM or KEY file)
!DATAFILE:original file (value of DATA)
!PROGFILE: Original PROGram file 
 !STARTED: invoked as; will return PROG, DATA, KEY or STEM

To do this, simply set target equal to the desired record number, and set varname equal to '#varname'. For example, varname='#*' will return the value of the entire record (assuming that you included * in the VARIABLEs option of the PROGram file).

Explicit record lookup is especially useful if you have occasion to search against several variables in your data. In such cases, you could use VARNAME searches, but that could be very slow. Instead, creation of a primary, and several secondary daemons (one for each of the variables you might search against) is recommended. With this strategy, it is best to specify variables only for the "primary" identifier. When a match is found, you could then query this primary identifier's daemon for the desired variable (using the record number returned by the appropriate secondary daemon).

Example:
   Suppose you have a 3 variable delimited dataset; each record contains
   a name, an id, and a  balance.
   Suppose you've launced two daemons:
        1) VAR1: The primary daemon, using ID as the identifier, and
                 specifying VARIABLES= * , ID 1 , NAME 2 , BALANCE 3
        2) VAR2: 1 secondary daemon, with NAME as the identifier, and NO
                VARIABLES specified
   You could then (the follwing is not an exhaustive list!):
        i) Search VAR1 for a matching ID, and return the value of the record.
            results=sref_find_index('VAR1',an_id)
        ii) Search VAR2 for a matching NAME, and then return the 
            matching record:
            irec=sref_find_index('VAR2',a_name,,,1)  
           if irec>0 then 
               avalue=sref_find_index('VAR1',irec,'#*',,2) 
       iii) Search VAR2 for matching NAME, and return her balance
            irec=sref_find_index('VAR2',a_name,,,1)  
           if irec>0 then 
                avalue=sref_find_index('VAR1',irec,'#BALANCE',,2)

Caution:When a "delimited" (one line per record) file is to be searched, SRE-Data will not create multi-line records, nor will it skip lines that begin with comment characters.

In general, be careful when using KEY files generated from data files that contain comment lines or that contain multi-line records.

DEM1SRED.CMD (examine your INITFILT.80 file) and DEM2SRED.CMD (examine U.S. county names) are designed to be run in stand-alone mode -- they do not work off of the web. For details on their useage, run either from an OS/2 command prompt and read the on-line help.

DATADEMO.HTM, which calls DATADEMO.CMD, provides an example of a web-interface to an SRE-data daemon. By default, it uses the U.S. county name database used by DEM2SRED.CMD.

Back to top

Kill a Daemon

Launch a Daemon