6 Feb 2002

                Using the SRE2002 Cache-Data Daemon

1)   Introduction
2)   The SRE_CACHE_DATA procedure
2.1)    SRE_CACHE_DATA ACTIONS
3)   The 7 character "special identifiers"
4)   Hint on saving non-text strings, or strings with tabs and CRLFS
5)   File structure

                        ---------------------

1) Introduction

The SRE2002 data-cache daemon provides permanent storage of small
to moderate sized collections of small data records. It's strengths
are relatively quick lookup and recording of data, using a caching
mechanism to reduce disk access.

Basically, this data-caching daemon is used by:
  1) defining a data series.  You can define where to store the data files.
  Later ...
  2) opening a data series (usually one created at an earlier date)
  3a) adding records (in an opened data series). Each record can
      contain several string variables (you can store numbers as strings).
  3b) looking up variables in previously defined records, 
  3c) modifying records (changing variable values)
  4) closing the series

Although fairly complete, this daemon is not designed to be efficient under
complicated changes. For example:
  a)  It is best used with relatively small records, and with variables whose
      size does not change.
  b)  Although record removal is relatively simple, space is only recovered when
      explicitily instructed.
Moreover, 
  c) it is not designed to work well with large datasets (say, over
     100,000 records).
  d) it has limited error recovery, hence should not be used with data for
     which guaranteed accuracy, and protection from accidental erasure, 
     is crucial

Features: 

  * You can specify several different data series, each associated with a 
    different permanent data file.

  * The data series files can be viewed, BUT NOT EDITED, with a text viewer. 

  * Records can be identified by an arbitrarily long text string, or
    by a 7 character identifier.

Usage:
  Although you can use SRE2002 daemon procecurs to communicate with the
  cache-data daemon, it is far more convenient to use the SRE_CACHE_DATA
  procedure. Hence, we now describe  SRE_CACHE_DATA...

                        ---------------------
       
2) The SRE_CACHE_DATA procedure:

Syntax:
    aresult=sre_cache_date(series,action [options],identifier,val1,..,val10)
where
      series:  a data series.  
      action: what to do.  The following actions are available:
            DEFINE : create a data series.  
            OPEN   : open a currently existing data series
            ADD    : add a  new record
            MODIFY : modify a record
            EXTRACT : Extract one or several records, by record number
            FIND   : find a record, or records
            REMOVE : remove a record
            CLOSE  : close the data series (save cached information)
            INFO   : info about a data series
            SAVE   : save any cached information
            CLEANUP : cleanup data
            REWRITE : rewrite the data file (fix odd records, remove old junk)
            DELETE  : delete a data series
        
       [options]  Space delimited list of optional action modifiers.

     identifier : an identifier 

     val1 .. val10 : up to 10 values, whose used depends on action
 
    aresult: depends on action.


A note on action modifiers:

 The ADD, MODIFY, FIND, and REMOVE actions all can take the ID7 and WRITE
 action modifiers:

  ID7 
    When an ID7 action modifier is specified, then the IDENTIFIER must be 
    a "special" 7 character identifier.
    The special identifiers are described in the 7 character "special 
    identifiers" section below.

  WRITE 
    When WRITE is used an action modifier, then the changes are immediately
    written to the data series file. Actually, all outstanding (not yet
    recorded) changes, for any open data series, are written to their
    respective data series files.
    Basically, this does the same thing as using a SAVE action after
    the ADD, etc. action.


2.1) SRE_CACHE_DATA ACTIONS

The following describes the usage of SRE_CACHE_DATA for each ACTION

DEFINE: define a data series

 Defining a data series.

    SERIES : series name
    ACTION: DEFINE
    IDENTIFER: Optional fully qualified file name
    VAL1: Optional space delimited list of variable names
    VAl2: Optional 60 char or shorter DESCRIPTION
    VAL3:  Optional "header" comment (can be arbitrarily long) 
    ASTAT:  0 for success, otherwise an error code

  When you DEFINE a data series, a new data file will be created.
  Unless otherwise specified, a "SRE2002 Data Series" file will be created
  in your x:\sre2002\DATA directory. Its name will be the SERIES variable,
  with a .SDS extension.

  To specify a file, set the IDENTIFIER variable to a fully qualified file name
  (if you don't give an extension, .SDS is appended).

  Notes:
     * SERIES names are case insensitive, and can contain A-Z,0-9,$, and _
       Their maximum length is 40 characters
     * If the drive you are using is FAT -- then the data series name can only
       be 8 characters long
     * If you use IDENTIFIER to assign a non-default filename, you'll have 
       to provide this filename whenever you OPEN the data series
    
  VAL1, .. ,VAL10 are optional variable names.

  Error values:
    The error values are:
        1  Could not open a data series file: file_name
        2  Data series name is too long (>8 character on a FAT drive)
        3  A variable name was too long
        4  A variable name had unallowed characters
        5  A series name had unallowed characters or is too long
        6  A data series with this name is already DEFINEd (you can
           DELETE, and then DEFINE again)

  A note on variable names:

     Variable names are associated with  the VALn argument they were found in.

     For example:
       astat=sre_cache_data('SERIES1','DEFINE',,'CATS','DOGS','GORILLAS')
     and later ...
       astat=sre_cache_data('SERIES1','OPEN')
       astat=sre_cache_data('SERIES1','RECORD','AF1','FELIX','Fido','MAGILLA')
       astat=sre_cache_data('SERIES1','RECORD','Gn1','FRITZ','SCOOBY DOO','K Kong')

     will result in the creation of 2 records:
        Id=AF1
     which will have the variables and values:
         CATS = FELIX
         DOGS = Fido
         GORILLAS = MAGILLA
     Id=Gn1   (note that ID variables can be case-sensitive)
         CATS = FRITZ
         DOGS = SCOOBY DOO
         GORILLAS = K Kong

  More Variable Name Notes:
      * after DEFINEing a data series, you have to OPEN it to work with it (say,
        to add records)
      * variable names can be up to 40 characters long; but shorter is better.
      * variable names are case insensitive, and can contain A-Z,0-9,$, and _
      * the default variable names are %VAR_1,..,%VAR_10 (note that % is NOT allowed
        in user defined variable names)

      * Extending the above example:

      astat=sre_cache_data('SERIES1','RECORD','Z2BC2', ,
                           'Garfield','Snoopy','Joe Young','Mothra')

     Id=Z2BC2   
         CATS = Garfield
         DOGS = Snoopy
         GORILLAS = Joe Young
         %VAR_4 = Mothra                                           

OPEN: open a data series
 
 Open a previously defined data series

    SERIES: Series name, or blank
    ACTION: Open
    IDENTIFIER: if DEFINEd with a file name, you MUST include it here
                Otherwise, identifier MUST NOT be used.
    VAL1,..  : not used
    ASTAT:  0 for success, otherwise an error code and message

  Error values:
    The error values are:
        1  Problem opening data series file
        2  Bad header information in data series file 
          (perhaps it's not a data series file)
        3 Too many records 
        4 Not enough records
        5 Improper series name
        6  Data series file does not exist 


  Notes:
    * when using OPEN and an explicit data series file, you do NOT need to
      use the same series name as the data series was DEFINEd by.
    * if you leave the SERIES name blank, and use an explicit data series
      file, the name it was defined by will be used. 

   Examples:
       aa=sre_cache_data('MYSERIES','OPEN')    


ADD:  Add new record to an open data series

    SERIES: series name
    ACTION: ADD
      Optional action modifiers: COMMENT, WRITE and ID7
          Comment means "add this as a comment". IDENTIFIER should
          containt the actual comment, which can be arbitraritly
          long. VAL1..VAL10 are ignored. The comment is written at
          the end of the data file.
    IDENTIFIER: either a string, or a special identifier (if ID7 specified)
    VAL1,..,VAL10: Values. These can be text strings of any length.
                   Note that TAB, CR, and LF characters are converted
                   to spaces, and trailing & leading spaces are stripped.
    ASTAT:  A 7 character identifier, or a 1 character error code
           If you explicitily defined a 7 character ID (using ID7), 
           this will be the same value (with upper case translation).
           Otherwise, one will automatically be generated.

 A note on defining values:
   We strongly recommend allocating as much space to these
   values as may be needed during future modifications. This may
   mean using dummy values; such as 'abcde' to set aside 5 bytes for
   a variable.  


 Error values:
    The error codes are:
        1  = Data series was not opened
        2  = Too many of this "family" of identifier.
        3  = Explicitily defined identifer does not have 7 characters
        4  = Explicitily defined identifier has an unallowed character
        6 =  Record with this  id already exists
        7 =  Problem adding to data series file
        8 =  Bad data series name
        9  = ADD specified with VAR or ID7

   Examples:
       aa=sre_cache_data('MYSERIES','ADD','/HOME/Index.1','1st house ','2nd dog')    
       aa=sre_cache_data('MYSERIES','ADD ID7','CXAQQE2','peanuts','cracker jacks')


MODIFY: Modify a record, or modify a portion of a record

   Change some or all of the variables in a record.

    SERIES: series name
    ACTION: MODIFY 
      Optional action modifiers: VAR, INCREMENT, ADD, WRITE, *, and ID7

      If the action modifiers contains VAR (VAR-mode):
          Then change the value of the selected variable
       If the action modifies contains INCREMENT (INC-mode)
           Then increment the value of the selected variable 
       Otherwise (ALL-mode)
           Change all the variables

       The ADD options means "if this record does not exist, create it".

       The * option means "if no exact match, try looking for a
       wildcard match". If no wildcard match, and ADD option is
       specified, then create a new record. Conversely, if there is a 
       wildcard match the ADD option is ignored.

       The ADD and * options can NOT be used with ID7.

    IDENTIFIER: either a string, or a special identifier
    VAL1,..,VAL10: Values. Depends on "mode".
        
        If VAR-mode:
              VAL1 = variable name
              VAL2 = new value
        If INC-mode
              VAL1 = variable name
              VAL2 = value to add (negative values allowed)
                     If val2 is not specified, a value of 1 is used.
        If ALL-mode:
           VAL1 .. VAL10  can be strings of any length.  

   
    ASTAT: If successful, and ALL-mode or VAR-mode: ID7. 
           If successful, and INC-mode: ID7' 'prior_count' 'most_recent_access
           Otherwise, a 1 character error code.
     
    The error codes are:
        1  = Data series was not defined
        2  = Both INCREMENT and VAR specified
        3  = Variable not found
        4  = Bad ID7 name
        5  = Record not found 
        6  = Attempting to increment with non-numeric value
        7  = Attempting to increment non-numeric variable
        8  = Bad data series name
        9  = ADD specified with ID7


 A note on defining values:
   We recommend not modifying a variable with a value 
   longer then what it was defined with. It will work, but it often
   requires creating a new record (and marking the old one for removal) --
   a set of actions that is somewhat time consuming.

   Examples:
       aa=sre_cache_data('MYSERIES','MODIFY WRITE','/HOME/Index.1','1st cat ','2nd mouse')    
       aa=sre_cache_data('MYSERIES','MODIFY ID7 VAR','CXAQQE2','%VAR_1','coated popcorn')


EXTRACT: Extract one or many records

    SERIES: series name
    ACTION: EXTRACT
    IDENTIFIER: a record number, or a range 
    VAL1..VAL10: not used
    ASTAT: If success,
              0 #RECORDS
              RECORD1
              ...
              RECORD_#RECORDS
        
           If falure
              1_digit_errorcode

   The error codes are:
        1  = Data series is not open
        2  = Bad range (misordered)
        3  = Bad range (smallest<1, or largest>#records in series)
        5  = Record could not be read
        8  = Bad data series name

  Notes:
    *  The returned records are have the following structure:
         ID7 '09'x  value1 '09'x ... value
       This will always be on one line (records can NOT contain crlfs)

    *  To specify a range, use N1 - N2.
        Thus, to read records 10 to 25, use "10 - 25 "

FIND: Find a record; return one, or several, variables
     
    SERIES: series name
    ACTION: FIND 
      Optional action modifiers:  ID7, VAR, ALL and *
    IDENTIFIER: either a string, or a special identifier
    VAL1: A space delimited list of variables to return, where
           >  %ID can be used to mean the "identifier" variable.
           >  * means %ID and all 10 variables (some of which may
              not be set).
           > '' to return basic information.
    VAL2: A variable name (to be searched).
          Only used if VAR is used as an action modifier
    ASTAT: If success (assuming J variables were specified):
              0 ID7 VarNames
              Var1 value
                ...
              VarJ value
           If success (assuming 0 variables were specified):
              0 ID7 
              If FIND7 was used, this ID7 is the same ad the IDENTIFIER
              (though upper case).
           If falure
              1_digit_errorcode

   The error codes are:
        1  = Data series is not open
        3  = Variable not found
        4  = Bad ID7  name
        5  = Record not found
        8  = Bad data series name

   For example, 
        astat=sre_cache_data('SERIES1','FIND','RECORD_12151','CATS %VAR4')
    might return:
        0 ABCDEF1 CATS %VAR4
        Garfield
        Mothra
    whereas:
        astat=sre_cache_data('SERIES1','FIND','Z2BC2','HORSES')
    might return:
        3


   Special actions: 
       VAR : Instead of looking at identifiers, examine the values of the
            variable named in VAL2 (if no such variable exists, a '3'
            error code is returned). Either the ID7 of the first match, or 
            the ID7s of all the  matches (see below), will be returned.

       ALL : Only used if VAR is a special action.  Causes a space delimited 
            list of ID7 of all the matches, not just an ID7 of the best or 
            first match. to be returned.        

        * : If there is no exact match (to the identifier or (selected 
            variable), a "wild card" match will be attempted. This entails 
            comparing the current identifier to all entries with an * in their
            identifier (or in the selected variable) -- and returning the 
            "best match" (or all matches).


SEARCH:  Search, on a variable, in all records for a match. 

   Returns one, or several, ID7s of matching records.
   This is MUCH slower then FIND, especially in large datasets,
   but it is more flexible.

    SERIES: series name
    ACTION: SEARCH
      option_list: ALL or *
           ALL : return ID7S of all matching records. Otherwise, return
                 the first match (or best match, if * is specified)
             * : do case insensitive wild card searches. Otherwise, do
                 case-sensitive exact matches. Wildcard search means that
                 * in a record's variable value means "wildcard". Note that
                 * in the identifier is NOT treated as a wildcard (though
                 it will match a * in record's variable).

    IDENTIFIER: a string to search for
    VAL1: Variable name, or %ID. The values of this variable, for all
          records in the series, will be compared to identifier.
          If %ID, then search record identifiers.

    ASTAT: If matches found.
              0 ID7_list
                 ID7_LIST will only have 1 element if ALL is
                 not specified in the option_list.
           If no matches found
              0
           If falure
              1_digit_errorcode

   The error codes are:
        1  = Data series is not open
        3  = Variable not found
        8  = Bad data series name


REMOVE: Remove a record
   SERIES: series name
   ACTION: REMOVE 
      Optional action modifiers:  ID7, WRITE
   IDENTIFIER: either a string, or a special identifier
   VAL1,..,VAL10 : Not used
   ASTAT: A 0 for success, or a 1 digit error code

   The error codes are:
        1  = Data series was not defined
        3 = No such record
        4 = Bad ID7
        5  = Record not found
        8  = Bad data series name

  Note that removal of a record does not remove space from the series data file.
  To actually remove the bytes, use the REWRITE action.

  Examples:
     aa=sre_cache_data('MYSERIES','REMOVE','/HOME/Index.1')


CLOSE: close one, or several, data series 

   Before closing, all cached information will be written to the
   data series file.

   After closing, you will NOT be able to find, add, etc. items in
   the data series. That is, you'll have to OPEN it again if you 
   want to to anything with it.


   SERIES: a series name, a space delimited list of series names, or *
           * means "all currently open series"
   ACTION: CLOSE
   IDENTIFIER: not used
   VAL1,..,VAL10 : Not used
   ASTAT: A 0 for success, or a 1 digit error code

   The error codes are:
        1  = Data series was not defined
        8  = Bad series name 


SAVE: save cached information in all open data series
   SERIES: not used
   ACTION: SAVE
   IDENTIFIER: not used
   VAL1,..,VAL10 : Not used
   ASTAT:  0 for success, or an integer # of charout errors

   Normally, cached information will be saved every few minutes.  You can
   use SAVE to force all information to be saved immediatly.  
 
   Note that CLOSE implies "SAVE first, and then CLOSE a specified
   data series".
   In contrast, SAVE keeps all the series open.

   The error codes are:
        2  = Problem saving a data series

INFO: info about an open data series
   SERIES: series name, or series names, or *
           * means "all currently open series"
           '' can be used if non-series specific parameters
           are requested
   ACTION: INFO
   Identifier: Not used
   VAL1 : Space delimited list of items to return
   ASTAT: If successful:
            0 series .. seriesn tab  ITEM1 ... ITEMK
            seriesname1 tab item1_value tab item2_value tab ..
            seriesname2 ...

          If not successful
             a 1 digit error code
         
     The error codes are:
        1  = Data series is not open
        2  = Invalid item
        8  = Bad data series name

     The valid series specific "items" to ask for (case insensitive)   
        SERIES: the series name used when the series was defined
        CREATION: the date & time the series was defined
        MODIFIED: the date & time the series was last modified (information 
                  written to the file)
        VARS: List of defined variables
        RECORDS:  Current number of records
        REMOVED:  Current number of removed records not yet deleted
        DESCRIPTION: The short description
        FILE: The name of the data series file

    There are also a few non-series specific parameters that can be
    obtained with info. 
        INCACHE:  Number of entries in cache. 
        CACHEHITS: A pair of comma delimited values:
                      Number of cache hits, number of non-hits 
                     (info gotten from data series file), 
                   since last CLEANUP.
        SERIES: Space delimited list of currently defined series
        DROPPED: Total number of "dropped" entries (that a 
                 CLEANUP will freeup)

CLEANUP: Cleanup internal data

   SERIES: not used
   ACTION: CLEANUP
      Optional action modifiers:  CACHE
   IDENTIFIER: not used
   VAL1,..,VAL10 : Not used
   ASTAT: A 0 for success, or a 1 digit error code

     The error codes are:
        1  = Data series was not defined
        2  = Bad header information in data series file 
        3  = Too many records (when reopening)
        4  = Not enough records (when reopening)
        8  = Bad series name 

   This will    
        1) close all open data series (first saving data to files).
        2) clear all data from memory.
        3) reopen these data series

   The net effect is to freeup memory that may be "orphaned", due
   to REMOVEing entries or CLOSEing data series.  However,
   since the cache is cleared, this will slow down response
   time for a while.
  
   If the CACHE action modifier is used, then ONLY the cache will be 
   shrunk, using a least-recently-used algorithim.  This does
   not free up as much memory, but it does retain the most useful
   cache entries.

   Note that the cache-data daemon will periodically do a cleanup 
   of both the cache, and the data (see the user configurable parameters 
   in SRE_DCSH.RXX for details).
       Thus, in most cases it is not necessary to use this command.

REWRITE: rewrite the data file (fix odd records, remove old junk)
   SERIES: series name
   ACTION: REWRITE
           Action modifier:  FIX
   IDENTIFIER: if DEFINEd with a file name, you MUST include it here
                Otherwise, identifier MUST NOT be used.
   VAL1,..,VAL10 : Not used
   ASTAT: A 0 for success, or a 1 digit error code
        
  The error codes are:
        1  = Data series was not defined
        2  = Problem saving data series
        3  = Problem rewriting data series

  Notes:
   * If the FIX action modifier is specified, then the file will be "fixed" --
     improperly sized entries will be fixed, and the number of records
     will be fixed. This is more time consuming then opening without 
     FIX.

   * If the selected series is currently open, it will first be SAVED.
   * 'x ' and '. ' orphan lines are removed
   * If a data line does not end properly (say, file ends too soon, or
     an expected contination line is not found), the data line (and
     it's known continuations) will be  ';' (commented) out


DELETE: delete a data series
   SERIES: series name
   ACTION: REWRITE
   IDENTIFIER: if DEFINEd with a file name, you MUST include it here
                Otherwise, identifier MUST NOT be used.
   VAL1,..,VAL10 : Not used
   ASTAT: A 0 for success, or a 1 digit error code

   This simply deletes a file. It is included as a convenience -- one could achieve
   the same effect by deleting the file from a command prompt.  
     
   Actually, if the selected series is open, it also closes it 
  (removing data from internal cache).

  The error codes are:
        1  = Problem finding data series file
        2  = Problem deleting data series

                        ---------------------

3) The 7 character "special identifiers":

    Each record is identified using a seven character identifier.
    Thie identifier is either explicitily specified by you (by using ID7
    as an action modifier), or it is automatically generated.

    The 7 character identifier has two components:
        The first 6 characters identify the "family"
        The 7th character identifies the individual within a family.

    For automatically generated identifiers, the distinction between
    family and individual is arbitary (that is, only meaningful
    internally).

    For explicitily defined identifiers, you can VIEW "all
    members" of a family, or just for one member.

    The identifiers can contain 38 different (case insensitive) 
    characters:  A-Z, 0-9, $, and _

    Note that when automatically creating an identifier (from a string),
    there is a chance that you will "run out of room" -- that the
    identifier will be assigned to a family for which there are already
    38 members. However, this should be an extremely rare occurence, at
    least for the relatively small (say, less then 100,000 records) databases
    this daemon is designed to work with.

                        ---------------------

4) Hint on saving non-text strings, or strings with tabs and CRLFS.

   It takes more time and space, but you can url-encode (or otherwise convert)
   the strings before saving, and then unconvert after retrieving the value
   (at some later date).

   For example, you can use SRE_MAKE_PACK64 to "pack the string" into text
   characters, and SRE_PACK64 to "unpack" it.

       Since the characters in a PACK64 string are all "legal"
       SRE_CACHE_DATA characters, this allows you to save any
       kind of string (not just strings with tabs and crlfs).

                        ---------------------

5) File structure:

    Note that SRE2002 data series files are NOT meant to be modified by the 
    user -- all changes to the data file should be done using the data-cache
    daemon.  However, these files can be read with a normal (80 character
    wide line) text viewer.

  * Each entry in a data series file consists of an 82 character line 
    (last 2 characters are always CRLF).

   The structure of the first line of a dara record is:
        7-char-id 
        1 space 
        bytes (length of info), 6 chars wide (hence truncated at 999999)
        space
        up to 65 characters of info
        crlf

   Continuation lines are:
        .
        space
        up to 78 characters of info
        crlf

   Other lines are
     Meta data:
       #
       space
       meta data (78 char wide)
       crlf
 
     Comments
       ;
       space
       comment (up to 78 characters)
       crlf
  
     Deleted lines (to be removed)
       x 
       space
       up to 78 characters of arbirary stuff
       crlf

  * Records consist of a least one entry, but can contain an unlimited number.

  * Values (stored in the info field of  records) can be arbitrarily long -- 
    they will be  stored in as many entries as needed. That is, values can span
    multiple entries.

  * The top of the file contains meta information in a name: value format.
    For example (ignoring the leading "#" code):
        SERIES: MYSERIES
        CREATED: 20 June 2001 10:45:01
        MODIFIED: 16 Feb 2002 15:31:22
        RECORDS: 203
        REMOVED: 0
        VAR1: DOGS
        VAR2: CATS

   * Values may contains spaces, but they may NOT contain tabs
     (or other non "text" characters).

           By default, a tab is used as a "value seperator" character.
           This allows for text viewers to read the file, but may cause some 
           wrapping problems. 
   
   * Comments lines start with a ;  
        Comment lines can be freely interspersed (even in the middle of a set
        of continuation lines).        
   
   * Removed records start with a "x" -- including removed continuation lines.


   * A special "identifier", *******, is used to store locations of all 
     entries that contain * characters in their identifiers. When a data
     series is created, an empty is generated and stored to the data series
     file.
