@need 3200

@node Hashing
@chapter Hashing

Hash values and hash tables are useful in many programs.  The
Hackerlab C Library provides:


@strong{hash utilities} -- functions for computing hash values in common
situations.  Hash functions for strings can be found elsewhere. (For
strings of 8-bit characters, see @ref{Computing Hash Values From Strings}.  For Unicode strings, see @emph{XREF}.)


@strong{an md5 implementation} -- functions to compute an md5 digest 
of an arbitrary string of bytes.


@strong{hash tables} -- a general-purpose implementation of variably sized
hash tables.

@menu
* Hash Utilities::
* MD5 Routines::
* Hash Table Trees::
@end menu

@need 3200

@node Hash Utilities
@section Hash Utilities
@noindent
@strong{#include <hackerlab/hash/hash-utils.h>}@*
@sp 1

The functions in this section provide tools useful for computing 
hash values.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hash_ul}@fuindex hash_ul
@*
@example
@group
unsigned long hash_ul (unsigned long n);
@end group
@group
@end group
@end example

Generate a hash value from an integer.


This function is slow, but attempts to give a good distribution of
hash values even for a series of @code{n} which are not particularly
random.


@code{slow} means that the function does rougly @code{3 * sizeof (n)} array
look-ups and lots of bit twiddling.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hash_pointers}@fuindex hash_pointers
@*
@example
@group
unsigned long hash_pointers (void * elts, size_t n_elts);
@end group
@group
@end group
@end example

Compute a hash value from an array of pointers.


This function is slow, but attempts to give a good distribution of
hash values even for a series of pointers which are not
particularly random.  Usually, pointers are not particularly
random.


@code{slow} means that the function does roughly @code{3 * sizeof (n)} array
look-ups and lots of bit twiddling, per pointer.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hash_mem}@fuindex hash_mem
@*
@example
@group
unsigned long hash_mem (t_uchar * elts, size_t n_bytes);
@end group
@group
@end group
@end example

Compute a hash value from an array of bytes.


This function is slow, but attempts to give a good distribution of
hash values even for a series of bytes which are not particularly
random.


@code{slow} means that the function does roughly @code{3 * sizeof (n)} array
look-ups and lots of bit twiddling, per @code{sizeof (unsigned long)}
bytes.




@need 3200

@node MD5 Routines
@section MD5 Routines

The md5 routines allow you to compute an MD5 message digest
According to the definition of MD5 in RFC @code{1321} from April @code{1992}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{make_md5_context}@fuindex make_md5_context
@*
@example
@group
md5_context_t make_md5_context (alloc_limits limits);
@end group
@group
@end group
@end example

Allocate and initialize an object which will keep track of the
state of an md5 digest computation.

@sp 1
@need 1750
@noindent
@b{* Function} @code{md5_context_reset}@fuindex md5_context_reset
@*
@example
@group
void md5_context_reset (md5_context_t ctx);
@end group
@group
@end group
@end example

Reinitialize an md5 state object.  This will 
undo the effects of any previous calls to
@code{md5_scan}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{free_md5_context}@fuindex free_md5_context
@*
@example
@group
void free_md5_context (alloc_limits limits, md5_context_t ctx);
@end group
@group
@end group
@end example

Free all resources associated with an md5 state object.

@sp 1
@need 1750
@noindent
@b{* Function} @code{md5_scan}@fuindex md5_scan
@*
@example
@group
void md5_scan (md5_context_t hd, t_uchar * inbuf, size_t inlen);
@end group
@group
@end group
@end example

Scan the next @code{inlen} bytes of @code{inbuf}, treating them as subsequent
bytes in a message for which we are computing an md5 digest.


This function may be called repeatedly on sequential "bursts" 
of a total message.

@sp 1
@need 1750
@noindent
@b{* Function} @code{md5_final}@fuindex md5_final
@*
@example
@group
void md5_final (t_uchar * result, md5_context_t state);
@end group
@group
@end group
@end example

Declare that a complete message has been scanned using
@code{state} and @code{md5_scan()}.


Return the 16-byte md5 digest in @code{result}, which must point to
storage for at least @code{16} bytes.


As a side-effect, @code{state} is reinitialized and may be used
again with @code{md5_scan ()} to process a new message.




@need 3200

@node Hash Table Trees
@section Hash Table Trees

The hashtree library implements in-core hash tables that are
automatically dynamically resized.  Callers are given complete
control over memory allocation of hash table data structures.

@menu
* Hashtree Data Structures::
* Creating Hashtree Rules::
* Hash Tree Allocation::
* Hash Tree Access::
@end menu

@need 3200

@node Hashtree Data Structures
@subsection Hashtree Data Structures
@menu
* How Hashtrees Work::
* Types for Hashtrees::
@end menu

@need 3200

@node How Hashtrees Work
@subsubsection How Hashtrees Work

A hash tree is an associative data structure mapping @geindex keys
@dfn{keys} to
@geindex values
@dfn{values}.  Both keys and values are stored as pointers of type
@code{void *} and may point to any type of value.


When creating a new hash table, programs specify how parts of the
hash table will be allocated and freed.  For example, programs
might simply use @code{malloc} and @code{free}, or they might use allocation
from a size-limited pool, or they might arrange so that parts of
the hashtree don't have to be allocated at all because they are
stored in hashtree keys.


Programs also specify, in advance, how keys are compared for
equality.


Programs do not specify in advance how hash values are computed for
keys.  Instead, when storing or looking up keys in a hash table,
programs first compute a hash value for the key and then call
either @code{hashtree_find} or @code{hashtree_store}.  Hash values are
@code{unsigned long} integers.


Internally, hashtrees are stored as trees.  Each tree node has @code{16}
children.  The children of leaf nodes are lists of key/value pairs.
At each level of the tree, four bits from the hash value are used
to select a child.  The minimum depth of the tree is two: a root
node at level @code{0}, lists of key/value pairs at level @code{1}.  The maximum
depth of the tree is:

@example
@group
     (2 * sizeof (unsigned long))
@end group
@group
@end group
@end example

If a list of key/value pairs grows too long (more than @code{5} elements),
and the node containing that list is not a maximum-depth node, the
node @geindex overflows
@dfn{overflows}.  When overflow occurs, the leaf node is made an
internal node with @code{16} subtrees.  Each of those subtrees is a new
leaf node.  All of the key/value pairs from the overflowing node
are redistributed among the new leaf nodes.


This style of hash tree makes reasonably efficient use of memory
for both small and moderately large numbers of key/value pairs.
Access times are determined by the depth of the tree and are
usually limited by the maximum depth.  In the exceptional
circumstance that a maximum-depth node contains a very large number
of key/value pairs (indicating a poor distribution of hash values),
access times for keys in that node grow linearly with the number of
keys in that node.


The cost of adding an element (possibly causing overflow) is most
commonly the same as the cost of adding an element.  The cost of
overflow is determined by the number of key/value pairs that must
be redistributed.  That number is limited by the number of
key/value pairs in a leaf node at not-maximal depth @code{(64 }== @code{16} lists
of key/value pairs * @code{4} pairs per list).  Once again, in the
exceptional circumstance of a maximum depth node with many
key/value pairs, the cost of inserting a key grows linearly with
the number of keys in that node.


@strong{WARNINGS:} 


Hashtrees are not a panacea.


If you know advance roughly how many key/value pairs
a hash table will contain, you can obtain better memory use and
better access times by using a flat, fixed-size hash table.


Hashtrees are most useful when the number of key/value pairs may 
vary over a wide range, when it is important that tables with only 
a few key/value pairs remain small, and when the cost of adding
an element (possibly causing overflow), must remain small.


Hashtrees are also a reasonable default choice when good hash-table
performance is desirable but optimal performance is not necessary.
In most situations, hashtrees will give at least good performance.




@need 3200

@node Types for Hashtrees
@subsubsection Types for Hashtrees
@sp 1
@need 1750
@noindent
@b{* Type} @code{hashtree}@tyindex hashtree
@*
@example
@group
struct hashtree;
@end group
@group
@end group
@end example

A @code{struct hashtree} represents a hash table.  It is an opaque
structure that should be allocated by @code{hashtree_new} or else
initialized by being filled with @code{0} bytes.

@sp 1
@need 1750
@noindent
@b{* Type} @code{hashtree_item}@tyindex hashtree_item
@*
@example
@group
struct hashtree_item
@end group
@group
@end group
@end example

A @code{struct hashtree_item} represents one key-value pair stored in a
hash table.  It contains (at least) the fields:

@example
@group
     void * key;
     void * binding;
@end group
@group
@end group
@end example

It is safe to modify either field, but modifications must not
change either the hash value of the key or its equality 
to other keys.

@sp 1
@need 1750
@noindent
@b{* Type} @code{hashtree_rules}@tyindex hashtree_rules
@*
@example
@group
struct hashtree_rules;
@end group
@group
@end group
@end example

A @code{struct hashtree_rules} contains function pointers, and an 
allocation limits.


It has these fields, in this order:

@example
@group
  // a function to compare keye:
  hashtree_eq_fn eq;
@end group
@group
@end group
@end example
@example
@group
  // a function to allocate tree nodes
  hash_alloc_fn hash_alloc;
@end group
@group
@end group
@end example
@example
@group
  // a function to free tree nodes:
  free_hashtree_fn free_hash;
@end group
@group
@end group
@end example
@example
@group
  // a function to allocate key/value pairs:
  hashtree_alloc_item_fn hash_item_alloc;
@end group
@group
@end group
@end example
@example
@group
  // a function to free key/value pairs:
  free_hashtree_item_fn free_hash_item;
@end group
@group
@end group
@end example
@example
@group
  // allocation limits that apply to hash tables
  // using these rules:
  alloc_limits limits;
@end group
@group
@end group
@end example
@example
@group
typedef int
(*hashtree_eq_fn) (void * key1,
                   void * key2,
                   struct hashtree_rules * rules);
@end group
@group

@end group
@end example

@code{hashtree_eq_fn} compares two keys for equality.  It returns @code{1} if they
are equal, @code{0} otherwise.

@example
@group
typedef struct hashtree * 
(*hashtree_alloc_fn) (struct hashtree_rules * rules);
@end group
@group
@end group
@end example

@code{hashtree_alloc_fn} allocates a new hash table tree node.  Hash tables
are nested to form trees; this function allocates one node of such
a tree.  This function may return @code{0} if allocation fails.

@example
@group
typedef void
(*free_hashtree_fn) (struct hashtree * node,
                     struct hashtree_rules * rules);
@end group
@group
@end group
@end example

@code{free_hashtree_fn} frees an empty hash table tree node.

@example
@group
typedef struct hashtree_item * 
(*hashtree_alloc_item_fn) (void * key, struct hashtree_rules * rules);
@end group
@group
@end group
@end example

@code{hashtree_alloc_item_fn} allocates a hash table item (key/value
pair) for the indicated key.  It may return @code{0} if allocation fails.


@code{hashtree_alloc_item_fn} must fill in the @code{key} and @code{binding}
fields of the @code{struct hashtree_item} that it returns.  Typically,
the field @code{key} is set equal to the parameter @code{key}, and the field
@code{binding} is initialized to @code{0}, indicating that the key initially
has no binding.


Sometimes, if a key will be stored in at most one hashtree, a
useful optimization is to store a @code{struct hashtree_item} within
each key.  In that case, @code{hashtree_alloc_item_fn} doesn't have to
allocate memory at all: it can return a pointer to the @code{struct
hashtree_item} in the key.

@example
@group
typedef void
(*free_hashtree_item_fn) (struct hashtree_item * node,
                          struct hashtree_rules * rules);
@end group
@group
@end group
@end example

@code{free_hashtree_item_fn} frees a hash table item (key/value pair).


The field @code{limits} points to allocation limits which are used
by the default implementations of these functions.  By convention,
non-default implementations should also use @code{limits} when performing
allocations.


Defaults are provided for the functions in a @code{struct
hashtree_rules}.  If any particular function pointer is @code{0} or if the
@code{struct hashtree_rules *} passed to @code{hashtree_alloc} is @code{0}, the
default implementations are used.


The default functions perform allocations by calling @code{lim_malloc}
with the allocation limits stored in the field @code{limits}.  (See
@ref{Allocation With Limitations}.)  If @code{limits} is @code{0}, or if the
@code{struct hashtree_rules *} pointer is @code{0}, the allocation limits
@code{lim_use_must_malloc} is used.  In that case, if allocation fails,
the process is exitted by calling @code{panic}.


The default implementation of @code{eq} compares pointers to keys using
@code{==}.







@need 3200

@node Creating Hashtree Rules
@subsection Creating Hashtree Rules
@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_make_rules}@fuindex hashtree_make_rules
@*
@example
@group
struct hashtree_rules * hashtree_make_rules (alloc_limits limits);
@end group
@group
@end group
@end example

Allocate a new @code{hashtree_rules} structure.  (See
@strong{hashtree_rules} in @ref{Types for Hashtrees}.)


The @code{limits} field of the allocated structure is initialized to @code{limits}.


The function pointers in the allocated structure point to
default implementations.  Those implementations will use @code{limits}
when performing allocations.


If an allocation failure occurs, this function returns @code{0}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_init_rules}@fuindex hashtree_init_rules
@*
@example
@group
void hashtree_init_rules (struct hashtree_rules * rules,
                          hashtree_eq_fn eq,
                          hashtree_alloc_fn hash_alloc,
                          free_hashtree_fn free_hash,
                          hashtree_alloc_item_fn hash_item_alloc,
                          free_hashtree_item_fn free_hash_item,
                          alloc_limits limits);
@end group
@group
@end group
@end example

Initialize a @code{struct hashtree_rules} structure.


It is also possible to initialize a rules structure by filling in
the fields directly.  The advantage of using @code{hashtree_init_rules}
is that all fields must be specified -- omitting a field will cause
a compilation error.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_free_limit_rules}@fuindex hashtree_free_limit_rules
@*
@example
@group
void hashtree_free_limit_rules (struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Free storage of a @code{struct hashtree_rules} allocated by
@code{hashtree_make_rules}.




@need 3200

@node Hash Tree Allocation
@subsection Hash Tree Allocation
@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_new}@fuindex hashtree_new
@*
@example
@group
struct hashtree * hashtree_new (struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Allocate a new hash table.


@code{rules} specifies how memory is allocated for this hash tree, and
how keys are compared for equality.  @code{rules} may be @code{0}, in which
case default implementations are used.  See @ref{Types for Hashtrees}.


If an allocation failure occurs, this function returns @code{0}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_free}@fuindex hashtree_free
@*
@example
@group
void hashtree_free (struct hashtree * it,
                    hashtree_free_data_fn freefn,
                    struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Free all storage allocated to a hash table, including the @code{struct
hashtree} itself.  (See @code{hashtree_free_static}.)


The @code{freefn} is:

@example
@group
  typedef void
  (*hashtree_free_data_fn) (struct hashtree_item * it,
                            struct hashtree_rules * rules);
@end group
@group
@end group
@end example

When called, it should free any storage associated with
the @code{key} and @code{binding} fields of @code{it}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_free_static}@fuindex hashtree_free_static
@*
@example
@group
void hashtree_free_static (struct hashtree * tab,
                           hashtree_free_data_fn freefn,
                           struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Free all storage allocated to a hash table, but not the @code{struct
hashtree} itself. (See @code{hashtree_free}.)




@need 3200

@node Hash Tree Access
@subsection Hash Tree Access
@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_find}@fuindex hashtree_find
@*
@example
@group
struct hashtree_item * hashtree_find (struct hashtree * table,
                                      unsigned long hash,
                                      void * key,
                                      struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Search for an entry for @code{key} in hash table @code{table} and return the
hash table item for that key (or @code{0}).  @code{hash} is the hash value for
@code{key}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_store}@fuindex hashtree_store
@*
@example
@group
struct hashtree_item * hashtree_store (struct hashtree * table,
                                       unsigned long hash,
                                       void * key,
                                       struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Ensure that there is an entry for @code{key} in hash table @code{table} and
return the hash table item for that key.  @code{hash} is the hash value
for @code{key}.


If @code{key} is not already present in @code{table}, this function uses
@code{rules->hash_item_alloc} to create a new key/value pair.


If an allocation failure occurs, this function returns @code{0}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{hashtree_delete}@fuindex hashtree_delete
@*
@example
@group
void hashtree_delete (struct hashtree_item * it,
                      struct hashtree_rules * rules);
@end group
@group
@end group
@end example

Remove hash table item @code{it} from its hash table.


To remove a particular key, use @code{hashtree_find} to retrieve
its key/value pair, and @code{hashtree_delete} to remove that pair
from the tree.


This function does not free any storage assocated with
the key or the binding.


This function does call @code{rules->free_hash_item} to free @code{it}.










