@need 3200

@node Unicode
@chapter Unicode

This chapter describes the foundation of support for the Unicode
character set in the Hackerlab C library.


This chapter is not a tutorial introduction to Unicode.  We presume
that readers are already somewhat familiar with Unicode.  A very brief
introduction can be found in @ref{An Absurdly Brief Introduction to Unicode}.

@menu
* Naming Unicode Coding Forms::
* Representing Unicode Strings::
* Unicode Character Properties::
@end menu

@need 3200

@node Naming Unicode Coding Forms
@section Naming Unicode Coding Forms
@sp 1
@need 1750
@noindent
@b{* Type} @code{uni_encoding_schemes}@tyindex uni_encoding_schemes
@*
@example
@group
enum uni_encoding_schemes;
@end group
@group
@end group
@end example

Values of the enumerated type @code{uni_encoding_schemes} are used in
interfaces throughout the Hackerlab C library to identify encoding
schemes for strings or streams of Unicode characters.  (See
@ref{An Absurdly Brief Introduction to Unicode}.)

@example
@group
     enum uni_encoding_schemes
      @{
        uni_iso8859_1,
        uni_utf8,
        uni_utf16,
        uni_utf16be,
        uni_utf16le,
      @};     
@end group
@group
@end group
@end example

@code{uni_iso8859_1} refers to a degenerate encoding scheme.  Each
character is stored in one byte.  Only characters in the
range @code{U+0000 .. U+00FF} can be represented.


@code{uni_utf8} refers to the UTF-8 encoding scheme.


@code{uni_utf16} refers to UTF-16 in the native byte order of
the machine.


@code{uni_utf16be} refers to UTF-16, explicitly in big-endian order.


@code{uni_utf16le} refers to UTF-16, explicitly in little-endian order.


Some low-level functions in the Hackerlab C library work with 
any of these five encodings.  Higher-level functions work
only with @code{uni_iso8859_1}, @code{uni_utf8}, and @code{uni_utf16}.


Code units in a @code{uni_utf8} string are of type @code{t_uchar} (unsigned,
8-bit integer).  Code units in a @code{uni_utf16} string are of type
@code{t_unichar} (unsigned 16-bit integer).  Unicode code points are
of type @code{t_unicode}.  (See @ref{Machine-Specific Definitions}.)




@need 3200

@node Representing Unicode Strings
@section Representing Unicode Strings

The Hackerlab C Library is designed to operate correctly for programs
which internally use any @emph{@strong{combination}} of the encodings iso8859-1,
utf-8, and utf-16.  (Future releases are likely to add support for
utf-32.)

@sp 1
@need 1750
@noindent
@b{* Type} @code{uni_string}@tyindex uni_string
@*
@example
@group
typedef struct uni__undefined_struct * uni_string;
@end group
@group
@end group
@end example

The type @code{uni_string} is pointer to a value of unknown size.  It is
used to represent the address of a Unicode string or an address
within a Unicode string.


Any two @code{uni_string} pointers may be compared for equality.


@code{uni_string} pointers within a single string may be compared
using any relational operator (@code{<}, @code{>}, etc.).


@code{uni_string} pointers are created from UTF-8 pointers (@code{t_uchar *})
and from UTF-16 pointers (@code{t_unichar *}) by means of a cast:

@example
@group
     uni_string s = (uni_string)utf_8_string;
     uni_string t = (uni_string)utf_16_string;
@end group
@group
@end group
@end example

By convention, all functions that operate on Unicode strings
accept two parameters for each string: an encoding form, and
a string pointer as in this function declaration:

@example
@group
     void uni_fn (enum uni_encoding_scheme encoding,
                  uni_string s);
@end group
@group
@end group
@end example

By convention, the length of a Unicode string is always measured
in code units, no matter what the size of those code units.
Integer string indexes are also measured in code units.

@need 3200

@subsection Basic Unicode String Functions

These functions were not ready for the current release of the
Hackerlab C Library.  They will be included in future releases.







@need 3200

@node Unicode Character Properties
@section Unicode Character Properties

The functions and macros in this chapter present programs with an
interface to various properties extracted from the Unicode
Character Database as published by the Unicode consortium.


For information about the version of the database used and the
implications of using these functions on program size, see
@ref{Data Sheet for the Hackerlab Unicode Database}.

@menu
* Assigned Code Points::
* General Category::
* Unicode Decimal Digit Values::
* Unicode Bidirectional Properties::
* Canonical Combining Class::
* Simple Unicode Case Conversions::
* Character Decomposition Mapping::
* Unicode Blocks::
* Unicode Category Bitsets::
@end menu

@need 3200

@node Assigned Code Points
@subsection Assigned Code Points
@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_is_assigned_code_point}@fuindex unidata_is_assigned_code_point
@*
@example
@group
int unidata_is_assigned_code_point (t_unicode c);
@end group
@group
@end group
@end example

Return @code{1} if @code{c} is an assigned code point, @code{0} otherwise.


A code point is @emph{assigned} if it has an entry in @code{unidata.txt}
or is part of a range of characters whose end-points are
defined in @code{unidata.txt}.




@need 3200

@node General Category
@subsection General Category
@sp 1
@need 1750
@noindent
@b{* Type} @code{enum unidata_general_category}@tyindex enum unidata_general_category
@*
@example
@group
enum uni_general_category;
@end group
@group
@end group
@end example

The General Category of a Unicode character is represented by
an enumerated value of this type.


The primary category values are:

@example
@group
     uni_general_category_Lu         Letter, uppercase
     uni_general_category_Ll         Letter, lowercase
     uni_general_category_Lt         Letter, titlecase
     uni_general_category_Lm         Letter, modifier
     uni_general_category_Lo         Letter, other"
@end group
@group
@end group
@end example
@example
@group
     uni_general_category_Mn         Mark, nonspacing
     uni_general_category_Mc         Mark, spacing combining
     uni_general_category_Me         Mark, enclosing
@end group
@group
@end group
@end example
@example
@group
     uni_general_category_Nd         Number, decimal digit
     uni_general_category_Nl         Number, letter
     uni_general_category_No         Number, other
@end group
@group
@end group
@end example
@example
@group
     uni_general_category_Zs         Separator, space
     uni_general_category_Zl         Separator, line
     uni_general_category_Zp         Separator, paragraph
@end group
@group
@end group
@end example
@example
@group
     uni_general_category_Cc         Other, control
     uni_general_category_Cf         Other, format
     uni_general_category_Cs         Other, surrogate
     uni_general_category_Co         Other, private use
     uni_general_category_Cn         Other, not assigned
@end group
@group
@end group
@end example
@example
@group
     uni_general_category_Pc         Punctuation, connector
     uni_general_category_Pd         Punctuation, dash
     uni_general_category_Ps         Punctuation, open
     uni_general_category_Pe         Punctuation, close
     uni_general_category_Pi         Punctuation, initial quote
     uni_general_category_Pf         Punctuation, final quote
     uni_general_category_Po         Punctuation, other
@end group
@group
@end group
@end example
@example
@group
     uni_general_category_Sm         Symbol, math
     uni_general_category_Sc         Symbol, currency
     uni_general_category_Sk         Symbol, modifier
     uni_general_category_So         Symbol, other
@end group
@group
@end group
@end example

Seven additional synthetic categories are defined.  These are:

@example
@group
     uni_general_category_L          Letter
     uni_general_category_M          Mark
     uni_general_category_N          Number
     uni_general_category_Z          Separator
     uni_general_category_C          Other
     uni_general_category_P          Punctuation
     uni_general_category_S          Symbol
@end group
@group
@end group
@end example

No character is given a synthetic category as its general category.
Rather, the synthetic categories are used in some interfaces to
refer to all characters having a general category within one of
the synthetic categories.

@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_general_category}@fuindex unidata_general_category
@*
@example
@group
enum uni_general_category unidata_general_category (t_unicode c);
@end group
@group
@end group
@end example

Return the general category of @code{c}.


The category returned for unassigned code points is
@code{uni_general_category_Cn} (Other, Not Assigned).




@need 3200

@node Unicode Decimal Digit Values
@subsection Unicode Decimal Digit Values
@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_decimal_digit_value}@fuindex unidata_decimal_digit_value
@*
@example
@group
int unidata_decimal_digit_value (t_unicode c);
@end group
@group
@end group
@end example

If @code{c} is a decimal digit (regardless of script) return
its digit value.  Otherwise, return @code{-1}.




@need 3200

@node Unicode Bidirectional Properties
@subsection Unicode Bidirectional Properties
@sp 1
@need 1750
@noindent
@b{* Type} @code{enum unidata_bidi_category}@tyindex enum unidata_bidi_category
@*
@example
@group
enum uni_bidi_category;
@end group
@group
@end group
@end example

The Bidrectional Category of a Unicode character is represented by
an enumerated value of this type.


The bidi category values are:

@example
@group
     uni_bidi_L      Left-to-Right
     uni_bidi_LRE    Left-to-Right Embedding
     uni_bidi_LRO    Left-to-Right Override
     uni_bidi_R      Right-to-Left
     uni_bidi_AL     Right-to-Left Arabic
@end group
@group
     uni_bidi_RLE    Right-to-Left Embedding
     uni_bidi_RLO    Right-to-Left Override
     uni_bidi_PDF    Pop Directional Format
     uni_bidi_EN     European Number
     uni_bidi_ES     European Number Separator
@end group
@group
     uni_bidi_ET     European Number Terminator
     uni_bidi_AN     Arabic Number
     uni_bidi_CS     Common Number Separator
     uni_bidi_NSM    Non-Spacing Mark
@end group
@group
     uni_bidi_BN     Boundary Neutral
     uni_bidi_B      Paragraph Separator
     uni_bidi_S      Segment Separator
     uni_bidi_WS     Whitspace
     uni_bidi_ON     Other Neutrals
@end group
@group
@end group
@end example
@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_bidi_category}@fuindex unidata_bidi_category
@*
@example
@group
enum uni_bidi_category unidata_bidi_category (t_unicode c);
@end group
@group
@end group
@end example

Return the bidirectional category of @code{c}.


The category returned for unassigned code points is 
@code{uni_bidi_ON} (other neutrals).

@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_is_mirrored}@fuindex unidata_is_mirrored
@*
@example
@group
int unidata_is_mirrored (t_unicode c);
@end group
@group
@end group
@end example

Return @code{1} if @code{c} is mirrored in bidirectional text, @code{0} 
otherwise.




@need 3200

@node Canonical Combining Class
@subsection Canonical Combining Class
@sp 1
@need 1750
@noindent
@b{* Macro} @code{unidata_canonical_combining_class}@maindex unidata_canonical_combining_class
@*
@example
@group
#define unidata_canonical_combining_class(C)
@end group
@group
@end group
@end example

Return the canonical combining class of a Unicode
character.


Combining classes are represented as unsigned 8-bit integers.




@need 3200

@node Simple Unicode Case Conversions
@subsection Simple Unicode Case Conversions

These functions use the case mappings in @code{unidata.txt}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_to_upper}@fuindex unidata_to_upper
@*
@example
@group
t_unicode unidata_to_upper (t_unicode c);
@end group
@group
@end group
@end example

If @code{c} has a default uppercase mapping, return that mapping.
Otherwise, return @code{c}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_to_lower}@fuindex unidata_to_lower
@*
@example
@group
t_unicode unidata_to_lower (t_unicode c);
@end group
@group
@end group
@end example

If @code{c} has a default lowercase mapping, return that mapping.
Otherwise, return @code{c}.

@sp 1
@need 1750
@noindent
@b{* Function} @code{unidata_to_title}@fuindex unidata_to_title
@*
@example
@group
t_unicode unidata_to_title (t_unicode c);
@end group
@group
@end group
@end example

If @code{c} has a default titlecase mapping, return that mapping.
Otherwise, return @code{c}.




@need 3200

@node Character Decomposition Mapping
@subsection Character Decomposition Mapping
@sp 1
@need 1750
@noindent
@b{* Type} @code{enum uni_decomposition_type}@tyindex enum uni_decomposition_type
@*
@example
@group
enum uni_decomposition_type;
@end group
@group
@end group
@end example

The decomposition mapping of a character is described by
values of this enumerated type:

@example
@group
     uni_decomposition_none
     uni_decomposition_canonical
     uni_decomposition_font
     uni_decomposition_noBreak
     uni_decomposition_initial
@end group
@group
     uni_decomposition_medial
     uni_decomposition_final
     uni_decomposition_isolated
     uni_decomposition_circle
     uni_decomposition_super
@end group
@group
     uni_decomposition_sub
     uni_decomposition_vertical
     uni_decomposition_wide
@end group
@group
     uni_decomposition_narrow
     uni_decomposition_small
     uni_decomposition_square
     uni_decomposition_fraction
     uni_decomposition_compat
@end group
@group
@end group
@end example

The value @code{uni_decomposition_none} indicates that a character
has no decomposition mapping.

@sp 1
@need 1750
@noindent
@b{* Type} @code{struct uni_decomposition_mapping}@tyindex struct uni_decomposition_mapping
@*
@example
@group
struct uni_decomposition_mapping;
@end group
@group
@end group
@end example

A character's decomposition mapping is described by this
structure.  It has the fields:

@example
@group
     enum uni_decomposition_type type;
     t_unicode * decomposition;
@end group
@group
@end group
@end example

@code{type} is the type of decomposition.


If @code{type} is not @code{uni_decomposition_none}, then @code{decomposition}
is a 0-termianted array of code points which are the decomposition
of the character.

@sp 1
@need 1750
@noindent
@b{* Macro} @code{unidata_character_decomposition_mapping}@maindex unidata_character_decomposition_mapping
@*
@example
@group
#define unidata_character_decomposition_mapping(C)
@end group
@group
@end group
@end example

Return the decomposition mapping of @code{C}.  This macro returns
a pointer to a @code{struct uni_decomposition_mapping}.




@need 3200

@node Unicode Blocks
@subsection Unicode Blocks
@sp 1
@need 1750
@noindent
@b{* Type} @code{struct uni_block}@tyindex struct uni_block
@*
@example
@group
struct uni_block;
@end group
@group
@end group
@end example

Structures of this type describe one of the standard blocks of
Unicode characters (@code{"Basic Latin"}, @code{"Latin-1 Supplement"}, etc.)

@example
@group

@end group
@group
struct uni_block
@{
  t_uchar * name;       /* name of the block */
  t_unichar start;      /* first character in the block */
  t_unichar end;        /* last character in the block */
@};
@end group
@group
@end group
@end example
@sp 1
@need 1750
@noindent
@b{* Variable} @code{uni_blocks}@vaindex uni_blocks
@*
@example
@group
extern struct uni_block uni_blocks[];
@end group
@group
@end group
@end example

The names of the standard Unicode blocks.  This array is sorted 
in code-point order, from least to greatest.


@code{n_uni_blocks} is the number of blocks in @code{uni_blocks}.

@example
@group
     uni_blocks[n_uni_blocks].name == 0
@end group
@group
@end group
@end example
@example
@group

@end group
@group
extern const struct uni_block uni_blocks[];
extern const int n_uni_blocks;
@end group
@group

@end group
@end example



@need 3200

@node Unicode Category Bitsets
@subsection Unicode Category Bitsets
@sp 1
@need 1750
@noindent
@b{* Function} @code{uni_universal_bitset}@fuindex uni_universal_bitset
@*
@example
@group
bits uni_universal_bitset (void);
@end group
@group
@end group
@end example

Return the set of all assigned code points which are not surrogate
code points and are not private use code points.  The set is
represented as a shared bitset tree.  (See @ref{Shared Bitset Trees}.)


The shared bitset tree returned by this function uses the tree
structure defined by @code{uni_bits_tree_rule}.  (See @ref{Unicode Character Bitsets}.)


Programs should not attempt to modify the set returned by
this function.

@sp 1
@need 1750
@noindent
@b{* Function} @code{uni_general_category_bitset}@fuindex uni_general_category_bitset
@*
@example
@group
bits uni_general_category_bitset (enum uni_general_category c);
@end group
@group
@end group
@end example

Return the set of all assigned code points having the indicated
general category or synthetic general category.  The set is
represented as a shared bitset tree.  (See @ref{Shared Bitset Trees}.)


@code{c} indicates which category to return.  It may be a Unicode
general category or a synthetic general category.  (See
@ref{General Category}.)


The shared bitset tree returned by this function uses the tree
structure defined by @code{uni_bits_tree_rule}.  (See @ref{Unicode Character Bitsets}.)


Programs should not attempt to modify the set returned by
this function.










