@need 3200

@node Comments on the XML Schema Regular Expression Syntax
@appendix Comments on the XML Schema Regular Expression Syntax

@geindex regular expression language

@geindex XML

@geindex W3C

@geindex Schema (XML)

@geindex XML Schema

@geindex errors (in the XML specification)



Appendix E of the W3C publication @emph{XML Schema Part 2: Datatypes} defines
a syntax for Unicode regular expressions.  That syntax is supported by
Rx. (See @ref{XML Regular Expression Functions}.)


This chapter describes some minor error's in W3C's specification (as
of the "24 October 2000" draft).  It will make the most sense if read
alongside the W3C specification.

@menu
* | is not Specified as a Metacharacter::
* The Specification of \i is Wrong::
* The Specification of \w is Wrong::
* Character Class Expression Syntax is Botched::
@end menu

@need 3200

@node | is not Specified as a Metacharacter
@section | is not Specified as a Metacharacter

Taking the grammar specification literally, the following
regular expression is ambiguous:

@example
@group
        a|b
@end group
@group
@end group
@end example

It could match one of:

@example
@group
        a
        b
@end group
@group
@end group
@end example

@noindent
or it could match only:

@example
@group
        a|b
@end group
@group
@end group
@end example

That's because @code{|} is defined both as the separator character for
branches, and as a normal character.  It is defined as a normal
character because it is an XML character that is not included in the
list of meta-characters.


@strong{Solution:} Add @code{|} to the list of meta-characters.  As a
side-effect, a new single character escape is created:

@example
@group
        \|
@end group
@group
@end group
@end example

@noindent
which denotes a literal @code{|}.




@need 3200

@node The Specification of \i is Wrong
@section The Specification of \i is Wrong

We presume that the intended meaning of the multi-character escape
@code{\i} is "Any character which may be the first character of an XML
Name".  That makes @code{\i} useful in conjunction with @code{\c}.


The specification says that multi-character escape @code{\i} is:

@example
@group
        [\p@{L@}\p@{Nl@}:_]
@end group
@group
@end group
@end example

@noindent
which doesn't agree with the XML definition of a name, and is besides
an odd definition.


@strong{Solution:} Define @code{\i} as the set of character that match the
production:

@example
@group
        NameInitial     ::= (Letter | '_' | ':')
@end group
@group
@end group
@end example

@noindent
if that production is added to the grammar for XML (in
@url{http://www.w3.org/TR/REC-xml} ).




@need 3200

@node The Specification of \w is Wrong
@section The Specification of \w is Wrong

The specification says that multi-character escape @code{\w} is:

@example
@group
    [&#x0000;-&#xFFFF;-[\p@{P@}\p@{S@}\p@{C@}]]
@end group
@group
@end group
@end example

@noindent
but also that it is:

@example
@group
    all characters except the set of "punctuation", "separator" and
    "control" characters
@end group
@group
@end group
@end example

The range @code{&#x0000;-&xFFFF} contains code points which are not valid
XML characters and which are not valid Unicode characters.
Furthermore, it does not contain code points (above @code{0xffff}) which
are valid XML and Unicode characters.


@strong{Solution:} Define @code{\w} as:

@example
@group
     [^\p@{P@}\p@{Z@}\p@{C@}]
@end group
@group
@end group
@end example



@need 3200

@node Character Class Expression Syntax is Botched
@section Character Class Expression Syntax is Botched

The syntax of character class expressions is ambiguous.  If the
ambiguity is ignored, the syntax requires an arbitrary amount of
look-ahead to parse correctly.


@strong{The Specified Syntax:} 


The basic syntax for a character class expression (as currently
specified) is:

@example
@group
     character_class_expression: '[' character_group ']'
         |    '[' character_group '-' character_class_expression ']'
@end group
@group
@end group
@end example
@example
@group
     character_group:   positive_character_group
         |    '^' positive_character_group
@end group
@group
@end group
@end example

A @code{positive_character_group} is defined as:

@example
@group
      one or more character ranges or character class escapes,
      concatenated together.
@end group
@group
@end group
@end example

A character range may be a single character escape, or any ordinary
XML character other than one of @code{[]\^-}, except that @code{^} is permitted
as the first character of a @code{positive_character_group} following the
@code{^} that indicates a negated character group, and @code{-} is permitted as
either the first or last character of any @code{positive_character_group}.


A character range is a pair of characters:

@example
@group
        s-e
@end group
@group
@end group
@end example

@noindent
where @code{s} may be a single character escape or any XML character
other than one of @code{^\}, except that @code{^} is permitted if @code{s} is not
the first character of a character class expression.


@code{e} may be a single character escape or any XML character which is
not one of @code{[\}.


Finally, the code point of @code{e} must be greater than or equal to the
code point of @code{s}.


@strong{The Problems:}


First, consider the ambiguous expression:

@example
@group
        [abc]-z[-def]
@end group
@group
@end group
@end example

On the one hand, the expression matches four characters such as:

@example
@group
        a-zd
        a-ze
        a-z-
@end group
@group
@end group
@end example

@noindent
because @code{[abc]} and @code{[-def]} are valid character class expressions.
Note that we are using the special case exception that @code{-} is
permitted as the first character of a positive character group.


On the other hand, the expression matches one character:

@example
@group
        a
        b
        c
        w
        x
        ]
        [
        ...
@end group
@group
@end group
@end example

@noindent
because @code{]-z} and @code{[-d} are valid character ranges.  Here we are using
the special case that the start character of a character range which
is not the first element of a character class expression may be any
XML character other than @code{\}.


Second, consider the expression:

@example
@group
       ['--9]
@end group
@group
@end group
@end example

The specification does not make it clear whether that is set
containing the range:

@example
@group
       '--
@end group
@group
@end group
@end example

@noindent
and the character:

@example
@group
       9
@end group
@group
@end group
@end example

@noindent
or a set containing the character:

@example
@group
      '
@end group
@group
@end group
@end example

@noindent
and the range:

@example
@group
      --9
@end group
@group
@end group
@end example

We are again using the special case rule that @code{-} is not a
meta-character in a character class expression if it occurs as an
end-point of a character class expression.


Third, consider the first four characters of an XML regular expression:

@example
@group
        [A-]
@end group
@group
@end group
@end example

Is that a two character character class?  It is if the entire regular
expression is:

@example
@group
        [A-]mistake
@end group
@group
@end group
@end example

@noindent
because in that case, @code{-} occurs as the last character of a
@code{positive_character_group}.


On the other hand, if the entire regular expression is:

@example
@group
        [A-]mistake]
@end group
@group
@end group
@end example

@noindent
then the @code{-} indicates a character range from @code{A} to @code{]}.


Overall, the special case rules concerning characters in @code{[]^-} create
ambiguities, make it difficult to write a correct parser, difficult to
read some expressions correctly, and easier than it needs to be to
write an un-intended pattern.


@strong{Proposed Solutions}


We propose several possible ways to solve the problems with character
class expressions.  All of our proposals are based on this
(incomplete) grammar:

@example
@group
     character_class_expression: '[' character_group ']'
         |      '[' character_group 
                    SUBTRACTION character_class_expression ']'
@end group
@group
@end group
@end example
@example
@group
     character_group:   positive_character_group
         |      '^' positive_character_group
@end group
@group
@end group
@end example
@example
@group
     positive_character_group:
                positive_character_subgroup
        |       positive_character_subgroup positive_character_group
@end group
@group
@end group
@end example
@example
@group
     positive_character_subgroup:
                multi_character_escape
        |       category_escape
        |       range_endpoint
        |       range_endpoint RANGE range_endpoint
@end group
@group
@end group
@end example
@example
@group
     range_endpoint:
                single_character_escape
        |       normal_range_character
@end group
@group
@end group
@end example
@example
@group
     normal_range_character:            [\p@{isXmlChar@}-SPECIALS]
@end group
@group
@end group
@end example

Three productions are missing from that grammar and we have several
ideas of how each may be defined.  Those productions are for:

@example
@group
        SUBTRACTION     the character class subtraction operator
@end group
@group
@end group
@end example
@example
@group
        RANGE           the character class range operator
@end group
@group
@end group
@end example
@example
@group
        SPECIALS        the set of characters that are
                        special in a character class expression.
                        These characters must be escaped by
                        `\' to have literal meaning in a
                        character class expression.
@end group
@group
@end group
@end example

For @code{SUBTRACTION}, we propose one of these alternatives:

@example
@group
        [s1]    SUBTRACTION: '^'
        [s2]    SUBTRACTION: '-'
@end group
@group
@end group
@end example

We include @code{[s1]} because that is the syntax used in @emph{Unicode
Technical Report #18} (available from @url{http://www.unicode.org} ).


For @code{RANGE}, we propose one of:

@example
@group
        [r1]    RANGE: '..'
        [r2]    RANGE: '-'
@end group
@group
@end group
@end example

For @code{SPECIALS}, we propose one of:

@example
@group
        [sp1]   SPECIALS: [\@{isMetaChar@} SUBTRACTION RANGE]
        [sp2]   SPECIALS: [\@{isCCMetaChar@} SUBTRACTION RANGE]
@end group
@group
@end group
@end example

@noindent
where the set @geindex MetaChar
@dfn{MetaChar} contains the characters that have special
meaning @emph{@strong{outside}} of character class expressions which are:

@example
@group
        . \ ? * + @{ @} ( ) [ ] |
@end group
@group
@end group
@end example

@noindent
where the set @geindex CCMetaChar
@dfn{CCMetaChar} contains the characters that have
special meaning @emph{@strong{inside}} of character class expressions in 
all of the proposed syntaxes.  These are:

@example
@group
        \ [ ]   
@end group
@group
@end group
@end example

@strong{Discussion:}


We have described a solution space of eight possible repairs
to the character class syntax.  Each solution in this 
space differs from the others in one or more of: the syntax
for character set subtraction; the syntax for character ranges;
the set of characters that must be escaped (with @code{\}) to have
literal meaning.


In our implementation of an XML regular expression matcher, the
default syntax we provided is:

@example
@group
        [s2]    SUBTRACTION: '-'
        [r2]    RANGE: '-'
        [sp2]   SPECIALS: [\@{isCCMetaChar@} SUBTRACTION RANGE]
@end group
@group
@end group
@end example

We made that the default because it most closely resembles the syntax
of the current specification, but removes the ambiguities.


If we had free reign to change the standard, we would instead choose:

@example
@group
        [s2]    SUBTRACTION: '-'
        [r1]    RANGE: '..'
        [sp1]   SPECIALS: [\@{isMetaChar@} SUBTRACTION RANGE]
@end group
@group
@end group
@end example

@noindent
for several reasons:


@strong{1} We believe that using @code{..} for ranges and @code{-} for subtraction
avoids confusion and is intuitive for most people.


@strong{2} We believe that requiring all meta-characters to be quoted
to have literal meaning in a character set expression is an easy
rule to understand and use.


@strong{3} Requiring meta-characters to be quoted leaves room for future
extensions in which unused meta-characters are given special meaning
in character class expressions.  For example, @code{^} could be used as a
"symmetric difference" ("xor") operator.  







