Paul Eggert <eggert@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at submit) by debbugs.gnu.org; 6 Jun 2014 04:02:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jun 06 00:02:10 2014
Received: from localhost ([127.0.0.1]:46357 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.80)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1WslM1-0003rP-Sc
for submit <at> debbugs.gnu.org; Fri, 06 Jun 2014 00:02:10 -0400
Received: from eggs.gnu.org ([208.118.235.92]:44382)
by debbugs.gnu.org with esmtp (Exim 4.80)
(envelope-from <behoffski@HIDDEN>) id 1WslLy-0003qk-Vk
for submit <at> debbugs.gnu.org; Fri, 06 Jun 2014 00:02:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <behoffski@HIDDEN>) id 1WslLZ-0002eM-A1
for submit <at> debbugs.gnu.org; Fri, 06 Jun 2014 00:02:01 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level:
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,
T_FILL_THIS_FORM_SHORT autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:56224)
by eggs.gnu.org with esmtp (Exim 4.71)
(envelope-from <behoffski@HIDDEN>) id 1WslLZ-0002dn-05
for submit <at> debbugs.gnu.org; Fri, 06 Jun 2014 00:01:41 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:46025)
by lists.gnu.org with esmtp (Exim 4.71)
(envelope-from <behoffski@HIDDEN>) id 1WslLE-00079Q-J6
for bug-grep@HIDDEN; Fri, 06 Jun 2014 00:01:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <behoffski@HIDDEN>) id 1WslKu-0002ZP-Ab
for bug-grep@HIDDEN; Fri, 06 Jun 2014 00:01:20 -0400
Received: from ipmail06.adl2.internode.on.net
([2001:44b8:8060:ff02:300:1:2:6]:4647)
by eggs.gnu.org with esmtp (Exim 4.71)
(envelope-from <behoffski@HIDDEN>) id 1WslKq-0002Z8-O5
for bug-grep@HIDDEN; Fri, 06 Jun 2014 00:01:00 -0400
X-IronPort-Anti-Spam-NotFiltered: toobig
Received: from ppp14-2-27-84.lns21.adl2.internode.on.net (HELO [192.168.1.1])
([14.2.27.84])
by ipmail06.adl2.internode.on.net with ESMTP; 06 Jun 2014 13:30:52 +0930
Message-ID: <53913CF2.2000404@HIDDEN>
Date: Fri, 06 Jun 2014 13:30:50 +0930
From: behoffski <behoffski@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: bug-grep@HIDDEN
Subject: Files created/modified by "untangle" script
Content-Type: multipart/mixed; boundary="------------050008000707080900010407"
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
recognized.
X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address
(bad octet value).
X-Received-From: 2001:4830:134:3::11
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
This is a multi-part message in MIME format.
--------------050008000707080900010407
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
G'day,
Running untangle, then make, in the grep/src directory, gives the
following output (I've added blank lines around the commands):
------
grep/src $ ./untangle
Creating charclass.h...
Creating charclass.c...
Creating fsatoken.h...
Creating fsatoken.c...
Creating mbcsets.h...
Creating mbcsets.c...
Creating proto-lexparse.h...
Creating fsalex.h...
Creating fsalex.c...
Creating fsaparse.h...
Creating fsaparse.c...
Creating fsamusts.h...
Creating fsamusts.c...
Generating dfa-prl.c by copying dfa.c and then applying edits
Modifying (if needed) Makefile.am; you may need to re-run automake...
grep/src $ make
cd .. && /bin/sh /home/grep-gnu/grep/build-aux/missing automake-1.13 --gnu src/Makefile
cd .. && /bin/sh ./config.status src/Makefile depfiles
config.status: creating src/Makefile
config.status: executing depfiles commands
CC charclass.o
CC fsatoken.o
CC fsalex.o
CC fsaparse.o
CC fsamusts.o
CC mbcsets.o
CC dfa-prl.o
CCLD grep
GEN egrep
GEN fgrep
grep/src $
------
Attached are the files created or modified by the script.
cheers,
behoffski (Brenton Hoff)
Programmer, Grouse Software
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="charclass.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="charclass.c"
/* charclass -- Tools to create and manipulate sets of C "char"s
This module provides tools to create, modify, store and retrieve character
classes, and provides tools tuned to the needs of RE lexical analysers.
The class itself is an opaque type, referenced by a pointer while under
construction, and later by an unique index when completed. The module
tries aggressively to reuse existing completed classes, rather than create
duplicates. Functions are provided to map between indexes and pointers.
Because of the deduplication effort, the index reported for a class upon
completion may map to a different pointer than the one supplied by new ().
Classes may be shared between different lexer instances, although, at the
time of writing (10 April 2014) it is not thread-safe. In many cases,
there might only be one class under construction at any time, with the
effort either completed or abandoned quickly. However, this module
recognises that sometimes multiple classes might be worked on in parallel,
and so explicitly marks each allocated class area as one of "unused",
"work" or "completed". This marking is done by an array of state bytes
dynamically allocated when the pool is created.
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* Always import environment-specific configuration items first. */
#include <config.h>
#include <assert.h>
#include "charclass.h"
#include <limits.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <stdio.h> /* for EOF assert test. */
#include <string.h>
#include <wchar.h> /* for WEOF assert test. */
#include "xalloc.h"
/* Lower bound for size of first pool in the list. */
/* ?? Set to 2 for pool debug; Use 10 in production? */
#define POOL_MINIMUM_INITIAL_SIZE 10
#ifndef MAX
# define MAX(a,b) ((a) > (b) ? (a) : (b))
#endif
#ifndef MIN
# define MIN(a,b) ((a) < (b) ? (a) : (b))
#endif
/* We maintain a list-of-pools here, choosing to malloc a new slab of
memory each time we run out, instead of a realloc strategy. This is so
that we can provide a guarantee to the user that any class pointer issued
remains valid for the lifetime of the module. */
typedef size_t pool_list_index_t;
/* Designator for each charclass in each pool. Note that enums are ints by
default, but we use a single unsigned char per class in our explicit
memory allocation. */
typedef enum
{
STATE_UNUSED = 0,
STATE_WORKING = 1,
STATE_COMPLETED = 2
} charclass_state_t;
typedef struct pool_info_struct {
charclass_index_t first_index;
size_t alloc; /* ?? Use pool_list_index_t type for these? */
size_t used;
charclass_t *classes;
/* Charclass designator byte array, one per item, allocated dynamically. */
unsigned char *class_state;
} pool_t;
static pool_list_index_t pool_list_used = 0;
static pool_list_index_t pool_list_alloc = 0;
static pool_t *pool_list = NULL;
/* While the header only guarantees a 3-bit gutter at each end of each
class, we use an entire word (typically 32 bits) for the gutter,
with at least 1 word placed at the start of each pool, 1 word as a
shared gutter between each class, and 1 word after the last
class. */
/* HPUX defines these as macros in sys/param.h. */
#ifdef setbit
# undef setbit
#endif
#ifdef clrbit
# undef clrbit
#endif
/* This represents part of a character class. It must be unsigned and
at least CHARCLASS_WORD_BITS wide. Any excess bits are zero. */
typedef unsigned int charclass_word;
/* The number of bits used in a charclass word. utf8_classes assumes
this is exactly 32. */
enum { CHARCLASS_WORD_BITS = 32 };
/* The maximum useful value of a charclass_word; all used bits are 1. */
#define CHARCLASS_WORD_MASK \
(((charclass_word) 1 << (CHARCLASS_WORD_BITS - 1) << 1) - 1)
/* Number of words required to hold a bit for every character. */
enum
{
CHARCLASS_WORDS = (CHARCLASS_NOTCHAR + CHARCLASS_WORD_BITS - 1) / CHARCLASS_WORD_BITS
};
/* Flesh out opaque charclass type given in the header */
/* The gutter element following the class member storage also serves as
the gutter element p-receding the next class in the list.
Note that since the "gutter" notion explicitly permits a restricted set
of negative indices, members need to be signed, not unsigned, so that
arithmetic shift right can be used where possible (e.g. -8 >> 8 == -1,
not -8 / 256 == 0). */
struct charclass_struct {
charclass_word members[CHARCLASS_WORDS];
charclass_word gutter_following;
};
/* Define class bit operations: test, set and clear a bit.
Grrrr. I wanted to exploit arithmetic right shift to come up with a
really cheap and neat way of reducing small negative bit values,
especially if b == EOF ==-1, to an index of -1 that falls neatly
into the gutter, but strict C conformance does not guarantee this.
The code below handles the two most likely scenarios, but, as with
anything that is undefined, this is playing with fire. */
#if INT_MAX == 2147483647
# define INT_BITS_LOG2 5 /* log2(sizeof(int)) + log2(CHARCLASS_WORD_BITS) */
#else
# error "Not implemented: Architectures with ints other than 32 bits"
#endif
#if ((~0 >> 1) < 0)
/* Arithmetic shift right: Both signed and unsigned cases are ok. */
# define ARITH_SHIFT_R_INT(b) ((b) >> INT_BITS_LOG2)
#else
/* Avoid using right shift if b is negative. The macro may evaluate b twice
in some circumstances. */
# define ARITH_SHIFT_R_INT(b) \
(((b) < 0) ? -1 : ((b) >> INT_BITS_LOG2))
#endif
bool _GL_ATTRIBUTE_PURE
charclass_tstbit (int b, charclass_t const *ccl)
{
return ccl->members[ARITH_SHIFT_R_INT(b)] >> b % CHARCLASS_WORD_BITS & 1;
}
void
charclass_setbit (int b, charclass_t *ccl)
{
ccl->members[ARITH_SHIFT_R_INT(b)] |= (charclass_word) 1 << b % CHARCLASS_WORD_BITS;
}
void
charclass_clrbit (int b, charclass_t *ccl)
{
ccl->members[ARITH_SHIFT_R_INT(b)] &= ~((charclass_word) 1
<< b % CHARCLASS_WORD_BITS);
}
void
charclass_setbit_range (int start, int end, charclass_t *ccl)
{
int bit;
/* Do nothing if the range doesn't make sense. */
if (end < start)
return;
if (start >= CHARCLASS_NOTCHAR)
return;
/* Clip the range to be in the interval [-1..NOTCHAR - 1] */
start = MAX(start, -1);
end = MAX(end, -1);
/* We know start is < CHARCLASS_NOTCHAR from the test above. */
end = MIN(end, CHARCLASS_NOTCHAR - 1);
/* ?? Could check that ccl is a valid class, but not at present. */
/* Okay, loop through the range, bit-by-bit, setting members. */
for (bit = start; bit <= end; bit++)
ccl->members[ARITH_SHIFT_R_INT(bit)] |= 1U << bit % CHARCLASS_WORD_BITS;
}
void
charclass_clrbit_range (int start, int end, charclass_t *ccl)
{
int bit;
/* Do nothing if the range doesn't make sense. */
if (end < start)
return;
if (start >= CHARCLASS_NOTCHAR)
return;
/* Clip the range to be in the interval [-1..NOTCHAR - 1] */
start = MAX(start, -1);
end = MAX(end, -1);
/* We know start is < CHARCLASS_NOTCHAR from the test above. */
end = MIN(end, CHARCLASS_NOTCHAR - 1);
/* ?? Could check that ccl is a valid class, but not at present. */
/* Okay, loop through the range, bit-by-bit, clearing members. */
for (bit = start; bit <= end; bit++)
ccl->members[ARITH_SHIFT_R_INT(bit)] &= ~(1U << bit % CHARCLASS_WORD_BITS);
}
/* Define whole-set operations: Copy, clear, invert, compare and union */
void
charclass_copyset (charclass_t const *src, charclass_t *dst)
{
memcpy (dst->members, src->members, sizeof(src->members));
}
void
charclass_zeroset (charclass_t *ccl)
{
memset (ccl->members, 0, sizeof(ccl->members));
}
void
charclass_notset (charclass_t *ccl)
{
int i;
for (i = 0; i < CHARCLASS_WORDS; ++i)
ccl->members[i] = CHARCLASS_WORD_MASK & ~ccl->members[i];
}
int _GL_ATTRIBUTE_PURE
charclass_equal (charclass_t const *ccl1, charclass_t const *ccl2)
{
return memcmp (ccl1->members, ccl2->members,
sizeof(ccl1->members)) == 0;
}
void
charclass_unionset (charclass_t const *src, charclass_t *dst)
{
int i;
for (i = 0; i < CHARCLASS_WORDS; ++i)
dst->members[i] |= src->members[i];
}
void
charclass_intersectset (charclass_t const *src, charclass_t *dst)
{
int i;
for (i = 0; i < CHARCLASS_WORDS; ++i)
dst->members[i] &= src->members[i];
}
/* #ifdef DEBUG */
/* Nybble (4bit)-to-char conversion array for little-bit-endian nybbles. */
static const char *disp_nybble = "084c2a6e195d3b7f";
/* Return a static string describing a class (Note: not reentrant). */
char *
charclass_describe (charclass_t const *ccl)
{
/* The string should probably be less than 90 chars, but overcompensate
for limited uncertainty introduced by the %p formatting operator. */
static char buf[256];
char *p_buf = buf;
int i;
p_buf += sprintf (p_buf, "0x%08lx:", (unsigned long) ccl);
for (i = 0; i < CHARCLASS_WORDS; i += 2)
{
int j = ccl->members[i];
*p_buf++ = ' ';
*p_buf++ = disp_nybble[(j >> 0) & 0x0f];
*p_buf++ = disp_nybble[(j >> 4) & 0x0f];
*p_buf++ = disp_nybble[(j >> 8) & 0x0f];
*p_buf++ = disp_nybble[(j >> 12) & 0x0f];
*p_buf++ = disp_nybble[(j >> 16) & 0x0f];
*p_buf++ = disp_nybble[(j >> 20) & 0x0f];
*p_buf++ = disp_nybble[(j >> 24) & 0x0f];
*p_buf++ = disp_nybble[(j >> 28) & 0x0f];
j = ccl->members[i + 1];
*p_buf++ = disp_nybble[(j >> 0) & 0x0f];
*p_buf++ = disp_nybble[(j >> 4) & 0x0f];
*p_buf++ = disp_nybble[(j >> 8) & 0x0f];
*p_buf++ = disp_nybble[(j >> 12) & 0x0f];
*p_buf++ = disp_nybble[(j >> 16) & 0x0f];
*p_buf++ = disp_nybble[(j >> 20) & 0x0f];
*p_buf++ = disp_nybble[(j >> 24) & 0x0f];
*p_buf++ = disp_nybble[(j >> 28) & 0x0f];
}
*p_buf++ = '\0';
return buf;
}
/* static */ void
debug_pools (const char *label, bool class_contents)
{
pool_list_index_t pool_nr;
size_t class_nr;
printf ("\nPool %p debug(%s): [alloc, used: %ld %ld]\n",
pool_list, label, pool_list_alloc, pool_list_used);
for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
{
pool_t *pool = &pool_list[pool_nr];
printf (
" %3ld: first_index, alloc, used, classes: %4ld %3lu %3lu %p\n",
pool_nr, pool->first_index, pool->alloc, pool->used, pool->classes);
printf (" class_states: ");
for (class_nr = 0; class_nr < pool->alloc; class_nr++)
switch (pool->class_state[class_nr]) {
case STATE_UNUSED: putchar ('.'); break;
case STATE_WORKING: putchar ('w'); break;
case STATE_COMPLETED: putchar ('C'); break;
default: printf ("?%02x", pool->class_state[class_nr]);
}
putchar ('\n');
}
/* If class contents requested, print them out as well. */
if (class_contents)
for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
{
pool_t *pool = &pool_list[pool_nr];
for (class_nr = 0; class_nr < pool->used; class_nr++)
printf ("%s\n",
charclass_describe (&pool->classes[class_nr]));
}
}
/* #endif * DEBUG */
static pool_t *
add_new_pool (void)
{
pool_t *prev, *pool;
size_t pool_class_alloc;
charclass_t *alloc_mem;
/* If the pools list is full, use x2nrealloc to expand its size. */
if (pool_list_used == pool_list_alloc)
pool_list = x2nrealloc (pool_list, &pool_list_alloc, sizeof (pool_t));
/* Find the size of the last charclass pool in the (old) list. Scale up
the size so that malloc activity will decrease as the number of pools
increases. Also, add 1 here as we knock off 1 to use as a gutter
later. */
prev = &pool_list[pool_list_used - 1];
pool_class_alloc = (prev->alloc * 5 / 2) + 1;
alloc_mem = XNMALLOC (pool_class_alloc, charclass_t);
/* Set up the new pool, shifting the alloc pointer to create the gutter
preceding the first class of the pool. */
pool = &pool_list[pool_list_used++];
pool->classes = alloc_mem + 1;
pool->first_index = prev->first_index + prev->alloc;
pool->alloc = pool_class_alloc - 1;
pool->used = 0;
pool->class_state = xzalloc (pool->alloc);
return pool;
}
charclass_t *
charclass_alloc (void)
{
pool_list_index_t pool_nr;
charclass_t *class;
pool_t *pool = NULL;
size_t class_nr;
size_t class_last_nr;
charclass_word *gutter_preceding;
/* Locate a pool with unused entries (if any). */
for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
{
pool = &pool_list[pool_nr];
/* Try use the earliest pool possible, first by filling in a hole
from a withdrawn class, or by grabbing an unused class from the
end of the list. */
class_last_nr = MIN(pool->used + 1, pool->alloc);
for (class_nr = 0; class_nr < class_last_nr; class_nr++)
{
if (pool->class_state[class_nr] == STATE_UNUSED)
goto found_pool_and_class;
}
}
/* No space found, so prepare a new pool and make this class its first
element. */
pool = add_new_pool ();
class_nr = 0;
/* FALLTHROUGH */
found_pool_and_class:
/* Mark the found class state as working, zero its elements, and return
class pointer to caller. Zeroing is needed as this class may have
been previously worked on, but then abandoned or withdrawn. */
pool->class_state[class_nr] = STATE_WORKING;
if (class_nr >= pool->used)
pool->used = class_nr + 1;
class = &pool->classes[class_nr];
/* Zero out the class' members, and also the gutters on each side. */
memset (class, 0, sizeof (*class));
gutter_preceding = ((charclass_word *) class) - 1;
*gutter_preceding = 0;
return class;
}
pool_t * _GL_ATTRIBUTE_PURE
find_class_pool (charclass_t const *ccl)
{
pool_list_index_t pool_nr;
pool_t *pool = NULL;
ptrdiff_t class_ptr_offset;
/* Locate the pool whose memory address space covers this class. */
/* ?? Perhaps check &pool->classes[pool->alloc] in this first loop, and
then check that the index is in the "used" portion later, so we can
diagnose malformed pointers more exactly. */
for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
{
pool = &pool_list[pool_nr];
if ((pool->classes <= ccl) && (ccl < &pool->classes[pool->alloc]))
goto found_pool;
}
/* No credible pool candidate was found. */
assert ("find_class_pool: no pool found");
return NULL;
found_pool:
/* Make sure the class clearly lies on an array boundary within the pool's
memory allocation. */
class_ptr_offset = (char *) ccl - (char *) pool->classes;
if ((class_ptr_offset % sizeof (charclass_t)) != 0)
{
/* Pointer does not lie at the start of a pool member. */
assert ("find_class_pool: pointer not aligned.");
return NULL;
}
return pool;
}
static void
withdraw_class (charclass_t *ccl, pool_t *class_pool)
{
pool_t *pool;
size_t class_nr;
int *gutter_preceding;
/* Use pool reference if given, otherwise work back from the class pointer
to find the associated pool. */
pool = (class_pool != NULL) ? class_pool : find_class_pool (ccl);
if (pool == NULL)
assert (!"Could not locate a pool for this charclass");
/* Zero out the gutters each side of the class. */
ccl->gutter_following = 0;
gutter_preceding = ((int *) ccl) - 1;
*gutter_preceding = 0;
/* Work out the class index in the pool. */
class_nr = ccl - pool->classes;
pool->class_state[class_nr] = STATE_UNUSED;
/* Is this the last item within the pool's class list? */
if (class_nr == pool->used - 1)
{
/* Yes, reduce the pool member count by 1. */
pool->used--;
return;
}
}
/* Finish off creating a class, and report an index that can be used
to reference the class. */
charclass_index_t
charclass_completed (charclass_t *ccl)
{
charclass_word *gutter_preceding;
pool_list_index_t pool_nr;
pool_t *pool;
charclass_t *found = NULL;
size_t class_nr;
pool_t *my_pool = NULL;
size_t my_class_nr = 0;
/* Search all pools for a completed class matching this class, and, if
found, use it in preference to the new one. While searching, also
record where the work class is located. If we can't find ourselves, the
pointer is invalid, and throw an assertion. */
for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
{
pool = &pool_list[pool_nr];
for (class_nr = 0; class_nr < pool->used; class_nr++)
{
charclass_t *search = &pool->classes[class_nr];
/* Have we found ourselves in the list? */
if (search == ccl)
{
/* Yes, remember this place in case no duplicate is found. */
my_pool = pool;
my_class_nr = class_nr;
}
if (pool->class_state[class_nr] != STATE_COMPLETED)
continue;
if (charclass_equal (search, ccl))
{
/* Another class, completeded, matches: Use it in preference
to potentially creating a duplicate. */
withdraw_class (ccl, my_pool);
found = search;
goto found_matching_class;
}
}
}
/* No duplicate found... but make sure the search pointer is known. */
assert (my_pool != NULL);
assert (my_pool->class_state[my_class_nr] == STATE_WORKING);
/* Prepare to convert the search (work) class into a completed class. */
pool = my_pool;
class_nr = my_class_nr;
found = &pool->classes[class_nr];
/* FALLTHROUGH */
found_matching_class:
/* Clear out the gutter integers each side of the class entry. */
gutter_preceding = found->members - 1;
*gutter_preceding = 0;
found->gutter_following = 0;
pool->class_state[class_nr] = STATE_COMPLETED;
/* Return the index of the class. */
return pool->first_index + class_nr;
}
void
charclass_abandon (charclass_t *ccl)
{
withdraw_class (ccl, NULL);
}
/* Additional functions to help clients work with classes. */
charclass_t * _GL_ATTRIBUTE_PURE
charclass_get_pointer (charclass_index_t const index)
{
pool_list_index_t pool_nr;
pool_t *pool;
/* Does this class match any class we've seen previously? */
for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
{
/* Is the index inside this pool? */
pool = &pool_list[pool_nr];
if (pool->first_index <= index
&& index < (pool->first_index + pool->used))
{
/* Yes, find the pointer within the pool and return it. */
return &pool->classes[index - pool->first_index];
}
}
/* The mapping above should never fail; we could return NULL, but we
choose to abort instead. */
assert (!"index-to-charclass mapping failed");
return NULL;
}
charclass_index_t _GL_ATTRIBUTE_PURE
charclass_get_index (charclass_t const *ccl)
{
pool_t *pool;
/* This code is similar to charclass_completed... perhaps merge? */
pool = find_class_pool (ccl);
if (pool == NULL)
return -1;
/* Report the index to the caller. */
return pool->first_index + (ccl - pool->classes);
}
/* Functions to initialise module on startup, and to shut down and
release acquired resources at exit. */
void
charclass_destroy_module (void)
{
pool_t *local_list;
int i;
charclass_t *alloc_mem;
/* Move the global list head into a local variable, and immediately clear
the global. This is a half-hearted attempt to avoid race conditions;
to do things properly, a system-wide atomic operation (locked,
including multi-CPU cache coherency) operation should be used. */
local_list = pool_list;
pool_list = NULL;
/* If the list is already empty, finish now. */
if (! local_list)
return;
/* First, discard the charclass memory associated with each pool,
including catering for the offset used upon creation. */
for (i = 0; i < pool_list_used; i++)
{
/* We skipped an initial class after allocation, in order to give
the first class in the pool a gutter, so undo the offset here
when freeing. */
/* REMOVEME: Was typed as "guitter" above, before fixing; how about
calling it some variant of "guitar"?! */
alloc_mem = local_list[i].classes;
free (alloc_mem - 1);
/* Also free up the class_state memory. */
free (local_list[i].class_state);
}
/* Second, free up the pool list itself. */
free (local_list);
pool_list_used = 0;
}
void
charclass_initialise (size_t initial_pool_size)
{
size_t initial_alloc;
charclass_t *alloc_mem;
pool_t *pool;
charclass_t *ccl;
charclass_index_t zeroclass_index;
/* Usually (gnulib assumption?) EOF = WEOF = -1, but the standard
merely states that EOF must be a negative integer, not necessarily
-1; furthermore, the definition for (ISO C90) WEOF states that it
need not be negative at all, let alone guaranteed to be -1.
In this demonstration/prototype code, we demand that both EOF and
WEOF be -1, as this value makes the gutters worthwhile: The user
can be relieved of some edge case processing if -1 is thrown
neatly into the gutter . */
assert (EOF == -1);
assert (WEOF == -1);
/* First, set up the list-of-pools structure with initial storage. */
pool_list_alloc = 4;
pool_list = (pool_t *) xnmalloc (pool_list_alloc, sizeof (pool_t));
/* If initial pool size is small, inflate it here as we prefer to
waste a little memory, rather than issue many calls to xmalloc ().
This minimum also ensures that our double-up pool size strategy
has a sane (perhaps overly generous?) starting point. */
initial_alloc = MAX(initial_pool_size, POOL_MINIMUM_INITIAL_SIZE);
/* Set up the first pool using our chosen first alloc size. Allocate
an extra class, and offset the pool by this amount, in order to
accommodate the initial gutter integer. (Note for the future: If
charclass alignment becomes significant, then sizeof (charclass)
and this offset may need to be changed, perhaps for SIMD
instructions.) */
pool_list_used = 1;
pool = &pool_list[0];
pool->first_index = 0;
pool->alloc = initial_alloc;
pool->used = 0;
alloc_mem = XNMALLOC (pool->alloc + 1, charclass_t);
pool->classes = alloc_mem + 1;
pool->class_state = xzalloc (pool->alloc);
/* Enforce the all-zeroes class to be the first class. This is needed
as "abandon" may leave a hole in a pool in some cases, and in these
cases we need to ensure that no-one else picks it up by accident
(as this would invalidate the guarantee that the module eliminates
all duplicates, from the point of view of the user). So, we set the
first class to all-zeroes, and also zero out abandoned classes where
a hole is unavoidable. */
ccl = charclass_alloc (); /* Alloc delivers an all-zeroes class. */
zeroclass_index = charclass_completed (ccl);
assert (zeroclass_index == CHARCLASS_ZEROCLASS_INDEX);
/* Finally, use atexit () to arrange for module cleanup on exit. */
atexit (charclass_destroy_module);
}
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="charclass.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="charclass.h"
/* charclass -- Tools to create and manipulate sets of characters (octets)
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* This module provides services to allocate, manipulate, consolidate
and discard 256-bit vectors, used to describe 8-bit (octet) sets.
Octet is used as the member name here, as "byte" or "char" can
sometimes refer to different bit sizes (e.g. char -> 6 bits on some
IBM/Cyber architectures; char -> 32 bits on some DSP architectures;
in C, sizeof (char) == 1 by definition on all architectures).
The connection between these "charclass" sets and set expression by
RE tools can be non-trivial: Many Unicode characters cannot fit
into 8 bits, and even where octet-based code pages are used,
nontrivial cases can appear (e.g. Code page 857, MS-DOS Turkish,
which has both a dotted and a dotless lowercase and uppercase "I").
On the storage side, things are slightly tricky and perhaps even
murky at times. The client starts by allocating a charclass,
working on it, and then either completing it (usually) or abandoning
it. The working class (pun intended) is represented by a pointer.
If not abandoned, this pointer is guaranteed to remain valid for the
lifetime of the module.
The module tries aggressively to eliminate duplicates; this is
perhaps the main function of the completion step. So, the pointer
that represents the class after completion may not be the working
pointer.
In addition to the pointer method of referring to a class, the
classes can be viewed as an array, with the first class receiving
index 0, the second receiving index 1, and so on. Functions are
provided to map pointers to indexes, and vice versa. The index
representation is handy as it is very compact (typically much fewer
than 24 bits), whereas pointers are architecture and OS-specific,
and may be 64 bits or more.
Index 0 is special; it will always represent the zero-class (no
members set). Users wanting to store a set of non-zeroclass classes
(e.g. utf8) can use this property as a sentinel (a value of 0 for a
static variable can mean "not initialised").
Finally, there are some "gutter" bits, at least 3 on each end of
the class, so that, to a limited extent (and especially for the
common case of EOF == -1), bits can be set and cleared without
causing problems, and the code does not need to include the overhead
of checks for out-of-bound bit numbers. These gutter bits are
cleared when the class is completed, so EOF (for instance) should
never be member of a class. */
#ifndef CHARCLASS_H
#define CHARCLASS_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
#include <limits.h>
#include <stdbool.h>
#include <stddef.h>
/* Define charclass as an opaque type. */
typedef struct charclass_struct charclass_t;
/* First integer value that is greater than any unibyte char. Each
class can model [0 .. CHARCLASS_NOTCHAR - 1] members. The user may
reference a few members above or below this range, thanks to the
explicit gutters, but the margins provided by this facility are
small. */
enum { CHARCLASS_NOTCHAR = 1 << CHAR_BIT };
/* Indices to valid charclasses are always non-negative; zero is
reserved as an index to the zerocass, and all other class entities
have positive indices. We use ptrdiff_t rather than size_t here
as -1 can be used as a sentinel in some places. */
typedef ptrdiff_t charclass_index_t;
/* Index zero is special: It is the zero-class (no bits set) class. */
enum
{
CHARCLASS_ZEROCLASS_INDEX = 0
};
/* Entire-module initialisation and destruction functions. The client
specifies starting size for the class pool. Destroy_module releases
all resources acquired by this module. */
extern void
charclass_initialise (size_t initial_pool_size);
extern void
charclass_destroy_module (void);
/* Single-bit operations (test, set, clear). */
extern bool _GL_ATTRIBUTE_PURE
charclass_tstbit (int b, charclass_t const *ccl);
extern void
charclass_setbit (int b, charclass_t *ccl);
extern void
charclass_clrbit (int b, charclass_t *ccl);
/* Range-of-bits set and clear operations. These are easier to read,
and also more efficient, than multiple single-bit calls. */
extern void
charclass_setbit_range (int start, int end, charclass_t *ccl);
extern void
charclass_clrbit_range (int start, int end, charclass_t *ccl);
/* Whole-of-set operations (copy, zero, invert, compare-equal). */
extern void
charclass_copyset (charclass_t const *src, charclass_t *dst);
extern void
charclass_zeroset (charclass_t *ccl);
extern void
charclass_notset (charclass_t *ccl);
extern int _GL_ATTRIBUTE_PURE
charclass_equal (charclass_t const *ccl1, charclass_t const *ccl2);
/* Add "unionset" and "intersectset" functions since whole-of-class
operations tend to be reasonably expressive and self-documenting.
In both cases, the source modifies the destination; ORed in, in the
case of unionset; ANDed in, in the case of intersectset. */
extern void
charclass_unionset (charclass_t const *src, charclass_t *dst);
extern void
charclass_intersectset (charclass_t const *src, charclass_t *dst);
/* Functions to allocate, complete and abandon charclasses. Note that
the module aggressively tries to reuse existing completed classes
rather than create new ones. The module returns an unique index
that can be used to reference the module; this index supercedes the
pointer used during the work phase, e.g.:
work_class_pointer = charclass_alloc ();
...(Set/clear members as required to construct a class in
work_class_pointer.)...
class_index = charclass_completed (work_class_pointer);
completed_class_pointer = charclass_get_pointer (class_index);
...(completed_class_pointer might not == work_class_pointer.)...
The aggressive-reuse policy also means that completed classes must
not undergo further modification. Another piece of coding hygiene is
that the pointer value used to construct the class should not be used
once the class is either completed or abandoned.
Allocating and then abandoning classes is useful where an operation
requires temporary classes for a while, but these do not need to be
maintained once the work is complete.
2 May 2014: Note to self and others: I was using the term "finalise"
instead of "completed" for the operation where a class has been constructed
and is now ready to be used; I've decided to change the terminology after
reading in various places how the term "finalise" is usually strongly
connected with end-of-life operations on an object and/or class, not a
ready-for-use operation. This note is a reminder to myself and a hint to
others about this change, in case vestiges of the earlier naming scheme
slip through my edits and/or appear in the code. */
extern charclass_t *
charclass_alloc (void);
extern charclass_index_t
charclass_completed (charclass_t *ccl);
extern void
charclass_abandon (charclass_t *ccl);
/* Functions to map between pointer references and index references for
a charclass. As explained above, the index is convenient as it is
typically an array reference, and is usually not much larger than the
number of classes that have been allocated. */
extern charclass_t * _GL_ATTRIBUTE_PURE
charclass_get_pointer (charclass_index_t const index);
extern charclass_index_t _GL_ATTRIBUTE_PURE
charclass_get_index (charclass_t const *ccl);
/* Return a static string describing a class (Note: not reentrant). */
extern char *
charclass_describe (charclass_t const *ccl);
#endif /* CHARCLASS_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="dfa-prl.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="dfa-prl.c"
/* dfa.c - deterministic extended regexp routines for GNU
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
#include <config.h>
#include "dfa.h"
#include <assert.h>
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <string.h>
#include <locale.h>
/* HOOK: Hack in interfaces to new charclass and fsa* modules. */
#include "charclass.h"
#include "fsatoken.h"
#include "fsalex.h"
#include "fsamusts.h"
#include "fsaparse.h"
#include "mbcsets.h"
#include "proto-lexparse.h"
/* HOOK: File handle for hook/parallel lex/parse debug/log messages */
FILE *pll_log = NULL;
#define HOOK_LOG_FILENAME "/tmp/parallel.log"
/* HOOK: Static variables to hold opaque parser and lexer contexts. */
static fsaparse_ctxt_t *parser = NULL;
static fsalex_ctxt_t *lexer = NULL;
/* HOOK: Store musts list as a static variable so we can free it correctly. */
static fsamusts_list_element_t *musts = NULL;
#define STREQ(a, b) (strcmp (a, b) == 0)
/* ISASCIIDIGIT differs from isdigit, as follows:
- Its arg may be any int or unsigned int; it need not be an unsigned char.
- It's guaranteed to evaluate its argument exactly once.
- It's typically faster.
Posix 1003.2-1992 section 2.5.2.1 page 50 lines 1556-1558 says that
only '0' through '9' are digits. Prefer ISASCIIDIGIT to isdigit unless
it's important to use the locale's definition of "digit" even when the
host does not conform to Posix. */
#define ISASCIIDIGIT(c) ((unsigned) (c) - '0' <= 9)
#include "gettext.h"
#define _(str) gettext (str)
#include <wchar.h>
#include <wctype.h>
#include "xalloc.h"
/* HPUX defines these as macros in sys/param.h. */
#ifdef setbit
# undef setbit
#endif
#ifdef clrbit
# undef clrbit
#endif
/* First integer value that is greater than any character code. */
enum { NOTCHAR = 1 << CHAR_BIT };
/* This represents part of a character class. It must be unsigned and
at least CHARCLASS_WORD_BITS wide. Any excess bits are zero. */
typedef unsigned int charclass_word;
/* The number of bits used in a charclass word. utf8_classes assumes
this is exactly 32. */
enum { CHARCLASS_WORD_BITS = 32 };
/* The maximum useful value of a charclass_word; all used bits are 1. */
#define CHARCLASS_WORD_MASK \
(((charclass_word) 1 << (CHARCLASS_WORD_BITS - 1) << 1) - 1)
/* Number of words required to hold a bit for every character. */
enum
{
CHARCLASS_WORDS = (NOTCHAR + CHARCLASS_WORD_BITS - 1) / CHARCLASS_WORD_BITS
};
/* Sets of unsigned characters are stored as bit vectors in arrays of ints. */
typedef charclass_word charclass[CHARCLASS_WORDS];
/* Convert a possibly-signed character to an unsigned character. This is
a bit safer than casting to unsigned char, since it catches some type
errors that the cast doesn't. */
static unsigned char
to_uchar (char ch)
{
return ch;
}
/* Contexts tell us whether a character is a newline or a word constituent.
Word-constituent characters are those that satisfy iswalnum, plus '_'.
Each character has a single CTX_* value; bitmasks of CTX_* values denote
a particular character class.
A state also stores a context value, which is a bitmask of CTX_* values.
A state's context represents a set of characters that the state's
predecessors must match. For example, a state whose context does not
include CTX_LETTER will never have transitions where the previous
character is a word constituent. A state whose context is CTX_ANY
might have transitions from any character. */
#define CTX_NONE 1
#define CTX_LETTER 2
#define CTX_NEWLINE 4
#define CTX_ANY 7
/* Sometimes characters can only be matched depending on the surrounding
context. Such context decisions depend on what the previous character
was, and the value of the current (lookahead) character. Context
dependent constraints are encoded as 8 bit integers. Each bit that
is set indicates that the constraint succeeds in the corresponding
context.
bit 8-11 - valid contexts when next character is CTX_NEWLINE
bit 4-7 - valid contexts when next character is CTX_LETTER
bit 0-3 - valid contexts when next character is CTX_NONE
The macro SUCCEEDS_IN_CONTEXT determines whether a given constraint
succeeds in a particular context. Prev is a bitmask of possible
context values for the previous character, curr is the (single-bit)
context value for the lookahead character. */
#define NEWLINE_CONSTRAINT(constraint) (((constraint) >> 8) & 0xf)
#define LETTER_CONSTRAINT(constraint) (((constraint) >> 4) & 0xf)
#define OTHER_CONSTRAINT(constraint) ((constraint) & 0xf)
#define SUCCEEDS_IN_CONTEXT(constraint, prev, curr) \
((((curr) & CTX_NONE ? OTHER_CONSTRAINT (constraint) : 0) \
| ((curr) & CTX_LETTER ? LETTER_CONSTRAINT (constraint) : 0) \
| ((curr) & CTX_NEWLINE ? NEWLINE_CONSTRAINT (constraint) : 0)) & (prev))
/* The following macros describe what a constraint depends on. */
#define PREV_NEWLINE_CONSTRAINT(constraint) (((constraint) >> 2) & 0x111)
#define PREV_LETTER_CONSTRAINT(constraint) (((constraint) >> 1) & 0x111)
#define PREV_OTHER_CONSTRAINT(constraint) ((constraint) & 0x111)
#define PREV_NEWLINE_DEPENDENT(constraint) \
(PREV_NEWLINE_CONSTRAINT (constraint) != PREV_OTHER_CONSTRAINT (constraint))
#define PREV_LETTER_DEPENDENT(constraint) \
(PREV_LETTER_CONSTRAINT (constraint) != PREV_OTHER_CONSTRAINT (constraint))
/* Tokens that match the empty string subject to some constraint actually
work by applying that constraint to determine what may follow them,
taking into account what has gone before. The following values are
the constraints corresponding to the special tokens previously defined. */
#define NO_CONSTRAINT 0x777
#define BEGLINE_CONSTRAINT 0x444
#define ENDLINE_CONSTRAINT 0x700
#define BEGWORD_CONSTRAINT 0x050
#define ENDWORD_CONSTRAINT 0x202
#define LIMWORD_CONSTRAINT 0x252
#define NOTLIMWORD_CONSTRAINT 0x525
/* The regexp is parsed into an array of tokens in postfix form. Some tokens
are operators and others are terminal symbols. Most (but not all) of these
codes are returned by the lexical analyzer. */
typedef ptrdiff_t token;
/* Predefined token values. */
enum
{
END = -1, /* END is a terminal symbol that matches the
end of input; any value of END or less in
the parse tree is such a symbol. Accepting
states of the DFA are those that would have
a transition on END. */
/* Ordinary character values are terminal symbols that match themselves. */
EMPTY = NOTCHAR, /* EMPTY is a terminal symbol that matches
the empty string. */
BACKREF, /* BACKREF is generated by \<digit>
or by any other construct that
is not completely handled. If the scanner
detects a transition on backref, it returns
a kind of "semi-success" indicating that
the match will have to be verified with
a backtracking matcher. */
BEGLINE, /* BEGLINE is a terminal symbol that matches
the empty string at the beginning of a
line. */
ENDLINE, /* ENDLINE is a terminal symbol that matches
the empty string at the end of a line. */
BEGWORD, /* BEGWORD is a terminal symbol that matches
the empty string at the beginning of a
word. */
ENDWORD, /* ENDWORD is a terminal symbol that matches
the empty string at the end of a word. */
LIMWORD, /* LIMWORD is a terminal symbol that matches
the empty string at the beginning or the
end of a word. */
NOTLIMWORD, /* NOTLIMWORD is a terminal symbol that
matches the empty string not at
the beginning or end of a word. */
QMARK, /* QMARK is an operator of one argument that
matches zero or one occurrences of its
argument. */
STAR, /* STAR is an operator of one argument that
matches the Kleene closure (zero or more
occurrences) of its argument. */
PLUS, /* PLUS is an operator of one argument that
matches the positive closure (one or more
occurrences) of its argument. */
REPMN, /* REPMN is a lexical token corresponding
to the {m,n} construct. REPMN never
appears in the compiled token vector. */
CAT, /* CAT is an operator of two arguments that
matches the concatenation of its
arguments. CAT is never returned by the
lexical analyzer. */
OR, /* OR is an operator of two arguments that
matches either of its arguments. */
LPAREN, /* LPAREN never appears in the parse tree,
it is only a lexeme. */
RPAREN, /* RPAREN never appears in the parse tree. */
ANYCHAR, /* ANYCHAR is a terminal symbol that matches
a valid multibyte (or single byte) character.
It is used only if MB_CUR_MAX > 1. */
MBCSET, /* MBCSET is similar to CSET, but for
multibyte characters. */
WCHAR, /* Only returned by lex. wctok contains
the wide character representation. */
CSET /* CSET and (and any value greater) is a
terminal symbol that matches any of a
class of characters. */
};
/* States of the recognizer correspond to sets of positions in the parse
tree, together with the constraints under which they may be matched.
So a position is encoded as an index into the parse tree together with
a constraint. */
typedef struct
{
size_t index; /* Index into the parse array. */
unsigned int constraint; /* Constraint for matching this position. */
} position;
/* Sets of positions are stored as arrays. */
typedef struct
{
position *elems; /* Elements of this position set. */
size_t nelem; /* Number of elements in this set. */
size_t alloc; /* Number of elements allocated in ELEMS. */
} position_set;
/* Sets of leaves are also stored as arrays. */
typedef struct
{
size_t *elems; /* Elements of this position set. */
size_t nelem; /* Number of elements in this set. */
} leaf_set;
/* A state of the dfa consists of a set of positions, some flags,
and the token value of the lowest-numbered position of the state that
contains an END token. */
typedef struct
{
size_t hash; /* Hash of the positions of this state. */
position_set elems; /* Positions this state could match. */
unsigned char context; /* Context from previous state. */
bool has_backref; /* This state matches a \<digit>. */
bool has_mbcset; /* This state matches a MBCSET. */
unsigned short constraint; /* Constraint for this state to accept. */
token first_end; /* Token value of the first END in elems. */
position_set mbps; /* Positions which can match multibyte
characters, e.g., period.
Used only if MB_CUR_MAX > 1. */
} dfa_state;
/* States are indexed by state_num values. These are normally
nonnegative but -1 is used as a special value. */
typedef ptrdiff_t state_num;
/* A bracket operator.
e.g., [a-c], [[:alpha:]], etc. */
struct mb_char_classes
{
ptrdiff_t cset;
bool invert;
wchar_t *chars; /* Normal characters. */
size_t nchars;
wctype_t *ch_classes; /* Character classes. */
size_t nch_classes;
struct /* Range characters. */
{
wchar_t beg; /* Range start. */
wchar_t end; /* Range end. */
} *ranges;
size_t nranges;
char **equivs; /* Equivalence classes. */
size_t nequivs;
char **coll_elems;
size_t ncoll_elems; /* Collating elements. */
};
/* A compiled regular expression. */
struct dfa
{
/* Fields filled by the scanner. */
charclass *charclasses; /* Array of character sets for CSET tokens. */
size_t cindex; /* Index for adding new charclasses. */
size_t calloc; /* Number of charclasses allocated. */
/* Fields filled by the parser. */
token *tokens; /* Postfix parse array. */
size_t tindex; /* Index for adding new tokens. */
size_t talloc; /* Number of tokens currently allocated. */
size_t depth; /* Depth required of an evaluation stack
used for depth-first traversal of the
parse tree. */
size_t nleaves; /* Number of leaves on the parse tree. */
size_t nregexps; /* Count of parallel regexps being built
with dfaparse. */
bool fast; /* The DFA is fast. */
bool multibyte; /* MB_CUR_MAX > 1. */
token utf8_anychar_classes[5]; /* To lower ANYCHAR in UTF-8 locales. */
mbstate_t mbs; /* Multibyte conversion state. */
/* The following are valid only if MB_CUR_MAX > 1. */
/* The value of multibyte_prop[i] is defined by following rule.
if tokens[i] < NOTCHAR
bit 0 : tokens[i] is the first byte of a character, including
single-byte characters.
bit 1 : tokens[i] is the last byte of a character, including
single-byte characters.
if tokens[i] = MBCSET
("the index of mbcsets corresponding to this operator" << 2) + 3
e.g.
tokens
= 'single_byte_a', 'multi_byte_A', single_byte_b'
= 'sb_a', 'mb_A(1st byte)', 'mb_A(2nd byte)', 'mb_A(3rd byte)', 'sb_b'
multibyte_prop
= 3 , 1 , 0 , 2 , 3
*/
int *multibyte_prop;
/* A table indexed by byte values that contains the corresponding wide
character (if any) for that byte. WEOF means the byte is not a
valid single-byte character. */
wint_t mbrtowc_cache[NOTCHAR];
/* Array of the bracket expression in the DFA. */
struct mb_char_classes *mbcsets;
size_t nmbcsets;
size_t mbcsets_alloc;
/* Fields filled by the superset. */
struct dfa *superset; /* Hint of the dfa. */
/* Fields filled by the state builder. */
dfa_state *states; /* States of the dfa. */
state_num sindex; /* Index for adding new states. */
size_t salloc; /* Number of states currently allocated. */
/* Fields filled by the parse tree->NFA conversion. */
position_set *follows; /* Array of follow sets, indexed by position
index. The follow of a position is the set
of positions containing characters that
could conceivably follow a character
matching the given position in a string
matching the regexp. Allocated to the
maximum possible position index. */
bool searchflag; /* We are supposed to build a searching
as opposed to an exact matcher. A searching
matcher finds the first and shortest string
matching a regexp anywhere in the buffer,
whereas an exact matcher finds the longest
string matching, but anchored to the
beginning of the buffer. */
/* Fields filled by dfaexec. */
state_num tralloc; /* Number of transition tables that have
slots so far, not counting trans[-1]. */
int trcount; /* Number of transition tables that have
actually been built. */
state_num **trans; /* Transition tables for states that can
never accept. If the transitions for a
state have not yet been computed, or the
state could possibly accept, its entry in
this table is NULL. This points to one
past the start of the allocated array,
and trans[-1] is always NULL. */
state_num **fails; /* Transition tables after failing to accept
on a state that potentially could do so. */
int *success; /* Table of acceptance conditions used in
dfaexec and computed in build_state. */
state_num *newlines; /* Transitions on newlines. The entry for a
newline in any transition table is always
-1 so we can count lines without wasting
too many cycles. The transition for a
newline is stored separately and handled
as a special case. Newline is also used
as a sentinel at the end of the buffer. */
struct dfamust *musts; /* List of strings, at least one of which
is known to appear in any r.e. matching
the dfa. */
position_set mb_follows; /* Follow set added by ANYCHAR and/or MBCSET
on demand. */
int *mb_match_lens; /* Array of length reduced by ANYCHAR and/or
MBCSET. Null if mb_follows.elems has not
been allocated. */
};
/* Some macros for user access to dfa internals. */
/* S could possibly be an accepting state of R. */
#define ACCEPTING(s, r) ((r).states[s].constraint)
/* STATE accepts in the specified context. */
#define ACCEPTS_IN_CONTEXT(prev, curr, state, dfa) \
SUCCEEDS_IN_CONTEXT ((dfa).states[state].constraint, prev, curr)
static void dfamust (struct dfa *dfa);
static void regexp (void);
static void
dfambcache (struct dfa *d)
{
int i;
for (i = CHAR_MIN; i <= CHAR_MAX; ++i)
{
char c = i;
unsigned char uc = i;
mbstate_t s = { 0 };
wchar_t wc;
d->mbrtowc_cache[uc] = mbrtowc (&wc, &c, 1, &s) <= 1 ? wc : WEOF;
}
}
/* Store into *PWC the result of converting the leading bytes of the
multibyte buffer S of length N bytes, using the mbrtowc_cache in *D
and updating the conversion state in *D. On conversion error,
convert just a single byte, to WEOF. Return the number of bytes
converted.
This differs from mbrtowc (PWC, S, N, &D->mbs) as follows:
* PWC points to wint_t, not to wchar_t.
* The last arg is a dfa *D instead of merely a multibyte conversion
state D->mbs. D also contains an mbrtowc_cache for speed.
* N must be at least 1.
* S[N - 1] must be a sentinel byte.
* Shift encodings are not supported.
* The return value is always in the range 1..N.
* D->mbs is always valid afterwards.
* *PWC is always set to something. */
static size_t
mbs_to_wchar (wint_t *pwc, char const *s, size_t n, struct dfa *d)
{
unsigned char uc = s[0];
wint_t wc = d->mbrtowc_cache[uc];
if (wc == WEOF)
{
wchar_t wch;
size_t nbytes = mbrtowc (&wch, s, n, &d->mbs);
if (0 < nbytes && nbytes < (size_t) -2)
{
*pwc = wch;
return nbytes;
}
memset (&d->mbs, 0, sizeof d->mbs);
}
*pwc = wc;
return 1;
}
#ifdef DEBUG
static void
prtok (token t)
{
char const *s;
if (t < 0)
fprintf (stderr, "END");
else if (t < NOTCHAR)
{
int ch = t;
fprintf (stderr, "%c", ch);
}
else
{
switch (t)
{
case EMPTY:
s = "EMPTY";
break;
case BACKREF:
s = "BACKREF";
break;
case BEGLINE:
s = "BEGLINE";
break;
case ENDLINE:
s = "ENDLINE";
break;
case BEGWORD:
s = "BEGWORD";
break;
case ENDWORD:
s = "ENDWORD";
break;
case LIMWORD:
s = "LIMWORD";
break;
case NOTLIMWORD:
s = "NOTLIMWORD";
break;
case QMARK:
s = "QMARK";
break;
case STAR:
s = "STAR";
break;
case PLUS:
s = "PLUS";
break;
case CAT:
s = "CAT";
break;
case OR:
s = "OR";
break;
case LPAREN:
s = "LPAREN";
break;
case RPAREN:
s = "RPAREN";
break;
case ANYCHAR:
s = "ANYCHAR";
break;
case MBCSET:
s = "MBCSET";
break;
default:
s = "CSET";
break;
}
fprintf (stderr, "%s", s);
}
}
#endif /* DEBUG */
/* Stuff pertaining to charclasses. */
static bool
tstbit (unsigned int b, charclass const c)
{
return c[b / CHARCLASS_WORD_BITS] >> b % CHARCLASS_WORD_BITS & 1;
}
static void
setbit (unsigned int b, charclass c)
{
c[b / CHARCLASS_WORD_BITS] |= (charclass_word) 1 << b % CHARCLASS_WORD_BITS;
}
static void
clrbit (unsigned int b, charclass c)
{
c[b / CHARCLASS_WORD_BITS] &= ~((charclass_word) 1
<< b % CHARCLASS_WORD_BITS);
}
static void
copyset (charclass const src, charclass dst)
{
memcpy (dst, src, sizeof (charclass));
}
static void
zeroset (charclass s)
{
memset (s, 0, sizeof (charclass));
}
static void
notset (charclass s)
{
int i;
for (i = 0; i < CHARCLASS_WORDS; ++i)
s[i] = CHARCLASS_WORD_MASK & ~s[i];
}
static bool
equal (charclass const s1, charclass const s2)
{
return memcmp (s1, s2, sizeof (charclass)) == 0;
}
/* Ensure that the array addressed by PTR holds at least NITEMS +
(PTR || !NITEMS) items. Either return PTR, or reallocate the array
and return its new address. Although PTR may be null, the returned
value is never null.
The array holds *NALLOC items; *NALLOC is updated on reallocation.
ITEMSIZE is the size of one item. Avoid O(N**2) behavior on arrays
growing linearly. */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
if (nitems < *nalloc)
return ptr;
*nalloc = nitems;
return x2nrealloc (ptr, nalloc, itemsize);
}
/* In DFA D, find the index of charclass S, or allocate a new one. */
static size_t
dfa_charclass_index (struct dfa *d, charclass const s)
{
size_t i;
for (i = 0; i < d->cindex; ++i)
if (equal (s, d->charclasses[i]))
return i;
d->charclasses = maybe_realloc (d->charclasses, d->cindex, &d->calloc,
sizeof *d->charclasses);
++d->cindex;
copyset (s, d->charclasses[i]);
return i;
}
/* A pointer to the current dfa is kept here during parsing. */
static struct dfa *dfa;
/* Find the index of charclass S in the current DFA, or allocate a new one. */
static size_t
charclass_index (charclass const s)
{
return dfa_charclass_index (dfa, s);
}
/* Syntax bits controlling the behavior of the lexical analyzer. */
static reg_syntax_t syntax_bits, syntax_bits_set;
/* Flag for case-folding letters into sets. */
static bool case_fold;
/* End-of-line byte in data. */
static unsigned char eolbyte;
/* Cache of char-context values. */
static int sbit[NOTCHAR];
/* Set of characters considered letters. */
static charclass letters;
/* Set of characters that are newline. */
static charclass newline;
/* Add this to the test for whether a byte is word-constituent, since on
BSD-based systems, many values in the 128..255 range are classified as
alphabetic, while on glibc-based systems, they are not. */
#ifdef __GLIBC__
# define is_valid_unibyte_character(c) 1
#else
# define is_valid_unibyte_character(c) (btowc (c) != WEOF)
#endif
/* C is a "word-constituent" byte. */
#define IS_WORD_CONSTITUENT(C) \
(is_valid_unibyte_character (C) && (isalnum (C) || (C) == '_'))
static int
char_context (unsigned char c)
{
if (c == eolbyte)
return CTX_NEWLINE;
if (IS_WORD_CONSTITUENT (c))
return CTX_LETTER;
return CTX_NONE;
}
static int
wchar_context (wint_t wc)
{
if (wc == (wchar_t) eolbyte || wc == 0)
return CTX_NEWLINE;
if (wc == L'_' || iswalnum (wc))
return CTX_LETTER;
return CTX_NONE;
}
typedef struct regex_name_mapping_struct
{
reg_syntax_t flag;
const char *name;
} regex_name_mapping_t;
static regex_name_mapping_t regex_names[] = {
{RE_BACKSLASH_ESCAPE_IN_LISTS, "backslash_escape_in_lists"},
{RE_BK_PLUS_QM, "bk_plus_qm"},
{RE_CHAR_CLASSES, "char_classes"},
{RE_CONTEXT_INDEP_ANCHORS, "context_indep_anchors"},
{RE_CONTEXT_INDEP_OPS, "context_indep_ops"},
{RE_CONTEXT_INVALID_OPS, "context_invalid_ops"},
{RE_DOT_NEWLINE, "dot_newline"},
{RE_DOT_NOT_NULL, "dot_not_null"},
{RE_HAT_LISTS_NOT_NEWLINE, "hat_lists_not_newline"},
{RE_INTERVALS, "intervals"},
{RE_LIMITED_OPS, "limited_ops"},
{RE_NEWLINE_ALT, "newline_alt"},
{RE_NO_BK_BRACES, "no_bk_braces"},
{RE_NO_BK_PARENS, "no_bk_parens"},
{RE_NO_BK_REFS, "no_bk_refs"},
{RE_NO_BK_VBAR, "no_bk_vbar"},
{RE_NO_EMPTY_RANGES, "no_empty_ranges"},
{RE_UNMATCHED_RIGHT_PAREN_ORD, "unmatched_right_paren_ord"},
{RE_NO_POSIX_BACKTRACKING, "no_posix_backtracking"},
{RE_NO_GNU_OPS, "no_gnu_ops"},
{RE_DEBUG, "debug"},
{RE_INVALID_INTERVAL_ORD, "invalid_interval_ord"},
{RE_ICASE, "icase"},
{RE_CARET_ANCHORS_HERE, "caret_anchors_here"},
{RE_CONTEXT_INVALID_DUP, "context_invalid_dup"},
{RE_NO_SUB, "no_sub"},
{0, NULL}
};
/* HOOK: When we abort, get rid of resources that we've acquired, before
finally handing control to the abort mechanism. We do this to minimise
resource allocation error reported by Valgrind; I believe that it is not
vital to do so otherwise (but I may be wrong). */
static void
HOOK_tear_down_additional_infrastructure (void)
{
/* Modules destroy themselves via a module-wide atexit () registration. */
#if 0
fsalex_destroy_module ();
fsaparse_destroy_module ();
charclass_destroy_module ();
#endif /* 0 */
parser = NULL;
lexer = NULL;
/* fsatoken, proto-lexparse and dfa-prl itself do not need
attention (yet?). */
/* Release the "musts" list we compiled earlier. */
{
fsamusts_list_element_t *mlist;
fsamusts_list_element_t *mnext;
for (mlist = musts; mlist; mlist = mnext)
{
mnext = mlist->next;
free (mlist->must);
free (mlist);
}
musts = NULL;
}
}
/* HOOK: Provide error/warning functions that include logging. */
static void
lexer_warn(const char *warning)
{
if (! pll_log)
pll_log = fopen (HOOK_LOG_FILENAME, "a");
fprintf (pll_log, "lexer warning: %s\n", warning);
fflush (NULL);
dfawarn (warning);
}
static _Noreturn void
lexer_error(const char *reason)
{
if (! pll_log)
pll_log = fopen (HOOK_LOG_FILENAME, "a");
fprintf (pll_log, "lexer error: %s\n", reason);
HOOK_tear_down_additional_infrastructure ();
fflush (NULL);
dfaerror (reason);
}
static void
parser_warn(const char *warning)
{
if (! pll_log)
pll_log = fopen (HOOK_LOG_FILENAME, "a");
fprintf (pll_log, "parser warning: %s\n", warning);
fflush (NULL);
dfawarn (warning);
}
static _Noreturn void
parser_error(const char *reason)
{
if (! pll_log)
pll_log = fopen (HOOK_LOG_FILENAME, "a");
fprintf (pll_log, "parser error: %s\n", reason);
HOOK_tear_down_additional_infrastructure ();
fflush (NULL);
dfaerror (reason);
}
static void
HOOK_set_up_fsa_stuff_if_not_done_already (void)
{
/* If lexer context is present, this function has been run previously. */
if (lexer != NULL)
return;
/* Initialise modules before creating instances. */
fsalex_initialise ();
fsaparse_initialise ();
/* Create a new lexer instance, and give it error/warning fns */
lexer = fsalex_new ();
fsalex_exception_fns (lexer, lexer_warn, lexer_error);
/* Start with a pool of 10 charclasses. */
charclass_initialise (10);
/* Iniialise multibyte-char-sets module. */
mbcsets_initialise ();
/* Create a new parser instance, and give it error/warning functions.
Defer connecting up the lexer until lexer syntax has been set up. */
parser = fsaparse_new ();
fsaparse_exception_fns (parser, parser_warn, parser_error);
}
/* Entry point to set syntax options. */
void
dfasyntax (reg_syntax_t bits, int fold, unsigned char eol)
{
unsigned int i;
/* Hook: Debug buffer to record search syntax specifications. */
static char buf[256];
char *p_buf;
char *locale;
syntax_bits_set = 1;
syntax_bits = bits;
case_fold = fold != 0;
eolbyte = eol;
HOOK_set_up_fsa_stuff_if_not_done_already ();
/* HOOK: Tell fsalex module about syntax selections. */
fsalex_syntax (lexer, bits, fold, eol);
/* HOOK: Record syntax selections in debug logfile. */
if (! pll_log)
pll_log = fopen (HOOK_LOG_FILENAME, "a");
locale = setlocale (LC_ALL, NULL);
fprintf (pll_log, "\nSyntax: Case fold: %d; eol char: %02x; locale: %s",
fold, (int) eol, locale);
p_buf = buf;
*p_buf++ = '\n';
*p_buf++ = ' ';
*p_buf = '\0';
for (i = 0; regex_names[i].name; i++)
{
char flag_char = (bits & regex_names[i].flag) ? '+' : '-';
p_buf += sprintf(p_buf, " %c%s", flag_char, regex_names[i].name);
if (strlen (buf) >= 82)
{
fprintf (pll_log, "%s", buf);
p_buf = &buf[2];
*p_buf = '\0';
}
}
fprintf (pll_log, "%s\n", buf);
/* Okay, now the parser can talk to a lexer that knows locale and syntax
configuration details. */
fsaparse_lexer (parser, lexer,
(proto_lexparse_lex_fn_t *) fsalex_lex,
(proto_lexparse_exchange_fn_t *) fsalex_exchange);
for (i = 0; i < NOTCHAR; ++i)
{
sbit[i] = char_context (i);
switch (sbit[i])
{
case CTX_LETTER:
setbit (i, letters);
break;
case CTX_NEWLINE:
setbit (i, newline);
break;
}
}
}
/* Set a bit in the charclass for the given wchar_t. Do nothing if WC
is represented by a multi-byte sequence. Even for MB_CUR_MAX == 1,
this may happen when folding case in weird Turkish locales where
dotless i/dotted I are not included in the chosen character set.
Return whether a bit was set in the charclass. */
static bool
setbit_wc (wint_t wc, charclass c)
{
int b = wctob (wc);
if (b == EOF)
return false;
setbit (b, c);
return true;
}
/* Set a bit for B and its case variants in the charclass C.
MB_CUR_MAX must be 1. */
static void
setbit_case_fold_c (int b, charclass c)
{
int ub = toupper (b);
int i;
for (i = 0; i < NOTCHAR; i++)
if (toupper (i) == ub)
setbit (i, c);
}
/* UTF-8 encoding allows some optimizations that we can't otherwise
assume in a multibyte encoding. */
int
using_utf8 (void)
{
static int utf8 = -1;
if (utf8 < 0)
{
wchar_t wc;
mbstate_t mbs = { 0 };
utf8 = mbrtowc (&wc, "\xc4\x80", 2, &mbs) == 2 && wc == 0x100;
}
return utf8;
}
/* The current locale is known to be a unibyte locale
without multicharacter collating sequences and where range
comparisons simply use the native encoding. These locales can be
processed more efficiently. */
static bool
using_simple_locale (void)
{
/* The native character set is known to be compatible with
the C locale. The following test isn't perfect, but it's good
enough in practice, as only ASCII and EBCDIC are in common use
and this test correctly accepts ASCII and rejects EBCDIC. */
enum { native_c_charset =
('\b' == 8 && '\t' == 9 && '\n' == 10 && '\v' == 11 && '\f' == 12
&& '\r' == 13 && ' ' == 32 && '!' == 33 && '"' == 34 && '#' == 35
&& '%' == 37 && '&' == 38 && '\'' == 39 && '(' == 40 && ')' == 41
&& '*' == 42 && '+' == 43 && ',' == 44 && '-' == 45 && '.' == 46
&& '/' == 47 && '0' == 48 && '9' == 57 && ':' == 58 && ';' == 59
&& '<' == 60 && '=' == 61 && '>' == 62 && '?' == 63 && 'A' == 65
&& 'Z' == 90 && '[' == 91 && '\\' == 92 && ']' == 93 && '^' == 94
&& '_' == 95 && 'a' == 97 && 'z' == 122 && '{' == 123 && '|' == 124
&& '}' == 125 && '~' == 126)
};
if (! native_c_charset || dfa->multibyte)
return false;
else
{
static int unibyte_c = -1;
if (unibyte_c < 0)
{
char const *locale = setlocale (LC_ALL, NULL);
unibyte_c = (!locale
|| STREQ (locale, "C")
|| STREQ (locale, "POSIX"));
}
return unibyte_c;
}
}
/* Lexical analyzer. All the dross that deals with the obnoxious
GNU Regex syntax bits is located here. The poor, suffering
reader is referred to the GNU Regex documentation for the
meaning of the @#%!@#%^!@ syntax bits. */
static char const *lexptr; /* Pointer to next input character. */
static size_t lexleft; /* Number of characters remaining. */
static token lasttok; /* Previous token returned; initially END. */
static bool laststart; /* We're separated from beginning or (,
| only by zero-width characters. */
static size_t parens; /* Count of outstanding left parens. */
static int minrep, maxrep; /* Repeat counts for {m,n}. */
static int cur_mb_len = 1; /* Length of the multibyte representation of
wctok. */
static wint_t wctok; /* Wide character representation of the current
multibyte character, or WEOF if there was
an encoding error. Used only if
MB_CUR_MAX > 1. */
/* Fetch the next lexical input character. Set C (of type int) to the
next input byte, except set C to EOF if the input is a multibyte
character of length greater than 1. Set WC (of type wint_t) to the
value of the input if it is a valid multibyte character (possibly
of length 1); otherwise set WC to WEOF. If there is no more input,
report EOFERR if EOFERR is not null, and return lasttok = END
otherwise. */
# define FETCH_WC(c, wc, eoferr) \
do { \
if (! lexleft) \
{ \
if ((eoferr) != 0) \
dfaerror (eoferr); \
else \
return lasttok = END; \
} \
else \
{ \
wint_t _wc; \
size_t nbytes = mbs_to_wchar (&_wc, lexptr, lexleft, dfa); \
cur_mb_len = nbytes; \
(wc) = _wc; \
(c) = nbytes == 1 ? to_uchar (*lexptr) : EOF; \
lexptr += nbytes; \
lexleft -= nbytes; \
} \
} while (0)
#ifndef MIN
# define MIN(a,b) ((a) < (b) ? (a) : (b))
#endif
/* The set of wchar_t values C such that there's a useful locale
somewhere where C != towupper (C) && C != towlower (towupper (C)).
For example, 0x00B5 (U+00B5 MICRO SIGN) is in this table, because
towupper (0x00B5) == 0x039C (U+039C GREEK CAPITAL LETTER MU), and
towlower (0x039C) == 0x03BC (U+03BC GREEK SMALL LETTER MU). */
static short const lonesome_lower[] =
{
0x00B5, 0x0131, 0x017F, 0x01C5, 0x01C8, 0x01CB, 0x01F2, 0x0345,
0x03C2, 0x03D0, 0x03D1, 0x03D5, 0x03D6, 0x03F0, 0x03F1,
/* U+03F2 GREEK LUNATE SIGMA SYMBOL lacks a specific uppercase
counterpart in locales predating Unicode 4.0.0 (April 2003). */
0x03F2,
0x03F5, 0x1E9B, 0x1FBE,
};
/* Maximum number of characters that can be the case-folded
counterparts of a single character, not counting the character
itself. This is 1 for towupper, 1 for towlower, and 1 for each
entry in LONESOME_LOWER. */
enum
{ CASE_FOLDED_BUFSIZE = 2 + sizeof lonesome_lower / sizeof *lonesome_lower };
/* Find the characters equal to C after case-folding, other than C
itself, and store them into FOLDED. Return the number of characters
stored. */
static int
case_folded_counterparts (wchar_t c, wchar_t folded[CASE_FOLDED_BUFSIZE])
{
int i;
int n = 0;
wint_t uc = towupper (c);
wint_t lc = towlower (uc);
if (uc != c)
folded[n++] = uc;
if (lc != uc && lc != c && towupper (lc) == uc)
folded[n++] = lc;
for (i = 0; i < sizeof lonesome_lower / sizeof *lonesome_lower; i++)
{
wint_t li = lonesome_lower[i];
if (li != lc && li != uc && li != c && towupper (li) == uc)
folded[n++] = li;
}
return n;
}
typedef int predicate (int);
/* The following list maps the names of the Posix named character classes
to predicate functions that determine whether a given character is in
the class. The leading [ has already been eaten by the lexical
analyzer. */
struct dfa_ctype
{
const char *name;
predicate *func;
bool single_byte_only;
};
static const struct dfa_ctype prednames[] = {
{"alpha", isalpha, false},
{"upper", isupper, false},
{"lower", islower, false},
{"digit", isdigit, true},
{"xdigit", isxdigit, false},
{"space", isspace, false},
{"punct", ispunct, false},
{"alnum", isalnum, false},
{"print", isprint, false},
{"graph", isgraph, false},
{"cntrl", iscntrl, false},
{"blank", isblank, false},
{NULL, NULL, false}
};
static const struct dfa_ctype *_GL_ATTRIBUTE_PURE
find_pred (const char *str)
{
unsigned int i;
for (i = 0; prednames[i].name; ++i)
if (STREQ (str, prednames[i].name))
break;
return &prednames[i];
}
/* Multibyte character handling sub-routine for lex.
Parse a bracket expression and build a struct mb_char_classes. */
static token
parse_bracket_exp (void)
{
bool invert;
int c, c1, c2;
charclass ccl;
/* This is a bracket expression that dfaexec is known to
process correctly. */
bool known_bracket_exp = true;
/* Used to warn about [:space:].
Bit 0 = first character is a colon.
Bit 1 = last character is a colon.
Bit 2 = includes any other character but a colon.
Bit 3 = includes ranges, char/equiv classes or collation elements. */
int colon_warning_state;
wint_t wc;
wint_t wc2;
wint_t wc1 = 0;
/* Work area to build a mb_char_classes. */
struct mb_char_classes *work_mbc;
size_t chars_al, ranges_al, ch_classes_al, equivs_al, coll_elems_al;
chars_al = ranges_al = ch_classes_al = equivs_al = coll_elems_al = 0;
if (dfa->multibyte)
{
dfa->mbcsets = maybe_realloc (dfa->mbcsets, dfa->nmbcsets,
&dfa->mbcsets_alloc,
sizeof *dfa->mbcsets);
/* dfa->multibyte_prop[] hold the index of dfa->mbcsets.
We will update dfa->multibyte_prop[] in addtok, because we can't
decide the index in dfa->tokens[]. */
/* Initialize work area. */
work_mbc = &(dfa->mbcsets[dfa->nmbcsets++]);
memset (work_mbc, 0, sizeof *work_mbc);
}
else
work_mbc = NULL;
memset (ccl, 0, sizeof ccl);
FETCH_WC (c, wc, _("unbalanced ["));
if (c == '^')
{
FETCH_WC (c, wc, _("unbalanced ["));
invert = true;
known_bracket_exp = using_simple_locale ();
}
else
invert = false;
colon_warning_state = (c == ':');
do
{
c1 = NOTCHAR; /* Mark c1 as not initialized. */
colon_warning_state &= ~2;
/* Note that if we're looking at some other [:...:] construct,
we just treat it as a bunch of ordinary characters. We can do
this because we assume regex has checked for syntax errors before
dfa is ever called. */
if (c == '[')
{
FETCH_WC (c1, wc1, _("unbalanced ["));
if ((c1 == ':' && (syntax_bits & RE_CHAR_CLASSES))
|| c1 == '.' || c1 == '=')
{
enum { MAX_BRACKET_STRING_LEN = 32 };
char str[MAX_BRACKET_STRING_LEN + 1];
size_t len = 0;
for (;;)
{
FETCH_WC (c, wc, _("unbalanced ["));
if ((c == c1 && *lexptr == ']') || lexleft == 0)
break;
if (len < MAX_BRACKET_STRING_LEN)
str[len++] = c;
else
/* This is in any case an invalid class name. */
str[0] = '\0';
}
str[len] = '\0';
/* Fetch bracket. */
FETCH_WC (c, wc, _("unbalanced ["));
if (c1 == ':')
/* Build character class. POSIX allows character
classes to match multicharacter collating elements,
but the regex code does not support that, so do not
worry about that possibility. */
{
char const *class
= (case_fold && (STREQ (str, "upper")
|| STREQ (str, "lower")) ? "alpha" : str);
const struct dfa_ctype *pred = find_pred (class);
if (!pred)
dfaerror (_("invalid character class"));
if (dfa->multibyte && !pred->single_byte_only)
{
/* Store the character class as wctype_t. */
wctype_t wt = wctype (class);
work_mbc->ch_classes
= maybe_realloc (work_mbc->ch_classes,
work_mbc->nch_classes, &ch_classes_al,
sizeof *work_mbc->ch_classes);
work_mbc->ch_classes[work_mbc->nch_classes++] = wt;
}
for (c2 = 0; c2 < NOTCHAR; ++c2)
if (pred->func (c2))
setbit (c2, ccl);
}
else
known_bracket_exp = false;
colon_warning_state |= 8;
/* Fetch new lookahead character. */
FETCH_WC (c1, wc1, _("unbalanced ["));
continue;
}
/* We treat '[' as a normal character here. c/c1/wc/wc1
are already set up. */
}
if (c == '\\' && (syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
FETCH_WC (c, wc, _("unbalanced ["));
if (c1 == NOTCHAR)
FETCH_WC (c1, wc1, _("unbalanced ["));
if (c1 == '-')
/* build range characters. */
{
FETCH_WC (c2, wc2, _("unbalanced ["));
/* A bracket expression like [a-[.aa.]] matches an unknown set.
Treat it like [-a[.aa.]] while parsing it, and
remember that the set is unknown. */
if (c2 == '[' && *lexptr == '.')
{
known_bracket_exp = false;
c2 = ']';
}
if (c2 != ']')
{
if (c2 == '\\' && (syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
FETCH_WC (c2, wc2, _("unbalanced ["));
if (dfa->multibyte)
{
/* When case folding map a range, say [m-z] (or even [M-z])
to the pair of ranges, [m-z] [M-Z]. Although this code
is wrong in multiple ways, it's never used in practice.
FIXME: Remove this (and related) unused code. */
if (wc != WEOF && wc2 != WEOF)
{
work_mbc->ranges
= maybe_realloc (work_mbc->ranges, work_mbc->nranges + 2,
&ranges_al, sizeof *work_mbc->ranges);
work_mbc->ranges[work_mbc->nranges].beg
= case_fold ? towlower (wc) : wc;
work_mbc->ranges[work_mbc->nranges++].end
= case_fold ? towlower (wc2) : wc2;
if (case_fold && (iswalpha (wc) || iswalpha (wc2)))
{
work_mbc->ranges[work_mbc->nranges].beg = towupper (wc);
work_mbc->ranges[work_mbc->nranges++].end
= towupper (wc2);
}
}
}
else if (using_simple_locale ())
{
for (c1 = c; c1 <= c2; c1++)
setbit (c1, ccl);
if (case_fold)
{
int uc = toupper (c);
int uc2 = toupper (c2);
for (c1 = 0; c1 < NOTCHAR; c1++)
{
int uc1 = toupper (c1);
if (uc <= uc1 && uc1 <= uc2)
setbit (c1, ccl);
}
}
}
else
known_bracket_exp = false;
colon_warning_state |= 8;
FETCH_WC (c1, wc1, _("unbalanced ["));
continue;
}
/* In the case [x-], the - is an ordinary hyphen,
which is left in c1, the lookahead character. */
lexptr -= cur_mb_len;
lexleft += cur_mb_len;
}
colon_warning_state |= (c == ':') ? 2 : 4;
if (!dfa->multibyte)
{
if (case_fold)
setbit_case_fold_c (c, ccl);
else
setbit (c, ccl);
continue;
}
if (wc == WEOF)
known_bracket_exp = false;
else
{
wchar_t folded[CASE_FOLDED_BUFSIZE + 1];
int i;
int n = (case_fold ? case_folded_counterparts (wc, folded + 1) + 1
: 1);
folded[0] = wc;
for (i = 0; i < n; i++)
if (!setbit_wc (folded[i], ccl))
{
work_mbc->chars
= maybe_realloc (work_mbc->chars, work_mbc->nchars,
&chars_al, sizeof *work_mbc->chars);
work_mbc->chars[work_mbc->nchars++] = folded[i];
}
}
}
while ((wc = wc1, (c = c1) != ']'));
if (colon_warning_state == 7)
dfawarn (_("character class syntax is [[:space:]], not [:space:]"));
if (! known_bracket_exp)
return BACKREF;
if (dfa->multibyte)
{
static charclass zeroclass;
work_mbc->invert = invert;
work_mbc->cset = equal (ccl, zeroclass) ? -1 : charclass_index (ccl);
return MBCSET;
}
if (invert)
{
assert (!dfa->multibyte);
notset (ccl);
if (syntax_bits & RE_HAT_LISTS_NOT_NEWLINE)
clrbit (eolbyte, ccl);
}
return CSET + charclass_index (ccl);
}
static token
original_lex (void)
{
int c, c2;
bool backslash = false;
charclass ccl;
int i;
/* Basic plan: We fetch a character. If it's a backslash,
we set the backslash flag and go through the loop again.
On the plus side, this avoids having a duplicate of the
main switch inside the backslash case. On the minus side,
it means that just about every case begins with
"if (backslash) ...". */
for (i = 0; i < 2; ++i)
{
FETCH_WC (c, wctok, NULL);
switch (c)
{
case '\\':
if (backslash)
goto normal_char;
if (lexleft == 0)
dfaerror (_("unfinished \\ escape"));
backslash = true;
break;
case '^':
if (backslash)
goto normal_char;
if (syntax_bits & RE_CONTEXT_INDEP_ANCHORS
|| lasttok == END || lasttok == LPAREN || lasttok == OR)
return lasttok = BEGLINE;
goto normal_char;
case '$':
if (backslash)
goto normal_char;
if (syntax_bits & RE_CONTEXT_INDEP_ANCHORS
|| lexleft == 0
|| (syntax_bits & RE_NO_BK_PARENS
? lexleft > 0 && *lexptr == ')'
: lexleft > 1 && lexptr[0] == '\\' && lexptr[1] == ')')
|| (syntax_bits & RE_NO_BK_VBAR
? lexleft > 0 && *lexptr == '|'
: lexleft > 1 && lexptr[0] == '\\' && lexptr[1] == '|')
|| ((syntax_bits & RE_NEWLINE_ALT)
&& lexleft > 0 && *lexptr == '\n'))
return lasttok = ENDLINE;
goto normal_char;
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9':
if (backslash && !(syntax_bits & RE_NO_BK_REFS))
{
laststart = false;
return lasttok = BACKREF;
}
goto normal_char;
case '`':
if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
return lasttok = BEGLINE; /* FIXME: should be beginning of string */
goto normal_char;
case '\'':
if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
return lasttok = ENDLINE; /* FIXME: should be end of string */
goto normal_char;
case '<':
if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
return lasttok = BEGWORD;
goto normal_char;
case '>':
if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
return lasttok = ENDWORD;
goto normal_char;
case 'b':
if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
return lasttok = LIMWORD;
goto normal_char;
case 'B':
if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
return lasttok = NOTLIMWORD;
goto normal_char;
case '?':
if (syntax_bits & RE_LIMITED_OPS)
goto normal_char;
if (backslash != ((syntax_bits & RE_BK_PLUS_QM) != 0))
goto normal_char;
if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
goto normal_char;
return lasttok = QMARK;
case '*':
if (backslash)
goto normal_char;
if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
goto normal_char;
return lasttok = STAR;
case '+':
if (syntax_bits & RE_LIMITED_OPS)
goto normal_char;
if (backslash != ((syntax_bits & RE_BK_PLUS_QM) != 0))
goto normal_char;
if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
goto normal_char;
return lasttok = PLUS;
case '{':
if (!(syntax_bits & RE_INTERVALS))
goto normal_char;
if (backslash != ((syntax_bits & RE_NO_BK_BRACES) == 0))
goto normal_char;
if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
goto normal_char;
/* Cases:
{M} - exact count
{M,} - minimum count, maximum is infinity
{,N} - 0 through N
{,} - 0 to infinity (same as '*')
{M,N} - M through N */
{
char const *p = lexptr;
char const *lim = p + lexleft;
minrep = maxrep = -1;
for (; p != lim && ISASCIIDIGIT (*p); p++)
{
if (minrep < 0)
minrep = *p - '0';
else
minrep = MIN (RE_DUP_MAX + 1, minrep * 10 + *p - '0');
}
if (p != lim)
{
if (*p != ',')
maxrep = minrep;
else
{
if (minrep < 0)
minrep = 0;
while (++p != lim && ISASCIIDIGIT (*p))
{
if (maxrep < 0)
maxrep = *p - '0';
else
maxrep = MIN (RE_DUP_MAX + 1, maxrep * 10 + *p - '0');
}
}
}
if (! ((! backslash || (p != lim && *p++ == '\\'))
&& p != lim && *p++ == '}'
&& 0 <= minrep && (maxrep < 0 || minrep <= maxrep)))
{
if (syntax_bits & RE_INVALID_INTERVAL_ORD)
goto normal_char;
dfaerror (_("invalid content of \\{\\}"));
}
if (RE_DUP_MAX < maxrep)
dfaerror (_("regular expression too big"));
lexptr = p;
lexleft = lim - p;
}
laststart = false;
return lasttok = REPMN;
case '|':
if (syntax_bits & RE_LIMITED_OPS)
goto normal_char;
if (backslash != ((syntax_bits & RE_NO_BK_VBAR) == 0))
goto normal_char;
laststart = true;
return lasttok = OR;
case '\n':
if (syntax_bits & RE_LIMITED_OPS
|| backslash || !(syntax_bits & RE_NEWLINE_ALT))
goto normal_char;
laststart = true;
return lasttok = OR;
case '(':
if (backslash != ((syntax_bits & RE_NO_BK_PARENS) == 0))
goto normal_char;
++parens;
laststart = true;
return lasttok = LPAREN;
case ')':
if (backslash != ((syntax_bits & RE_NO_BK_PARENS) == 0))
goto normal_char;
if (parens == 0 && syntax_bits & RE_UNMATCHED_RIGHT_PAREN_ORD)
goto normal_char;
--parens;
laststart = false;
return lasttok = RPAREN;
case '.':
if (backslash)
goto normal_char;
if (dfa->multibyte)
{
/* In multibyte environment period must match with a single
character not a byte. So we use ANYCHAR. */
laststart = false;
return lasttok = ANYCHAR;
}
zeroset (ccl);
notset (ccl);
if (!(syntax_bits & RE_DOT_NEWLINE))
clrbit (eolbyte, ccl);
if (syntax_bits & RE_DOT_NOT_NULL)
clrbit ('\0', ccl);
laststart = false;
return lasttok = CSET + charclass_index (ccl);
case 's':
case 'S':
if (!backslash || (syntax_bits & RE_NO_GNU_OPS))
goto normal_char;
if (!dfa->multibyte)
{
zeroset (ccl);
for (c2 = 0; c2 < NOTCHAR; ++c2)
if (isspace (c2))
setbit (c2, ccl);
if (c == 'S')
notset (ccl);
laststart = false;
return lasttok = CSET + charclass_index (ccl);
}
#define PUSH_LEX_STATE(s) \
do \
{ \
char const *lexptr_saved = lexptr; \
size_t lexleft_saved = lexleft; \
lexptr = (s); \
lexleft = strlen (lexptr)
#define POP_LEX_STATE() \
lexptr = lexptr_saved; \
lexleft = lexleft_saved; \
} \
while (0)
/* FIXME: see if optimizing this, as is done with ANYCHAR and
add_utf8_anychar, makes sense. */
/* \s and \S are documented to be equivalent to [[:space:]] and
[^[:space:]] respectively, so tell the lexer to process those
strings, each minus its "already processed" '['. */
PUSH_LEX_STATE (c == 's' ? "[:space:]]" : "^[:space:]]");
lasttok = parse_bracket_exp ();
POP_LEX_STATE ();
laststart = false;
return lasttok;
case 'w':
case 'W':
if (!backslash || (syntax_bits & RE_NO_GNU_OPS))
goto normal_char;
zeroset (ccl);
for (c2 = 0; c2 < NOTCHAR; ++c2)
if (IS_WORD_CONSTITUENT (c2))
setbit (c2, ccl);
if (c == 'W')
notset (ccl);
laststart = false;
return lasttok = CSET + charclass_index (ccl);
case '[':
if (backslash)
goto normal_char;
laststart = false;
return lasttok = parse_bracket_exp ();
default:
normal_char:
laststart = false;
/* For multibyte character sets, folding is done in atom. Always
return WCHAR. */
if (dfa->multibyte)
return lasttok = WCHAR;
if (case_fold && isalpha (c))
{
zeroset (ccl);
setbit_case_fold_c (c, ccl);
return lasttok = CSET + charclass_index (ccl);
}
return lasttok = c;
}
}
/* The above loop should consume at most a backslash
and some other character. */
abort ();
return END; /* keeps pedantic compilers happy. */
}
static token
lex (void)
{
token original_token;
fsatoken_token_t fsalex_token;
const char *alarm;
original_token = original_lex ();
fsalex_token = fsalex_lex (lexer);
alarm = "";
if ((original_token != fsalex_token) && (original_token < CSET))
alarm = " *** ERROR: Token mismatch";
fprintf (pll_log, "Token debug: Original, fsalex: %08lx %08lx %s\n",
original_token, fsalex_token, alarm);
if (fsalex_token == FSATOKEN_TK_REPMN)
{
int x_minrep, x_maxrep;
x_minrep = fsalex_exchange(lexer,
PROTO_LEXPARSE_OP_GET_REPMN_MIN, NULL);
x_maxrep = fsalex_exchange(lexer,
PROTO_LEXPARSE_OP_GET_REPMN_MAX, NULL);
fprintf (pll_log, " Original REPMN{%d,%d}; ", minrep, maxrep);
fprintf (pll_log, " FSATOKEN_TK_REPMN{%d,%d}\n", x_minrep, x_maxrep);
if ((x_minrep != minrep) || (x_maxrep != maxrep))
fprintf (pll_log, " *** ERROR: Min, max rep doesn't match\n");
}
else if (fsalex_token >= FSATOKEN_TK_CSET)
{
size_t index;
unsigned int * orig_ccl;
int i;
charclass_t *charset;
char *description;
static char buf[256];
char *p_buf;
/* Nybble (4bit)-to-char conversion array for little-bit-endian nybbles. */
static const char *disp_nybble = "084c2a6e195d3b7f";
/* Report details of the original charclas produced by dfa.c. */
index = original_token - CSET;
p_buf = buf;
orig_ccl = dfa->charclasses[index];
for (i = 0; i < CHARCLASS_WORDS; i += 2)
{
charclass_word j = orig_ccl[i];
*p_buf++ = ' ';
*p_buf++ = disp_nybble[(j >> 0) & 0x0f];
*p_buf++ = disp_nybble[(j >> 4) & 0x0f];
*p_buf++ = disp_nybble[(j >> 8) & 0x0f];
*p_buf++ = disp_nybble[(j >> 12) & 0x0f];
*p_buf++ = disp_nybble[(j >> 16) & 0x0f];
*p_buf++ = disp_nybble[(j >> 20) & 0x0f];
*p_buf++ = disp_nybble[(j >> 24) & 0x0f];
*p_buf++ = disp_nybble[(j >> 28) & 0x0f];
j = orig_ccl[i + 1];
*p_buf++ = disp_nybble[(j >> 0) & 0x0f];
*p_buf++ = disp_nybble[(j >> 4) & 0x0f];
*p_buf++ = disp_nybble[(j >> 8) & 0x0f];
*p_buf++ = disp_nybble[(j >> 12) & 0x0f];
*p_buf++ = disp_nybble[(j >> 16) & 0x0f];
*p_buf++ = disp_nybble[(j >> 20) & 0x0f];
*p_buf++ = disp_nybble[(j >> 24) & 0x0f];
*p_buf++ = disp_nybble[(j >> 28) & 0x0f];
}
*p_buf++ = '\0';
fprintf (pll_log, " original [%3lu]:%s\n", index, buf);
/* Also report the charclass member details from fsalex etc. */
index = fsalex_token - FSATOKEN_TK_CSET;
charset = charclass_get_pointer (index);
description = charclass_describe (charset);
index = charclass_get_index (charset);
fprintf (pll_log, " fsalex: [%3lu] %s\n", index, description);
if (! charclass_equal (charset, (charclass_t const *) orig_ccl))
fprintf (pll_log, " *** ERROR: Charclass mismatch\n");
}
return original_token;
}
static void
show_musts (const char *title, fsamusts_list_element_t *list)
{
fsamusts_list_element_t *elem;
static char buf[256];
char *p_buf;
fprintf (pll_log, "\n%s:\n", title);
p_buf = buf;
for (elem = list; elem != NULL; elem = elem->next)
{
if (((p_buf - buf) + 4 + strlen (elem->must)) > 72)
{
fprintf(pll_log, " %s\n", buf);
p_buf = buf;
}
p_buf += sprintf (p_buf, " (%s) >%s<",
elem->exact ? "Entire" : "partial",
elem->must);
}
fprintf (pll_log, "%s\n", buf);
}
/* Recursive descent parser for regular expressions. */
static token tok; /* Lookahead token. */
static size_t depth; /* Current depth of a hypothetical stack
holding deferred productions. This is
used to determine the depth that will be
required of the real stack later on in
dfaanalyze. */
static void
addtok_mb (token t, int mbprop)
{
if (dfa->talloc == dfa->tindex)
{
dfa->tokens = x2nrealloc (dfa->tokens, &dfa->talloc,
sizeof *dfa->tokens);
if (dfa->multibyte)
dfa->multibyte_prop = xnrealloc (dfa->multibyte_prop, dfa->talloc,
sizeof *dfa->multibyte_prop);
}
if (dfa->multibyte)
dfa->multibyte_prop[dfa->tindex] = mbprop;
dfa->tokens[dfa->tindex++] = t;
switch (t)
{
case QMARK:
case STAR:
case PLUS:
break;
case CAT:
case OR:
--depth;
break;
case BACKREF:
dfa->fast = false;
/* fallthrough */
default:
++dfa->nleaves;
/* fallthrough */
case EMPTY:
++depth;
break;
}
if (depth > dfa->depth)
dfa->depth = depth;
}
static void addtok_wc (wint_t wc);
/* Add the given token to the parse tree, maintaining the depth count and
updating the maximum depth if necessary. */
static void
addtok (token t)
{
if (dfa->multibyte && t == MBCSET)
{
bool need_or = false;
struct mb_char_classes *work_mbc = &dfa->mbcsets[dfa->nmbcsets - 1];
/* Extract wide characters into alternations for better performance.
This does not require UTF-8. */
if (!work_mbc->invert)
{
size_t i;
for (i = 0; i < work_mbc->nchars; i++)
{
addtok_wc (work_mbc->chars[i]);
if (need_or)
addtok (OR);
need_or = true;
}
work_mbc->nchars = 0;
}
/* If the MBCSET is non-inverted and doesn't include neither
character classes including multibyte characters, range
expressions, equivalence classes nor collating elements,
it can be replaced to a simple CSET. */
if (work_mbc->invert
|| work_mbc->nch_classes != 0
|| work_mbc->nranges != 0
|| work_mbc->nequivs != 0 || work_mbc->ncoll_elems != 0)
{
addtok_mb (MBCSET, ((dfa->nmbcsets - 1) << 2) + 3);
if (need_or)
addtok (OR);
}
else
{
/* Characters have been handled above, so it is possible
that the mbcset is empty now. Do nothing in that case. */
if (work_mbc->cset != -1)
{
addtok (CSET + work_mbc->cset);
if (need_or)
addtok (OR);
}
}
}
else
{
addtok_mb (t, 3);
}
}
/* We treat a multibyte character as a single atom, so that DFA
can treat a multibyte character as a single expression.
e.g., we construct the following tree from "<mb1><mb2>".
<mb1(1st-byte)><mb1(2nd-byte)><CAT><mb1(3rd-byte)><CAT>
<mb2(1st-byte)><mb2(2nd-byte)><CAT><mb2(3rd-byte)><CAT><CAT> */
static void
addtok_wc (wint_t wc)
{
unsigned char buf[MB_LEN_MAX];
mbstate_t s = { 0 };
int i;
size_t stored_bytes = wcrtomb ((char *) buf, wc, &s);
if (stored_bytes != (size_t) -1)
cur_mb_len = stored_bytes;
else
{
/* This is merely stop-gap. buf[0] is undefined, yet skipping
the addtok_mb call altogether can corrupt the heap. */
cur_mb_len = 1;
buf[0] = 0;
}
addtok_mb (buf[0], cur_mb_len == 1 ? 3 : 1);
for (i = 1; i < cur_mb_len; i++)
{
addtok_mb (buf[i], i == cur_mb_len - 1 ? 2 : 0);
addtok (CAT);
}
}
static void
add_utf8_anychar (void)
{
static const charclass utf8_classes[5] = {
/* 80-bf: non-leading bytes. */
{0, 0, 0, 0, CHARCLASS_WORD_MASK, CHARCLASS_WORD_MASK, 0, 0},
/* 00-7f: 1-byte sequence. */
{CHARCLASS_WORD_MASK, CHARCLASS_WORD_MASK, CHARCLASS_WORD_MASK,
CHARCLASS_WORD_MASK, 0, 0, 0, 0},
/* c2-df: 2-byte sequence. */
{0, 0, 0, 0, 0, 0, ~3 & CHARCLASS_WORD_MASK, 0},
/* e0-ef: 3-byte sequence. */
{0, 0, 0, 0, 0, 0, 0, 0xffff},
/* f0-f7: 4-byte sequence. */
{0, 0, 0, 0, 0, 0, 0, 0xff0000}
};
const unsigned int n = sizeof (utf8_classes) / sizeof (utf8_classes[0]);
unsigned int i;
/* Define the five character classes that are needed below. */
if (dfa->utf8_anychar_classes[0] == 0)
for (i = 0; i < n; i++)
{
charclass c;
copyset (utf8_classes[i], c);
if (i == 1)
{
if (!(syntax_bits & RE_DOT_NEWLINE))
clrbit (eolbyte, c);
if (syntax_bits & RE_DOT_NOT_NULL)
clrbit ('\0', c);
}
dfa->utf8_anychar_classes[i] = CSET + charclass_index (c);
}
/* A valid UTF-8 character is
([0x00-0x7f]
|[0xc2-0xdf][0x80-0xbf]
|[0xe0-0xef[0x80-0xbf][0x80-0xbf]
|[0xf0-f7][0x80-0xbf][0x80-0xbf][0x80-0xbf])
which I'll write more concisely "B|CA|DAA|EAAA". Factor the [0x00-0x7f]
and you get "B|(C|(D|EA)A)A". And since the token buffer is in reverse
Polish notation, you get "B C D E A CAT OR A CAT OR A CAT OR". */
for (i = 1; i < n; i++)
addtok (dfa->utf8_anychar_classes[i]);
while (--i > 1)
{
addtok (dfa->utf8_anychar_classes[0]);
addtok (CAT);
addtok (OR);
}
}
/* The grammar understood by the parser is as follows.
regexp:
regexp OR branch
branch
branch:
branch closure
closure
closure:
closure QMARK
closure STAR
closure PLUS
closure REPMN
atom
atom:
<normal character>
<multibyte character>
ANYCHAR
MBCSET
CSET
BACKREF
BEGLINE
ENDLINE
BEGWORD
ENDWORD
LIMWORD
NOTLIMWORD
LPAREN regexp RPAREN
<empty>
The parser builds a parse tree in postfix form in an array of tokens. */
static void
atom (void)
{
if (tok == WCHAR)
{
if (wctok == WEOF)
addtok (BACKREF);
else
{
addtok_wc (wctok);
if (case_fold)
{
wchar_t folded[CASE_FOLDED_BUFSIZE];
int i, n = case_folded_counterparts (wctok, folded);
for (i = 0; i < n; i++)
{
addtok_wc (folded[i]);
addtok (OR);
}
}
}
tok = lex ();
}
else if (tok == ANYCHAR && using_utf8 ())
{
/* For UTF-8 expand the period to a series of CSETs that define a valid
UTF-8 character. This avoids using the slow multibyte path. I'm
pretty sure it would be both profitable and correct to do it for
any encoding; however, the optimization must be done manually as
it is done above in add_utf8_anychar. So, let's start with
UTF-8: it is the most used, and the structure of the encoding
makes the correctness more obvious. */
add_utf8_anychar ();
tok = lex ();
}
else if ((tok >= 0 && tok < NOTCHAR) || tok >= CSET || tok == BACKREF
|| tok == BEGLINE || tok == ENDLINE || tok == BEGWORD
|| tok == ANYCHAR || tok == MBCSET
|| tok == ENDWORD || tok == LIMWORD || tok == NOTLIMWORD)
{
addtok (tok);
tok = lex ();
}
else if (tok == LPAREN)
{
tok = lex ();
regexp ();
if (tok != RPAREN)
dfaerror (_("unbalanced ("));
tok = lex ();
}
else
addtok (EMPTY);
}
/* Return the number of tokens in the given subexpression. */
static size_t _GL_ATTRIBUTE_PURE
nsubtoks (size_t tindex)
{
size_t ntoks1;
switch (dfa->tokens[tindex - 1])
{
default:
return 1;
case QMARK:
case STAR:
case PLUS:
return 1 + nsubtoks (tindex - 1);
case CAT:
case OR:
ntoks1 = nsubtoks (tindex - 1);
return 1 + ntoks1 + nsubtoks (tindex - 1 - ntoks1);
}
}
/* Copy the given subexpression to the top of the tree. */
static void
copytoks (size_t tindex, size_t ntokens)
{
size_t i;
if (dfa->multibyte)
for (i = 0; i < ntokens; ++i)
addtok_mb (dfa->tokens[tindex + i], dfa->multibyte_prop[tindex + i]);
else
for (i = 0; i < ntokens; ++i)
addtok_mb (dfa->tokens[tindex + i], 3);
}
static void
closure (void)
{
int i;
size_t tindex, ntokens;
atom ();
while (tok == QMARK || tok == STAR || tok == PLUS || tok == REPMN)
if (tok == REPMN && (minrep || maxrep))
{
ntokens = nsubtoks (dfa->tindex);
tindex = dfa->tindex - ntokens;
if (maxrep < 0)
addtok (PLUS);
if (minrep == 0)
addtok (QMARK);
for (i = 1; i < minrep; ++i)
{
copytoks (tindex, ntokens);
addtok (CAT);
}
for (; i < maxrep; ++i)
{
copytoks (tindex, ntokens);
addtok (QMARK);
addtok (CAT);
}
tok = lex ();
}
else if (tok == REPMN)
{
dfa->tindex -= nsubtoks (dfa->tindex);
tok = lex ();
closure ();
}
else
{
addtok (tok);
tok = lex ();
}
}
static void
branch (void)
{
closure ();
while (tok != RPAREN && tok != OR && tok >= 0)
{
closure ();
addtok (CAT);
}
}
static void
regexp (void)
{
branch ();
while (tok == OR)
{
tok = lex ();
branch ();
addtok (OR);
}
}
/* Main entry point for the parser. S is a string to be parsed, len is the
length of the string, so s can include NUL characters. D is a pointer to
the struct dfa to parse into. */
void
dfaparse (char const *s, size_t len, struct dfa *d)
{
dfa = d;
lexptr = s;
lexleft = len;
lasttok = END;
laststart = true;
parens = 0;
if (dfa->multibyte)
{
cur_mb_len = 0;
memset (&d->mbs, 0, sizeof d->mbs);
}
if (!syntax_bits_set)
dfaerror (_("no syntax specified"));
tok = lex ();
depth = d->depth;
regexp ();
if (tok != END)
dfaerror (_("unbalanced )"));
addtok (END - d->nregexps);
addtok (CAT);
if (d->nregexps)
addtok (OR);
++d->nregexps;
}
/* Some primitives for operating on sets of positions. */
/* Copy one set to another. */
static void
copy (position_set const *src, position_set * dst)
{
if (dst->alloc < src->nelem)
{
free (dst->elems);
dst->alloc = src->nelem;
dst->elems = x2nrealloc (NULL, &dst->alloc, sizeof *dst->elems);
}
memcpy (dst->elems, src->elems, src->nelem * sizeof *dst->elems);
dst->nelem = src->nelem;
}
static void
alloc_position_set (position_set * s, size_t size)
{
s->elems = xnmalloc (size, sizeof *s->elems);
s->alloc = size;
s->nelem = 0;
}
/* Insert position P in set S. S is maintained in sorted order on
decreasing index. If there is already an entry in S with P.index
then merge (logically-OR) P's constraints into the one in S.
S->elems must point to an array large enough to hold the resulting set. */
static void
insert (position p, position_set * s)
{
size_t count = s->nelem;
size_t lo = 0, hi = count;
size_t i;
while (lo < hi)
{
size_t mid = (lo + hi) >> 1;
if (s->elems[mid].index > p.index)
lo = mid + 1;
else
hi = mid;
}
if (lo < count && p.index == s->elems[lo].index)
{
s->elems[lo].constraint |= p.constraint;
return;
}
s->elems = maybe_realloc (s->elems, count, &s->alloc, sizeof *s->elems);
for (i = count; i > lo; i--)
s->elems[i] = s->elems[i - 1];
s->elems[lo] = p;
++s->nelem;
}
/* Merge two sets of positions into a third. The result is exactly as if
the positions of both sets were inserted into an initially empty set. */
static void
merge (position_set const *s1, position_set const *s2, position_set * m)
{
size_t i = 0, j = 0;
if (m->alloc < s1->nelem + s2->nelem)
{
free (m->elems);
m->elems = maybe_realloc (NULL, s1->nelem + s2->nelem, &m->alloc,
sizeof *m->elems);
}
m->nelem = 0;
while (i < s1->nelem && j < s2->nelem)
if (s1->elems[i].index > s2->elems[j].index)
m->elems[m->nelem++] = s1->elems[i++];
else if (s1->elems[i].index < s2->elems[j].index)
m->elems[m->nelem++] = s2->elems[j++];
else
{
m->elems[m->nelem] = s1->elems[i++];
m->elems[m->nelem++].constraint |= s2->elems[j++].constraint;
}
while (i < s1->nelem)
m->elems[m->nelem++] = s1->elems[i++];
while (j < s2->nelem)
m->elems[m->nelem++] = s2->elems[j++];
}
/* Delete a position from a set. */
static void
delete (position p, position_set * s)
{
size_t i;
for (i = 0; i < s->nelem; ++i)
if (p.index == s->elems[i].index)
break;
if (i < s->nelem)
for (--s->nelem; i < s->nelem; ++i)
s->elems[i] = s->elems[i + 1];
}
/* Find the index of the state corresponding to the given position set with
the given preceding context, or create a new state if there is no such
state. Context tells whether we got here on a newline or letter. */
static state_num
state_index (struct dfa *d, position_set const *s, int context)
{
size_t hash = 0;
int constraint;
state_num i, j;
for (i = 0; i < s->nelem; ++i)
hash ^= s->elems[i].index + s->elems[i].constraint;
/* Try to find a state that exactly matches the proposed one. */
for (i = 0; i < d->sindex; ++i)
{
if (hash != d->states[i].hash || s->nelem != d->states[i].elems.nelem
|| context != d->states[i].context)
continue;
for (j = 0; j < s->nelem; ++j)
if (s->elems[j].constraint
!= d->states[i].elems.elems[j].constraint
|| s->elems[j].index != d->states[i].elems.elems[j].index)
break;
if (j == s->nelem)
return i;
}
/* We'll have to create a new state. */
d->states = maybe_realloc (d->states, d->sindex, &d->salloc,
sizeof *d->states);
d->states[i].hash = hash;
alloc_position_set (&d->states[i].elems, s->nelem);
copy (s, &d->states[i].elems);
d->states[i].context = context;
d->states[i].has_backref = false;
d->states[i].has_mbcset = false;
d->states[i].constraint = 0;
d->states[i].first_end = 0;
d->states[i].mbps.nelem = 0;
d->states[i].mbps.elems = NULL;
for (j = 0; j < s->nelem; ++j)
if (d->tokens[s->elems[j].index] < 0)
{
constraint = s->elems[j].constraint;
if (SUCCEEDS_IN_CONTEXT (constraint, context, CTX_ANY))
d->states[i].constraint |= constraint;
if (!d->states[i].first_end)
d->states[i].first_end = d->tokens[s->elems[j].index];
}
else if (d->tokens[s->elems[j].index] == BACKREF)
{
d->states[i].constraint = NO_CONSTRAINT;
d->states[i].has_backref = true;
}
++d->sindex;
return i;
}
/* Find the epsilon closure of a set of positions. If any position of the set
contains a symbol that matches the empty string in some context, replace
that position with the elements of its follow labeled with an appropriate
constraint. Repeat exhaustively until no funny positions are left.
S->elems must be large enough to hold the result. */
static void
epsclosure (position_set *s, struct dfa const *d, char *visited)
{
size_t i, j;
position p, old;
bool initialized = false;
for (i = 0; i < s->nelem; ++i)
if (d->tokens[s->elems[i].index] >= NOTCHAR
&& d->tokens[s->elems[i].index] != BACKREF
&& d->tokens[s->elems[i].index] != ANYCHAR
&& d->tokens[s->elems[i].index] != MBCSET
&& d->tokens[s->elems[i].index] < CSET)
{
if (!initialized)
{
memset (visited, 0, d->tindex * sizeof (*visited));
initialized = true;
}
old = s->elems[i];
p.constraint = old.constraint;
delete (s->elems[i], s);
if (visited[old.index])
{
--i;
continue;
}
visited[old.index] = 1;
switch (d->tokens[old.index])
{
case BEGLINE:
p.constraint &= BEGLINE_CONSTRAINT;
break;
case ENDLINE:
p.constraint &= ENDLINE_CONSTRAINT;
break;
case BEGWORD:
p.constraint &= BEGWORD_CONSTRAINT;
break;
case ENDWORD:
p.constraint &= ENDWORD_CONSTRAINT;
break;
case LIMWORD:
p.constraint &= LIMWORD_CONSTRAINT;
break;
case NOTLIMWORD:
p.constraint &= NOTLIMWORD_CONSTRAINT;
break;
default:
break;
}
for (j = 0; j < d->follows[old.index].nelem; ++j)
{
p.index = d->follows[old.index].elems[j].index;
insert (p, s);
}
/* Force rescan to start at the beginning. */
i = -1;
}
}
/* Returns the set of contexts for which there is at least one
character included in C. */
static int
charclass_context (charclass c)
{
int context = 0;
unsigned int j;
if (tstbit (eolbyte, c))
context |= CTX_NEWLINE;
for (j = 0; j < CHARCLASS_WORDS; ++j)
{
if (c[j] & letters[j])
context |= CTX_LETTER;
if (c[j] & ~(letters[j] | newline[j]))
context |= CTX_NONE;
}
return context;
}
/* Returns the contexts on which the position set S depends. Each context
in the set of returned contexts (let's call it SC) may have a different
follow set than other contexts in SC, and also different from the
follow set of the complement set (sc ^ CTX_ANY). However, all contexts
in the complement set will have the same follow set. */
static int _GL_ATTRIBUTE_PURE
state_separate_contexts (position_set const *s)
{
int separate_contexts = 0;
size_t j;
for (j = 0; j < s->nelem; ++j)
{
if (PREV_NEWLINE_DEPENDENT (s->elems[j].constraint))
separate_contexts |= CTX_NEWLINE;
if (PREV_LETTER_DEPENDENT (s->elems[j].constraint))
separate_contexts |= CTX_LETTER;
}
return separate_contexts;
}
/* Perform bottom-up analysis on the parse tree, computing various functions.
Note that at this point, we're pretending constructs like \< are real
characters rather than constraints on what can follow them.
Nullable: A node is nullable if it is at the root of a regexp that can
match the empty string.
* EMPTY leaves are nullable.
* No other leaf is nullable.
* A QMARK or STAR node is nullable.
* A PLUS node is nullable if its argument is nullable.
* A CAT node is nullable if both its arguments are nullable.
* An OR node is nullable if either argument is nullable.
Firstpos: The firstpos of a node is the set of positions (nonempty leaves)
that could correspond to the first character of a string matching the
regexp rooted at the given node.
* EMPTY leaves have empty firstpos.
* The firstpos of a nonempty leaf is that leaf itself.
* The firstpos of a QMARK, STAR, or PLUS node is the firstpos of its
argument.
* The firstpos of a CAT node is the firstpos of the left argument, union
the firstpos of the right if the left argument is nullable.
* The firstpos of an OR node is the union of firstpos of each argument.
Lastpos: The lastpos of a node is the set of positions that could
correspond to the last character of a string matching the regexp at
the given node.
* EMPTY leaves have empty lastpos.
* The lastpos of a nonempty leaf is that leaf itself.
* The lastpos of a QMARK, STAR, or PLUS node is the lastpos of its
argument.
* The lastpos of a CAT node is the lastpos of its right argument, union
the lastpos of the left if the right argument is nullable.
* The lastpos of an OR node is the union of the lastpos of each argument.
Follow: The follow of a position is the set of positions that could
correspond to the character following a character matching the node in
a string matching the regexp. At this point we consider special symbols
that match the empty string in some context to be just normal characters.
Later, if we find that a special symbol is in a follow set, we will
replace it with the elements of its follow, labeled with an appropriate
constraint.
* Every node in the firstpos of the argument of a STAR or PLUS node is in
the follow of every node in the lastpos.
* Every node in the firstpos of the second argument of a CAT node is in
the follow of every node in the lastpos of the first argument.
Because of the postfix representation of the parse tree, the depth-first
analysis is conveniently done by a linear scan with the aid of a stack.
Sets are stored as arrays of the elements, obeying a stack-like allocation
scheme; the number of elements in each set deeper in the stack can be
used to determine the address of a particular set's array. */
void
dfaanalyze (struct dfa *d, int searchflag)
{
/* Array allocated to hold position sets. */
position *posalloc = xnmalloc (d->nleaves, 2 * sizeof *posalloc);
/* Firstpos and lastpos elements. */
position *firstpos = posalloc + d->nleaves;
position *lastpos = firstpos + d->nleaves;
/* Stack for element counts and nullable flags. */
struct
{
/* Whether the entry is nullable. */
bool nullable;
/* Counts of firstpos and lastpos sets. */
size_t nfirstpos;
size_t nlastpos;
} *stkalloc = xnmalloc (d->depth, sizeof *stkalloc), *stk = stkalloc;
position_set tmp; /* Temporary set for merging sets. */
position_set merged; /* Result of merging sets. */
int separate_contexts; /* Context wanted by some position. */
size_t i, j;
position *pos;
char *visited = xnmalloc (d->tindex, sizeof *visited);
#ifdef DEBUG
fprintf (stderr, "dfaanalyze:\n");
for (i = 0; i < d->tindex; ++i)
{
fprintf (stderr, " %zd:", i);
prtok (d->tokens[i]);
}
putc ('\n', stderr);
#endif
d->searchflag = searchflag != 0;
alloc_position_set (&merged, d->nleaves);
d->follows = xcalloc (d->tindex, sizeof *d->follows);
for (i = 0; i < d->tindex; ++i)
{
switch (d->tokens[i])
{
case EMPTY:
/* The empty set is nullable. */
stk->nullable = true;
/* The firstpos and lastpos of the empty leaf are both empty. */
stk->nfirstpos = stk->nlastpos = 0;
stk++;
break;
case STAR:
case PLUS:
/* Every element in the firstpos of the argument is in the follow
of every element in the lastpos. */
tmp.nelem = stk[-1].nfirstpos;
tmp.elems = firstpos;
pos = lastpos;
for (j = 0; j < stk[-1].nlastpos; ++j)
{
merge (&tmp, &d->follows[pos[j].index], &merged);
copy (&merged, &d->follows[pos[j].index]);
}
/* fallthrough */
case QMARK:
/* A QMARK or STAR node is automatically nullable. */
if (d->tokens[i] != PLUS)
stk[-1].nullable = true;
break;
case CAT:
/* Every element in the firstpos of the second argument is in the
follow of every element in the lastpos of the first argument. */
tmp.nelem = stk[-1].nfirstpos;
tmp.elems = firstpos;
pos = lastpos + stk[-1].nlastpos;
for (j = 0; j < stk[-2].nlastpos; ++j)
{
merge (&tmp, &d->follows[pos[j].index], &merged);
copy (&merged, &d->follows[pos[j].index]);
}
/* The firstpos of a CAT node is the firstpos of the first argument,
union that of the second argument if the first is nullable. */
if (stk[-2].nullable)
stk[-2].nfirstpos += stk[-1].nfirstpos;
else
firstpos += stk[-1].nfirstpos;
/* The lastpos of a CAT node is the lastpos of the second argument,
union that of the first argument if the second is nullable. */
if (stk[-1].nullable)
stk[-2].nlastpos += stk[-1].nlastpos;
else
{
pos = lastpos + stk[-2].nlastpos;
for (j = stk[-1].nlastpos; j-- > 0;)
pos[j] = lastpos[j];
lastpos += stk[-2].nlastpos;
stk[-2].nlastpos = stk[-1].nlastpos;
}
/* A CAT node is nullable if both arguments are nullable. */
stk[-2].nullable &= stk[-1].nullable;
stk--;
break;
case OR:
/* The firstpos is the union of the firstpos of each argument. */
stk[-2].nfirstpos += stk[-1].nfirstpos;
/* The lastpos is the union of the lastpos of each argument. */
stk[-2].nlastpos += stk[-1].nlastpos;
/* An OR node is nullable if either argument is nullable. */
stk[-2].nullable |= stk[-1].nullable;
stk--;
break;
default:
/* Anything else is a nonempty position. (Note that special
constructs like \< are treated as nonempty strings here;
an "epsilon closure" effectively makes them nullable later.
Backreferences have to get a real position so we can detect
transitions on them later. But they are nullable. */
stk->nullable = d->tokens[i] == BACKREF;
/* This position is in its own firstpos and lastpos. */
stk->nfirstpos = stk->nlastpos = 1;
stk++;
--firstpos, --lastpos;
firstpos->index = lastpos->index = i;
firstpos->constraint = lastpos->constraint = NO_CONSTRAINT;
/* Allocate the follow set for this position. */
alloc_position_set (&d->follows[i], 1);
break;
}
#ifdef DEBUG
/* ... balance the above nonsyntactic #ifdef goo... */
fprintf (stderr, "node %zd:", i);
prtok (d->tokens[i]);
putc ('\n', stderr);
fprintf (stderr,
stk[-1].nullable ? " nullable: yes\n" : " nullable: no\n");
fprintf (stderr, " firstpos:");
for (j = stk[-1].nfirstpos; j-- > 0;)
{
fprintf (stderr, " %zd:", firstpos[j].index);
prtok (d->tokens[firstpos[j].index]);
}
fprintf (stderr, "\n lastpos:");
for (j = stk[-1].nlastpos; j-- > 0;)
{
fprintf (stderr, " %zd:", lastpos[j].index);
prtok (d->tokens[lastpos[j].index]);
}
putc ('\n', stderr);
#endif
}
/* For each follow set that is the follow set of a real position, replace
it with its epsilon closure. */
for (i = 0; i < d->tindex; ++i)
if (d->tokens[i] < NOTCHAR || d->tokens[i] == BACKREF
|| d->tokens[i] == ANYCHAR || d->tokens[i] == MBCSET
|| d->tokens[i] >= CSET)
{
#ifdef DEBUG
fprintf (stderr, "follows(%zd:", i);
prtok (d->tokens[i]);
fprintf (stderr, "):");
for (j = d->follows[i].nelem; j-- > 0;)
{
fprintf (stderr, " %zd:", d->follows[i].elems[j].index);
prtok (d->tokens[d->follows[i].elems[j].index]);
}
putc ('\n', stderr);
#endif
copy (&d->follows[i], &merged);
epsclosure (&merged, d, visited);
copy (&merged, &d->follows[i]);
}
/* Get the epsilon closure of the firstpos of the regexp. The result will
be the set of positions of state 0. */
merged.nelem = 0;
for (i = 0; i < stk[-1].nfirstpos; ++i)
insert (firstpos[i], &merged);
epsclosure (&merged, d, visited);
/* Build the initial state. */
separate_contexts = state_separate_contexts (&merged);
state_index (d, &merged,
(separate_contexts & CTX_NEWLINE
? CTX_NEWLINE : separate_contexts ^ CTX_ANY));
free (posalloc);
free (stkalloc);
free (merged.elems);
free (visited);
}
/* Find, for each character, the transition out of state s of d, and store
it in the appropriate slot of trans.
We divide the positions of s into groups (positions can appear in more
than one group). Each group is labeled with a set of characters that
every position in the group matches (taking into account, if necessary,
preceding context information of s). For each group, find the union
of the its elements' follows. This set is the set of positions of the
new state. For each character in the group's label, set the transition
on this character to be to a state corresponding to the set's positions,
and its associated backward context information, if necessary.
If we are building a searching matcher, we include the positions of state
0 in every state.
The collection of groups is constructed by building an equivalence-class
partition of the positions of s.
For each position, find the set of characters C that it matches. Eliminate
any characters from C that fail on grounds of backward context.
Search through the groups, looking for a group whose label L has nonempty
intersection with C. If L - C is nonempty, create a new group labeled
L - C and having the same positions as the current group, and set L to
the intersection of L and C. Insert the position in this group, set
C = C - L, and resume scanning.
If after comparing with every group there are characters remaining in C,
create a new group labeled with the characters of C and insert this
position in that group. */
void
dfastate (state_num s, struct dfa *d, state_num trans[])
{
leaf_set grps[NOTCHAR]; /* As many as will ever be needed. */
charclass labels[NOTCHAR]; /* Labels corresponding to the groups. */
size_t ngrps = 0; /* Number of groups actually used. */
position pos; /* Current position being considered. */
charclass matches; /* Set of matching characters. */
charclass_word matchesf; /* Nonzero if matches is nonempty. */
charclass intersect; /* Intersection with some label set. */
charclass_word intersectf; /* Nonzero if intersect is nonempty. */
charclass leftovers; /* Stuff in the label that didn't match. */
charclass_word leftoversf; /* Nonzero if leftovers is nonempty. */
position_set follows; /* Union of the follows of some group. */
position_set tmp; /* Temporary space for merging sets. */
int possible_contexts; /* Contexts that this group can match. */
int separate_contexts; /* Context that new state wants to know. */
state_num state; /* New state. */
state_num state_newline; /* New state on a newline transition. */
state_num state_letter; /* New state on a letter transition. */
bool next_isnt_1st_byte = false; /* We can't add state0. */
size_t i, j, k;
zeroset (matches);
for (i = 0; i < d->states[s].elems.nelem; ++i)
{
pos = d->states[s].elems.elems[i];
if (d->tokens[pos.index] >= 0 && d->tokens[pos.index] < NOTCHAR)
setbit (d->tokens[pos.index], matches);
else if (d->tokens[pos.index] >= CSET)
copyset (d->charclasses[d->tokens[pos.index] - CSET], matches);
else
{
if (d->tokens[pos.index] == MBCSET
|| d->tokens[pos.index] == ANYCHAR)
{
/* MB_CUR_MAX > 1 */
if (d->tokens[pos.index] == MBCSET)
d->states[s].has_mbcset = true;
/* ANYCHAR and MBCSET must match with a single character, so we
must put it to d->states[s].mbps, which contains the positions
which can match with a single character not a byte. */
if (d->states[s].mbps.nelem == 0)
alloc_position_set (&d->states[s].mbps, 1);
insert (pos, &(d->states[s].mbps));
}
continue;
}
/* Some characters may need to be eliminated from matches because
they fail in the current context. */
if (pos.constraint != NO_CONSTRAINT)
{
if (!SUCCEEDS_IN_CONTEXT (pos.constraint,
d->states[s].context, CTX_NEWLINE))
for (j = 0; j < CHARCLASS_WORDS; ++j)
matches[j] &= ~newline[j];
if (!SUCCEEDS_IN_CONTEXT (pos.constraint,
d->states[s].context, CTX_LETTER))
for (j = 0; j < CHARCLASS_WORDS; ++j)
matches[j] &= ~letters[j];
if (!SUCCEEDS_IN_CONTEXT (pos.constraint,
d->states[s].context, CTX_NONE))
for (j = 0; j < CHARCLASS_WORDS; ++j)
matches[j] &= letters[j] | newline[j];
/* If there are no characters left, there's no point in going on. */
for (j = 0; j < CHARCLASS_WORDS && !matches[j]; ++j)
continue;
if (j == CHARCLASS_WORDS)
continue;
}
for (j = 0; j < ngrps; ++j)
{
/* If matches contains a single character only, and the current
group's label doesn't contain that character, go on to the
next group. */
if (d->tokens[pos.index] >= 0 && d->tokens[pos.index] < NOTCHAR
&& !tstbit (d->tokens[pos.index], labels[j]))
continue;
/* Check if this group's label has a nonempty intersection with
matches. */
intersectf = 0;
for (k = 0; k < CHARCLASS_WORDS; ++k)
intersectf |= intersect[k] = matches[k] & labels[j][k];
if (!intersectf)
continue;
/* It does; now find the set differences both ways. */
leftoversf = matchesf = 0;
for (k = 0; k < CHARCLASS_WORDS; ++k)
{
/* Even an optimizing compiler can't know this for sure. */
charclass_word match = matches[k], label = labels[j][k];
leftoversf |= leftovers[k] = ~match & label;
matchesf |= matches[k] = match & ~label;
}
/* If there were leftovers, create a new group labeled with them. */
if (leftoversf)
{
copyset (leftovers, labels[ngrps]);
copyset (intersect, labels[j]);
grps[ngrps].elems = xnmalloc (d->nleaves,
sizeof *grps[ngrps].elems);
memcpy (grps[ngrps].elems, grps[j].elems,
sizeof (grps[j].elems[0]) * grps[j].nelem);
grps[ngrps].nelem = grps[j].nelem;
++ngrps;
}
/* Put the position in the current group. The constraint is
irrelevant here. */
grps[j].elems[grps[j].nelem++] = pos.index;
/* If every character matching the current position has been
accounted for, we're done. */
if (!matchesf)
break;
}
/* If we've passed the last group, and there are still characters
unaccounted for, then we'll have to create a new group. */
if (j == ngrps)
{
copyset (matches, labels[ngrps]);
zeroset (matches);
grps[ngrps].elems = xnmalloc (d->nleaves, sizeof *grps[ngrps].elems);
grps[ngrps].nelem = 1;
grps[ngrps].elems[0] = pos.index;
++ngrps;
}
}
alloc_position_set (&follows, d->nleaves);
alloc_position_set (&tmp, d->nleaves);
/* If we are a searching matcher, the default transition is to a state
containing the positions of state 0, otherwise the default transition
is to fail miserably. */
if (d->searchflag)
{
/* Find the state(s) corresponding to the positions of state 0. */
copy (&d->states[0].elems, &follows);
separate_contexts = state_separate_contexts (&follows);
state = state_index (d, &follows, separate_contexts ^ CTX_ANY);
if (separate_contexts & CTX_NEWLINE)
state_newline = state_index (d, &follows, CTX_NEWLINE);
else
state_newline = state;
if (separate_contexts & CTX_LETTER)
state_letter = state_index (d, &follows, CTX_LETTER);
else
state_letter = state;
for (i = 0; i < NOTCHAR; ++i)
trans[i] = (IS_WORD_CONSTITUENT (i)) ? state_letter : state;
trans[eolbyte] = state_newline;
}
else
for (i = 0; i < NOTCHAR; ++i)
trans[i] = -1;
for (i = 0; i < ngrps; ++i)
{
follows.nelem = 0;
/* Find the union of the follows of the positions of the group.
This is a hideously inefficient loop. Fix it someday. */
for (j = 0; j < grps[i].nelem; ++j)
for (k = 0; k < d->follows[grps[i].elems[j]].nelem; ++k)
insert (d->follows[grps[i].elems[j]].elems[k], &follows);
if (d->multibyte)
{
/* If a token in follows.elems is not 1st byte of a multibyte
character, or the states of follows must accept the bytes
which are not 1st byte of the multibyte character.
Then, if a state of follows encounter a byte, it must not be
a 1st byte of a multibyte character nor single byte character.
We cansel to add state[0].follows to next state, because
state[0] must accept 1st-byte
For example, we assume <sb a> is a certain single byte
character, <mb A> is a certain multibyte character, and the
codepoint of <sb a> equals the 2nd byte of the codepoint of
<mb A>.
When state[0] accepts <sb a>, state[i] transit to state[i+1]
by accepting accepts 1st byte of <mb A>, and state[i+1]
accepts 2nd byte of <mb A>, if state[i+1] encounter the
codepoint of <sb a>, it must not be <sb a> but 2nd byte of
<mb A>, so we cannot add state[0]. */
next_isnt_1st_byte = false;
for (j = 0; j < follows.nelem; ++j)
{
if (!(d->multibyte_prop[follows.elems[j].index] & 1))
{
next_isnt_1st_byte = true;
break;
}
}
}
/* If we are building a searching matcher, throw in the positions
of state 0 as well. */
if (d->searchflag && (!d->multibyte || !next_isnt_1st_byte))
{
merge (&d->states[0].elems, &follows, &tmp);
copy (&tmp, &follows);
}
/* Find out if the new state will want any context information. */
possible_contexts = charclass_context (labels[i]);
separate_contexts = state_separate_contexts (&follows);
/* Find the state(s) corresponding to the union of the follows. */
if ((separate_contexts & possible_contexts) != possible_contexts)
state = state_index (d, &follows, separate_contexts ^ CTX_ANY);
else
state = -1;
if (separate_contexts & possible_contexts & CTX_NEWLINE)
state_newline = state_index (d, &follows, CTX_NEWLINE);
else
state_newline = state;
if (separate_contexts & possible_contexts & CTX_LETTER)
state_letter = state_index (d, &follows, CTX_LETTER);
else
state_letter = state;
/* Set the transitions for each character in the current label. */
for (j = 0; j < CHARCLASS_WORDS; ++j)
for (k = 0; k < CHARCLASS_WORD_BITS; ++k)
if (labels[i][j] >> k & 1)
{
int c = j * CHARCLASS_WORD_BITS + k;
if (c == eolbyte)
trans[c] = state_newline;
else if (IS_WORD_CONSTITUENT (c))
trans[c] = state_letter;
else if (c < NOTCHAR)
trans[c] = state;
}
}
for (i = 0; i < ngrps; ++i)
free (grps[i].elems);
free (follows.elems);
free (tmp.elems);
}
/* Make sure D's state arrays are large enough to hold NEW_STATE. */
static void
realloc_trans_if_necessary (struct dfa *d, state_num new_state)
{
state_num oldalloc = d->tralloc;
if (oldalloc <= new_state)
{
state_num **realtrans = d->trans ? d->trans - 1 : NULL;
size_t newalloc, newalloc1;
newalloc1 = new_state + 1;
realtrans = x2nrealloc (realtrans, &newalloc1, sizeof *realtrans);
realtrans[0] = NULL;
d->trans = realtrans + 1;
d->tralloc = newalloc = newalloc1 - 1;
d->fails = xnrealloc (d->fails, newalloc, sizeof *d->fails);
d->success = xnrealloc (d->success, newalloc, sizeof *d->success);
d->newlines = xnrealloc (d->newlines, newalloc, sizeof *d->newlines);
for (; oldalloc < newalloc; oldalloc++)
{
d->trans[oldalloc] = NULL;
d->fails[oldalloc] = NULL;
}
}
}
/* Some routines for manipulating a compiled dfa's transition tables.
Each state may or may not have a transition table; if it does, and it
is a non-accepting state, then d->trans[state] points to its table.
If it is an accepting state then d->fails[state] points to its table.
If it has no table at all, then d->trans[state] is NULL.
TODO: Improve this comment, get rid of the unnecessary redundancy. */
static void
build_state (state_num s, struct dfa *d)
{
state_num *trans; /* The new transition table. */
state_num i, maxstate;
/* Set an upper limit on the number of transition tables that will ever
exist at once. 1024 is arbitrary. The idea is that the frequently
used transition tables will be quickly rebuilt, whereas the ones that
were only needed once or twice will be cleared away. However, do
not clear the initial state, as it's always used. */
if (d->trcount >= 1024)
{
for (i = 1; i < d->tralloc; ++i)
{
free (d->trans[i]);
free (d->fails[i]);
d->trans[i] = d->fails[i] = NULL;
}
d->trcount = 1;
}
++d->trcount;
/* Set up the success bits for this state. */
d->success[s] = 0;
if (ACCEPTS_IN_CONTEXT (d->states[s].context, CTX_NEWLINE, s, *d))
d->success[s] |= CTX_NEWLINE;
if (ACCEPTS_IN_CONTEXT (d->states[s].context, CTX_LETTER, s, *d))
d->success[s] |= CTX_LETTER;
if (ACCEPTS_IN_CONTEXT (d->states[s].context, CTX_NONE, s, *d))
d->success[s] |= CTX_NONE;
trans = xmalloc (NOTCHAR * sizeof *trans);
dfastate (s, d, trans);
/* Now go through the new transition table, and make sure that the trans
and fail arrays are allocated large enough to hold a pointer for the
largest state mentioned in the table. */
maxstate = -1;
for (i = 0; i < NOTCHAR; ++i)
if (maxstate < trans[i])
maxstate = trans[i];
realloc_trans_if_necessary (d, maxstate);
/* Keep the newline transition in a special place so we can use it as
a sentinel. */
d->newlines[s] = trans[eolbyte];
trans[eolbyte] = -1;
if (ACCEPTING (s, *d))
d->fails[s] = trans;
else
d->trans[s] = trans;
}
/* Multibyte character handling sub-routines for dfaexec. */
/* Return values of transit_state_singlebyte, and
transit_state_consume_1char. */
typedef enum
{
TRANSIT_STATE_IN_PROGRESS, /* State transition has not finished. */
TRANSIT_STATE_DONE, /* State transition has finished. */
TRANSIT_STATE_END_BUFFER /* Reach the end of the buffer. */
} status_transit_state;
/* Consume a single byte and transit state from 's' to '*next_state'.
This function is almost same as the state transition routin in dfaexec.
But state transition is done just once, otherwise matching succeed or
reach the end of the buffer. */
static status_transit_state
transit_state_singlebyte (struct dfa *d, state_num s, unsigned char const *p,
state_num * next_state)
{
state_num *t;
state_num works = s;
status_transit_state rval = TRANSIT_STATE_IN_PROGRESS;
while (rval == TRANSIT_STATE_IN_PROGRESS)
{
if ((t = d->trans[works]) != NULL)
{
works = t[*p];
rval = TRANSIT_STATE_DONE;
if (works < 0)
works = 0;
}
else if (works < 0)
works = 0;
else if (d->fails[works])
{
works = d->fails[works][*p];
rval = TRANSIT_STATE_DONE;
}
else
{
build_state (works, d);
}
}
*next_state = works;
return rval;
}
/* Match a "." against the current context. Return the length of the
match, in bytes. POS is the position of the ".". */
static int
match_anychar (struct dfa *d, state_num s, position pos,
wint_t wc, size_t mbclen)
{
int context;
/* Check syntax bits. */
if (wc == (wchar_t) eolbyte)
{
if (!(syntax_bits & RE_DOT_NEWLINE))
return 0;
}
else if (wc == (wchar_t) '\0')
{
if (syntax_bits & RE_DOT_NOT_NULL)
return 0;
}
else if (wc == WEOF)
return 0;
context = wchar_context (wc);
if (!SUCCEEDS_IN_CONTEXT (pos.constraint, d->states[s].context, context))
return 0;
return mbclen;
}
/* Match a bracket expression against the current context.
Return the length of the match, in bytes.
POS is the position of the bracket expression. */
static int
match_mb_charset (struct dfa *d, state_num s, position pos,
char const *p, wint_t wc, size_t match_len)
{
size_t i;
bool match; /* Matching succeeded. */
int op_len; /* Length of the operator. */
char buffer[128];
/* Pointer to the structure to which we are currently referring. */
struct mb_char_classes *work_mbc;
int context;
/* Check syntax bits. */
if (wc == (wchar_t) eolbyte)
{
if (!(syntax_bits & RE_DOT_NEWLINE))
return 0;
}
else if (wc == (wchar_t) '\0')
{
if (syntax_bits & RE_DOT_NOT_NULL)
return 0;
}
else if (wc == WEOF)
return 0;
context = wchar_context (wc);
if (!SUCCEEDS_IN_CONTEXT (pos.constraint, d->states[s].context, context))
return 0;
/* Assign the current referring operator to work_mbc. */
work_mbc = &(d->mbcsets[(d->multibyte_prop[pos.index]) >> 2]);
match = !work_mbc->invert;
/* Match in range 0-255? */
if (wc < NOTCHAR && work_mbc->cset != -1
&& tstbit (to_uchar (wc), d->charclasses[work_mbc->cset]))
goto charset_matched;
/* match with a character class? */
for (i = 0; i < work_mbc->nch_classes; i++)
{
if (iswctype ((wint_t) wc, work_mbc->ch_classes[i]))
goto charset_matched;
}
strncpy (buffer, p, match_len);
buffer[match_len] = '\0';
/* match with an equivalence class? */
for (i = 0; i < work_mbc->nequivs; i++)
{
op_len = strlen (work_mbc->equivs[i]);
strncpy (buffer, p, op_len);
buffer[op_len] = '\0';
if (strcoll (work_mbc->equivs[i], buffer) == 0)
{
match_len = op_len;
goto charset_matched;
}
}
/* match with a collating element? */
for (i = 0; i < work_mbc->ncoll_elems; i++)
{
op_len = strlen (work_mbc->coll_elems[i]);
strncpy (buffer, p, op_len);
buffer[op_len] = '\0';
if (strcoll (work_mbc->coll_elems[i], buffer) == 0)
{
match_len = op_len;
goto charset_matched;
}
}
/* match with a range? */
for (i = 0; i < work_mbc->nranges; i++)
{
if (work_mbc->ranges[i].beg <= wc && wc <= work_mbc->ranges[i].end)
goto charset_matched;
}
/* match with a character? */
for (i = 0; i < work_mbc->nchars; i++)
{
if (wc == work_mbc->chars[i])
goto charset_matched;
}
match = !match;
charset_matched:
return match ? match_len : 0;
}
/* Check whether each of 'd->states[s].mbps.elem' can match. Then return the
array which corresponds to 'd->states[s].mbps.elem'; each element of the
array contains the number of bytes with which the element can match.
The caller MUST free the array which this function return. */
static int *
check_matching_with_multibyte_ops (struct dfa *d, state_num s,
char const *p, wint_t wc, size_t mbclen)
{
size_t i;
int *rarray;
rarray = d->mb_match_lens;
for (i = 0; i < d->states[s].mbps.nelem; ++i)
{
position pos = d->states[s].mbps.elems[i];
switch (d->tokens[pos.index])
{
case ANYCHAR:
rarray[i] = match_anychar (d, s, pos, wc, mbclen);
break;
case MBCSET:
rarray[i] = match_mb_charset (d, s, pos, p, wc, mbclen);
break;
default:
break; /* cannot happen. */
}
}
return rarray;
}
/* Consume a single character and enumerate all of the positions which can
be the next position from the state 's'.
'match_lens' is the input. It can be NULL, but it can also be the output
of check_matching_with_multibyte_ops for optimization.
'mbclen' and 'pps' are the output. 'mbclen' is the length of the
character consumed, and 'pps' is the set this function enumerates. */
static status_transit_state
transit_state_consume_1char (struct dfa *d, state_num s,
unsigned char const **pp,
wint_t wc, size_t mbclen,
int *match_lens)
{
size_t i, j;
int k;
state_num s1, s2;
status_transit_state rs = TRANSIT_STATE_DONE;
if (! match_lens && d->states[s].mbps.nelem != 0)
match_lens = check_matching_with_multibyte_ops (d, s, (char const *) *pp,
wc, mbclen);
/* Calculate the state which can be reached from the state 's' by
consuming 'mbclen' single bytes from the buffer. */
s1 = s;
for (k = 0; k < mbclen; k++)
{
s2 = s1;
rs = transit_state_singlebyte (d, s2, (*pp)++, &s1);
}
copy (&d->states[s1].elems, &d->mb_follows);
/* Add all of the positions which can be reached from 's' by consuming
a single character. */
for (i = 0; i < d->states[s].mbps.nelem; i++)
{
if (match_lens[i] == mbclen)
for (j = 0; j < d->follows[d->states[s].mbps.elems[i].index].nelem;
j++)
insert (d->follows[d->states[s].mbps.elems[i].index].elems[j],
&d->mb_follows);
}
/* FIXME: this return value is always ignored. */
return rs;
}
/* Transit state from s, then return new state and update the pointer of the
buffer. This function is for some operator which can match with a multi-
byte character or a collating element (which may be multi characters). */
static state_num
transit_state (struct dfa *d, state_num s, unsigned char const **pp,
unsigned char const *end)
{
state_num s1;
int mbclen; /* The length of current input multibyte character. */
int maxlen = 0;
size_t i, j;
int *match_lens = NULL;
size_t nelem = d->states[s].mbps.nelem; /* Just a alias. */
unsigned char const *p1 = *pp;
wint_t wc;
if (nelem > 0)
/* This state has (a) multibyte operator(s).
We check whether each of them can match or not. */
{
/* Note: caller must free the return value of this function. */
mbclen = mbs_to_wchar (&wc, (char const *) *pp, end - *pp, d);
match_lens = check_matching_with_multibyte_ops (d, s, (char const *) *pp,
wc, mbclen);
for (i = 0; i < nelem; i++)
/* Search the operator which match the longest string,
in this state. */
{
if (match_lens[i] > maxlen)
maxlen = match_lens[i];
}
}
if (nelem == 0 || maxlen == 0)
/* This state has no multibyte operator which can match.
We need to check only one single byte character. */
{
status_transit_state rs;
rs = transit_state_singlebyte (d, s, *pp, &s1);
/* We must update the pointer if state transition succeeded. */
if (rs == TRANSIT_STATE_DONE)
++*pp;
return s1;
}
/* This state has some operators which can match a multibyte character. */
d->mb_follows.nelem = 0;
/* 'maxlen' may be longer than the length of a character, because it may
not be a character but a (multi character) collating element.
We enumerate all of the positions which 's' can reach by consuming
'maxlen' bytes. */
transit_state_consume_1char (d, s, pp, wc, mbclen, match_lens);
s1 = state_index (d, &d->mb_follows, wchar_context (wc));
realloc_trans_if_necessary (d, s1);
while (*pp - p1 < maxlen)
{
mbclen = mbs_to_wchar (&wc, (char const *) *pp, end - *pp, d);
transit_state_consume_1char (d, s1, pp, wc, mbclen, NULL);
for (i = 0; i < nelem; i++)
{
if (match_lens[i] == *pp - p1)
for (j = 0;
j < d->follows[d->states[s1].mbps.elems[i].index].nelem; j++)
insert (d->follows[d->states[s1].mbps.elems[i].index].elems[j],
&d->mb_follows);
}
s1 = state_index (d, &d->mb_follows, wchar_context (wc));
realloc_trans_if_necessary (d, s1);
}
return s1;
}
/* Search through a buffer looking for a match to the given struct dfa.
Find the first occurrence of a string matching the regexp in the
buffer, and the shortest possible version thereof. Return a pointer to
the first character after the match, or NULL if none is found. BEGIN
points to the beginning of the buffer, and END points to the first byte
after its end. Note however that we store a sentinel byte (usually
newline) in *END, so the actual buffer must be one byte longer.
When ALLOW_NL is nonzero, newlines may appear in the matching string.
If COUNT is non-NULL, increment *COUNT once for each newline processed.
Finally, if BACKREF is non-NULL set *BACKREF to indicate whether we
encountered a back-reference (1) or not (0). The caller may use this
to decide whether to fall back on a backtracking matcher. */
char *
dfaexec (struct dfa *d, char const *begin, char *end,
int allow_nl, size_t *count, int *backref)
{
state_num s, s1; /* Current state. */
unsigned char const *p, *mbp; /* Current input character. */
state_num **trans, *t; /* Copy of d->trans so it can be optimized
into a register. */
unsigned char eol = eolbyte; /* Likewise for eolbyte. */
unsigned char saved_end;
size_t nlcount = 0;
if (!d->tralloc)
{
realloc_trans_if_necessary (d, 1);
build_state (0, d);
}
s = s1 = 0;
p = mbp = (unsigned char const *) begin;
trans = d->trans;
saved_end = *(unsigned char *) end;
*end = eol;
if (d->multibyte)
{
memset (&d->mbs, 0, sizeof d->mbs);
if (! d->mb_match_lens)
{
d->mb_match_lens = xnmalloc (d->nleaves, sizeof *d->mb_match_lens);
alloc_position_set (&d->mb_follows, d->nleaves);
}
}
for (;;)
{
if (d->multibyte)
{
while ((t = trans[s]) != NULL)
{
s1 = s;
if (s == 0)
{
/* The initial state may encounter a byte which is not
a single byte character nor the first byte of a
multibyte character. But it is incorrect for the
initial state to accept such a byte. For example,
in Shift JIS the regular expression "\\" accepts
the codepoint 0x5c, but should not accept the second
byte of the codepoint 0x815c. Then the initial
state must skip the bytes that are not a single
byte character nor the first byte of a multibyte
character. */
wint_t wc;
while (mbp < p)
mbp += mbs_to_wchar (&wc, (char const *) mbp,
end - (char const *) mbp, d);
p = mbp;
if ((char *) p > end)
{
p = NULL;
goto done;
}
}
if (d->states[s].mbps.nelem == 0)
{
s = t[*p++];
continue;
}
/* Falling back to the glibc matcher in this case gives
better performance (up to 25% better on [a-z], for
example) and enables support for collating symbols and
equivalence classes. */
if (d->states[s].has_mbcset && backref)
{
*backref = 1;
goto done;
}
/* Can match with a multibyte character (and multi character
collating element). Transition table might be updated. */
s = transit_state (d, s, &p, (unsigned char *) end);
mbp = p;
trans = d->trans;
}
}
else
{
while ((t = trans[s]) != NULL)
{
s1 = t[*p++];
if ((t = trans[s1]) == NULL)
{
state_num tmp = s;
s = s1;
s1 = tmp; /* swap */
break;
}
s = t[*p++];
}
}
if ((char *) p > end)
{
p = NULL;
goto done;
}
if (s >= 0 && d->fails[s])
{
if (d->success[s] & sbit[*p])
{
if (backref)
*backref = d->states[s].has_backref;
goto done;
}
s1 = s;
if (d->multibyte)
{
/* Can match with a multibyte character (and multicharacter
collating element). Transition table might be updated. */
s = transit_state (d, s, &p, (unsigned char *) end);
mbp = p;
trans = d->trans;
}
else
s = d->fails[s][*p++];
continue;
}
/* If the previous character was a newline, count it, and skip
checking of multibyte character boundary until here. */
if (p[-1] == eol)
{
nlcount++;
mbp = p;
}
if (s >= 0)
{
if (!d->trans[s])
build_state (s, d);
trans = d->trans;
continue;
}
if (p[-1] == eol && allow_nl)
{
s = d->newlines[s1];
continue;
}
s = 0;
}
done:
if (count)
*count += nlcount;
*end = saved_end;
return (char *) p;
}
struct dfa *
dfasuperset (struct dfa const *d)
{
return d->superset;
}
bool
dfaisfast (struct dfa const *d)
{
return d->fast;
}
static void
free_mbdata (struct dfa *d)
{
size_t i;
free (d->multibyte_prop);
for (i = 0; i < d->nmbcsets; ++i)
{
size_t j;
struct mb_char_classes *p = &(d->mbcsets[i]);
free (p->chars);
free (p->ch_classes);
free (p->ranges);
for (j = 0; j < p->nequivs; ++j)
free (p->equivs[j]);
free (p->equivs);
for (j = 0; j < p->ncoll_elems; ++j)
free (p->coll_elems[j]);
free (p->coll_elems);
}
free (d->mbcsets);
free (d->mb_follows.elems);
free (d->mb_match_lens);
d->mb_match_lens = NULL;
}
/* Initialize the components of a dfa that the other routines don't
initialize for themselves. */
void
dfainit (struct dfa *d)
{
memset (d, 0, sizeof *d);
d->multibyte = MB_CUR_MAX > 1;
d->fast = !d->multibyte;
}
static void
dfaoptimize (struct dfa *d)
{
size_t i;
bool have_backref = false;
if (!using_utf8 ())
return;
for (i = 0; i < d->tindex; ++i)
{
switch (d->tokens[i])
{
case ANYCHAR:
/* Lowered. */
abort ();
case BACKREF:
have_backref = true;
break;
case MBCSET:
/* Requires multi-byte algorithm. */
return;
default:
break;
}
}
if (!have_backref && d->superset)
{
/* The superset DFA is not likely to be much faster, so remove it. */
dfafree (d->superset);
free (d->superset);
d->superset = NULL;
}
free_mbdata (d);
d->multibyte = false;
}
static void
dfassbuild (struct dfa *d)
{
size_t i, j;
charclass ccl;
bool have_achar = false;
bool have_nchar = false;
struct dfa *sup = dfaalloc ();
*sup = *d;
sup->multibyte = false;
sup->multibyte_prop = NULL;
sup->mbcsets = NULL;
sup->superset = NULL;
sup->states = NULL;
sup->sindex = 0;
sup->follows = NULL;
sup->tralloc = 0;
sup->trans = NULL;
sup->fails = NULL;
sup->success = NULL;
sup->newlines = NULL;
sup->musts = NULL;
sup->charclasses = xnmalloc (sup->calloc, sizeof *sup->charclasses);
memcpy (sup->charclasses, d->charclasses,
d->cindex * sizeof *sup->charclasses);
sup->tokens = xnmalloc (d->tindex, 2 * sizeof *sup->tokens);
sup->talloc = d->tindex * 2;
for (i = j = 0; i < d->tindex; i++)
{
switch (d->tokens[i])
{
case ANYCHAR:
case MBCSET:
case BACKREF:
zeroset (ccl);
notset (ccl);
sup->tokens[j++] = CSET + dfa_charclass_index (sup, ccl);
sup->tokens[j++] = STAR;
if (d->tokens[i + 1] == QMARK || d->tokens[i + 1] == STAR
|| d->tokens[i + 1] == PLUS)
i++;
have_achar = true;
break;
case BEGWORD:
case ENDWORD:
case LIMWORD:
case NOTLIMWORD:
if (d->multibyte)
{
/* These constraints aren't supported in a multibyte locale.
Ignore them in the superset DFA, and treat them as
backreferences in the main DFA. */
sup->tokens[j++] = EMPTY;
d->tokens[i] = BACKREF;
break;
}
default:
sup->tokens[j++] = d->tokens[i];
if ((0 <= d->tokens[i] && d->tokens[i] < NOTCHAR)
|| d->tokens[i] >= CSET)
have_nchar = true;
break;
}
}
sup->tindex = j;
if (have_nchar && (have_achar || d->multibyte))
d->superset = sup;
else
{
dfafree (sup);
free (sup);
}
}
/* Run the fsa* etc modules (parsing/lexing/tokens/charclass) first, then
run the original dfa code, with a hook added that the parser/lexer
interface in the original code calls the new code in parallel, so that
the veracity of the lexical token streams can be inspected (they should
be identical). After these runs, the parser token list from each side
is extracted and compared, and should also be identical. */
/* Parse and analyze a single string of the given length. */
void
dfacomp (char const *s, size_t len, struct dfa *d, int searchflag)
{
size_t i;
HOOK_set_up_fsa_stuff_if_not_done_already ();
/* HOOK: Write the search pattern to the logfile.. */
if (! pll_log)
pll_log = fopen(HOOK_LOG_FILENAME, "a");
fprintf (pll_log, "Pattern:");
for (i = 0; i < len; i++)
fprintf (pll_log, " %c", isprint (s[i]) ? s[i] : ' ');
fprintf (pll_log, "\n ");
for (i = 0; i < len; i++)
fprintf (pll_log, " %02x", ((unsigned) s[i]) & 0xff);
fprintf (pll_log, "\n");
/* HOOK: Run the new code first so any dependencies on old code are more
likely to bring up errors. */
fsalex_pattern (lexer, s, len);
fsaparse_parse (parser);
/* Re-prime fsalex () to analyse the pattern a second time, so that
lexical tokens from both systems can be compared side-by-side.
This lexical-token-check watches the activity at the parser/lexer
boundary; the code below compares the outputs of the original parser
and fsaparse, and they should also be functionally identical (some
differences can occur in the indices added to CSET tokens, but
the referenced set should be identical). */
fsalex_pattern (lexer, s, len);
dfainit (d);
dfambcache (d);
dfaparse (s, len, d);
dfamust (d);
/* YET ANOTHER HACK, 16 April 2014 (was it related to the lunar eclipse
last night?? !!?? )
Compare, side-by-side, the list of tokens generated by dfa.c and by
fsaparse, and write these to the debug log file. As elsewhere, these
should be identical, as the modularised code starts as a functional
clone of the original code. (Later, if/when tokens are reworked to
maintain abstractions at a higher level, the token lists will
differ.) */
{
size_t nr_tokens;
fsatoken_token_t *token_list;
size_t j;
size_t max_index;
fsaparse_get_token_list (parser, &nr_tokens, &token_list);
fprintf (pll_log, "\ntokens: original fsaparse\n");
max_index = d->tindex > nr_tokens ? d->tindex : nr_tokens;
for (j = 0; j < max_index; ++j)
{
static char buf[256];
if (j < d->tindex)
{
sprintf (buf, "%02lx", d->tokens[j]);
fprintf (pll_log, "%17s ", buf);
}
else
fprintf (pll_log, "%17s", "");
if (j < nr_tokens)
{
sprintf (buf, "%02lx", token_list[j]);
fprintf (pll_log, "%9s", buf);
}
fprintf (pll_log, "\n");
}
/* And finally, see how extracting musts from dfa.c compares to extracting
musts via the fsa/charclass family of functions; again, these should
be identical. */
musts = (fsamusts_list_element_t *) d->musts;
show_musts ("original dfa.c", musts);
/* ANOTHER UGLY HACK: Rely on dfa.c's case_fold and unibyte locale when
instructing dfamust how to operate; an "Exchange" function might be
more appropriate in the short-to-mid-term, but in the longer term,
the token vocabluary should get more expressive, so that information
can be conveyed directly. */
musts = fsamusts_must (NULL, nr_tokens, token_list,
/* dfa.c copy: */ case_fold,
/* current (dfa.c) locale: */ MB_CUR_MAX == 1);
show_musts ("fsa* et al functions", musts);
}
dfassbuild (d);
dfaoptimize (d);
dfaanalyze (d, searchflag);
if (d->superset)
{
d->fast = true;
dfaanalyze (d->superset, searchflag);
}
}
/* Free the storage held by the components of a dfa. */
void
dfafree (struct dfa *d)
{
size_t i;
struct dfamust *dm, *ndm;
free (d->charclasses);
free (d->tokens);
if (d->multibyte)
free_mbdata (d);
for (i = 0; i < d->sindex; ++i)
{
free (d->states[i].elems.elems);
free (d->states[i].mbps.elems);
}
free (d->states);
if (d->follows)
{
for (i = 0; i < d->tindex; ++i)
free (d->follows[i].elems);
free (d->follows);
}
if (d->trans)
{
for (i = 0; i < d->tralloc; ++i)
{
free (d->trans[i]);
free (d->fails[i]);
}
free (d->trans - 1);
free (d->fails);
free (d->newlines);
free (d->success);
}
for (dm = d->musts; dm; dm = ndm)
{
ndm = dm->next;
free (dm->must);
free (dm);
}
if (d->superset)
dfafree (d->superset);
/* HOOK: Tear down all the infrastructure added for parallel operation. */
HOOK_tear_down_additional_infrastructure ();
}
/* Having found the postfix representation of the regular expression,
try to find a long sequence of characters that must appear in any line
containing the r.e.
Finding a "longest" sequence is beyond the scope here;
we take an easy way out and hope for the best.
(Take "(ab|a)b"--please.)
We do a bottom-up calculation of sequences of characters that must appear
in matches of r.e.'s represented by trees rooted at the nodes of the postfix
representation:
sequences that must appear at the left of the match ("left")
sequences that must appear at the right of the match ("right")
lists of sequences that must appear somewhere in the match ("in")
sequences that must constitute the match ("is")
When we get to the root of the tree, we use one of the longest of its
calculated "in" sequences as our answer. The sequence we find is returned in
d->must (where "d" is the single argument passed to "dfamust");
the length of the sequence is returned in d->mustn.
The sequences calculated for the various types of node (in pseudo ANSI c)
are shown below. "p" is the operand of unary operators (and the left-hand
operand of binary operators); "q" is the right-hand operand of binary
operators.
"ZERO" means "a zero-length sequence" below.
Type left right is in
---- ---- ----- -- --
char c # c # c # c # c
ANYCHAR ZERO ZERO ZERO ZERO
MBCSET ZERO ZERO ZERO ZERO
CSET ZERO ZERO ZERO ZERO
STAR ZERO ZERO ZERO ZERO
QMARK ZERO ZERO ZERO ZERO
PLUS p->left p->right ZERO p->in
CAT (p->is==ZERO)? (q->is==ZERO)? (p->is!=ZERO && p->in plus
p->left : q->right : q->is!=ZERO) ? q->in plus
p->is##q->left p->right##q->is p->is##q->is : p->right##q->left
ZERO
OR longest common longest common (do p->is and substrings common to
leading trailing q->is have same p->in and q->in
(sub)sequence (sub)sequence length and
of p->left of p->right content) ?
and q->left and q->right p->is : NULL
If there's anything else we recognize in the tree, all four sequences get set
to zero-length sequences. If there's something we don't recognize in the
tree, we just return a zero-length sequence.
Break ties in favor of infrequent letters (choosing 'zzz' in preference to
'aaa')?
And ... is it here or someplace that we might ponder "optimizations" such as
egrep 'psi|epsilon' -> egrep 'psi'
egrep 'pepsi|epsilon' -> egrep 'epsi'
(Yes, we now find "epsi" as a "string
that must occur", but we might also
simplify the *entire* r.e. being sought)
grep '[c]' -> grep 'c'
grep '(ab|a)b' -> grep 'ab'
grep 'ab*' -> grep 'a'
grep 'a*b' -> grep 'b'
There are several issues:
Is optimization easy (enough)?
Does optimization actually accomplish anything,
or is the automaton you get from "psi|epsilon" (for example)
the same as the one you get from "psi" (for example)?
Are optimizable r.e.'s likely to be used in real-life situations
(something like 'ab*' is probably unlikely; something like is
'psi|epsilon' is likelier)? */
static char *
icatalloc (char *old, char const *new)
{
char *result;
size_t oldsize;
size_t newsize = strlen (new);
if (newsize == 0)
return old;
oldsize = strlen (old);
result = xrealloc (old, oldsize + newsize + 1);
memcpy (result + oldsize, new, newsize + 1);
return result;
}
static char *_GL_ATTRIBUTE_PURE
istrstr (char const *lookin, char const *lookfor)
{
char const *cp;
size_t len;
len = strlen (lookfor);
for (cp = lookin; *cp != '\0'; ++cp)
if (strncmp (cp, lookfor, len) == 0)
return (char *) cp;
return NULL;
}
static void
freelist (char **cpp)
{
while (*cpp)
free (*cpp++);
}
static char **
enlist (char **cpp, char *new, size_t len)
{
size_t i, j;
new = memcpy (xmalloc (len + 1), new, len);
new[len] = '\0';
/* Is there already something in the list that's new (or longer)? */
for (i = 0; cpp[i] != NULL; ++i)
if (istrstr (cpp[i], new) != NULL)
{
free (new);
return cpp;
}
/* Eliminate any obsoleted strings. */
j = 0;
while (cpp[j] != NULL)
if (istrstr (new, cpp[j]) == NULL)
++j;
else
{
free (cpp[j]);
if (--i == j)
break;
cpp[j] = cpp[i];
cpp[i] = NULL;
}
/* Add the new string. */
cpp = xnrealloc (cpp, i + 2, sizeof *cpp);
cpp[i] = new;
cpp[i + 1] = NULL;
return cpp;
}
/* Given pointers to two strings, return a pointer to an allocated
list of their distinct common substrings. */
static char **
comsubs (char *left, char const *right)
{
char **cpp = xzalloc (sizeof *cpp);
char *lcp;
for (lcp = left; *lcp != '\0'; ++lcp)
{
size_t len = 0;
char *rcp = strchr (right, *lcp);
while (rcp != NULL)
{
size_t i;
for (i = 1; lcp[i] != '\0' && lcp[i] == rcp[i]; ++i)
continue;
if (i > len)
len = i;
rcp = strchr (rcp + 1, *lcp);
}
if (len != 0)
cpp = enlist (cpp, lcp, len);
}
return cpp;
}
static char **
addlists (char **old, char **new)
{
for (; *new; new++)
old = enlist (old, *new, strlen (*new));
return old;
}
/* Given two lists of substrings, return a new list giving substrings
common to both. */
static char **
inboth (char **left, char **right)
{
char **both = xzalloc (sizeof *both);
size_t lnum, rnum;
for (lnum = 0; left[lnum] != NULL; ++lnum)
{
for (rnum = 0; right[rnum] != NULL; ++rnum)
{
char **temp = comsubs (left[lnum], right[rnum]);
both = addlists (both, temp);
freelist (temp);
free (temp);
}
}
return both;
}
typedef struct must must;
struct must
{
char **in;
char *left;
char *right;
char *is;
bool begline;
bool endline;
must *prev;
};
static must *
allocmust (must *mp)
{
must *new_mp = xmalloc (sizeof *new_mp);
new_mp->in = xzalloc (sizeof *new_mp->in);
new_mp->left = xzalloc (2);
new_mp->right = xzalloc (2);
new_mp->is = xzalloc (2);
new_mp->begline = false;
new_mp->endline = false;
new_mp->prev = mp;
return new_mp;
}
static void
resetmust (must *mp)
{
freelist (mp->in);
mp->in[0] = NULL;
mp->left[0] = mp->right[0] = mp->is[0] = '\0';
mp->begline = false;
mp->endline = false;
}
static void
freemust (must *mp)
{
freelist (mp->in);
free (mp->in);
free (mp->left);
free (mp->right);
free (mp->is);
free (mp);
}
static void
dfamust (struct dfa *d)
{
must *mp = NULL;
char const *result = "";
size_t ri;
size_t i;
bool exact = false;
bool begline = false;
bool endline = false;
struct dfamust *dm;
for (ri = 0; ri < d->tindex; ++ri)
{
token t = d->tokens[ri];
switch (t)
{
case BEGLINE:
mp = allocmust (mp);
mp->begline = true;
break;
case ENDLINE:
mp = allocmust (mp);
mp->endline = true;
break;
case LPAREN:
case RPAREN:
assert (!"neither LPAREN nor RPAREN may appear here");
case EMPTY:
case BEGWORD:
case ENDWORD:
case LIMWORD:
case NOTLIMWORD:
case BACKREF:
case ANYCHAR:
case MBCSET:
mp = allocmust (mp);
break;
case STAR:
case QMARK:
resetmust (mp);
break;
case OR:
{
char **new;
must *rmp = mp;
must *lmp = mp = mp->prev;
size_t j, ln, rn, n;
/* Guaranteed to be. Unlikely, but ... */
if (STREQ (lmp->is, rmp->is))
{
lmp->begline &= rmp->begline;
lmp->endline &= rmp->endline;
}
else
{
lmp->is[0] = '\0';
lmp->begline = false;
lmp->endline = false;
}
/* Left side--easy */
i = 0;
while (lmp->left[i] != '\0' && lmp->left[i] == rmp->left[i])
++i;
lmp->left[i] = '\0';
/* Right side */
ln = strlen (lmp->right);
rn = strlen (rmp->right);
n = ln;
if (n > rn)
n = rn;
for (i = 0; i < n; ++i)
if (lmp->right[ln - i - 1] != rmp->right[rn - i - 1])
break;
for (j = 0; j < i; ++j)
lmp->right[j] = lmp->right[(ln - i) + j];
lmp->right[j] = '\0';
new = inboth (lmp->in, rmp->in);
freelist (lmp->in);
free (lmp->in);
lmp->in = new;
freemust (rmp);
}
break;
case PLUS:
mp->is[0] = '\0';
break;
case END:
assert (!mp->prev);
for (i = 0; mp->in[i] != NULL; ++i)
if (strlen (mp->in[i]) > strlen (result))
result = mp->in[i];
if (STREQ (result, mp->is))
{
exact = true;
begline = mp->begline;
endline = mp->endline;
}
goto done;
case CAT:
{
must *rmp = mp;
must *lmp = mp = mp->prev;
/* In. Everything in left, plus everything in
right, plus concatenation of
left's right and right's left. */
lmp->in = addlists (lmp->in, rmp->in);
if (lmp->right[0] != '\0' && rmp->left[0] != '\0')
{
size_t lrlen = strlen (lmp->right);
size_t rllen = strlen (rmp->left);
char *tp = xmalloc (lrlen + rllen);
memcpy (tp, lmp->right, lrlen);
memcpy (tp + lrlen, rmp->left, rllen);
lmp->in = enlist (lmp->in, tp, lrlen + rllen);
free (tp);
}
/* Left-hand */
if (lmp->is[0] != '\0')
lmp->left = icatalloc (lmp->left, rmp->left);
/* Right-hand */
if (rmp->is[0] == '\0')
lmp->right[0] = '\0';
lmp->right = icatalloc (lmp->right, rmp->right);
/* Guaranteed to be */
if ((lmp->is[0] != '\0' || lmp->begline)
&& (rmp->is[0] != '\0' || rmp->endline))
{
lmp->is = icatalloc (lmp->is, rmp->is);
lmp->endline = rmp->endline;
}
else
{
lmp->is[0] = '\0';
lmp->begline = false;
lmp->endline = false;
}
freemust (rmp);
}
break;
case '\0':
/* Not on *my* shift. */
goto done;
default:
mp = allocmust (mp);
if (CSET <= t)
{
/* If T is a singleton, or if case-folding in a unibyte
locale and T's members all case-fold to the same char,
convert T to one of its members. Otherwise, do
nothing further with T. */
charclass *ccl = &d->charclasses[t - CSET];
int j;
for (j = 0; j < NOTCHAR; j++)
if (tstbit (j, *ccl))
break;
if (! (j < NOTCHAR))
break;
t = j;
while (++j < NOTCHAR)
if (tstbit (j, *ccl)
&& ! (case_fold && !d->multibyte
&& toupper (j) == toupper (t)))
break;
if (j < NOTCHAR)
break;
}
mp->is[0] = mp->left[0] = mp->right[0]
= case_fold && !d->multibyte ? toupper (t) : t;
mp->is[1] = mp->left[1] = mp->right[1] = '\0';
mp->in = enlist (mp->in, mp->is, 1);
break;
}
}
done:
if (*result)
{
dm = xmalloc (sizeof *dm);
dm->exact = exact;
dm->begline = begline;
dm->endline = endline;
dm->must = xstrdup (result);
dm->next = d->musts;
d->musts = dm;
}
while (mp)
{
must *prev = mp->prev;
freemust (mp);
mp = prev;
}
}
struct dfa *
dfaalloc (void)
{
return xmalloc (sizeof (struct dfa));
}
struct dfamust *_GL_ATTRIBUTE_PURE
dfamusts (struct dfa const *d)
{
return d->musts;
}
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="fsalex.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsalex.c"
/* fsalex - Repackage pattern text as compact, expressive tokens
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* Always import environment-specific configuration items first. */
#include <config.h> /* define _GNU_SOURCE for regex extensions. */
#include <assert.h>
#include "charclass.h"
#include <ctype.h>
#include "fsalex.h"
#include "fsatoken.h"
#include <limits.h>
#include <locale.h>
#include "mbcsets.h"
#include "proto-lexparse.h"
#include <regex.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <wctype.h>
#include "xalloc.h"
#include "gettext.h"
#define _(str) gettext (str)
/* ISASCIIDIGIT differs from isdigit, as follows:
- Its arg may be any int or unsigned int; it need not be an unsigned char.
- It's guaranteed to evaluate its argument exactly once.
- It's typically faster.
Posix 1003.2-1992 section 2.5.2.1 page 50 lines 1556-1558 says that
only '0' through '9' are digits. Prefer ISASCIIDIGIT to isdigit unless
it's important to use the locale's definition of "digit" even when the
host does not conform to Posix. */
#define ISASCIIDIGIT(c) ((unsigned) (c) - '0' <= 9)
#define STREQ(a, b) (strcmp (a, b) == 0)
#ifndef MIN
# define MIN(a,b) ((a) < (b) ? (a) : (b))
#endif
/* The following list maps the names of the Posix named character
classes to predicate functions that determine whether a given
character is in the class. The leading [ has already been eaten by
the lexical analyzer. Additional objects are provided to assist
the client: wchar_desc for multibyte matching, and class for octet
matching. Lazy evaluation and caching are used to minimise
processing costs, so these additional items are only valid after a
class has been located using find_pred (). */
typedef int predicate_t (wint_t, wctype_t);
typedef struct predicate_entry_struct
{
const char *name;
wctype_t wchar_desc;
charclass_t *class;
bool may_be_multibyte;
} predicate_entry_t;
/* This list is a template, copied into each lexer's state, and
interrogated and updated from there. The membership of a class can
vary due to locale and other settings, so each lexer must maintain
its own list. Duplicate class sharing across different lexer
instances is facilitated by checks in charclass_completed. */
/* Locale portability note: We may need to record the entire explicit
locale when syntax () is called, and then use isalpha_l () etc.
here, as otherwise the wchar_desc may be interpreted at runtime in
the context of the current locale. If a complete copy of the locale
needs to be stored locally, see uselocale(3) and duplocale(3) (and
free the copy with freelocale(3)). */
static predicate_entry_t template_predicate_list[] = {
{"alpha", 0, NULL, true},
{"alnum", 0, NULL, true},
{"blank", 0, NULL, true},
{"cntrl", 0, NULL, true},
{"digit", 0, NULL, false},
{"graph", 0, NULL, true},
{"lower", 0, NULL, true},
{"print", 0, NULL, true},
{"punct", 0, NULL, true},
{"space", 0, NULL, true},
{"upper", 0, NULL, true},
{"xdigit", 0, NULL, true},
{NULL, 0, NULL}
};
#define PREDICATE_TEMPLATE_ITEMS \
(sizeof template_predicate_list / sizeof *template_predicate_list)
/* Flesh out the opaque instance context type given in the header. */
struct fsalex_ctxt_struct
{
/* Singly-linked list of all lexer instances, so destroy_module can
release all resources by traversing the list. */
fsalex_ctxt_t *next_instance;
/* Using the lexer without setting the syntax is a fatal error, so
use a flag so we can report such errors in a direct fashion. */
bool syntax_initialised;
/* Syntax flags/characters directing how to interpret the pattern. */
reg_syntax_t syntax_bits;
bool case_fold;
unsigned char eolbyte;
/* Exception handling is done by explicit callbacks. */
fsalex_warn_callback_fn *warn_client;
fsalex_error_callback_fn *abandon_with_error;
/* Pattern pointer/length, updated as pattern is consumed. */
char const *lexptr;
size_t lexleft;
/* Break out some regex syntax bits into boolean vars. Do this for
the ones that are heavily used, and/or where the nature of the
bitmask flag test tends to clutter the lexer code. */
bool re_gnu_ops; /* GNU regex operators are allowed. */
/* Carry dotclass here, as it's easier for clients (UTF-8) to perform
class operations with this class, rather than to know intimate
details of the regex syntax configuration bits and items such as
eolbyte. */
charclass_t *dotclass; /* ".": All chars except eolbyte and/or
NUL, depending on syntax flags. */
charclass_index_t dotclass_index;
/* Work variables to help organise lexer operation. */
fsatoken_token_t lasttok;
bool laststart;
size_t parens;
/* Character class predicate mapping/caching table. */
predicate_entry_t predicates[PREDICATE_TEMPLATE_ITEMS];
/* Minrep and maxrep are actually associated with the REPMN token,
and need to be accessible outside this module (by the parser),
perhaps by an explicit interface call. In the far, far future,
a completely-reworked token list may see these values properly
become integrated into the token stream (perhaps by a pair of
"Parameter" tokens? Perhaps by a MINREP token with 1 parameter,
followed by a MAXREP token with a corresponding parameter?) */
int minrep, maxrep;
/* Booleans to simplify unibyte/multibyte code selection paths. In
addition, other flags are placed here to summarise properties of
the locale in a concise fashion that can useful when
implementing optimisations. There may be some overlap and/or
redundancy here; flag names are chosen to let the user be direct
and concise, but we only provide names for the more popular cases,
not all. */
char *locale_name;
bool unibyte_locale;
bool multibyte_locale;
bool ascii_7bit_encoding;
bool using_simple_locale;
/* REVIEWME: Wide-character support variables. */
int cur_mb_len; /* Length (in bytes) of the last character
fetched; this is needed when backing up
during lexing. In a non-multibyte situation
(locale?), this variable remains at 1;
otherwise, it is updated as required by
FETCH_WC. */
/* These variables are used only if in a multibyte locale. */
wchar_t wctok; /* Storage for a single multibyte character,
used both during lexing, and as the implied
parameter of a WCHAR token returned by the
lexer. */
mbstate_t mbrtowc_state; /* State management area for mbrtowc to
use. */
/* A table indexed by byte values that contains the corresponding wide
character (if any) for that byte. WEOF means the byte is not a
valid single-byte character. */
wint_t mbrtowc_cache[FSATOKEN_NOTCHAR];
mbcsets_set_t **mbcsets;
size_t nmbcsets;
size_t mbcsets_alloc;
};
/* Linked list of all instances created by this module. */
static fsalex_ctxt_t *fsalex_instance_list_head = NULL;
/* Ensure that the array addressed by PTR holds at least NITEMS +
(PTR || !NITEMS) items. Either return PTR, or reallocate the array
and return its new address. Although PTR may be null, the returned
value is never null.
The array holds *NALLOC items; *NALLOC is updated on reallocation.
ITEMSIZE is the size of one item. Avoid O(N**2) behavior on arrays
growing linearly. */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
if (nitems < *nalloc)
return ptr;
*nalloc = nitems;
return x2nrealloc (ptr, nalloc, itemsize);
}
/* Set a bit in the charclass for the given wchar_t. Do nothing if WC
is represented by a multi-byte sequence. Even in unibyte locales,
this may happen when folding case in weird Turkish locales where
dotless i/dotted I are not included in the chosen character set.
Return whether a bit was set in the charclass. */
/* Set a bit for B and its case variants in the charclass C.
We must be in an unibyte locale. */
static void
setbit_case_fold_c (int b, charclass_t *c)
{
int ub = toupper (b);
int i;
for (i = 0; i < FSATOKEN_NOTCHAR; i++)
if (toupper (i) == ub)
charclass_setbit (i, c);
}
/* Convert a possibly-signed character to an unsigned character. This is
a bit safer than casting to unsigned char, since it catches some type
errors that the cast doesn't. */
static unsigned char
to_uchar (char ch)
{
return ch;
}
static void
initialise_uchar_to_wc_cache (fsalex_ctxt_t *lexer)
{
int i;
for (i = CHAR_MIN; i <= CHAR_MAX; ++i)
{
char c = i;
unsigned char uc = i;
mbstate_t s = { 0 };
wchar_t wc;
lexer->mbrtowc_cache[uc] = mbrtowc (&wc, &c, 1, &s) <= 1 ? wc : WEOF;
}
}
/* This function is intimately connected with multibyte (wide-char)
handling in the macro FETCH_WC below, in the case where
FETCH_SINGLE_CHAR has run but the result has been found to be
inconclusive. It works by unwinding the FETCH_SINGLE_CHAR
side-effects (lexptr/lexleft), then calling mbrtowc on the pattern
space, and communicates mbrtowc's understanding of the octet
stream back to the caller:
- If a valid multibyte octet sequence is next, then the wide
character associated with this sequence is written back to
*p_wchar, and the number of octets consumed is returned; or
- If the sequence is invalid for any reason, the mbrtowc working
state is reset (zeroed), *p_wchar is not modified, and 1 is
returned.
Lexer state variables, including cur_mb_len, mbs, lexleft and
lexptr, are updated as appropriate by this function (mainly if
mbrtowc succeeds). The wide char NUL is unusual as it is a
1-octet sequence, the length returned is 0, we report it as length
1, but write the converted wide character in temp_wchar to the
caller. */
/* Additional notes: This code, in partnership with the macro
FETCH_WC, is closely related to mbs_to_wchar in dfa.c. There is
documentation there (e.g. pattern must end in a sentinel, shift
encodings not supported, plus other comments/guarantees) that is
important, but I'm deferring writing up anything at present until
I see how this code is received. */
static size_t
fetch_offset_wide_char (fsalex_ctxt_t *lexer, wchar_t *p_wchar)
{
size_t nbytes;
wchar_t temp_wchar;
nbytes = mbrtowc (&temp_wchar,
lexer->lexptr - 1, lexer->lexleft + 1,
&lexer->mbrtowc_state);
switch (nbytes)
{
case (size_t) -2:
case (size_t) -1:
/* Conversion failed: Incomplete (-2) or invalid (-1)
sequence. */
memset (&lexer->mbrtowc_state, 0, sizeof (lexer->mbrtowc_state));
return 1;
case (size_t) 0:
/* This is the wide NUL character, actually 1 byte long. */
nbytes = 1;
break;
default:
/* Converted character is in temp_wchar, and nbytes is a byte
count. */
break;
}
/* We converted 1 or more bytes, tell result to caller. */
*p_wchar = temp_wchar;
/* Update the number of bytes consumed (offset by 1 since
FETCH_SINGLE_CHAR grabbed one earlier). */
lexer->lexptr += nbytes - 1;
lexer->lexleft -= nbytes - 1;
return nbytes;
}
/* Single-character input fetch, with EOF/error handling. Note that
characters become unsigned here. If no characters are available,
the macro either returns END or reports an error, depending on
eoferr. Otherwise, one character is consumed (lexptr/lexleft),
the char is converted into an unsigned char, and is written into
the parameter c. */
#define FETCH_SINGLE_CHAR(lexer, c, eoferr) \
do { \
if (! (lexer)->lexleft) \
{ \
if ((eoferr) != 0) \
(lexer)->abandon_with_error (eoferr); \
else \
return FSATOKEN_TK_END; \
} \
(c) = to_uchar (*(lexer)->lexptr++); \
(lexer)->lexleft--; \
} while (0)
/* Do the fetch in stages: Single char, octet+multibyte cache
check, and possible wide char fetch if the cache result indicates
that the input sequence is longer than a single octet. The first
fetch handles end-of-input cases (if this happens, control never
reaches the rest of the macro); otherwise, it returns temp_uchar
which is used in the cache lookup, and may be the single-octet
result. A cache result of WEOF means that the octet is not a
complete sequence by itself, so a second fetch tweaks lexptr/lexleft
to undo the single-char-fetch side-effects, and, depending on
mbrtowc valid/invalid result, propagates either the multichar fetch
or the single-char fetch back to the caller. */
# define FETCH_WC(lexer, c, wc, eoferr) \
do { \
wchar_t temp_wc; \
unsigned char temp_uchar; \
(lexer)->cur_mb_len = 1; \
FETCH_SINGLE_CHAR ((lexer), temp_uchar, (eoferr)); \
temp_wc = (lexer)->mbrtowc_cache[temp_uchar]; \
if (temp_wc != WEOF) \
{ \
(c) = temp_uchar; \
(wc) = temp_wc; \
} \
else \
{ \
size_t nbytes; \
temp_wc = temp_uchar; \
nbytes = fetch_offset_wide_char ((lexer), &temp_wc); \
(wc) = temp_wc; \
(c) = nbytes == 1 ? temp_uchar : EOF; \
(lexer)->cur_mb_len = nbytes; \
} \
} while (0)
/* Given a predicate name, find it in a list, and report the list
entry to the caller. If the name is not recognised, the function
returns NULL. The list entry includes a charclass set and (if
relevant) a wide-char descriptor for testing for the predicate.
Lazy evaluation and caching are used to keep processing costs
down. */
static predicate_entry_t *
find_pred (fsalex_ctxt_t *lexer, const char *str)
{
predicate_entry_t *p_entry;
charclass_t *work_class;
for (p_entry = lexer->predicates; p_entry->name; p_entry++)
{
if (STREQ (str, p_entry->name))
break;
}
/* If there was no matching predicate name found, return NULL. */
if (! p_entry->name)
return NULL;
/* Is the charclass pointer NULL for this entry? */
if (p_entry->class == NULL)
{
/* Yes, allocate, set up and cache a charclass for this
predicate. Note that the wchar_desc entries were set up in
fsalex_syntax (). */
int i;
charclass_index_t index;
wctype_t wctype_desc;
wctype_desc = p_entry->wchar_desc;
work_class = charclass_alloc ();
for (i = 0; i < FSATOKEN_NOTCHAR; i++)
{
wchar_t wc;
/* Try integer->unsigned char->wide char using lexer's
mbrtowc_cache array, and, if successful, test for class
membership, and set the bit in the class if the value is
a member. */
/* FIXME: iswctype depends on the *current* locale
(LC_CTYPE); this breaks the long-term goal of having each
lexer instance use only the locale in force when
fsalex_syntax was called. We can fix this by using
isalnum_l () etc here, but carrying around a full copy of
the locale in the lexer instance may be expensive or
possibly non-portable, so it's being avoided while this
code is still in an experimental/proof-of-concept
form. */
wc = lexer->mbrtowc_cache[i];
if (iswctype (wc, wctype_desc))
charclass_setbit (i, work_class);
}
/* Mark the class as completed, and obtain a persistent class
pointer. */
index = charclass_completed (work_class);
p_entry->class = charclass_get_pointer (index);
}
/* Return predicate entry to the caller. */
return p_entry;
}
/* The set of wchar_t values C such that there's a useful locale
somewhere where C != towupper (C) && C != towlower (towupper (C)).
For example, 0x00B5 (U+00B5 MICRO SIGN) is in this table, because
towupper (0x00B5) == 0x039C (U+039C GREEK CAPITAL LETTER MU), and
towlower (0x039C) == 0x03BC (U+03BC GREEK SMALL LETTER MU). */
static short const lonesome_lower[] =
{
0x00B5, 0x0131, 0x017F, 0x01C5, 0x01C8, 0x01CB, 0x01F2, 0x0345,
0x03C2, 0x03D0, 0x03D1, 0x03D5, 0x03D6, 0x03F0, 0x03F1,
/* U+03F2 GREEK LUNATE SIGMA SYMBOL lacks a specific uppercase
counterpart in locales predating Unicode 4.0.0 (April 2003). */
0x03F2,
0x03F5, 0x1E9B, 0x1FBE,
};
/* Maximum number of wide characters that can be the case-folded
counterparts of a single wide character, not counting the character
itself. This is 1 for towupper, 1 for towlower, and 1 for each
entry in lonesome_lower[]. */
enum { CASE_FOLDED_BUFSIZE = 1 + 1 + 19 };
/* Given a wide character c, and working to a search specification
that demands that case-folded variants of the character must match
wherever that character could match, generate a list of all
possible match counterparts to the specified character, and return
that list, the characters via a parameter pointer, and the list
length (possibly 0) via the function return value. */
/* ?? CHECKME: Should parameter "c" in, and/or "folded" out, be wint_t? */
static int
case_folded_counterparts (fsalex_ctxt_t *lexer,
wchar_t c,
wchar_t folded[CASE_FOLDED_BUFSIZE])
{
int i;
int n = 0;
wint_t uc = towupper (c);
wint_t lc = towlower (uc);
if (uc != c)
folded[n++] = uc;
if (lc != uc && lc != c && towupper (lc) == uc)
folded[n++] = lc;
for (i = 0; i < sizeof lonesome_lower / sizeof *lonesome_lower; i++)
{
wint_t li = lonesome_lower[i];
if (li != lc && li != uc && li != c && towupper (li) == uc)
folded[n++] = li;
}
return n;
}
/* Multibyte character handling sub-routine for lex.
Parse a bracket expression and build a struct mb_char_classes. */
static fsatoken_token_t
parse_bracket_exp (fsalex_ctxt_t *lexer)
{
bool invert;
int c, c1, c2;
charclass_t *ccl;
mbcsets_set_t *work_mbc = NULL;
/* This is a bracket expression that dfaexec is known to
process correctly. */
bool known_bracket_exp = true;
/* Used to warn about [:space:].
Bit 0 = first character is a colon.
Bit 1 = last character is a colon.
Bit 2 = includes any other character but a colon.
Bit 3 = includes ranges, char/equiv classes or collation elements. */
int colon_warning_state;
wint_t wc;
wint_t wc2;
wint_t wc1 = 0;
if (lexer->multibyte_locale)
{
/* Ensure we have space to store the new set. */
lexer->mbcsets = maybe_realloc (lexer->mbcsets,
lexer->nmbcsets,
&lexer->mbcsets_alloc,
sizeof work_mbc);
/* Initialize work area. */
work_mbc = mbcsets_new ();
lexer->mbcsets[lexer->nmbcsets++] = work_mbc;
}
ccl = charclass_alloc ();
FETCH_WC (lexer, c, wc, _("unbalanced ["));
if (c == '^')
{
FETCH_WC (lexer, c, wc, _("unbalanced ["));
invert = true;
known_bracket_exp = lexer->using_simple_locale;
}
else
invert = false;
colon_warning_state = (c == ':');
do
{
c1 = FSATOKEN_NOTCHAR; /* Mark c1 as not initialized. */
colon_warning_state &= ~2;
/* Note that if we're looking at some other [:...:] construct,
we just treat it as a bunch of ordinary characters. We can do
this because we assume regex has checked for syntax errors before
dfa is ever called. */
if (c == '[')
{
FETCH_WC (lexer, c1, wc1, _("unbalanced ["));
if ((c1 == ':' && (lexer->syntax_bits & RE_CHAR_CLASSES))
|| c1 == '.' || c1 == '=')
{
enum { MAX_BRACKET_STRING_LEN = 32 };
char str[MAX_BRACKET_STRING_LEN + 1];
size_t len = 0;
for (;;)
{
FETCH_WC (lexer, c, wc, _("unbalanced ["));
if ((c == c1 && *lexer->lexptr == ']')
|| lexer->lexleft == 0)
break;
if (len < MAX_BRACKET_STRING_LEN)
str[len++] = c;
else
/* This is in any case an invalid class name. */
str[0] = '\0';
}
str[len] = '\0';
/* Fetch bracket. */
FETCH_WC (lexer, c, wc, _("unbalanced ["));
if (c1 == ':')
/* Find and merge named character class. POSIX allows
character classes to match multicharacter collating
elements, but the regex code does not support that,
so do not worry about that possibility. */
{
char const *class;
predicate_entry_t *pred;
class = str;
if (lexer->case_fold && (STREQ (class, "upper")
|| STREQ (class, "lower")))
class = "alpha";
pred = find_pred (lexer, class);
if (! pred)
lexer->abandon_with_error (_("invalid character class"));
charclass_unionset (pred->class, ccl);
/* Does this class have a wide-char type descriptor? */
if (lexer->multibyte_locale && pred->may_be_multibyte)
{
mbcsets_add_class(work_mbc, pred->wchar_desc);
}
}
else
known_bracket_exp = false;
colon_warning_state |= 8;
/* Fetch new lookahead character. */
FETCH_WC (lexer, c1, wc1, _("unbalanced ["));
continue;
}
/* We treat '[' as a normal character here. c/c1/wc/wc1
are already set up. */
}
if (c == '\\' && (lexer->syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
FETCH_WC (lexer, c, wc, _("unbalanced ["));
if (c1 == FSATOKEN_NOTCHAR)
FETCH_WC (lexer, c1, wc1, _("unbalanced ["));
if (c1 == '-')
/* build range characters. */
{
FETCH_WC (lexer, c2, wc2, _("unbalanced ["));
/* A bracket expression like [a-[.aa.]] matches an unknown
set. Treat it like [-a[.aa.]] while parsing it, and
remember that the set is unknown. */
if (c2 == '[' && *lexer->lexptr == '.')
{
known_bracket_exp = false;
c2 = ']';
}
if (c2 != ']')
{
if (c2 == '\\' && (lexer->syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
FETCH_WC (lexer, c2, wc2, _("unbalanced ["));
if (lexer->multibyte_locale)
{
/* Merely record range as given; don't try to handle the
case-folding variants here for now. */
if (wc != WEOF && wc2 != WEOF)
{
mbcsets_add_range(work_mbc, wc, wc2);
/* FIXME, part 2: Since the canonical measure of case
equivalence is uppercase(A) == uppercase(B), perhaps
we could do this transformation above if lexer->case_fold
is true... but need to be careful not to throw away
information. */
}
}
else if (lexer->using_simple_locale)
{
charclass_setbit_range (c, c2, ccl);
if (lexer->case_fold)
{
int uc = toupper (c);
int uc2 = toupper (c2);
for (c1 = 0; c1 < FSATOKEN_NOTCHAR; c1++)
{
int uc1 = toupper (c1);
if (uc <= uc1 && uc1 <= uc2)
charclass_setbit (c1, ccl);
}
}
}
else
known_bracket_exp = false;
colon_warning_state |= 8;
FETCH_WC (lexer, c1, wc1, _("unbalanced ["));
continue;
}
/* In the case [x-], the - is an ordinary hyphen,
which is left in c1, the lookahead character. */
lexer->lexptr -= lexer->cur_mb_len;
lexer->lexleft += lexer->cur_mb_len;
}
colon_warning_state |= (c == ':') ? 2 : 4;
if (lexer->unibyte_locale)
{
if (lexer->case_fold)
setbit_case_fold_c (c, ccl);
else
charclass_setbit (c, ccl);
continue;
}
if (wc == WEOF)
known_bracket_exp = false;
else
{
wchar_t folded[CASE_FOLDED_BUFSIZE + 1];
int n = 1;
folded[0] = wc;
if (lexer->case_fold)
n += case_folded_counterparts (lexer, wc, &folded[1]);
mbcsets_add_wchar_list (work_mbc, n, folded);
}
}
while ((wc = wc1, (c = c1) != ']'));
if (colon_warning_state == 7)
lexer->warn_client (_("character class syntax is [[:space:]], not [:space:]"));
if (! known_bracket_exp)
return FSATOKEN_TK_BACKREF;
if (lexer->multibyte_locale)
{
mbcsets_set_match_sense (work_mbc, invert);
mbcsets_receive_incomplete_charclass (work_mbc, ccl);
mbcsets_completed (work_mbc);
return FSATOKEN_TK_MBCSET;
}
if (invert)
{
charclass_notset (ccl);
if (lexer->syntax_bits & RE_HAT_LISTS_NOT_NEWLINE)
charclass_clrbit (lexer->eolbyte, ccl);
}
return FSATOKEN_TK_CSET + charclass_completed (ccl);
}
fsatoken_token_t
fsalex_lex (fsalex_ctxt_t *lexer)
{
int c;
bool backslash = false;
int i;
predicate_entry_t *predicate;
charclass_t *work_class;
charclass_index_t class_index;
mbcsets_set_t *work_mbc;
bool invert;
/* Ensure that syntax () has been called on this lexer instance; many
things will fail if this isn't done. */
assert (lexer->syntax_initialised);
/* Basic plan: We fetch a character. If it's a backslash,
we set the backslash flag and go through the loop again.
On the plus side, this avoids having a duplicate of the
main switch inside the backslash case. On the minus side,
it means that just about every case begins with
"if (backslash) ...". */
for (i = 0; i < 2; ++i)
{
FETCH_WC (lexer, c, lexer->wctok, NULL);
switch (c)
{
case '\\':
if (backslash)
goto normal_char;
if (lexer->lexleft == 0)
lexer->abandon_with_error (_("unfinished \\ escape"));
backslash = true;
break;
case '^':
if (backslash)
goto normal_char;
if (lexer->syntax_bits & RE_CONTEXT_INDEP_ANCHORS
|| lexer->lasttok == FSATOKEN_TK_END || lexer->lasttok == FSATOKEN_TK_LPAREN || lexer->lasttok == FSATOKEN_TK_OR)
return lexer->lasttok = FSATOKEN_TK_BEGLINE;
goto normal_char;
case '$':
if (backslash)
goto normal_char;
if (lexer->syntax_bits & RE_CONTEXT_INDEP_ANCHORS
|| lexer->lexleft == 0
|| (lexer->syntax_bits & RE_NO_BK_PARENS
? lexer->lexleft > 0 && *lexer->lexptr == ')'
: lexer->lexleft > 1 && lexer->lexptr[0] == '\\' && lexer->lexptr[1] == ')')
|| (lexer->syntax_bits & RE_NO_BK_VBAR
? lexer->lexleft > 0 && *lexer->lexptr == '|'
: lexer->lexleft > 1 && lexer->lexptr[0] == '\\' && lexer->lexptr[1] == '|')
|| ((lexer->syntax_bits & RE_NEWLINE_ALT)
&& lexer->lexleft > 0 && *lexer->lexptr == '\n'))
return lexer->lasttok = FSATOKEN_TK_ENDLINE;
goto normal_char;
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9':
if (backslash && !(lexer->syntax_bits & RE_NO_BK_REFS))
{
lexer->laststart = false;
return lexer->lasttok = FSATOKEN_TK_BACKREF;
}
goto normal_char;
case '`':
if (backslash && lexer->re_gnu_ops)
return lexer->lasttok = FSATOKEN_TK_BEGLINE; /* FIXME: should be beginning of string */
goto normal_char;
case '\'':
if (backslash && lexer->re_gnu_ops)
return lexer->lasttok = FSATOKEN_TK_ENDLINE; /* FIXME: should be end of string */
goto normal_char;
case '<':
if (backslash && lexer->re_gnu_ops)
return lexer->lasttok = FSATOKEN_TK_BEGWORD;
goto normal_char;
case '>':
if (backslash && lexer->re_gnu_ops)
return lexer->lasttok = FSATOKEN_TK_ENDWORD;
goto normal_char;
case 'b':
if (backslash && lexer->re_gnu_ops)
return lexer->lasttok = FSATOKEN_TK_LIMWORD;
goto normal_char;
case 'B':
if (backslash && lexer->re_gnu_ops)
return lexer->lasttok = FSATOKEN_TK_NOTLIMWORD;
goto normal_char;
case '?':
if (lexer->syntax_bits & RE_LIMITED_OPS)
goto normal_char;
if (backslash != ((lexer->syntax_bits & RE_BK_PLUS_QM) != 0))
goto normal_char;
if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
goto normal_char;
return lexer->lasttok = FSATOKEN_TK_QMARK;
case '*':
if (backslash)
goto normal_char;
if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
goto normal_char;
return lexer->lasttok = FSATOKEN_TK_STAR;
case '+':
if (lexer->syntax_bits & RE_LIMITED_OPS)
goto normal_char;
if (backslash != ((lexer->syntax_bits & RE_BK_PLUS_QM) != 0))
goto normal_char;
if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
goto normal_char;
return lexer->lasttok = FSATOKEN_TK_PLUS;
case '{':
if (!(lexer->syntax_bits & RE_INTERVALS))
goto normal_char;
if (backslash != ((lexer->syntax_bits & RE_NO_BK_BRACES) == 0))
goto normal_char;
if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
goto normal_char;
/* Cases:
{M} - exact count
{M,} - minimum count, maximum is infinity
{,N} - 0 through N
{,} - 0 to infinity (same as '*')
{M,N} - M through N */
{
char const *p = lexer->lexptr;
char const *lim = p + lexer->lexleft;
int minrep = -1;
int maxrep = -1;
for (; p != lim && ISASCIIDIGIT (*p); p++)
{
if (minrep < 0)
minrep = *p - '0';
else
minrep = MIN (RE_DUP_MAX + 1, minrep * 10 + *p - '0');
}
if (p != lim)
{
if (*p != ',')
maxrep = minrep;
else
{
if (minrep < 0)
minrep = 0;
while (++p != lim && ISASCIIDIGIT (*p))
{
if (maxrep < 0)
maxrep = *p - '0';
else
maxrep = MIN (RE_DUP_MAX + 1, maxrep * 10 + *p - '0');
}
}
}
if (! ((! backslash || (p != lim && *p++ == '\\'))
&& p != lim && *p++ == '}'
&& 0 <= minrep && (maxrep < 0 || minrep <= maxrep)))
{
if (lexer->syntax_bits & RE_INVALID_INTERVAL_ORD)
goto normal_char;
lexer->abandon_with_error (_("invalid content of \\{\\}"));
}
if (RE_DUP_MAX < maxrep)
lexer->abandon_with_error (_("regular expression too big"));
lexer->lexptr = p;
lexer->lexleft = lim - p;
lexer->minrep = minrep;
lexer->maxrep = maxrep;
}
lexer->laststart = false;
return lexer->lasttok = FSATOKEN_TK_REPMN;
case '|':
if (lexer->syntax_bits & RE_LIMITED_OPS)
goto normal_char;
if (backslash != ((lexer->syntax_bits & RE_NO_BK_VBAR) == 0))
goto normal_char;
lexer->laststart = true;
return lexer->lasttok = FSATOKEN_TK_OR;
case '\n':
if (lexer->syntax_bits & RE_LIMITED_OPS
|| backslash || !(lexer->syntax_bits & RE_NEWLINE_ALT))
goto normal_char;
lexer->laststart = true;
return lexer->lasttok = FSATOKEN_TK_OR;
case '(':
if (backslash != ((lexer->syntax_bits & RE_NO_BK_PARENS) == 0))
goto normal_char;
++lexer->parens;
lexer->laststart = true;
return lexer->lasttok = FSATOKEN_TK_LPAREN;
case ')':
if (backslash != ((lexer->syntax_bits & RE_NO_BK_PARENS) == 0))
goto normal_char;
if (lexer->parens == 0 && lexer->syntax_bits & RE_UNMATCHED_RIGHT_PAREN_ORD)
goto normal_char;
--lexer->parens;
lexer->laststart = false;
return lexer->lasttok = FSATOKEN_TK_RPAREN;
case '.':
if (backslash)
goto normal_char;
lexer->laststart = false;
if (lexer->multibyte_locale)
{
/* In multibyte environment period must match with a single
character not a byte. So we use FSATOKEN_TK_ANYCHAR. */
return lexer->lasttok = FSATOKEN_TK_ANYCHAR;
}
return lexer->lasttok = FSATOKEN_TK_CSET + lexer->dotclass_index;
case 's':
case 'S':
/* "\s" == "[[:space:]]"; "\S" == "[^[:space:]]". */
if (! (backslash && lexer->re_gnu_ops))
goto normal_char;
lexer->laststart = false;
invert = c == 'S';
predicate = find_pred (lexer, "space");
if (c == 's')
class_index = charclass_get_index (predicate->class);
else
{
/* Invert the predicate class in a work class. */
work_class = charclass_alloc ();
charclass_copyset (predicate->class, work_class);
charclass_notset (work_class);
class_index = charclass_completed (work_class);
}
if (lexer->unibyte_locale)
return lexer->lasttok = FSATOKEN_TK_CSET + class_index;
/* FIXME: see if optimizing this, as is done with FSATOKEN_TK_ANYCHAR and
add_utf8_anychar, makes sense. */
/* Multibyte locale: Fill out an entire set description. */
lexer->mbcsets = maybe_realloc (lexer->mbcsets, lexer->nmbcsets,
&lexer->mbcsets_alloc,
sizeof work_mbc);
work_mbc = mbcsets_new ();
lexer->mbcsets[lexer->nmbcsets++] = work_mbc;
mbcsets_set_match_sense (work_mbc, invert);
mbcsets_add_class (work_mbc, predicate->wchar_desc);
/* ?? REVIEWME: "invert" in parse_bracket_exp leads to FSATOKEN_TK_BACKREF
(== "this is too hard for me") token via known_bracket_exp
flag. */
if ((c == 'S') && !lexer->using_simple_locale)
return lexer->lasttok = FSATOKEN_TK_BACKREF;
else
return lexer->lasttok = FSATOKEN_TK_MBCSET;
case 'w':
case 'W':
/* Can mean "[_[:alnum:]]" (\w) or its inverse (\W). */
if (! (backslash && lexer->re_gnu_ops))
goto normal_char;
lexer->laststart = false;
predicate = find_pred (lexer, "alnum");
work_class = charclass_alloc ();
charclass_copyset (predicate->class, work_class);
charclass_setbit ('_', work_class);
if (c == 'W')
charclass_notset (work_class);
return lexer->lasttok = FSATOKEN_TK_CSET
+ charclass_completed (work_class);
case '[':
if (backslash)
goto normal_char;
lexer->laststart = false;
return lexer->lasttok = parse_bracket_exp (lexer);
default:
normal_char:
lexer->laststart = false;
/* For multibyte character sets, folding is done in atom, so
always return FSATOKEN_TK_WCHAR. */
if (lexer->multibyte_locale)
return lexer->lasttok = FSATOKEN_TK_WCHAR;
if (lexer->case_fold && isalpha (c))
{
charclass_t *ccl = charclass_alloc ();
setbit_case_fold_c (c, ccl);
return lexer->lasttok = FSATOKEN_TK_CSET
+ charclass_completed (ccl);
}
return lexer->lasttok = c;
}
}
/* The above loop should consume at most a backslash
and some other character. */
abort ();
return FSATOKEN_TK_END; /* keeps pedantic compilers happy. */
}
/* Receive the pattern, and reset the lexical analyser state. The
interpretation of the chars (octets?) in the pattern (ASCII chars?
variable-length UTF-8 sequences? Simplified Chinese? etc.) depends on
the locale that was in force when fsalex_syntax () was called. NULs may
be present amongst the codes, which is why the length is given
explicitly, rather than relying on strlen(3). */
void
fsalex_pattern (fsalex_ctxt_t *lexer,
char const *pattern, size_t const pattern_len)
{
/* Copy parameters to internal state variables. */
lexer->lexptr = pattern;
lexer->lexleft = pattern_len;
/* Reset lexical scanner state. */
lexer->lasttok = FSATOKEN_TK_END;
lexer->laststart = 1;
lexer->parens = 0;
/* Reset multibyte parsing state. */
lexer->cur_mb_len = 1;
memset (&lexer->mbrtowc_state, 0, sizeof (lexer->mbrtowc_state));
}
/* Report whether codes 0-127 conform to ASCII encoding. This is handy
for optimisation, particularly lower-case and upper-case characters,
as the codes are contiguous, unlike EBCDIC. ASCII is also a common
denominator in many other unibyte code pages (locales), and also in
the multibyte UTF-8 locale. */
static bool _GL_ATTRIBUTE_PURE
char_encoding_7bits_is_ascii (void)
{
/* True if the native character set is known to be compatible with
the C locale. The following test isn't perfect, but it's good
enough in practice, as only ASCII and EBCDIC are in common use
and this test correctly accepts ASCII and rejects EBCDIC. */
const bool is_ascii =
'\b' == 8 && '\t' == 9 && '\n' == 10 && '\v' == 11
&& '\f' == 12 && '\r' == 13 && ' ' == 32 && '!' == 33
&& '"' == 34 && '#' == 35 && '%' == 37 && '&' == 38
&& '\'' == 39 && '(' == 40 && ')' == 41 && '*' == 42
&& '+' == 43 && ',' == 44 && '-' == 45 && '.' == 46
&& '/' == 47 && '0' == 48 && '9' == 57 && ':' == 58
&& ';' == 59 && '<' == 60 && '=' == 61 && '>' == 62
&& '?' == 63 && 'A' == 65 && 'Z' == 90 && '[' == 91
&& '\\' == 92 && ']' == 93 && '^' == 94 && '_' == 95
&& 'a' == 97 && 'z' == 122 && '{' == 123 && '|' == 124
&& '}' == 125 && '~' == 126;
return is_ascii;
}
/* Receive syntax directives, and other pattern interpretation
instructions such as case folding and end-of-line character.
In addition, this function configures various internal structures
based on the locale in force. */
void
fsalex_syntax (fsalex_ctxt_t *lexer,
reg_syntax_t bits, int fold, unsigned char eol)
{
charclass_t *work_class;
predicate_entry_t *pred;
char const *locale_name;
/* Set a flag noting that this lexer has had its syntax params
set. */
lexer->syntax_initialised = true;
/* Record the function parameters in our local context. */
lexer->syntax_bits = bits;
lexer->case_fold = fold;
lexer->eolbyte = eol;
/* Set up unibyte/multibyte flags, based on MB_CUR_MAX, which
depends on the current locale. We capture this information here
as the locale may change later. At present, we don't capture
MB_CUR_MAX itself. */
if (MB_CUR_MAX > 1)
{
/* Multibyte locale: Prepare booleans to make code easier to
read */
lexer->unibyte_locale = false;
lexer->multibyte_locale = true;
/* Discard any earlier storage we may have acquired. */
free (lexer->mbcsets);
lexer->mbcsets = NULL;
/* Set up an array of structures to hold multibyte character
sets. */
lexer->nmbcsets = 0;
lexer->mbcsets_alloc = 2;
lexer->mbcsets = xzalloc (sizeof (*lexer->mbcsets)
* lexer->mbcsets_alloc);
}
else
{
/* Unibyte locale: Prepare booleans to make code easier to
read */
lexer->unibyte_locale = true;
lexer->multibyte_locale = false;
}
/* Charclass guarantees that class index 0 is zeroclass, so we don't
need to set it up here. */
/* Set up a character class to match anychar ('.'), tailored to
accommodate options from the regex syntax. */
work_class = charclass_alloc ();
charclass_notset (work_class);
if (! (lexer->syntax_bits & RE_DOT_NEWLINE))
{
charclass_clrbit (lexer->eolbyte, work_class);
}
if (lexer->syntax_bits & RE_DOT_NOT_NULL)
{
charclass_clrbit (0, work_class);
}
lexer->dotclass_index = charclass_completed (work_class);
lexer->dotclass = charclass_get_pointer (lexer->dotclass_index);
/* Testing for the absence of RE_NO_GNU_OPS in syntax_bits happens
often, so set a direct flag variable: This makes code more
readable. */
lexer->re_gnu_ops = ! (lexer->syntax_bits & RE_NO_GNU_OPS);
/* Initialise cache and other tables that have syntax and/or locale
influences. */
/* Set up the wchar_desc fields of the predicate table. */
for (pred = lexer->predicates; pred->name != NULL; pred++)
pred->wchar_desc = wctype (pred->name);
/* Record the name of the current locale. Note that setlocale can
return NULL for the name. */
lexer->locale_name = NULL;
locale_name = setlocale (LC_ALL, NULL);
if (locale_name != NULL)
lexer->locale_name = xstrdup (locale_name);
/* Determine if the locale's treatment of codes 0-127 matches ASCII,
as some operations (e.g. "[c-v]") become easier to handle. */
lexer->ascii_7bit_encoding = char_encoding_7bits_is_ascii ();
/* Further optimisations are possible if the locale is an unibyte
ASCII 7-bit C/POSIX environment, and this is a common case, so
spend some effort looking for this situation. */
lexer->using_simple_locale = false;
if (lexer->unibyte_locale && lexer->ascii_7bit_encoding)
{
lexer->using_simple_locale = ! lexer->locale_name
|| STREQ (lexer->locale_name, "C")
|| STREQ (lexer->locale_name, "POSIX");
}
/* Initialise cache that distinguishes between unibyte and multibyte
(wide) characters based on the leading octet. */
initialise_uchar_to_wc_cache (lexer);
/* ?? If a complete copy of the locale needs to be stored within
this instance, see uselocale(3) and duplocale(3) (and free the
copy with freelocale(3)). */
}
/* Define external function to do non-core data exchanges between the lexer
and the parser. This function must conform to
proto_lexparse_exchange_fn_t. This interface lets two instances
communicate without requiring help from an outside party. */
int
fsalex_exchange (fsalex_ctxt_t *lexer,
proto_lexparse_opcode_t opcode,
void *param)
{
switch (opcode)
{
case PROTO_LEXPARSE_OP_GET_IS_MULTIBYTE_LOCALE:
return (int) lexer->multibyte_locale;
case PROTO_LEXPARSE_OP_GET_REPMN_MIN:
return lexer->minrep;
case PROTO_LEXPARSE_OP_GET_REPMN_MAX:
return lexer->maxrep;
case PROTO_LEXPARSE_OP_GET_WIDE_CHAR_LIST_MAX:
/* Original token, plus upper/lowercase, plus extended variants. */
return 1 + CASE_FOLDED_BUFSIZE;
case PROTO_LEXPARSE_OP_GET_WIDE_CHARS:
{
wchar_t *p_wc = (wchar_t *) param;
/* Record the original token (wctok) implicitly associated with
WCHAR. */
*p_wc++ = lexer->wctok;
/* Add in direct and/or indirect case equivalents, as appropriate. */
if (lexer->case_fold)
p_wc += case_folded_counterparts (lexer, lexer->wctok, p_wc);
/* Return how many wide chars (at least 1) are in the list. */
return p_wc - (wchar_t *) param;
}
break;
case PROTO_LEXPARSE_OP_GET_DOTCLASS:
*((charclass_t **) param) = lexer->dotclass;
break;
case PROTO_LEXPARSE_OP_GET_MBCSET:
/* Return the set associated with the most recent MBCSET token. */
*((mbcsets_set_t **) param) = lexer->mbcsets[lexer->nmbcsets - 1];
break;
default:
/* ?? Not sure if we should complain/assert or merely ignore an
opcode that we don't recognise here. */
assert (!"unrecognised PROTO_LEXPARSE opcode");
/* NOTREACHED */
break;
}
/* If we reach here, return value is unimportant, so just say 0. */
return 0;
}
/* Receive functions to deal with exceptions detected by the lexer:
Warnings and errors. Internally, we add the _Noreturn attribute
to the error callback, to help the compiler with code flow
analysis. */
void
fsalex_exception_fns (fsalex_ctxt_t *lexer,
fsalex_warn_callback_fn *warningfn,
fsalex_error_callback_fn *errorfn)
{
/* Record the provided functions in the lexer's context. */
lexer->warn_client = warningfn;
lexer->abandon_with_error = errorfn;
}
/* Add "not provided!" stub function that gets called if the client
fails to provide proper resources. This is a hack, merely to get
the module started; better treatment needs to be added later. */
static void
no_function_provided (void *unused)
{
assert (!"fsalex: Plug-in function required, but not provided.");
}
/* Generate a new instance of an FSA lexer. */
fsalex_ctxt_t *
fsalex_new (void)
{
fsalex_ctxt_t *new_context;
/* Acquire zeroed memory for new lexer context. */
new_context = XZALLOC (fsalex_ctxt_t);
/* ?? Point warning and error functions to a "you need to tell me
these first!" function? */
new_context->warn_client = (fsalex_warn_callback_fn *)
no_function_provided;
new_context->abandon_with_error = (fsalex_error_callback_fn *)
no_function_provided;
/* Default to working in a non-multibyte locale. In some cases,
FETCH_WC never sets this variable (as it's assumed to be 1), so
fulfil this expectation here. */
new_context->cur_mb_len = 1;
/* Copy the template predicate list into this context, so that we
can have lexer-specific named predicate classes. */
memcpy (new_context->predicates, template_predicate_list,
sizeof (new_context->predicates));
/* Default to unibyte locale at first; the final locale setting is
made according to what's in force when fsalex_syntax () is
called. */
new_context->unibyte_locale = true;
new_context->multibyte_locale = false;
/* Many things depend on decisions made in fsalex_syntax (), so note
here that it hasn't been called yet, and fail gracefully later if
the client hasn't called the function before commencing work. */
new_context->syntax_initialised = false;
/* Add this instance at the head of the module's list. */
new_context->next_instance = fsalex_instance_list_head;
fsalex_instance_list_head = new_context;
return new_context;
}
/* Internal function to free all resources directly or indirectly used
by an instance. The pointer is no longer valid after this call. */
static void
free_instance (fsalex_ctxt_t *lexer)
{
/* Free top-level structures. */
free (lexer->locale_name);
free (lexer->mbcsets);
free (lexer);
/* Note that multibyte sets are no longer stored explicitly in
the instance; apart from the pointer list (freed above), the
set contents are freed separately when mbcsets module is
destroyed. */
}
/* Destroy all lexer instances, plus any associated resources owned by the
module. */
void
fsalex_destroy_module (void)
{
fsalex_ctxt_t *p_list;
fsalex_ctxt_t *p_next;
/* Move the global list head into a local variable, and immediately
clear the global. This is a half-hearted attempt to avoid race
conditions; to do things properly, a system-wide atomic
operation (locked, including multi-CPU cache coherency) operation
should be used. */
p_list = fsalex_instance_list_head;
fsalex_instance_list_head = NULL;
/* Traverse the list of instances, releasing all resources
associated with each one. */
while (p_list)
{
p_next = p_list->next_instance;
free_instance (p_list);
p_list = p_next;
}
}
/* Prepare module for operation. */
void
fsalex_initialise (void)
{
/* Initialise the linked list of instances created by this module. */
fsalex_instance_list_head = NULL;
atexit (fsalex_destroy_module);
}
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="fsalex.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsalex.h"
/* fsalex - Repackage pattern text as compact, expressive tokens
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
#ifndef FSALEX_H
#define FSALEX_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
#include "fsatoken.h"
#include "proto-lexparse.h"
#include <regex.h>
/* Multiple lexer instances can exist in parallel, so define an
opaque type to collect together all the context relating to each
instance. */
typedef struct fsalex_ctxt_struct fsalex_ctxt_t;
/* Prepare module for operation. */
extern void
fsalex_initialise (void);
/* Destroy all lexer instances, plus any associated resources owned by the
module. */
extern void
fsalex_destroy_module (void);
/* Generate a new instance of an FSA lexer. */
extern fsalex_ctxt_t *
fsalex_new (void);
/* ?? Reserve "fsalex_discard ()" for getting rid of a lexer
instance. */
/* Receive syntax directives, and other pattern interpretation
instructions such as case folding and end-of-line character.
In addition, this function configures various internal structures
based on the locale in force. */
extern void
fsalex_syntax (fsalex_ctxt_t *lexer,
reg_syntax_t bits, int fold, unsigned char eol);
/* Receive the pattern, and reset the lexical analyser state. The
interpretation of the chars (octets?) in the pattern (ASCII chars?
variable-length UTF-8 sequences? Simplified Chinese? etc.) depends on
the locale that was in force when fsalex_syntax () was called. NULs may
be present amongst the codes, which is why the length is given
explicitly, rather than relying on strlen(3). */
extern void
fsalex_pattern (fsalex_ctxt_t *lexer,
char const *pattern, size_t const pattern_len);
/* Define function prototypes for warning and error callbacks. */
typedef void
fsalex_warn_callback_fn (const char *);
typedef void /* ?? _Noreturn? */
fsalex_error_callback_fn (const char *);
/* Receive functions to deal with exceptions detected by the lexer:
Warnings and errors. Internally, we add the _Noreturn attribute
to the error callback, to help the compiler with code flow
analysis. */
extern void
fsalex_exception_fns (fsalex_ctxt_t *lexer,
fsalex_warn_callback_fn *warningfn,
fsalex_error_callback_fn *errorfn);
/* Main function to incrementally consume and interpret the pattern
text, and return a token describing a single lexical element as a
token, perhaps with implied parameters such as character classes
for CSET tokens, and {min,max} values for each REPMN token. The
user should call this function repeatedly, receiving one token each
time, until the lexer detects a fatal error, or returns the END
token. */
/* This function must conform to proto_lexparse_lex_fn_t. */
extern fsatoken_token_t
fsalex_lex (fsalex_ctxt_t *lexer);
/* Define external function to do non-core data exchanges between the lexer
and the parser. This function must conform to
proto_lexparse_exchange_fn_t. This interface lets two instances
communicate without requiring help from an outside party. */
extern int
fsalex_exchange (fsalex_ctxt_t *lexer,
proto_lexparse_opcode_t opcode,
void *param);
#endif /* FSALEX_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="fsamusts.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsamusts.c"
/* fsamusts -- Report a list of must-have simple strings in the pattern
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* (?? Long description/discussion goes here...) */
/* Always import environment-specific configuration items first. */
#include <config.h>
#include <assert.h>
#include "charclass.h"
#include <ctype.h>
#include "fsamusts.h"
#include "fsatoken.h"
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include "xalloc.h"
#if DEBUG
#include <stdio.h>
#endif /* DEBUG */
#define STREQ(a, b) (strcmp (a, b) == 0)
/* Having found the postfix representation of the regular expression,
try to find a long sequence of characters that must appear in any line
containing the r.e.
Finding a "longest" sequence is beyond the scope here;
we take an easy way out and hope for the best.
(Take "(ab|a)b"--please.)
We do a bottom-up calculation of sequences of characters that must appear
in matches of r.e.'s represented by trees rooted at the nodes of the postfix
representation:
sequences that must appear at the left of the match ("left")
sequences that must appear at the right of the match ("right")
lists of sequences that must appear somewhere in the match ("in")
sequences that must constitute the match ("is")
When we get to the root of the tree, we use one of the longest of its
calculated "in" sequences as our answer. The sequence we find is returned in
d->must (where "d" is the single argument passed to "dfamust");
the length of the sequence is returned in d->mustn.
The sequences calculated for the various types of node (in pseudo ANSI c)
are shown below. "p" is the operand of unary operators (and the left-hand
operand of binary operators); "q" is the right-hand operand of binary
operators.
"ZERO" means "a zero-length sequence" below.
Type left right is in
---- ---- ----- -- --
char c # c # c # c # c
ANYCHAR ZERO ZERO ZERO ZERO
MBCSET ZERO ZERO ZERO ZERO
CSET ZERO ZERO ZERO ZERO
STAR ZERO ZERO ZERO ZERO
QMARK ZERO ZERO ZERO ZERO
PLUS p->left p->right ZERO p->in
CAT (p->is==ZERO)? (q->is==ZERO)? (p->is!=ZERO && p->in plus
p->left : q->right : q->is!=ZERO) ? q->in plus
p->is##q->left p->right##q->is p->is##q->is : p->right##q->left
ZERO
OR longest common longest common (do p->is and substrings common to
leading trailing q->is have same p->in and q->in
(sub)sequence (sub)sequence length and
of p->left of p->right content) ?
and q->left and q->right p->is : NULL
If there's anything else we recognize in the tree, all four sequences get set
to zero-length sequences. If there's something we don't recognize in the
tree, we just return a zero-length sequence.
Break ties in favor of infrequent letters (choosing 'zzz' in preference to
'aaa')?
And ... is it here or someplace that we might ponder "optimizations" such as
egrep 'psi|epsilon' -> egrep 'psi'
egrep 'pepsi|epsilon' -> egrep 'epsi'
(Yes, we now find "epsi" as a "string
that must occur", but we might also
simplify the *entire* r.e. being sought)
grep '[c]' -> grep 'c'
grep '(ab|a)b' -> grep 'ab'
grep 'ab*' -> grep 'a'
grep 'a*b' -> grep 'b'
There are several issues:
Is optimization easy (enough)?
Does optimization actually accomplish anything,
or is the automaton you get from "psi|epsilon" (for example)
the same as the one you get from "psi" (for example)?
Are optimizable r.e.'s likely to be used in real-life situations
(something like 'ab*' is probably unlikely; something like is
'psi|epsilon' is likelier)? */
static char *
icatalloc (char *old, char const *new)
{
char *result;
size_t oldsize;
size_t newsize = strlen (new);
if (newsize == 0)
return old;
oldsize = strlen (old);
result = xrealloc (old, oldsize + newsize + 1);
memcpy (result + oldsize, new, newsize + 1);
return result;
}
static char *_GL_ATTRIBUTE_PURE
istrstr (char const *lookin, char const *lookfor)
{
char const *cp;
size_t len;
len = strlen (lookfor);
for (cp = lookin; *cp != '\0'; ++cp)
if (strncmp (cp, lookfor, len) == 0)
return (char *) cp;
return NULL;
}
static void
freelist (char **cpp)
{
while (*cpp)
free (*cpp++);
}
static char **
enlist (char **cpp, char *new, size_t len)
{
size_t i, j;
new = memcpy (xmalloc (len + 1), new, len);
new[len] = '\0';
/* Is there already something in the list that's new (or longer)? */
for (i = 0; cpp[i] != NULL; ++i)
if (istrstr (cpp[i], new) != NULL)
{
free (new);
return cpp;
}
/* Eliminate any obsoleted strings. */
j = 0;
while (cpp[j] != NULL)
if (istrstr (new, cpp[j]) == NULL)
++j;
else
{
free (cpp[j]);
if (--i == j)
break;
cpp[j] = cpp[i];
cpp[i] = NULL;
}
/* Add the new string. */
cpp = xnrealloc (cpp, i + 2, sizeof *cpp);
cpp[i] = new;
cpp[i + 1] = NULL;
return cpp;
}
/* Given pointers to two strings, return a pointer to an allocated
list of their distinct common substrings. */
static char **
comsubs (char *left, char const *right)
{
char **cpp = xzalloc (sizeof *cpp);
char *lcp;
for (lcp = left; *lcp != '\0'; ++lcp)
{
size_t len = 0;
char *rcp = strchr (right, *lcp);
while (rcp != NULL)
{
size_t i;
for (i = 1; lcp[i] != '\0' && lcp[i] == rcp[i]; ++i)
continue;
if (i > len)
len = i;
rcp = strchr (rcp + 1, *lcp);
}
if (len != 0)
cpp = enlist (cpp, lcp, len);
}
return cpp;
}
static char **
addlists (char **old, char **new)
{
for (; *new; new++)
old = enlist (old, *new, strlen (*new));
return old;
}
/* Given two lists of substrings, return a new list giving substrings
common to both. */
static char **
inboth (char **left, char **right)
{
char **both = xzalloc (sizeof *both);
size_t lnum, rnum;
for (lnum = 0; left[lnum] != NULL; ++lnum)
{
for (rnum = 0; right[rnum] != NULL; ++rnum)
{
char **temp = comsubs (left[lnum], right[rnum]);
both = addlists (both, temp);
freelist (temp);
free (temp);
}
}
return both;
}
typedef struct must must;
struct must
{
char **in;
char *left;
char *right;
char *is;
bool begline;
bool endline;
must *prev;
};
static must *
allocmust (must *mp)
{
must *new_mp = xmalloc (sizeof *new_mp);
new_mp->in = xzalloc (sizeof *new_mp->in);
new_mp->left = xzalloc (2);
new_mp->right = xzalloc (2);
new_mp->is = xzalloc (2);
new_mp->begline = false;
new_mp->endline = false;
new_mp->prev = mp;
return new_mp;
}
static void
resetmust (must *mp)
{
freelist (mp->in);
mp->in[0] = NULL;
mp->left[0] = mp->right[0] = mp->is[0] = '\0';
mp->begline = false;
mp->endline = false;
}
static void
freemust (must *mp)
{
freelist (mp->in);
free (mp->in);
free (mp->left);
free (mp->right);
free (mp->is);
free (mp);
}
/* Receive an existing list (possibly empty) of must-have strings, together
with a list of the tokens for the current FSA (postfix tree order), and
if there are any more must-have strings in the token list, add them to
the must-have list. Returns the possibly-modified list to the caller.
Locale and syntax items are partially covered here by the case_fold and
unibyte_locale flags, but this is incomplete, and should be addressed by
Stage 2 (improving the expressiveness of tokens). */
fsamusts_list_element_t *
fsamusts_must (fsamusts_list_element_t *must_list,
size_t nr_tokens, fsatoken_token_t *token_list,
bool case_fold, bool unibyte_locale)
{
must *mp = NULL;
char const *result = "";
size_t ri;
size_t i;
bool exact = false;
bool begline = false;
bool endline = false;
for (ri = 0; ri < nr_tokens; ++ri)
{
fsatoken_token_t t = token_list[ri];
switch (t)
{
case FSATOKEN_TK_BEGLINE:
mp = allocmust (mp);
mp->begline = true;
break;
case FSATOKEN_TK_ENDLINE:
mp = allocmust (mp);
mp->endline = true;
break;
case FSATOKEN_TK_LPAREN:
case FSATOKEN_TK_RPAREN:
assert (!"neither FSATOKEN_TK_LPAREN nor FSATOKEN_TK_RPAREN may appear here");
case FSATOKEN_TK_EMPTY:
case FSATOKEN_TK_BEGWORD:
case FSATOKEN_TK_ENDWORD:
case FSATOKEN_TK_LIMWORD:
case FSATOKEN_TK_NOTLIMWORD:
case FSATOKEN_TK_BACKREF:
case FSATOKEN_TK_ANYCHAR:
case FSATOKEN_TK_MBCSET:
mp = allocmust (mp);
break;
case FSATOKEN_TK_STAR:
case FSATOKEN_TK_QMARK:
resetmust (mp);
break;
case FSATOKEN_TK_OR:
{
char **new;
must *rmp = mp;
must *lmp = mp = mp->prev;
size_t j, ln, rn, n;
/* Guaranteed to be. Unlikely, but ... */
if (STREQ (lmp->is, rmp->is))
{
lmp->begline &= rmp->begline;
lmp->endline &= rmp->endline;
}
else
{
lmp->is[0] = '\0';
lmp->begline = false;
lmp->endline = false;
}
/* Left side--easy */
i = 0;
while (lmp->left[i] != '\0' && lmp->left[i] == rmp->left[i])
++i;
lmp->left[i] = '\0';
/* Right side */
ln = strlen (lmp->right);
rn = strlen (rmp->right);
n = ln;
if (n > rn)
n = rn;
for (i = 0; i < n; ++i)
if (lmp->right[ln - i - 1] != rmp->right[rn - i - 1])
break;
for (j = 0; j < i; ++j)
lmp->right[j] = lmp->right[(ln - i) + j];
lmp->right[j] = '\0';
new = inboth (lmp->in, rmp->in);
freelist (lmp->in);
free (lmp->in);
lmp->in = new;
freemust (rmp);
}
break;
case FSATOKEN_TK_PLUS:
mp->is[0] = '\0';
break;
case FSATOKEN_TK_END:
assert (!mp->prev);
for (i = 0; mp->in[i] != NULL; ++i)
if (strlen (mp->in[i]) > strlen (result))
result = mp->in[i];
if (STREQ (result, mp->is))
{
exact = true;
begline = mp->begline;
endline = mp->endline;
}
goto done;
case FSATOKEN_TK_CAT:
{
must *rmp = mp;
must *lmp = mp = mp->prev;
/* In. Everything in left, plus everything in
right, plus concatenation of
left's right and right's left. */
lmp->in = addlists (lmp->in, rmp->in);
if (lmp->right[0] != '\0' && rmp->left[0] != '\0')
{
size_t lrlen = strlen (lmp->right);
size_t rllen = strlen (rmp->left);
char *tp = xmalloc (lrlen + rllen);
memcpy (tp, lmp->right, lrlen);
memcpy (tp + lrlen, rmp->left, rllen);
lmp->in = enlist (lmp->in, tp, lrlen + rllen);
free (tp);
}
/* Left-hand */
if (lmp->is[0] != '\0')
lmp->left = icatalloc (lmp->left, rmp->left);
/* Right-hand */
if (rmp->is[0] == '\0')
lmp->right[0] = '\0';
lmp->right = icatalloc (lmp->right, rmp->right);
/* Guaranteed to be */
if ((lmp->is[0] != '\0' || lmp->begline)
&& (rmp->is[0] != '\0' || rmp->endline))
{
lmp->is = icatalloc (lmp->is, rmp->is);
lmp->endline = rmp->endline;
}
else
{
lmp->is[0] = '\0';
lmp->begline = false;
lmp->endline = false;
}
freemust (rmp);
}
break;
case '\0':
/* Not on *my* shift. */
goto done;
default:
mp = allocmust (mp);
if (FSATOKEN_TK_CSET <= t)
{
/* If T is a singleton, or if case-folding in a unibyte
locale and T's members all case-fold to the same char,
convert T to one of its members. Otherwise, do
nothing further with T. */
charclass_t *ccl = charclass_get_pointer (t - FSATOKEN_TK_CSET);
int j;
for (j = 0; j < FSATOKEN_NOTCHAR; j++)
if (charclass_tstbit (j, ccl))
break;
if (! (j < FSATOKEN_NOTCHAR))
break;
t = j;
while (++j < FSATOKEN_NOTCHAR)
if (charclass_tstbit (j, ccl)
&& ! (case_fold && unibyte_locale
&& toupper (j) == toupper (t)))
break;
if (j < FSATOKEN_NOTCHAR)
break;
}
mp->is[0] = mp->left[0] = mp->right[0]
= case_fold && unibyte_locale ? toupper (t) : t;
mp->is[1] = mp->left[1] = mp->right[1] = '\0';
mp->in = enlist (mp->in, mp->is, 1);
break;
}
}
done:
if (strlen (result))
{
fsamusts_list_element_t *dm;
dm = xmalloc (sizeof *dm);
dm->exact = exact;
dm->begline = begline;
dm->endline = endline;
dm->must = xstrdup (result);
dm->next = must_list;
must_list = dm;
}
while (mp)
{
must *prev = mp->prev;
freemust (mp);
mp = prev;
}
return must_list;
}
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="fsamusts.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsamusts.h"
/* fsamusts -- Report a list of must-have simple strings in the pattern
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* ?? Insert long description/discussion here. */
#ifndef FSAMUSTS_H
#define FSAMUSTS_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
#include "fsatoken.h"
/* Element of a list of strings, at least one of which is known to
appear in any R.E. matching the DFA. */
typedef struct fsamusts_list_element
{
bool exact;
bool begline;
bool endline;
char *must;
struct fsamusts_list_element *next;
} fsamusts_list_element_t;
/* Receive an existing list (possibly empty) of must-have strings, together
with a list of the tokens for the current FSA (postfix tree order), and
if there are any more must-have strings in the token list, add them to
the must-have list. Returns the possibly-modified list to the caller.
Locale and syntax items are partially covered here by the case_fold and
unibyte_locale flags, but this is incomplete, and should be addressed by
Stage 2 (improving the expressiveness of tokens). */
extern fsamusts_list_element_t *
fsamusts_must (fsamusts_list_element_t *must_list,
size_t nr_tokens, fsatoken_token_t *token_list,
bool case_fold, bool unibyte_locale);
#endif /* FSAMUSTS_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="fsaparse.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsaparse.c"
/* fsaparse -- Build a structure naming relationships (sequences,
alternatives, backreferences, options and precedence)
of tokens
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* This function receives a stream of tokens from fsalex, and processes
them to impose precedence rules and to describe complex pattern
elements that are beyond the capability of the simple lexer. In
addition to the cases explicit in the syntax (e.g."(ab|c)",
variable-length multibyte encodings (UTF-8; codesets including
modifiers and/or shift items) also require these enhanced
facilities. */
/* Always import environment-specific configuration items first. */
#include <config.h>
#include <assert.h>
#include "charclass.h"
#include "fsaparse.h"
#include "fsalex.h"
#include "fsatoken.h"
#include "mbcsets.h"
#include "proto-lexparse.h"
#include <stdbool.h>
#include <stdlib.h>
#include "xalloc.h"
/* gettext.h ensures that we don't use gettext if ENABLE_NLS is not defined */
#include "gettext.h"
#define _(str) gettext (str)
#include <wchar.h>
#include <wctype.h>
#if HAVE_LANGINFO_CODESET
# include <langinfo.h>
#endif
/* fsaparse_ctxt: Gather all the context to do with the parser into a
single struct. We do this mainly because it make it easier to
contemplate having multiple instances of this module running in
parallel, but also because it makes translating from "dfa->" easier.
This definition fleshes out the opaque type given in the module
header. */
struct fsaparse_ctxt_struct
{
/* Singly-linked list of all parser instances, so destroy_module can
release all resources by traversing the list. */
fsaparse_ctxt_t *next_instance;
/* Warning and abort functions provided by client. */
fsalex_warn_callback_fn *warn_client;
fsalex_error_callback_fn *abandon_with_error;
/* Plug-in functions and context to deal with lexer at arm's
length. */
proto_lexparse_lex_fn_t *lexer;
proto_lexparse_exchange_fn_t *lex_exchange;
void *lex_context;
/* Information about locale (needs to sync with lexer...?) */
bool multibyte_locale;
bool unibyte_locale;
/* Variable to store a dynamically-allocated list of wide characters.
The parser needs to fetch this list whenever a WCHAR token is
received. */
int wide_chars_max;
wchar_t *wide_char_list;
fsatoken_token_t lookahead_token;
size_t current_depth; /* Current depth of a hypothetical stack
holding deferred productions. This is
used to determine the depth that will be
required of the real stack later on in
dfaanalyze. */
/* Fields filled by the parser. */
fsatoken_token_t *tokens; /* Postfix parse array. */
size_t tindex; /* Index for adding new tokens. */
size_t talloc; /* Number of tokens currently allocated. */
size_t depth; /* Depth required of an evaluation stack
used for depth-first traversal of the
parse tree. */
size_t nleaves; /* Number of leaves on the parse tree. */
size_t nregexps; /* Count of parallel regexps being built
with dfaparse. */
bool fast; /* The DFA is fast. */
bool multibyte; /* MB_CUR_MAX > 1. */
fsatoken_token_t utf8_anychar_classes[5]; /* To lower ANYCHAR in UTF-8 locales. */
mbstate_t mbs; /* Multibyte conversion state. */
/* The following are valid only if MB_CUR_MAX > 1. */
/* The value of multibyte_prop[i] is defined by following rule.
if tokens[i] < NOTCHAR
bit 0 : tokens[i] is the first byte of a character, including
single-byte characters.
bit 1 : tokens[i] is the last byte of a character, including
single-byte characters.
if tokens[i] = MBCSET
("the index of mbcsets corresponding to this operator" << 2) + 3
e.g.
tokens
= 'single_byte_a', 'multi_byte_A', single_byte_b'
= 'sb_a', 'mb_A(1st byte)', 'mb_A(2nd byte)', 'mb_A(3rd byte)', 'sb_b'
multibyte_prop
= 3 , 1 , 0 , 2 , 3
*/
int *multibyte_prop;
mbcsets_set_t **mbcsets;
size_t nmbcsets;
size_t mbcsets_alloc;
};
/* Linked list of all instances created by this module. */
static fsaparse_ctxt_t *fsaparse_instance_list_head = NULL;
/* Ensure that the array addressed by PTR holds at least NITEMS +
(PTR || !NITEMS) items. Either return PTR, or reallocate the array
and return its new address. Although PTR may be null, the returned
value is never null.
The array holds *NALLOC items; *NALLOC is updated on reallocation.
ITEMSIZE is the size of one item. Avoid O(N**2) behavior on arrays
growing linearly. */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
if (nitems < *nalloc)
return ptr;
*nalloc = nitems;
return x2nrealloc (ptr, nalloc, itemsize);
}
/* UTF-8 encoding allows some optimizations that we can't otherwise
assume in a multibyte encoding. */
static int
using_utf8 (void)
{
static int utf8 = -1;
if (utf8 < 0)
{
wchar_t wc;
mbstate_t mbs = { 0 };
utf8 = mbrtowc (&wc, "\xc4\x80", 2, &mbs) == 2 && wc == 0x100;
}
return utf8;
}
/* Recursive descent parser for regular expressions. */
static void
addtok_mb (fsaparse_ctxt_t *parser, fsatoken_token_t t, int mbprop)
{
if (parser->talloc == parser->tindex)
{
parser->tokens = x2nrealloc (parser->tokens, &parser->talloc,
sizeof *parser->tokens);
if (parser->multibyte)
parser->multibyte_prop = xnrealloc (parser->multibyte_prop, parser->talloc,
sizeof *parser->multibyte_prop);
}
if (parser->multibyte)
parser->multibyte_prop[parser->tindex] = mbprop;
parser->tokens[parser->tindex++] = t;
switch (t)
{
case FSATOKEN_TK_QMARK:
case FSATOKEN_TK_STAR:
case FSATOKEN_TK_PLUS:
break;
case FSATOKEN_TK_CAT:
case FSATOKEN_TK_OR:
--parser->current_depth;
break;
case FSATOKEN_TK_BACKREF:
parser->fast = false;
/* fallthrough */
default:
++parser->nleaves;
/* fallthrough */
case FSATOKEN_TK_EMPTY:
++parser->current_depth;
break;
}
if (parser->depth < parser->current_depth)
parser->depth = parser->current_depth;
}
static void addtok_wc (fsaparse_ctxt_t *parser, wint_t wc);
/* Sigh. Mbcsets is so disruptive (intimate details hidden, by
design), that we are writing addtok here from scratch. */
static void
addtok (fsaparse_ctxt_t *parser, fsatoken_token_t t)
{
bool need_or = false;
mbcsets_set_t *work_mbc;
bool invert;
charclass_t *charclass;
size_t nchars;
size_t nch_classes;
size_t nranges;
size_t nequivs;
size_t ncoll_elems;
if (t != FSATOKEN_TK_MBCSET || parser->unibyte_locale)
{
addtok_mb (parser, t, 3);
return;
}
work_mbc = parser->mbcsets[parser->nmbcsets - 1];
mbcsets_get_characteristics (work_mbc,
&invert,
&charclass,
&nchars,
&nch_classes,
&nranges,
&nequivs,
&ncoll_elems);
if (nchars && !invert)
{
size_t i;
wchar_t *char_list;
char_list = xnmalloc (nchars, sizeof *char_list);
mbcsets_get_chars (work_mbc, char_list);
addtok_wc (parser, char_list[0]);
for (i = 1; i < nchars; i++)
{
addtok_wc (parser, char_list[i]);
addtok (parser, FSATOKEN_TK_OR);
}
free (char_list);
need_or = true;
}
/* If the set contains non-trivial components, it's too hard to deal
with here, so hand it on to higher-up machinery. */
if (invert || nch_classes || nranges || ncoll_elems)
{
addtok_mb (parser, FSATOKEN_TK_MBCSET,
((parser->nmbcsets - 1) << 2) + 3);
if (need_or)
addtok (parser, FSATOKEN_TK_OR);
}
/* Individual characters have been handled above, so the only case
remaining is where codes have been simplified into charclass
members. */
else if (charclass)
{
addtok (parser, FSATOKEN_TK_CSET + charclass_get_index (charclass));
if (need_or)
addtok (parser, FSATOKEN_TK_OR);
}
}
/* We treat a multibyte character as a single atom, so that DFA
can treat a multibyte character as a single expression.
e.g., we construct the following tree from "<mb1><mb2>".
<mb1(1st-byte)><mb1(2nd-byte)><FSATOKEN_TK_CAT><mb1(3rd-byte)><FSATOKEN_TK_CAT>
<mb2(1st-byte)><mb2(2nd-byte)><FSATOKEN_TK_CAT><mb2(3rd-byte)><FSATOKEN_TK_CAT><FSATOKEN_TK_CAT> */
static void
addtok_wc (fsaparse_ctxt_t *parser, wint_t wc)
{
unsigned char buf[MB_LEN_MAX];
mbstate_t s = { 0 };
int i;
int cur_mb_len;
size_t stored_bytes = wcrtomb ((char *) buf, wc, &s);
if (stored_bytes != (size_t) -1)
cur_mb_len = stored_bytes;
else
{
/* This is merely stop-gap. buf[0] is undefined, yet skipping
the addtok_mb call altogether can corrupt the heap. */
cur_mb_len = 1;
buf[0] = 0;
}
addtok_mb (parser, buf[0], cur_mb_len == 1 ? 3 : 1);
for (i = 1; i < cur_mb_len; i++)
{
addtok_mb (parser, buf[i], i == cur_mb_len - 1 ? 2 : 0);
addtok (parser, FSATOKEN_TK_CAT);
}
}
static void
add_utf8_anychar (fsaparse_ctxt_t *parser)
{
unsigned int i;
/* Have we set up the classes for the 1-byte to 4-byte sequence types? */
if (parser->utf8_anychar_classes[0] == 0)
{
/* No, first time we've been called, so set them up now. */
charclass_t *ccl;
const charclass_t *dotclass;
/* Index 0: 80-bf -- Non-leading bytes. */
ccl = charclass_alloc ();
charclass_setbit_range (0x80, 0xbf, ccl);
parser->utf8_anychar_classes[0] = charclass_completed (ccl);
/* Index 1: 00-7f -- 1-byte leading seq, minus dotclass exceptions. */
ccl = charclass_alloc ();
charclass_setbit_range (0x00, 0x7f, ccl);
fsalex_exchange (parser->lex_context, PROTO_LEXPARSE_OP_GET_DOTCLASS,
&dotclass);
charclass_intersectset (dotclass, ccl);
parser->utf8_anychar_classes[1] = charclass_completed (ccl);
/* Index 2: c2-df -- 2-byte sequence. */
ccl = charclass_alloc ();
charclass_setbit_range (0xc2, 0xdf, ccl);
parser->utf8_anychar_classes[2] = charclass_completed (ccl);
/* Index 2: e0-ef -- 3-byte sequence. */
ccl = charclass_alloc ();
charclass_setbit_range (0xe0, 0xef, ccl);
parser->utf8_anychar_classes[3] = charclass_completed (ccl);
/* Index 2: f0-f7 -- 4-byte sequence. */
ccl = charclass_alloc ();
charclass_setbit_range (0xf0, 0xf7, ccl);
parser->utf8_anychar_classes[4] = charclass_completed (ccl);
}
/* A valid UTF-8 character is
([0x00-0x7f]
|[0xe0-0xef][0x80-0xbf][0x80-0xbf]xc2-0xdf][0x80-0xbf]
|[0xe0-0xef][0x80-0xbf][0x80-0xbf]xe0-0xef[0x80-0xbf][0x80-0xbf]
|[0xe0-0xef][0x80-0xbf][0x80-0xbf]xf0-f7][0x80-0xbf][0x80-0xbf][0x80-0xbf])
which I'll write more concisely "B|CA|DAA|EAAA". Factor the [0x00-0x7f]
and you get "B|(C|(D|EA)A)A". And since the token buffer is in reverse
Polish notation, you get "B C D E A CAT OR A CAT OR A CAT OR". */
/* Write out leaf tokens for each of the four possible starting bytes. */
for (i = 1; i < 5; i++)
addtok (parser, FSATOKEN_TK_CSET + parser->utf8_anychar_classes[i]);
/* Add follow-on classes, plus tokens to build a postfix tree
covering all four alternatives of valid UTF-8 sequences. */
for (i = 1; i <= 3; i++)
{
addtok (parser, FSATOKEN_TK_CSET + parser->utf8_anychar_classes[0]);
addtok (parser, FSATOKEN_TK_CAT);
addtok (parser, FSATOKEN_TK_OR);
}
}
/* The grammar understood by the parser is as follows.
regexp:
regexp FSATOKEN_TK_OR branch
branch
branch:
branch closure
closure
closure:
closure FSATOKEN_TK_QMARK
closure FSATOKEN_TK_STAR
closure FSATOKEN_TK_PLUS
closure FSATOKEN_TK_REPMN
atom
atom:
<normal character>
<multibyte character>
FSATOKEN_TK_ANYCHAR
FSATOKEN_TK_MBCSET
FSATOKEN_TK_CSET
FSATOKEN_TK_BACKREF
FSATOKEN_TK_BEGLINE
FSATOKEN_TK_ENDLINE
FSATOKEN_TK_BEGWORD
FSATOKEN_TK_ENDWORD
FSATOKEN_TK_LIMWORD
FSATOKEN_TK_NOTLIMWORD
FSATOKEN_TK_LPAREN regexp FSATOKEN_TK_RPAREN
<empty>
The parser builds a parse tree in postfix form in an array of tokens. */
/* Provide a forward declaration for regexp, as it is at the top of
the parse tree, but is referenced by atom, at the bottom of the
tree. */
static void regexp (fsaparse_ctxt_t *parser);
/* 3 June 2014: I must be a sucker for punishment, and/or going crazy;
whatever. I've looked at the partially-edited code for "atom",
looking at it in the (incomplete) light of introducing the mbcsets
change... and have decided, just as I previously did with
closure () below, to rewrite it from scratch. So, here goes... */
static void
atom (fsaparse_ctxt_t *parser)
{
fsatoken_token_t tok = parser->lookahead_token;
/* For a unibyte character, it is its own token. */
if (tok >= 0 && tok < FSATOKEN_NOTCHAR)
{
addtok (parser, tok);
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
}
/* For a unibyte character set (CSET + index), it is its own token. */
if (tok >= FSATOKEN_TK_CSET)
{
addtok (parser, tok);
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
}
/* For ANYCHAR in an UTF-8 setting, use a hierarchy of CSETs. */
if (tok == FSATOKEN_TK_ANYCHAR && using_utf8 ())
{
/* For UTF-8, expand the period to a series of CSETs that define
a valid UTF-8 character. This avoids using the slow multibyte
path. I'm pretty sure it would be both profitable and correct
to do it for any encoding; however, the optimization must be
done manually as it is done above in add_utf8_anychar. So,
let's start with UTF-8: it is the most used, and the structure
of the encoding makes the correctness more obvious. */
add_utf8_anychar (parser);
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
}
/* Use switch to carve up the remaining cases into groups. */
switch (tok)
{
case FSATOKEN_TK_WCHAR:
{
/* Wide character, possibly including equivalent characters
(probably due to selection of case folding). */
int nr_wide_chars;
nr_wide_chars = fsalex_exchange (parser->lex_context,
PROTO_LEXPARSE_OP_GET_WIDE_CHARS,
parser->wide_char_list);
assert(nr_wide_chars >= 1);
if (parser->wide_char_list[0] == WEOF)
addtok (parser, FSATOKEN_TK_BACKREF);
else
{
int i;
addtok_wc (parser, parser->wide_char_list[0]);
for (i = 1; i < nr_wide_chars; i++)
{
addtok_wc (parser, parser->wide_char_list[i]);
addtok (parser, FSATOKEN_TK_OR);
}
}
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
}
case FSATOKEN_TK_MBCSET:
{
mbcsets_set_t *work_mbc;
/* Acquire space to store set pointer. Sadly (?), fsaparse
maintains a duplicate list to fsalex at present. */
parser->mbcsets = maybe_realloc (parser->mbcsets,
parser->nmbcsets,
&parser->mbcsets_alloc,
sizeof work_mbc);
/* Get set pointer from fsalex, and add it to the list. */
fsalex_exchange (parser->lex_context,
PROTO_LEXPARSE_OP_GET_MBCSET,
&work_mbc);
parser->mbcsets[parser->nmbcsets++] = work_mbc;
/* We could fall-through case labels to the addtok code below,
but am keeping code sections separate due to an overabundance
of caution. */
addtok (parser, tok);
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
}
case FSATOKEN_TK_ANYCHAR:
case FSATOKEN_TK_BACKREF:
case FSATOKEN_TK_BEGLINE:
case FSATOKEN_TK_ENDLINE:
case FSATOKEN_TK_BEGWORD:
case FSATOKEN_TK_ENDWORD:
case FSATOKEN_TK_LIMWORD:
case FSATOKEN_TK_NOTLIMWORD:
addtok (parser, tok);
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
case FSATOKEN_TK_LPAREN:
parser->lookahead_token = parser->lexer (parser->lex_context);
regexp (parser);
if (parser->lookahead_token != FSATOKEN_TK_RPAREN)
parser->abandon_with_error (_("unbalanced ("));
parser->lookahead_token = parser->lexer (parser->lex_context);
return;
default:
addtok (parser, FSATOKEN_TK_EMPTY);
return;
}
/* NOTREACHED */
}
/* Return the number of tokens in the given subexpression. */
static size_t _GL_ATTRIBUTE_PURE
nsubtoks (fsaparse_ctxt_t *parser, size_t tindex)
{
size_t ntoks1;
switch (parser->tokens[tindex - 1])
{
default:
return 1;
case FSATOKEN_TK_QMARK:
case FSATOKEN_TK_STAR:
case FSATOKEN_TK_PLUS:
return 1 + nsubtoks (parser, tindex - 1);
case FSATOKEN_TK_CAT:
case FSATOKEN_TK_OR:
ntoks1 = nsubtoks (parser, tindex - 1);
return 1 + ntoks1 + nsubtoks (parser, tindex - 1 - ntoks1);
}
}
/* Copy the given subexpression to the top of the tree. */
static void
copytoks (fsaparse_ctxt_t *parser, size_t tindex, size_t ntokens)
{
size_t i;
if (parser->multibyte)
for (i = 0; i < ntokens; ++i)
addtok_mb (parser, parser->tokens[tindex + i], parser->multibyte_prop[tindex + i]);
else
for (i = 0; i < ntokens; ++i)
addtok_mb (parser, parser->tokens[tindex + i], 3);
}
/* Rewriting fsaparse:closure () from scratch; original is clever but a
little tricky to follow, so I'm trying to break up a while + compound-if
loop into a simpler construct (more like a finite-state machine). Also,
edits such as replacing "dfa->" with "parser->" are done here, adding
"parser" as a parameter in lots of places, as well as the long-winded
FSATOKEN_TK_" prefix.
I'm not sure if this version is an improvement over the original; the
need to use "parser->lookahead_token" instead of "tok" influenced my
decision to try this... but the jury is still out. */
static void
closure (fsaparse_ctxt_t *parser)
{
restart_closure:
atom (parser);
for (;;)
{
switch (parser->lookahead_token)
{
case FSATOKEN_TK_QMARK:
case FSATOKEN_TK_STAR:
case FSATOKEN_TK_PLUS:
addtok (parser, parser->lookahead_token);
parser->lookahead_token = parser->lexer (parser->lex_context);
continue;
case FSATOKEN_TK_REPMN:
/* REPMN needs extra work; move outside the switch statement. */
break;
default:
/* Merely let the intial atom call stand as our return result. */
return;
}
/* Deal with REPMN{min, max} cases in a separate block. */
{
int i;
size_t prev_sub_index, ntokens;
int minrep, maxrep;
/* Get the {min, max} pair decoded by the lexer. */
minrep = parser->lex_exchange (parser->lex_context,
PROTO_LEXPARSE_OP_GET_REPMN_MIN,
NULL);
maxrep = parser->lex_exchange (parser->lex_context,
PROTO_LEXPARSE_OP_GET_REPMN_MAX,
NULL);
/* Find out how many tokens are in the peceding token list that are
covered by this REPMN directive. This involves carefully working
backwards through the linear, postfix token ordering. */
ntokens = nsubtoks (parser, parser->tindex);
/* If min and max are both zero, merely remove preceding
subexpression, get a new token, and restart the atom/closure
processing from the top of the function. Not sure if people will
like this goto statement, but we'll give it a whirl. */
if (minrep == 0 && maxrep == 0)
{
parser->tindex -= ntokens;
parser->lookahead_token = parser->lexer (parser->lex_context);
goto restart_closure;
}
/* Non-zero min or max, defined as follows:
{n} The preceding item is matched exactly n times.
{n,} The preceding item is matched n or more times.
{,m} The preceding item is matched at most m times (GNU ext.)
{n,m} The preceding item is matched at least n, but not more
than m times.
For {n,} and {,m} cases, the omitted parameter is reported here
as a negative value. */
prev_sub_index = parser->tindex - ntokens;
if (maxrep < 0)
addtok (parser, FSATOKEN_TK_PLUS);
if (minrep == 0)
addtok (parser, FSATOKEN_TK_QMARK);
for (i = 1; i < minrep; ++i)
{
copytoks (parser, prev_sub_index, ntokens);
addtok (parser, FSATOKEN_TK_CAT);
}
for (; i < maxrep; ++i)
{
copytoks (parser, prev_sub_index, ntokens);
addtok (parser, FSATOKEN_TK_QMARK);
addtok (parser, FSATOKEN_TK_CAT);
}
/* Prime the parser with the next token after REPMN and loop. */
parser->lookahead_token = parser->lexer (parser->lex_context);
}
}
}
static void
branch (fsaparse_ctxt_t *parser)
{
fsatoken_token_t tok;
closure (parser);
tok = parser->lookahead_token;
while (tok != FSATOKEN_TK_RPAREN && tok != FSATOKEN_TK_OR && tok >= 0)
{
closure (parser);
tok = parser->lookahead_token;
addtok (parser, FSATOKEN_TK_CAT);
}
}
static void
regexp (fsaparse_ctxt_t *parser)
{
branch (parser);
while (parser->lookahead_token == FSATOKEN_TK_OR)
{
parser->lookahead_token = parser->lexer (parser->lex_context);
branch (parser);
addtok (parser, FSATOKEN_TK_OR);
}
}
/* Main entry point for the parser. Parser is a pointer to a parser
context struct created by fsaparse_new. Before calling this function,
the parser instance must be supplied with a lexer (fsaparse_lexer), and
also with callback functions to receive warning and error reports
(fsaparse_esception_fns). */
void
fsaparse_parse (fsaparse_ctxt_t *parser)
{
/* Obtain an initial token for lookahead, and keep tracking tree depth. */
parser->lookahead_token = parser->lexer (parser->lex_context);
parser->current_depth = parser->depth;
/* Run regexp to manage the next level of parsing. */
regexp (parser);
if (parser->lookahead_token != FSATOKEN_TK_END)
parser->abandon_with_error (_("unbalanced )"));
/* If multiple expressions are parsed, second and subsequent patters are
presented as alternatives to preceding patterns. */
addtok (parser, FSATOKEN_TK_END - parser->nregexps);
addtok (parser, FSATOKEN_TK_CAT);
if (parser->nregexps)
addtok (parser, FSATOKEN_TK_OR);
++parser->nregexps;
}
/* Receive functions to deal with exceptions detected by the parser:
Warnings and errors. Internally, we add the _Noreturn attribute
to the error callback, to help the compiler with code flow
analysis. */
extern void
fsaparse_exception_fns (fsaparse_ctxt_t *parser,
fsaparse_warn_callback_fn *warningfn,
fsaparse_error_callback_fn *errorfn)
{
/* Exception handling is done by explicit callbacks. */
parser->warn_client = warningfn;
parser->abandon_with_error = errorfn;
}
/* Add "not provided!" stub function that gets called if the client
fails to provide proper resources. This is a hack, merely to get the
module started; better treatment needs to be added later. */
static void
no_function_provided (void *unused)
{
assert (!"fsaparse: Plug-in function required, but not provided.");
}
/* Receiver a lexer function, plus lexer instance context pointer, for use by
the parser. Although not needed initially, this plug-in architecture may
be useful in the future, and it breaks up some of the intricate
connections that made the original dfa.c code so daunting. */
void
fsaparse_lexer (fsaparse_ctxt_t *parser,
void *lexer_context,
proto_lexparse_lex_fn_t *lex_fn,
proto_lexparse_exchange_fn_t *lex_exchange_fn)
{
bool is_multibyte;
int wchars_max;
/* Record supplied lexer function and context for use later. */
parser->lex_context = lexer_context;
parser->lexer = lex_fn;
parser->lex_exchange = lex_exchange_fn;
/* Query lexer to get multibyte nature of this locale. */
is_multibyte
= lex_exchange_fn (lexer_context,
PROTO_LEXPARSE_OP_GET_IS_MULTIBYTE_LOCALE,
NULL);
parser->multibyte_locale = is_multibyte;
parser->unibyte_locale = ! is_multibyte;
/* Set up WCHAR token infrastructure: Wide-char list/array/whatever. We
free the wide_char_list first, in case someone calls this function
multiple times, so that earlier allocations are correctly freed up, and
we use the value advised by the most recent lexer presented to us. */
wchars_max = lex_exchange_fn (lexer_context,
PROTO_LEXPARSE_OP_GET_WIDE_CHAR_LIST_MAX,
NULL);
parser->wide_chars_max = wchars_max;
free (parser->wide_char_list);
parser->wide_char_list = NULL;
if (wchars_max)
parser->wide_char_list = xnmalloc (wchars_max, sizeof (wchar_t));
}
/* Generate a new instance of an FSA parser. */
fsaparse_ctxt_t *
fsaparse_new (void)
{
fsaparse_ctxt_t *new_context;
/* Acquire zeroed memory for new parser context. */
new_context = XZALLOC (fsaparse_ctxt_t);
/* ?? Point warning, error and lexer functions to a "you need to tell me
these first!" function? */
new_context->warn_client = (fsaparse_warn_callback_fn *)
no_function_provided;
new_context->abandon_with_error = (fsaparse_error_callback_fn *)
no_function_provided;
new_context->lexer = (fsaparse_lexer_fn_t *)
no_function_provided;
/* Default to unibyte locale... but we synchronise with the lexer later. */
new_context->multibyte_locale = false;
new_context->unibyte_locale = true;
/* Add this instance at the head of the module's list. */
new_context->next_instance = fsaparse_instance_list_head;
fsaparse_instance_list_head = new_context;
return new_context;
}
/* After parsing, report a list of tokens describing the pattern. Complex
structures such as alternation, backreferences, and locale-induced
complexity such as variable-length utf8 sequences are described here by
appending operators that apply to the preceding item(s) (postfix
notation). */
void
fsaparse_get_token_list (fsaparse_ctxt_t *parser,
size_t *nr_tokens,
fsatoken_token_t **token_list)
{
*nr_tokens = parser->tindex;
*token_list = parser->tokens;
}
/* Internal function to free all resources directly or indirectly used by an
instance. The pointer is no longer valid after this call. */
static void
free_instance (fsaparse_ctxt_t *parser)
{
free (parser->wide_char_list);
free (parser->tokens);
free (parser->multibyte_prop);
free (parser);
}
/* Destroy all parser instances, plus any associated resources owned by the
module. */
void
fsaparse_destroy_module (void)
{
fsaparse_ctxt_t *p_list;
fsaparse_ctxt_t *p_next;
/* Move the global list head into a local variable, and immediately clear
the global. This is a half-hearted attempt to avoid race conditions;
to do things properly, a system-wide atomic operation (locked,
including multi-CPU cache coherency) operation should be used. */
p_list = fsaparse_instance_list_head;
fsaparse_instance_list_head = NULL;
/* Traverse the list of instances, releasing all resources associated
with each one. */
while (p_list)
{
p_next = p_list->next_instance;
free_instance (p_list);
p_list = p_next;
}
}
/* Prepare module for operation. */
void
fsaparse_initialise (void)
{
/* Initialise the linked list of instances created by this module. */
fsaparse_instance_list_head = NULL;
/* Clean up resources upon exit. */
atexit (fsaparse_destroy_module);
}
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="fsaparse.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsaparse.h"
/* fsaparse -- Build a structure naming relationships (sequences,
alternatives, options and precedence) of tokens
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* This function receives a stream of tokens from fsalex, and
processes them to impose precedence rules and to describe complex
pattern elements that are beyond the capability of the simple
lexer. In addition to the cases explicit in the syntax (e.g.
"(ab|c)", variable-length multibyte encodings (UTF-8; codesets
including modifiers and/or shift items) also require these enhanced
facilities. */
#ifndef FSAPARSE_H
#define FSAPARSE_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
#include "fsatoken.h"
#include "proto-lexparse.h"
/* Multiple parser instances can exist in parallel, so define an opaque
type to collect together all the context relating to each instance. */
typedef struct fsaparse_ctxt_struct fsaparse_ctxt_t;
/* Allow configurable parser/lexer combinations by using a plugin interface
for lexer invocation. */
typedef fsatoken_token_t
fsaparse_lexer_fn_t (void *lexer_context);
/* Functions to initialise and tear down entire module and all associated
instances. */
/* Prepare module for operation. */
extern void
fsaparse_initialise (void);
/* Destroy all parser instances, plus any associated resources owned by the
module. */
extern void
fsaparse_destroy_module (void);
/* Functions to create a new parser instance, and to connect it with
a compatible lexer instance. */
/* Generate a new instance of an FSA parser. */
extern fsaparse_ctxt_t *
fsaparse_new (void);
/* Receiver a lexer function, plus lexer instance context pointer, for use by
the parser. Although not needed initially, this plug-in architecture may
be useful in the future, and it breaks up some of the intricate
connections that made the original dfa.c code so daunting. */
extern void
fsaparse_lexer (fsaparse_ctxt_t *parser,
void *lexer_context,
proto_lexparse_lex_fn_t *lex_fn,
proto_lexparse_exchange_fn_t *lex_exchange_fn);
/* Define function prototypes for warning and error callbacks. */
typedef void
fsaparse_warn_callback_fn (const char *);
typedef void /* ?? _Noreturn? */
fsaparse_error_callback_fn (const char *);
/* Receive functions to deal with exceptions detected by the parser:
Warnings and errors. Internally, we add the _Noreturn attribute
to the error callback, to help the compiler with code flow
analysis. */
extern void
fsaparse_exception_fns (fsaparse_ctxt_t *parser,
fsaparse_warn_callback_fn *warningfn,
fsaparse_error_callback_fn *errorfn);
/* Main entry point for the parser. Parser is a pointer to a parser
context struct created by fsaparse_new. Before calling this function,
the parser instance must be supplied with a lexer (fsaparse_lexer), and
also with callback functions to receive warning and error reports
(fsaparse_esception_fns). */
extern void
fsaparse_parse (fsaparse_ctxt_t *parser);
/* After parsing, report a list of tokens describing the pattern. Complex
structures such as alternation, backreferences, and locale-induced
complexity such as variable-length utf8 sequences are described here by
appending operators that apply to the preceding item(s) (postfix
notation). */
extern void
fsaparse_get_token_list (fsaparse_ctxt_t *parser,
size_t *nr_tokens,
fsatoken_token_t **token_list);
#endif /* FSAPARSE_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="fsatoken.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsatoken.c"
/* fsatoken - Support routines specific to token definitions
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* The majority of the fsatoken[ch] module is in fsatoken.h, as it is
shared by other modules. This file provides token-specific support
functions, such as functions to print tokens (for debugging).
Although there is a relationship between some generic constructs
such as character classes and the CSET token defined here, the
generic items are defined in a separate support library, not in this
module. This is because these tokens are very FSA/grep-specific,
whereas the generic consructs are potentially widely useable, and may
even be amenable to hardware-specific optimisations (such as
superscalar opcodes such as:
- and/or/set/clear/test-and-set/test-and-clear; and/or
- bit counting
operations). */
/* Always import environment-specific configuration items first. */
#include <config.h>
#include "fsatoken.h"
#include <stdio.h>
#ifdef DEBUG
void
fsatoken_prtok (fsatoken_token_t t)
{
char const *s;
if (t < 0)
fprintf (stderr, "FSATOKEN_TK_END");
else if (t < FSATOKEN_NOTCHAR)
{
int ch = t;
fprintf (stderr, "%c", ch);
}
else
{
switch (t)
{
case FSATOKEN_TK_EMPTY:
s = "FSATOKEN_TK_EMPTY";
break;
case FSATOKEN_TK_BACKREF:
s = "FSATOKEN_TK_BACKREF";
break;
case FSATOKEN_TK_BEGLINE:
s = "FSATOKEN_TK_BEGLINE";
break;
case FSATOKEN_TK_ENDLINE:
s = "FSATOKEN_TK_ENDLINE";
break;
case FSATOKEN_TK_BEGWORD:
s = "FSATOKEN_TK_BEGWORD";
break;
case FSATOKEN_TK_ENDWORD:
s = "FSATOKEN_TK_ENDWORD";
break;
case FSATOKEN_TK_LIMWORD:
s = "FSATOKEN_TK_LIMWORD";
break;
case FSATOKEN_TK_NOTLIMWORD:
s = "FSATOKEN_TK_NOTLIMWORD";
break;
case FSATOKEN_TK_QMARK:
s = "FSATOKEN_TK_QMARK";
break;
case FSATOKEN_TK_STAR:
s = "FSATOKEN_TK_STAR";
break;
case FSATOKEN_TK_PLUS:
s = "FSATOKEN_TK_PLUS";
break;
case FSATOKEN_TK_CAT:
s = "FSATOKEN_TK_CAT";
break;
case FSATOKEN_TK_OR:
s = "FSATOKEN_TK_OR";
break;
case FSATOKEN_TK_LPAREN:
s = "FSATOKEN_TK_LPAREN";
break;
case FSATOKEN_TK_RPAREN:
s = "FSATOKEN_TK_RPAREN";
break;
case FSATOKEN_TK_ANYCHAR:
s = "FSATOKEN_TK_ANYCHAR";
break;
case FSATOKEN_TK_MBCSET:
s = "FSATOKEN_TK_MBCSET";
break;
default:
s = "FSATOKEN_TK_CSET";
break;
}
fprintf (stderr, "%s", s);
}
}
#endif /* DEBUG */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="fsatoken.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="fsatoken.h"
/* fsatoken - Create tokens for a compact, coherent regular expression language
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* Regular expression patterns are presented as text, possibly ASCII;
the format is very expressive, but this comes at the cost of being
somewhat expensive to interpret (including identifying invalid
patterns). By tokenising the pattern, we make life much easier for
the parser and other search machinery that follows.
This file defines the tokens that we use, both for the benefit of
the lexer/parser/dfa analyser that share this information, and for
other machinery (such as the C compiler) that may need to store
and/or manipulate these items. */
#ifndef FSATOKEN_H
#define FSATOKEN_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
/* Obtain defn. of ptrdiff_t from stddef.h, and CHAR_BIT from limits.h */
#include <limits.h>
#include <stddef.h>
/* C stream octets, and non-stream EOF, are self-representing tokens.
We need to include stdio.h to obtain the definition of EOF. */
#include <stdio.h>
/* First integer value that is greater than any character code. */
enum { FSATOKEN_NOTCHAR = 1 << CHAR_BIT };
/* The regexp is parsed into an array of tokens in postfix form. Some tokens
are operators and others are terminal symbols. Most (but not all) of these
codes are returned by the lexical analyzer. */
typedef ptrdiff_t fsatoken_token_t;
/* Predefined token values. */
enum
{
FSATOKEN_TK_END = -1, /* FSATOKEN_TK_END is a terminal symbol that matches the
end of input; any value of FSATOKEN_TK_END or less in
the parse tree is such a symbol. Accepting
states of the DFA are those that would have
a transition on FSATOKEN_TK_END. */
/* Ordinary character values are terminal symbols that match themselves. */
FSATOKEN_TK_EMPTY = FSATOKEN_NOTCHAR, /* FSATOKEN_TK_EMPTY is a terminal symbol that matches
the empty string. */
FSATOKEN_TK_BACKREF, /* FSATOKEN_TK_BACKREF is generated by \<digit>
or by any other construct that
is not completely handled. If the scanner
detects a transition on backref, it returns
a kind of "semi-success" indicating that
the match will have to be verified with
a backtracking matcher. */
FSATOKEN_TK_BEGLINE, /* FSATOKEN_TK_BEGLINE is a terminal symbol that matches
the empty string at the beginning of a
line. */
FSATOKEN_TK_ENDLINE, /* FSATOKEN_TK_ENDLINE is a terminal symbol that matches
the empty string at the end of a line. */
FSATOKEN_TK_BEGWORD, /* FSATOKEN_TK_BEGWORD is a terminal symbol that matches
the empty string at the beginning of a
word. */
FSATOKEN_TK_ENDWORD, /* FSATOKEN_TK_ENDWORD is a terminal symbol that matches
the empty string at the end of a word. */
FSATOKEN_TK_LIMWORD, /* FSATOKEN_TK_LIMWORD is a terminal symbol that matches
the empty string at the beginning or the
end of a word. */
FSATOKEN_TK_NOTLIMWORD, /* FSATOKEN_TK_NOTLIMWORD is a terminal symbol that
matches the empty string not at
the beginning or end of a word. */
FSATOKEN_TK_QMARK, /* FSATOKEN_TK_QMARK is an operator of one argument that
matches zero or one occurrences of its
argument. */
FSATOKEN_TK_STAR, /* FSATOKEN_TK_STAR is an operator of one argument that
matches the Kleene closure (zero or more
occurrences) of its argument. */
FSATOKEN_TK_PLUS, /* FSATOKEN_TK_PLUS is an operator of one argument that
matches the positive closure (one or more
occurrences) of its argument. */
FSATOKEN_TK_REPMN, /* FSATOKEN_TK_REPMN is a lexical token corresponding
to the {m,n} construct. FSATOKEN_TK_REPMN never
appears in the compiled token vector. */
FSATOKEN_TK_CAT, /* FSATOKEN_TK_CAT is an operator of two arguments that
matches the concatenation of its
arguments. FSATOKEN_TK_CAT is never returned by the
lexical analyzer. */
FSATOKEN_TK_OR, /* FSATOKEN_TK_OR is an operator of two arguments that
matches either of its arguments. */
FSATOKEN_TK_LPAREN, /* FSATOKEN_TK_LPAREN never appears in the parse tree,
it is only a lexeme. */
FSATOKEN_TK_RPAREN, /* FSATOKEN_TK_RPAREN never appears in the parse tree. */
FSATOKEN_TK_ANYCHAR, /* FSATOKEN_TK_ANYCHAR is a terminal symbol that matches
a valid multibyte (or single byte) character.
It is used only if MB_CUR_MAX > 1. */
FSATOKEN_TK_MBCSET, /* FSATOKEN_TK_MBCSET is similar to FSATOKEN_TK_CSET, but for
multibyte characters. */
FSATOKEN_TK_WCHAR, /* Only returned by lex. wctok contains
the wide character representation. */
FSATOKEN_TK_CSET /* FSATOKEN_TK_CSET and (and any value greater) is a
terminal symbol that matches any of a
class of characters. */
};
/* prtok - Display token name (for debugging) */
#ifdef DEBUG
extern void
fsatoken_prtok (fsatoken_token_t t);
#endif /* DEBUG */
#endif /* FSATOKEN_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-csrc;
name="mbcsets.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="mbcsets.c"
/* mbcsets -- Handle multi-byte and/or locale-dependent sets of chars.
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Repackaged by "untangle" script, written by behoffski. */
/* ?? Need to document things here...
*/
/* Always import environment-specific configuration items first. */
#include <config.h>
#include "charclass.h"
#include "mbcsets.h"
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <wctype.h>
#include "xalloc.h"
/* Flesh out opaque type given in the header. */
struct mbcsets_set_struct
{
/* Singly-linked list of all instances, so destroy_module can release
all resources by traversing the list. */
mbcsets_set_t *next_instance;
charclass_t *charclass;
bool invert;
wchar_t *chars; /* Normal characters. */
size_t nchars;
size_t chars_alloc;
wctype_t *ch_classes; /* Character classes. */
size_t nch_classes;
size_t ch_classes_alloc;
struct /* Range characters. */
{
wchar_t beg; /* Range start. */
wchar_t end; /* Range end. */
} *ranges;
size_t nranges;
size_t ranges_alloc;
char **equivs; /* Equivalence classes. */
size_t nequivs;
size_t equivs_alloc;
char **coll_elems;
size_t ncoll_elems; /* Collating elements. */
size_t coll_alloc;
};
/* Linked list of all instances created by this module. */
static mbcsets_set_t *mbcsets_instances_list_head = NULL;
/* Ensure that the array addressed by PTR holds at least NITEMS +
(PTR || !NITEMS) items. Either return PTR, or reallocate the array
and return its new address. Although PTR may be null, the returned
value is never null.
The array holds *NALLOC items; *NALLOC is updated on reallocation.
ITEMSIZE is the size of one item. Avoid O(N**2) behavior on arrays
growing linearly. */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
if (nitems < *nalloc)
return ptr;
*nalloc = nitems;
return x2nrealloc (ptr, nalloc, itemsize);
}
/* By default, classes match the specified characters. Regular
expressions allow this sense to be inverted, usually by the
convention of "^" being the first character of a bracketed class.
By default, positive sense is selected; this function lets the
user specify the sense, probably to specify inverted matching. */
void
mbcsets_set_match_sense (mbcsets_set_t *mbc, bool invert)
{
mbc->invert = invert;
}
/* Individual wide characters. */
void
mbcsets_add_wchar (mbcsets_set_t *mbc, wint_t wc)
{
/* ?? Quietly ignore WEOF... is this reasonable? */
if (wc == WEOF)
return;
/* Could this character fit into a charclass set? */
/* ?? We don't know directly the signedness of wint_t; in gnulib, it
is unsigned, and so testing for "wc >= 0" is flagged as an error
as it is redundant. Have removed the test here for now, but not
confident that this is the best treatment. */
if (/* wc >= 0 && */ wc < CHARCLASS_NOTCHAR)
{
/* Yes, does this wide char have a valid unichar representation? */
/* ?? The documentation states that wctob should not be used;
perhaps the unibyte cache mbrtowc_cache in fsalex, built by
initialise_uchar_to_wc_cache (), might be relevant here? */
int b = wctob (wc);
if (b != EOF)
{
/* Yes, add the char (byte?) (octet?) to the charclass set. */
charclass_setbit (b, mbc->charclass);
return;
}
}
/* Ensure we have space to store another character. */
mbc->chars = maybe_realloc(mbc->chars, mbc->nchars,
&mbc->chars_alloc,
sizeof *mbc->chars);
/* Add the character to the list. */
mbc->chars[mbc->nchars++] = wc;
}
/* Add a list of wide characters (note: not wide integers). */
void
mbcsets_add_wchar_list (mbcsets_set_t *mbc,
size_t len, wchar_t *wc_list)
{
size_t i;
/* Ensure we have space to store the incoming list element(s). */
mbc->chars = maybe_realloc(mbc->chars, mbc->nchars + len,
&mbc->chars_alloc,
sizeof *mbc->chars);
/* Add all the characters to the list. (?? use memcpy here?) */
for (i = 0; i < len; i++)
mbc->chars[mbc->nchars++] = wc_list[i];
}
/* Common character classes, e.g. alpha, digit, punct etc. */
void
mbcsets_add_class (mbcsets_set_t *mbc, wctype_t wchar_class)
{
/* ?? We don't check that the descriptor is valid. */
/* Ensure we have space to store another class descriptor. */
mbc->ch_classes = maybe_realloc(mbc->ch_classes, mbc->nch_classes,
&mbc->ch_classes_alloc,
sizeof *mbc->ch_classes);
/* Add the class descriptor to the list. */
mbc->ch_classes[mbc->nch_classes++] = wchar_class;
}
/* Explicit character ranges. */
void
mbcsets_add_range (mbcsets_set_t *mbc, wint_t beg, wint_t end)
{
/* ?? We don't check that the begin/end chars are valid. */
/* Ensure we have space to store another begin/end char pair. */
mbc->ranges = maybe_realloc(mbc->ranges, mbc->nranges + 1,
&mbc->ranges_alloc,
sizeof *mbc->ranges);
/* Add the range to the list. */
mbc->ranges[mbc->nranges].beg = beg;
mbc->ranges[mbc->nranges++].end = end;
}
/* Receive an "in-work" character class, which may or may not have
members. Mbcset takes ownership of this set, and, depending on
the circumstances, either maintains it internally, or else copies
its contents (if any) to its internals, and releases (abandons)
the supplied set. This function must not applied to a set that
has been completed. */
void
mbcsets_receive_incomplete_charclass (mbcsets_set_t *mbc,
charclass_t *ccl)
{
charclass_unionset (ccl, mbc->charclass);
charclass_abandon (ccl);
}
/* Mark a set as completed; the implementation may also analyse and
optimise the set at this point (e.g. use charclasses to represent
unibyte characters; merge overlapping ranges; remove the
individual listing of a character if it is covered by a range,
etc.)
In addition, note that no further changes (e.g. receive another
incomplete charclass) are allowed for this set, once "completed"
is called. */
void
mbcsets_completed (mbcsets_set_t *mbc)
{
charclass_t *zeroset;
/* Did we end up putting anything into the charclass? */
zeroset = charclass_get_pointer (0);
if (charclass_equal (mbc->charclass, zeroset))
{
/* No, abandon the class, and use NULL as our sentinel. */
charclass_abandon (mbc->charclass);
mbc->charclass = NULL;
}
else
{
/* Yes, complete the class, and obtain a persistent pointer. */
charclass_index_t index;
index = charclass_completed (mbc->charclass);
mbc->charclass = charclass_get_pointer (index);
}
}
/* Retrieve high-level information about the class, which is useful
(in fsaparse) for deciding on how to deal with it. We are forced
to provide significant query resources since we demand that the
type internal remain opaque (even though the initial
implementation may do a poor job of this effort). */
void
mbcsets_get_characteristics (mbcsets_set_t *mbc,
bool *p_invert,
charclass_t **pp_charclass,
size_t *p_nchars,
size_t *p_nch_classes,
size_t *p_nranges,
size_t *p_nequivs,
size_t *p_ncoll_elems)
{
*p_invert = mbc->invert;
*pp_charclass = mbc->charclass;
*p_nchars = mbc->nchars;
*p_nch_classes = mbc->nch_classes;
*p_nranges = mbc->nranges;
*p_nequivs = mbc->nequivs;
*p_ncoll_elems = mbc->ncoll_elems;
}
/* Copy wide char list to caller's work area. */
void
mbcsets_get_chars (mbcsets_set_t *mbc, wchar_t *char_list)
{
memcpy (char_list, mbc->chars, mbc->nchars * sizeof(*char_list));
}
/* Prepare module for operation. */
void
mbcsets_initialise (void)
{
/* Initialise the linked list of instances created by this module. */
mbcsets_instances_list_head = NULL;
atexit (mbcsets_destroy_module);
}
/* Internal function to free all resources directly or indirectly used by
an instance. The pointer is no longer valid after this call. */
static void
free_instance (mbcsets_set_t *mbc)
{
size_t i;
free (mbc->chars);
free (mbc->ch_classes);
free (mbc->ranges);
for (i = 0; i < mbc->nequivs; ++i)
free (mbc->equivs[i]);
free (mbc->equivs);
for (i = 0; i < mbc->ncoll_elems; ++i)
free (mbc->coll_elems[i]);
free (mbc->coll_elems);
free (mbc);
}
/* Destroy all classes, plus any associated resources owned by the
module. */
void
mbcsets_destroy_module (void)
{
mbcsets_set_t *p_list;
mbcsets_set_t *p_next;
/* Move the global list head into a local variable, and immediately
clear the global. This is a half-hearted attempt to avoid race
conditions; to do things properly, a system-wide atomic operation
(locked, including multi-CPU cache coherency) operation should be
used. */
p_list = mbcsets_instances_list_head;
mbcsets_instances_list_head = NULL;
/* Traverse the list of instances, releasing all resources associated
with each one. */
while (p_list)
{
p_next = p_list->next_instance;
free_instance (p_list);
p_list = p_next;
}
}
/* Generate a new instance of a multibyte-character set descriptor. */
mbcsets_set_t *
mbcsets_new (void)
{
mbcsets_set_t *new_set;
/* Allocate memory for new instance. */
new_set = xzalloc (sizeof (*new_set));
/* Lint new instance into list of instances created by this module. */
new_set->next_instance = mbcsets_instances_list_head;
mbcsets_instances_list_head = new_set;
/* Allocate a charclass set to the instance, so we can easily push
codes into that representation if possible. */
new_set->charclass = charclass_alloc ();
/* Report created instance to the caller. */
return new_set;
}
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="mbcsets.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="mbcsets.h"
/* mbcsets -- Handle multi-byte and/or locale-dependent sets of chars.
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Created by "untangle" script, written by behoffski.
?? more stuff here
*/
#ifndef MBCSETS_H
#define MBCSETS_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
#include "charclass.h"
#include <stddef.h>
#include <wctype.h>
/* Define the multibyte-set descriptor as an opaque type. */
typedef struct mbcsets_set_struct mbcsets_set_t;
/* Prepare module for operation. */
extern void
mbcsets_initialise (void);
/* Destroy all classes, plus any associated resources owned by the
module. */
extern void
mbcsets_destroy_module (void);
/* Generate a new instance of a multibyte-character set descriptor. */
extern mbcsets_set_t *
mbcsets_new (void);
/* By default, classes match the specified characters. Regular
expressions allow this sense to be inverted, usually by the
convention of "^" being the first character of a bracketed class.
By default, positive sense is selected; this function lets the
user specify the sense, probably to specify inverted matching. */
extern void
mbcsets_set_match_sense (mbcsets_set_t *mbc, bool invert);
/* Individual wide characters. */
extern void
mbcsets_add_wchar (mbcsets_set_t *mbc, wint_t wc);
/* Add a list of wide characters (note: not wide integers). */
extern void
mbcsets_add_wchar_list (mbcsets_set_t *mbc,
size_t len, wchar_t *wc_list);
/* Common character classes, e.g. alpha, digit, punct etc. */
extern void
mbcsets_add_class (mbcsets_set_t *mbc, wctype_t wchar_class);
/* Explicit character ranges. */
extern void
mbcsets_add_range (mbcsets_set_t *mbc, wint_t beg, wint_t end);
/* Receive an "in-work" character class, which may or may not have
members. Mbcset takes ownership of this set, and, depending on
the circumstances, either maintains it internally, or else copies
its contents (if any) to its internals, and releases (abandons)
the supplied set. This function must not applied to a set that
has been completed. */
extern void
mbcsets_receive_incomplete_charclass (mbcsets_set_t *mbc,
charclass_t *ccl);
/* Retrieve high-level information about the class, which is useful
(in fsaparse) for deciding on how to deal with it. We are forced
to provide significant query resources since we demand that the
type internal remain opaque (even though the initial
implementation may do a poor job of this effort). */
extern void
mbcsets_get_characteristics (mbcsets_set_t *mbc,
bool *p_invert,
charclass_t **pp_charclass,
size_t *p_nchars,
size_t *p_nch_classes,
size_t *p_nranges,
size_t *p_nequivs,
size_t *p_ncoll_elems);
/* Copy wide char list to caller's work area. */
extern void
mbcsets_get_chars (mbcsets_set_t *mbc, wchar_t *char_list);
/* Mark a set as completed; the implementation may also analyse and
optimise the set at this point (e.g. use charclasses to represent
unibyte characters; merge overlapping ranges; remove the
individual listing of a character if it is covered by a range,
etc.)
In addition, note that no further changes (e.g. receive another
incomplete charclass) are allowed for this set, once "completed"
is called. */
extern void
mbcsets_completed (mbcsets_set_t *mbc);
#endif /* MBCSETS_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/x-chdr;
name="proto-lexparse.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="proto-lexparse.h"
/* proto-lexparse -- Define how lexer and parser can interact.
Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc.,
51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA */
/* Written June, 1988 by Mike Haertel
Modified July, 1988 by Arthur David Olson to assist BMG speedups */
/* 2014: Created by "untangle" script, written by behoffski.
(EVENTUALLYREWRITEME: A very lengthy editorial/discussion of this module
follows, partially interspersed with a description of its intended
function. Comments and criticism are welcome; if adopted, I expect that
this comment block will be rewritten to be much more focussed and much
shorter.)
This file has been added very, very late in the non-linear process of
developing this code. It addresses a feature that was not anticipated early
on, but which I believe is worth exploring in the future: The presence of
pluggable lexers to supply to the parser, or, equivalently, breaking up the
master/slave relationship between the parser and the lexer, and replacing it
with a peer-to-peer conversation (perhaps the software equivalent of a
shared bus).
Quite a bit of the need for this protocol/interface module is that I've
demanded that the individual modules strictly hide their internals, and only
share information and/or control of resources (such as the parser calling
lex ()) in an explicit fashion. Early on, the nature of connections between
the lexer and the parser appeared with the need to communicate {min,max}
values as part of the REPMN token; then the need to communicate the wide
character implied by the WCHAR token.
The crisis came arose from an early decision that at the time seemed fairly
mild, but gradually grew and grew over time: when I decided to extend the
meaning of the fsalex_syntax () call to not only include the parameters
named in the call (reg_syntax_t, case folding, eol char), but to also
capture the locale in force at that point, and make the lexer's behaviour
obey that locale, even if the locale was subsequently changed.
Another example of data interchange is that the lexer builds structures to
model multibyte character sets, but (at the time or writing) does not provide
an interface for clients to access this information. The need for many and
varied interface function has been steadily growing, and this is just another
example.
The breaking point came when I saw that the parser needed to know if it was
in a unibyte or multibyte locale (currently done by testing MB_CUR_MAX > 1).
If the lexer was the authority on the locale, then the parser needed to
query the lexer, since, because of the pluggable-lexer architecture I'd
created, the lexer was already the authority, but needed to share its
information with others. The number of specialised get/set functions needed
was out of control, and, if a pluggable architecture was to be maintained,the
number had to be drastically reduced.
Hence, this module, that defines the interface/protocol that any lexer must
serve, and which allows a lexer and its client to exchange information and
negotiate in a flexible fashion. This comes at the extent of type safety in
many cases, but makes data exchange more formal.
This module only models essential exchanges between the parser and lexer
instances, and encourages the user to negotiate directly with the modules
for unrelated topics. For example, the parser does not need to know what the
original plain-text pattern was; it receives a full description of the
pattern via the lexer tokens and associated parameters/structures. So, there
is no "pattern" interchange described here: The client can work directly
with the lexer.
The design is to split interactions between modules into two sets: For code
that is so central to the design of the modules on each side, where an
indirect approach would tend to significantly complicate the exchange, and/or
where efficiency is a central concern, then a direct function is provided to
facilitate each such possible exchange.
The other set consists of the majority of cases (by type, but usually not by
runtime volume): For these cases, a generic "exchange" function is provided
by the lexer, and is plugged into the parser. This function has parameters
and return values that permit the parties to exchange information, although
some strict type safety is sacrificed where memory pointers can cross the
boundary. A slightly simplified definition of the exchange function is:
int lexer_provided_exchange (void *lexer_instance_context,
exchange_opcode_enum opcode,
void *generic_parameter);
The opcode defines what meaning (if any) is assigned to the parameter and/or
to the return value. The exchange function, these opcodes, and the required
meaning the values involved are formally defined by this module, and both the
lexer and the parser must conform to this protocol/interface. By having this
module as an independent entity, it reinforces the independence of each party
in the arrangement, and can ease the creation of alternatives (e.g. perhaps
use a much simpler lexer for "grep -F" in a unibyte locale?).
On the downside, some of the initial opcodes/exchanges I'm writing here are
obviously directly lifted from the existing dfa.c lexer/parser interactions
(e.g. the multibyte sets struct(s)). I'm hoping that this will not be a
fatal flaw, but in the final analysis this module is a side-effect of
requiring a pluggable lexer architecture, and this may not be acceptable to
others. */
#ifndef PROTO_LEXPARSE_H
#define PROTO_LEXPARSE_H 1
/* Always import environment-specific configuration items first. */
#include <config.h>
/* The lexer returns a token, defined by fsatoken. */
#include "fsatoken.h"
/* Define opcodes for lexer/parser exchanges. */
typedef enum proto_lexparse_opcode_enum
{
/* Possible future opcode: PROTO_LEXPARSE_OP_GET_LOCALE, entire
locale from uselocale/duplocale */
PROTO_LEXPARSE_OP_GET_IS_MULTIBYTE_LOCALE,
PROTO_LEXPARSE_OP_GET_REPMN_MIN,
PROTO_LEXPARSE_OP_GET_REPMN_MAX,
PROTO_LEXPARSE_OP_GET_WIDE_CHAR_LIST_MAX,
PROTO_LEXPARSE_OP_GET_WIDE_CHARS,
PROTO_LEXPARSE_OP_GET_DOTCLASS,
PROTO_LEXPARSE_OP_GET_MBCSET,
} proto_lexparse_opcode_t;
/* Most opcodes above are self-documenting; the "WIDE_CHAR(S)"
items require a little more explanation:
1. When the lexer returns a WCHAR token, there is at least one
explicit character (specified in the pattern), plus a list of
zero or more alternate characters that also match (usually
because case folding is selected). The parser should use
GET_WIDE_CHARS in response to a WCHAR token to fetch this list
(including the explicit character as the first element of the
list); and
2. The GET_WIDE_CHAR_LIST_MAX is provided so that the parser can
allocate storage for the longest possible list, either as a
once-off allocation during startup, or on a case-by-case
basis. */
/* Declare prototypes for main lex function (lex (), fetch a token),
and the exchange function. */
typedef fsatoken_token_t proto_lexparse_lex_fn_t (void *lexer_context);
typedef int proto_lexparse_exchange_fn_t (void *lexer_context,
proto_lexparse_opcode_t opcode,
void *parameter);
#endif /* PROTO_LEXPARSE_H */
/* vim:set shiftwidth=2: */
--------------050008000707080900010407
Content-Type: text/plain; charset=us-ascii;
name="Makefile.am"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="Makefile.am"
IyMgUHJvY2VzcyB0aGlzIGZpbGUgd2l0aCBhdXRvbWFrZSB0byBjcmVhdGUgTWFrZWZpbGUu
aW4KIyBDb3B5cmlnaHQgMTk5Ny0xOTk4LCAyMDA1LTIwMTQgRnJlZSBTb2Z0d2FyZSBGb3Vu
ZGF0aW9uLCBJbmMuCiMKIyBUaGlzIHByb2dyYW0gaXMgZnJlZSBzb2Z0d2FyZTsgeW91IGNh
biByZWRpc3RyaWJ1dGUgaXQgYW5kL29yIG1vZGlmeQojIGl0IHVuZGVyIHRoZSB0ZXJtcyBv
ZiB0aGUgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgYXMgcHVibGlzaGVkIGJ5CiMgdGhl
IEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbjsgZWl0aGVyIHZlcnNpb24gMywgb3IgKGF0IHlv
dXIgb3B0aW9uKQojIGFueSBsYXRlciB2ZXJzaW9uLgojCiMgVGhpcyBwcm9ncmFtIGlzIGRp
c3RyaWJ1dGVkIGluIHRoZSBob3BlIHRoYXQgaXQgd2lsbCBiZSB1c2VmdWwsCiMgYnV0IFdJ
VEhPVVQgQU5ZIFdBUlJBTlRZOyB3aXRob3V0IGV2ZW4gdGhlIGltcGxpZWQgd2FycmFudHkg
b2YKIyBNRVJDSEFOVEFCSUxJVFkgb3IgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFSIFBVUlBP
U0UuICBTZWUgdGhlCiMgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgZm9yIG1vcmUgZGV0
YWlscy4KIwojIFlvdSBzaG91bGQgaGF2ZSByZWNlaXZlZCBhIGNvcHkgb2YgdGhlIEdOVSBH
ZW5lcmFsIFB1YmxpYyBMaWNlbnNlCiMgYWxvbmcgd2l0aCB0aGlzIHByb2dyYW0uICBJZiBu
b3QsIHNlZSA8aHR0cDovL3d3dy5nbnUub3JnL2xpY2Vuc2VzLz4uCgpMTiA9IGxuCgpBTV9D
RkxBR1MgPSAkKFdBUk5fQ0ZMQUdTKSAkKFdFUlJPUl9DRkxBR1MpCgojIFRlbGwgdGhlIGxp
bmtlciB0byBvbWl0IHJlZmVyZW5jZXMgdG8gdW51c2VkIHNoYXJlZCBsaWJyYXJpZXMuCkFN
X0xERkxBR1MgPSAkKElHTk9SRV9VTlVTRURfTElCUkFSSUVTX0NGTEFHUykKCmJpbl9QUk9H
UkFNUyA9IGdyZXAKYmluX1NDUklQVFMgPSBlZ3JlcCBmZ3JlcApncmVwX1NPVVJDRVMgPSBn
cmVwLmMgc2VhcmNodXRpbHMuYyBcCiAgICAgICAgICBjaGFyY2xhc3MuYyBmc2F0b2tlbi5j
IGZzYWxleC5jIGZzYXBhcnNlLmMgZnNhbXVzdHMuYyBtYmNzZXRzLmMgZGZhLXBybC5jIGRm
YXNlYXJjaC5jIFwKICAgICAgICAgIGt3c2V0LmMga3dzZWFyY2guYyBcCiAgICAgICAgICBw
Y3Jlc2VhcmNoLmMKbm9pbnN0X0hFQURFUlMgPSBncmVwLmggZGZhLmgga3dzZXQuaCBzZWFy
Y2guaCBzeXN0ZW0uaAoKIyBTb21ldGltZXMsIHRoZSBleHBhbnNpb24gb2YgJChMSUJJTlRM
KSBpbmNsdWRlcyAtbGMgd2hpY2ggbWF5CiMgaW5jbHVkZSBtb2R1bGVzIGRlZmluaW5nIHZh
cmlhYmxlcyBsaWtlICdvcHRpbmQnLCBzbyBsaWJncmVwdXRpbHMuYQojIG11c3QgcHJlY2Vk
ZSAkKExJQklOVEwpIGluIG9yZGVyIHRvIGVuc3VyZSB3ZSB1c2UgR05VIGdldG9wdC4KIyBC
dXQgbGliZ3JlcHV0aWxzLmEgbXVzdCBhbHNvIGZvbGxvdyAkKExJQklOVEwpLCBzaW5jZSBs
aWJpbnRsIHVzZXMKIyByZXBsYWNlbWVudCBmdW5jdGlvbnMgZGVmaW5lZCBpbiBsaWJncmVw
dXRpbHMuYS4KTERBREQgPSBcCiAgLi4vbGliL2xpYmdyZXB1dGlscy5hICQoTElCSU5UTCkg
Li4vbGliL2xpYmdyZXB1dGlscy5hICQoTElCSUNPTlYpIFwKICAkKExJQlRIUkVBRCkKCmdy
ZXBfTERBREQgPSAkKExEQUREKSAkKExJQl9QQ1JFKQpsb2NhbGVkaXIgPSAkKGRhdGFkaXIp
L2xvY2FsZQpBTV9DUFBGTEFHUyA9IC1JJCh0b3BfYnVpbGRkaXIpL2xpYiAtSSQodG9wX3Ny
Y2RpcikvbGliCgpFWFRSQV9ESVNUID0gZG9zYnVmLmMgZWdyZXAuc2gKCmVncmVwIGZncmVw
OiBlZ3JlcC5zaCBNYWtlZmlsZQoJJChBTV9WX0dFTilncmVwPWBlY2hvIGdyZXAgfCBzZWQg
LWUgJyQodHJhbnNmb3JtKSdgCSYmIFwKCWNhc2UgJEAgaW4gZWdyZXApIG9wdGlvbj0tRTs7
IGZncmVwKSBvcHRpb249LUY7OyBlc2FjCSYmIFwKCXNoZWxsX2RvZXNfc3Vic3RyaW5ncz0n
c2V0IHgveSAmJiBkPSQkezElLyp9ICYmIHRlc3QgIiQkZCIgPSB4JyAmJiBcCglpZiAkKFNI
RUxMKSAtYyAiJCRzaGVsbF9kb2VzX3N1YnN0cmluZ3MiIDI+L2Rldi9udWxsOyB0aGVuIFwK
CSAgZWRpdF9zdWJzdHJpbmc9J3MsWCxYLCc7IFwKCWVsc2UgXAoJICBlZGl0X3N1YnN0cmlu
Zz0ncyxcJCR7MCUvXCp9LGBleHByICJYJCQwIiA6ICdcJydYXFwoLipcXCkvJ1wnJ2AsZyc7
IFwKCWZpICYmIFwKCXNlZCAtZSAnc3xbQF1TSEVMTEB8JChTSEVMTCl8ZycgXAoJICAgIC1l
ICIkJGVkaXRfc3Vic3RyaW5nIiBcCgkgICAgLWUgInN8W0BdZ3JlcEB8JCRncmVwfGciIFwK
CSAgICAtZSAic3xbQF1vcHRpb25AfCQkb3B0aW9ufGciIDwkKHNyY2RpcikvZWdyZXAuc2gg
PiRALXQKCSQoQU1fVl9hdCljaG1vZCAreCAkQC10CgkkKEFNX1ZfYXQpbXYgJEAtdCAkQAoK
Q0xFQU5GSUxFUyA9IGVncmVwIGZncmVwICotdAo=
--------------050008000707080900010407--
behoffski <behoffski@HIDDEN>:bug-grep@HIDDEN.
Full text available.bug-grep@HIDDEN:bug#17712; Package grep.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.