GNU logs - #17712, boring messages

Message sent to bug-grep@HIDDEN:

Subject: bug#17712: Files created/modified by "untangle" script
Resent-From: behoffski <behoffski@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-grep@HIDDEN
Resent-Date: Fri, 06 Jun 2014 04:03:02 +0000
Resent-Message-ID: <handler.17712.B.140202733014848 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 17712 <at> debbugs.gnu.org
Message-ID: <53913CF2.2000404@HIDDEN>
Date: Fri, 06 Jun 2014 13:30:50 +0930
From: behoffski <behoffski@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="------------050008000707080900010407"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

This is a multi-part message in MIME format.
--------------050008000707080900010407
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

G'day,

Running untangle, then make, in the grep/src directory, gives the
following output (I've added blank lines around the commands):

------

grep/src $ ./untangle

Creating charclass.h...
Creating charclass.c...
Creating fsatoken.h...
Creating fsatoken.c...
Creating mbcsets.h...
Creating mbcsets.c...
Creating proto-lexparse.h...
Creating fsalex.h...
Creating fsalex.c...
Creating fsaparse.h...
Creating fsaparse.c...
Creating fsamusts.h...
Creating fsamusts.c...
Generating dfa-prl.c by copying dfa.c and then applying edits
Modifying (if needed) Makefile.am; you may need to re-run automake...

grep/src $ make

  cd .. && /bin/sh /home/grep-gnu/grep/build-aux/missing automake-1.13 --gnu src/Makefile
  cd .. && /bin/sh ./config.status src/Makefile depfiles
config.status: creating src/Makefile
config.status: executing depfiles commands
   CC       charclass.o
   CC       fsatoken.o
   CC       fsalex.o
   CC       fsaparse.o
   CC       fsamusts.o
   CC       mbcsets.o
   CC       dfa-prl.o
   CCLD     grep
   GEN      egrep
   GEN      fgrep

grep/src $

------

Attached are the files created or modified by the script.

cheers,

behoffski (Brenton Hoff)
Programmer, Grouse Software

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="charclass.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="charclass.c"

/* charclass -- Tools to create and manipulate sets of C "char"s

This module provides tools to create, modify, store and retrieve character
classes, and provides tools tuned to the needs of RE lexical analysers.

The class itself is an opaque type, referenced by a pointer while under
construction, and later by an unique index when completed.  The module
tries aggressively to reuse existing completed classes, rather than create
duplicates.  Functions are provided to map between indexes and pointers.
Because of the deduplication effort, the index reported for a class upon
completion may map to a different pointer than the one supplied by new ().

Classes may be shared between different lexer instances, although, at the
time of writing (10 April 2014) it is not thread-safe.  In many cases,
there might only be one class under construction at any time, with the
effort either completed or abandoned quickly.  However, this module
recognises that sometimes multiple classes might be worked on in parallel,
and so explicitly marks each allocated class area as one of "unused",
"work" or "completed".  This marking is done by an array of state bytes
dynamically allocated when the pool is created.

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include <assert.h>
#include "charclass.h"
#include <limits.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <stdio.h> /* for EOF assert test.  */
#include <string.h>
#include <wchar.h> /* for WEOF assert test.  */
#include "xalloc.h"

/* Lower bound for size of first pool in the list.  */
/* ?? Set to 2 for pool debug;  Use 10 in production?  */
#define POOL_MINIMUM_INITIAL_SIZE          10

#ifndef MAX
# define MAX(a,b) ((a) > (b) ? (a) : (b))
#endif

#ifndef MIN
# define MIN(a,b) ((a) < (b) ? (a) : (b))
#endif

/* We maintain a list-of-pools here, choosing to malloc a new slab of
   memory each time we run out, instead of a realloc strategy.  This is so
   that we can provide a guarantee to the user that any class pointer issued
   remains valid for the lifetime of the module.  */

typedef size_t pool_list_index_t;

/* Designator for each charclass in each pool.  Note that enums are ints by
   default, but we use a single unsigned char per class in our explicit
   memory allocation.  */
typedef enum
{
  STATE_UNUSED = 0,
  STATE_WORKING = 1,
  STATE_COMPLETED = 2
} charclass_state_t;

typedef struct pool_info_struct {
  charclass_index_t first_index;
  size_t alloc;      /* ?? Use pool_list_index_t type for these?  */
  size_t used;
  charclass_t *classes;

  /* Charclass designator byte array, one per item, allocated dynamically.  */
  unsigned char *class_state;
} pool_t;

static pool_list_index_t pool_list_used  = 0;
static pool_list_index_t pool_list_alloc = 0;
static pool_t *pool_list = NULL;

/* While the header only guarantees a 3-bit gutter at each end of each
   class, we use an entire word (typically 32 bits) for the gutter,
   with at least 1 word placed at the start of each pool, 1 word as a
   shared gutter between each class, and 1 word after the last
   class.  */

/* HPUX defines these as macros in sys/param.h.  */
#ifdef setbit
# undef setbit
#endif
#ifdef clrbit
# undef clrbit
#endif

/* This represents part of a character class.  It must be unsigned and
   at least CHARCLASS_WORD_BITS wide.  Any excess bits are zero.  */
typedef unsigned int charclass_word;

/* The number of bits used in a charclass word.  utf8_classes assumes
   this is exactly 32.  */
enum { CHARCLASS_WORD_BITS = 32 };

/* The maximum useful value of a charclass_word; all used bits are 1.  */
#define CHARCLASS_WORD_MASK \
  (((charclass_word) 1 << (CHARCLASS_WORD_BITS - 1) << 1) - 1)

/* Number of words required to hold a bit for every character.  */
enum
{
  CHARCLASS_WORDS = (CHARCLASS_NOTCHAR + CHARCLASS_WORD_BITS - 1) / CHARCLASS_WORD_BITS
};

/* Flesh out opaque charclass type given in the header  */
/* The gutter element following the class member storage also serves as
   the gutter element p-receding the next class in the list.

   Note that since the "gutter" notion explicitly permits a restricted set
   of negative indices, members need to be signed, not unsigned, so that
   arithmetic shift right can be used where possible (e.g. -8 >> 8 == -1,
   not -8 / 256 == 0).  */

struct charclass_struct {
   charclass_word members[CHARCLASS_WORDS];
   charclass_word gutter_following;
};

/* Define class bit operations: test, set and clear a bit.

   Grrrr.  I wanted to exploit arithmetic right shift to come up with a
   really cheap and neat way of reducing small negative bit values,
   especially if b == EOF ==-1, to an index of -1 that falls neatly
   into the gutter, but strict C conformance does not guarantee this.
   The code below handles the two most likely scenarios, but, as with
   anything that is undefined, this is playing with fire.  */

#if INT_MAX == 2147483647
# define INT_BITS_LOG2 5   /* log2(sizeof(int)) + log2(CHARCLASS_WORD_BITS) */
#else
# error "Not implemented: Architectures with ints other than 32 bits"
#endif

#if ((~0 >> 1) < 0)
  /* Arithmetic shift right: Both signed and unsigned cases are ok.  */
# define ARITH_SHIFT_R_INT(b) ((b) >> INT_BITS_LOG2)
#else
  /* Avoid using right shift if b is negative.  The macro may evaluate b twice
     in some circumstances.  */
# define ARITH_SHIFT_R_INT(b) \
      (((b) < 0) ? -1 : ((b) >> INT_BITS_LOG2))
#endif

bool _GL_ATTRIBUTE_PURE
charclass_tstbit (int b, charclass_t const *ccl)
{
  return ccl->members[ARITH_SHIFT_R_INT(b)] >> b % CHARCLASS_WORD_BITS & 1;
}

void
charclass_setbit (int b, charclass_t *ccl)
{
  ccl->members[ARITH_SHIFT_R_INT(b)] |= (charclass_word) 1 << b % CHARCLASS_WORD_BITS;
}

void
charclass_clrbit (int b, charclass_t *ccl)
{
  ccl->members[ARITH_SHIFT_R_INT(b)] &= ~((charclass_word) 1
                                  << b % CHARCLASS_WORD_BITS);
}

void
charclass_setbit_range (int start, int end, charclass_t *ccl)

{
  int bit;

  /* Do nothing if the range doesn't make sense.  */
  if (end < start)
    return;
  if (start >= CHARCLASS_NOTCHAR)
    return;

  /* Clip the range to be in the interval [-1..NOTCHAR - 1] */
  start = MAX(start, -1);
  end   = MAX(end,   -1);
  /* We know start is < CHARCLASS_NOTCHAR from the test above.  */
  end   = MIN(end,   CHARCLASS_NOTCHAR - 1);

  /* ?? Could check that ccl is a valid class, but not at present.  */

  /* Okay, loop through the range, bit-by-bit, setting members.  */
  for (bit = start; bit <= end; bit++)
    ccl->members[ARITH_SHIFT_R_INT(bit)] |= 1U << bit % CHARCLASS_WORD_BITS;
}

void
charclass_clrbit_range (int start, int end, charclass_t *ccl)

{
  int bit;

  /* Do nothing if the range doesn't make sense.  */
  if (end < start)
    return;
  if (start >= CHARCLASS_NOTCHAR)
    return;

  /* Clip the range to be in the interval [-1..NOTCHAR - 1] */
  start = MAX(start, -1);
  end   = MAX(end,   -1);
  /* We know start is < CHARCLASS_NOTCHAR from the test above.  */
  end   = MIN(end,   CHARCLASS_NOTCHAR - 1);

  /* ?? Could check that ccl is a valid class, but not at present.  */

  /* Okay, loop through the range, bit-by-bit, clearing members.  */
  for (bit = start; bit <= end; bit++)
    ccl->members[ARITH_SHIFT_R_INT(bit)] &= ~(1U << bit % CHARCLASS_WORD_BITS);
}

/* Define whole-set operations: Copy, clear, invert, compare and union  */

void
charclass_copyset (charclass_t const *src, charclass_t *dst)
{
  memcpy (dst->members, src->members, sizeof(src->members));
}

void
charclass_zeroset (charclass_t *ccl)
{
  memset (ccl->members, 0, sizeof(ccl->members));
}

void
charclass_notset (charclass_t *ccl)
{
  int i;

  for (i = 0; i < CHARCLASS_WORDS; ++i)
    ccl->members[i] = CHARCLASS_WORD_MASK & ~ccl->members[i];
}

int _GL_ATTRIBUTE_PURE
charclass_equal (charclass_t const *ccl1, charclass_t const *ccl2)
{
  return memcmp (ccl1->members, ccl2->members,
       sizeof(ccl1->members)) == 0;
}

void
charclass_unionset (charclass_t const *src, charclass_t *dst)
{
  int i;

  for (i = 0; i < CHARCLASS_WORDS; ++i)
    dst->members[i] |= src->members[i];
}

void
charclass_intersectset (charclass_t const *src, charclass_t *dst)
{
  int i;

  for (i = 0; i < CHARCLASS_WORDS; ++i)
    dst->members[i] &= src->members[i];
}

/* #ifdef DEBUG */

/* Nybble (4bit)-to-char conversion array for little-bit-endian nybbles.  */
static const char *disp_nybble = "084c2a6e195d3b7f";

/* Return a static string describing a class (Note: not reentrant).  */
char *
charclass_describe (charclass_t const *ccl)
{
  /* The string should probably be less than 90 chars, but overcompensate
     for limited uncertainty introduced by the %p formatting operator.  */
  static char buf[256];
  char *p_buf = buf;
  int i;

  p_buf += sprintf (p_buf, "0x%08lx:", (unsigned long) ccl);
  for (i = 0; i < CHARCLASS_WORDS; i += 2)
    {
      int j = ccl->members[i];
      *p_buf++ = ' ';
      *p_buf++ = disp_nybble[(j >>  0) & 0x0f];
      *p_buf++ = disp_nybble[(j >>  4) & 0x0f];
      *p_buf++ = disp_nybble[(j >>  8) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 12) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 16) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 20) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 24) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 28) & 0x0f];

      j = ccl->members[i + 1];
      *p_buf++ = disp_nybble[(j >>  0) & 0x0f];
      *p_buf++ = disp_nybble[(j >>  4) & 0x0f];
      *p_buf++ = disp_nybble[(j >>  8) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 12) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 16) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 20) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 24) & 0x0f];
      *p_buf++ = disp_nybble[(j >> 28) & 0x0f];
    }
  *p_buf++ = '\0';
  return buf;
}

/* static */ void
debug_pools (const char *label, bool class_contents)
{
  pool_list_index_t pool_nr;
  size_t class_nr;

  printf ("\nPool %p debug(%s): [alloc, used: %ld %ld]\n",
          pool_list, label, pool_list_alloc, pool_list_used);
  for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
    {
      pool_t *pool = &pool_list[pool_nr];
      printf (
" %3ld: first_index, alloc, used, classes: %4ld %3lu %3lu %p\n",
pool_nr, pool->first_index, pool->alloc, pool->used, pool->classes);
      printf ("     class_states: ");
      for (class_nr = 0; class_nr < pool->alloc; class_nr++)
        switch (pool->class_state[class_nr]) {
          case STATE_UNUSED:    putchar ('.'); break;
          case STATE_WORKING:   putchar ('w'); break;
          case STATE_COMPLETED: putchar ('C'); break;
          default: printf ("?%02x", pool->class_state[class_nr]);
        }
      putchar ('\n');
    }

  /* If class contents requested, print them out as well.  */
  if (class_contents)
    for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
      {
        pool_t *pool = &pool_list[pool_nr];
        for (class_nr = 0; class_nr < pool->used; class_nr++)
          printf ("%s\n",
                  charclass_describe (&pool->classes[class_nr]));
      }
}

/* #endif * DEBUG */

static pool_t *
add_new_pool (void)
{
  pool_t *prev, *pool;
  size_t pool_class_alloc;
  charclass_t *alloc_mem;

  /* If the pools list is full, use x2nrealloc to expand its size.  */
  if (pool_list_used == pool_list_alloc)
      pool_list = x2nrealloc (pool_list, &pool_list_alloc, sizeof (pool_t));

  /* Find the size of the last charclass pool in the (old) list.  Scale up
     the size so that malloc activity will decrease as the number of pools
     increases.  Also, add 1 here as we knock off 1 to use as a gutter
     later.  */
  prev = &pool_list[pool_list_used - 1];
  pool_class_alloc = (prev->alloc * 5 / 2) + 1;
  alloc_mem = XNMALLOC (pool_class_alloc, charclass_t);

  /* Set up the new pool, shifting the alloc pointer to create the gutter
     preceding the first class of the pool.  */
  pool = &pool_list[pool_list_used++];
  pool->classes = alloc_mem + 1;
  pool->first_index = prev->first_index + prev->alloc;
  pool->alloc = pool_class_alloc - 1;
  pool->used = 0;
  pool->class_state = xzalloc (pool->alloc);

  return pool;
}

charclass_t *
charclass_alloc (void)
{
  pool_list_index_t pool_nr;
  charclass_t *class;
  pool_t *pool = NULL;
  size_t class_nr;
  size_t class_last_nr;
  charclass_word *gutter_preceding;

  /* Locate a pool with unused entries (if any).  */
  for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
    {
      pool = &pool_list[pool_nr];

      /* Try use the earliest pool possible, first by filling in a hole
         from a withdrawn class, or by grabbing an unused class from the
         end of the list.  */
      class_last_nr = MIN(pool->used + 1, pool->alloc);
      for (class_nr = 0; class_nr < class_last_nr; class_nr++)
       {
          if (pool->class_state[class_nr] == STATE_UNUSED)
            goto found_pool_and_class;
       }
    }

  /* No space found, so prepare a new pool and make this class its first
     element.  */
  pool = add_new_pool ();
  class_nr = 0;
  /* FALLTHROUGH */

found_pool_and_class:
  /* Mark the found class state as working, zero its elements, and return
       class pointer to caller.  Zeroing is needed as this class may have
       been previously worked on, but then abandoned or withdrawn.  */
  pool->class_state[class_nr] = STATE_WORKING;
  if (class_nr >= pool->used)
    pool->used = class_nr + 1;
  class = &pool->classes[class_nr];

  /* Zero out the class' members, and also the gutters on each side.  */
  memset (class, 0, sizeof (*class));
  gutter_preceding = ((charclass_word *) class) - 1;
  *gutter_preceding = 0;

  return class;
}

pool_t * _GL_ATTRIBUTE_PURE
find_class_pool (charclass_t const *ccl)
{
  pool_list_index_t pool_nr;
  pool_t *pool = NULL;
  ptrdiff_t class_ptr_offset;

  /* Locate the pool whose memory address space covers this class.  */
  /* ?? Perhaps check &pool->classes[pool->alloc] in this first loop, and
     then check that the index is in the "used" portion later, so we can
     diagnose malformed pointers more exactly.  */
  for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
    {
      pool = &pool_list[pool_nr];
      if ((pool->classes <= ccl) && (ccl < &pool->classes[pool->alloc]))
        goto found_pool;
    }

  /* No credible pool candidate was found.  */
  assert ("find_class_pool: no pool found");
  return NULL;

found_pool:
  /* Make sure the class clearly lies on an array boundary within the pool's
     memory allocation.  */
  class_ptr_offset = (char *) ccl - (char *) pool->classes;
  if ((class_ptr_offset % sizeof (charclass_t)) != 0)
    {
      /* Pointer does not lie at the start of a pool member.  */
      assert ("find_class_pool: pointer not aligned.");
      return NULL;
    }

  return pool;
}

static void
withdraw_class (charclass_t *ccl, pool_t *class_pool)
{
  pool_t *pool;
  size_t class_nr;
  int *gutter_preceding;

  /* Use pool reference if given, otherwise work back from the class pointer
     to find the associated pool.  */
  pool = (class_pool != NULL) ? class_pool : find_class_pool (ccl);

  if (pool == NULL)
    assert (!"Could not locate a pool for this charclass");

  /* Zero out the gutters each side of the class.  */
  ccl->gutter_following = 0;
  gutter_preceding = ((int *) ccl) - 1;
  *gutter_preceding = 0;

  /* Work out the class index in the pool.  */
  class_nr = ccl - pool->classes;
  pool->class_state[class_nr] = STATE_UNUSED;

  /* Is this the last item within the pool's class list? */
  if (class_nr == pool->used - 1)
    {
      /* Yes, reduce the pool member count by 1.  */
      pool->used--;
      return;
    }
}

/* Finish off creating a class, and report an index that can be used
   to reference the class.  */
charclass_index_t
charclass_completed (charclass_t *ccl)
{
  charclass_word *gutter_preceding;
  pool_list_index_t pool_nr;
  pool_t *pool;
  charclass_t *found = NULL;
  size_t class_nr;
  pool_t *my_pool = NULL;
  size_t my_class_nr = 0;

  /* Search all pools for a completed class matching this class, and, if
     found, use it in preference to the new one.  While searching, also
     record where the work class is located.  If we can't find ourselves, the
     pointer is invalid, and throw an assertion.   */
  for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
    {
      pool = &pool_list[pool_nr];
      for (class_nr = 0; class_nr < pool->used; class_nr++)
        {
          charclass_t *search = &pool->classes[class_nr];
          /* Have we found ourselves in the list? */
          if (search == ccl)
            {
              /* Yes, remember this place in case no duplicate is found.  */
              my_pool = pool;
              my_class_nr = class_nr;
          }
          if (pool->class_state[class_nr] != STATE_COMPLETED)
            continue;
          if (charclass_equal (search, ccl))
            {
              /* Another class, completeded, matches:  Use it in preference
                 to potentially creating a duplicate.  */
              withdraw_class (ccl, my_pool);
              found = search;
              goto found_matching_class;
            }
        }
    }

  /* No duplicate found... but make sure the search pointer is known. */
  assert (my_pool != NULL);
  assert (my_pool->class_state[my_class_nr] == STATE_WORKING);

  /* Prepare to convert the search (work) class into a completed class.  */
  pool = my_pool;
  class_nr = my_class_nr;
  found = &pool->classes[class_nr];
  /* FALLTHROUGH */

found_matching_class:
  /* Clear out the gutter integers each side of the class entry.  */
  gutter_preceding = found->members - 1;
  *gutter_preceding = 0;
  found->gutter_following = 0;
  pool->class_state[class_nr] = STATE_COMPLETED;

  /* Return the index of the class.  */
  return pool->first_index + class_nr;
}

void
charclass_abandon (charclass_t *ccl)
{
  withdraw_class (ccl, NULL);
}

/* Additional functions to help clients work with classes.  */

charclass_t * _GL_ATTRIBUTE_PURE
charclass_get_pointer (charclass_index_t const index)
{
  pool_list_index_t pool_nr;
  pool_t *pool;

  /* Does this class match any class we've seen previously? */
  for (pool_nr = 0; pool_nr < pool_list_used; pool_nr++)
    {
      /* Is the index inside this pool? */
      pool = &pool_list[pool_nr];
      if (pool->first_index <= index
              && index < (pool->first_index + pool->used))
        {
          /* Yes, find the pointer within the pool and return it.  */
          return &pool->classes[index - pool->first_index];
        }
    }

  /* The mapping above should never fail; we could return NULL, but we
     choose to abort instead.  */
  assert (!"index-to-charclass mapping failed");
  return NULL;
}

charclass_index_t _GL_ATTRIBUTE_PURE
charclass_get_index (charclass_t const *ccl)
{
  pool_t *pool;

  /* This code is similar to charclass_completed... perhaps merge? */
  pool = find_class_pool (ccl);
  if (pool == NULL)
    return -1;

  /* Report the index to the caller.  */
  return pool->first_index + (ccl - pool->classes);
}

/* Functions to initialise module on startup, and to shut down and
   release acquired resources at exit.  */

void
charclass_destroy_module (void)
{
  pool_t *local_list;
  int i;
  charclass_t *alloc_mem;

  /* Move the global list head into a local variable, and immediately clear
     the global.  This is a half-hearted attempt to avoid race conditions;
     to do things properly, a system-wide atomic operation (locked,
     including multi-CPU cache coherency) operation should be used.  */
  local_list = pool_list;
  pool_list = NULL;

  /* If the list is already empty, finish now.  */
  if (! local_list)
    return;

  /* First, discard the charclass memory associated with each pool,
     including catering for the offset used upon creation.  */
  for (i = 0; i < pool_list_used; i++)
    {
      /* We skipped an initial class after allocation, in order to give
         the first class in the pool a gutter, so undo the offset here
         when freeing.  */
      /* REMOVEME: Was typed as "guitter" above, before fixing; how about
         calling it some variant of "guitar"?! */
      alloc_mem = local_list[i].classes;
      free (alloc_mem - 1);

      /* Also free up the class_state memory.  */
      free (local_list[i].class_state);
    }

  /* Second, free up the pool list itself.  */
  free (local_list);
  pool_list_used = 0;
}

void
charclass_initialise (size_t initial_pool_size)
{
  size_t initial_alloc;
  charclass_t *alloc_mem;
  pool_t *pool;
  charclass_t *ccl;
  charclass_index_t zeroclass_index;

  /* Usually (gnulib assumption?) EOF = WEOF = -1, but the standard
     merely states that EOF must be a negative integer, not necessarily
     -1; furthermore, the definition for (ISO C90) WEOF states that it
     need not be negative at all, let alone guaranteed to be -1.

     In this demonstration/prototype code, we demand that both EOF and
     WEOF be -1, as this value makes the gutters worthwhile:  The user
     can be relieved of some edge case processing if -1 is thrown
     neatly into the gutter .  */
  assert (EOF == -1);
  assert (WEOF == -1);

  /* First, set up the list-of-pools structure with initial storage.  */
  pool_list_alloc = 4;
  pool_list = (pool_t *) xnmalloc (pool_list_alloc, sizeof (pool_t));

  /* If initial pool size is small, inflate it here as we prefer to
     waste a little memory, rather than issue many calls to xmalloc ().
     This minimum also ensures that our double-up pool size strategy
     has a sane (perhaps overly generous?) starting point.  */
  initial_alloc = MAX(initial_pool_size, POOL_MINIMUM_INITIAL_SIZE);

  /* Set up the first pool using our chosen first alloc size.  Allocate
     an extra class, and offset the pool by this amount, in order to
     accommodate the initial gutter integer.  (Note for the future:  If
     charclass alignment becomes significant, then sizeof (charclass)
     and this offset may need to be changed, perhaps for SIMD
     instructions.)  */
  pool_list_used = 1;
  pool = &pool_list[0];
  pool->first_index = 0;
  pool->alloc = initial_alloc;
  pool->used = 0;
  alloc_mem = XNMALLOC (pool->alloc + 1, charclass_t);
  pool->classes = alloc_mem + 1;
  pool->class_state = xzalloc (pool->alloc);

  /* Enforce the all-zeroes class to be the first class.  This is needed
     as "abandon" may leave a hole in a pool in some cases, and in these
     cases we need to ensure that no-one else picks it up by accident
     (as this would invalidate the guarantee that the module eliminates
     all duplicates, from the point of view of the user).  So, we set the
     first class to all-zeroes, and also zero out abandoned classes where
     a hole is unavoidable.  */
  ccl = charclass_alloc (); /* Alloc delivers an all-zeroes class.  */
  zeroclass_index = charclass_completed (ccl);
  assert (zeroclass_index == CHARCLASS_ZEROCLASS_INDEX);

  /* Finally, use atexit () to arrange for module cleanup on exit.  */
  atexit (charclass_destroy_module);
}

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="charclass.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="charclass.h"

/* charclass -- Tools to create and manipulate sets of characters (octets)

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* This module provides services to allocate, manipulate, consolidate
   and discard 256-bit vectors, used to describe 8-bit (octet) sets.
   Octet is used as the member name here, as "byte" or "char" can
   sometimes refer to different bit sizes (e.g. char -> 6 bits on some
   IBM/Cyber architectures; char -> 32 bits on some DSP architectures;
   in C, sizeof (char) == 1 by definition on all architectures).

   The connection between these "charclass" sets and set expression by
   RE tools can be non-trivial:  Many Unicode characters cannot fit
   into 8 bits, and even where octet-based code pages are used,
   nontrivial cases can appear (e.g. Code page 857, MS-DOS Turkish,
   which has both a dotted and a dotless lowercase and uppercase "I").

   On the storage side, things are slightly tricky and perhaps even
   murky at times.  The client starts by allocating a charclass,
   working on it, and then either completing it (usually) or abandoning
   it.  The working class (pun intended) is represented by a pointer.
   If not abandoned, this pointer is guaranteed to remain valid for the
   lifetime of the module.

   The module tries aggressively to eliminate duplicates; this is
   perhaps the main function of the completion step.  So, the pointer
   that represents the class after completion may not be the working
   pointer.

   In addition to the pointer method of referring to a class, the
   classes can be viewed as an array, with the first class receiving
   index 0, the second receiving index 1, and so on.  Functions are
   provided to map pointers to indexes, and vice versa.  The index
   representation is handy as it is very compact (typically much fewer
   than 24 bits), whereas pointers are architecture and OS-specific,
   and may be 64 bits or more.

   Index 0 is special; it will always represent the zero-class (no
   members set).  Users wanting to store a set of non-zeroclass classes
   (e.g. utf8) can use this property as a sentinel (a value of 0 for a
   static variable can mean "not initialised").

   Finally, there are some "gutter" bits, at least 3 on each end of
   the class, so that, to a limited extent (and especially for the
   common case of EOF == -1), bits can be set and cleared without
   causing problems, and the code does not need to include the overhead
   of checks for out-of-bound bit numbers.  These gutter bits are
   cleared when the class is completed, so EOF (for instance) should
   never be member of a class.  */


#ifndef CHARCLASS_H
#define CHARCLASS_H 1

/* Always import environment-specific configuration items first. */
#include <config.h>

#include <limits.h>
#include <stdbool.h>
#include <stddef.h>

/* Define charclass as an opaque type.  */
typedef struct charclass_struct charclass_t;

/* First integer value that is greater than any unibyte char.  Each
   class can model [0 .. CHARCLASS_NOTCHAR - 1] members.  The user may
   reference a few members above or below this range, thanks to the
   explicit gutters, but the margins provided by this facility are
   small.  */
enum { CHARCLASS_NOTCHAR = 1 << CHAR_BIT };

/* Indices to valid charclasses are always non-negative; zero is
   reserved as an index to the zerocass, and all other class entities
   have positive indices.  We use ptrdiff_t rather than size_t here
   as -1 can be used as a sentinel in some places.  */
typedef ptrdiff_t charclass_index_t;

/* Index zero is special:  It is the zero-class (no bits set) class.  */
enum
{
  CHARCLASS_ZEROCLASS_INDEX = 0
};

/* Entire-module initialisation and destruction functions.  The client
   specifies starting size for the class pool.  Destroy_module releases
   all resources acquired by this module.  */

extern void
charclass_initialise (size_t initial_pool_size);

extern void
charclass_destroy_module (void);

/* Single-bit operations (test, set, clear).  */

extern bool _GL_ATTRIBUTE_PURE
charclass_tstbit (int b, charclass_t const *ccl);

extern void
charclass_setbit (int b, charclass_t *ccl);

extern void
charclass_clrbit (int b, charclass_t *ccl);

/* Range-of-bits set and clear operations.  These are easier to read,
   and also more efficient, than multiple single-bit calls.  */

extern void
charclass_setbit_range (int start, int end, charclass_t *ccl);

extern void
charclass_clrbit_range (int start, int end, charclass_t *ccl);

/* Whole-of-set operations (copy, zero, invert, compare-equal).  */

extern void
charclass_copyset (charclass_t const *src, charclass_t *dst);

extern void
charclass_zeroset (charclass_t *ccl);

extern void
charclass_notset (charclass_t *ccl);

extern int _GL_ATTRIBUTE_PURE
charclass_equal (charclass_t const *ccl1, charclass_t const *ccl2);

/* Add "unionset" and "intersectset" functions since whole-of-class
   operations tend to be reasonably expressive and self-documenting.
   In both cases, the source modifies the destination; ORed in, in the
   case of unionset; ANDed in, in the case of intersectset.  */
extern void
charclass_unionset (charclass_t const *src, charclass_t *dst);

extern void
charclass_intersectset (charclass_t const *src, charclass_t *dst);

/* Functions to allocate, complete and abandon charclasses.  Note that
   the module aggressively tries to reuse existing completed classes
   rather than create new ones.  The module returns an unique index
   that can be used to reference the module; this index supercedes the
   pointer used during the work phase, e.g.:

        work_class_pointer = charclass_alloc ();
        ...(Set/clear members as required to construct a class in
             work_class_pointer.)...
        class_index = charclass_completed (work_class_pointer);
        completed_class_pointer = charclass_get_pointer (class_index);
        ...(completed_class_pointer might not == work_class_pointer.)...

   The aggressive-reuse policy also means that completed classes must
   not undergo further modification.  Another piece of coding hygiene is
   that the pointer value used to construct the class should not be used
   once the class is either completed or abandoned.

   Allocating and then abandoning classes is useful where an operation
   requires temporary classes for a while, but these do not need to be
   maintained once the work is complete.

   2 May 2014: Note to self and others: I was using the term "finalise"
   instead of "completed" for the operation where a class has been constructed
   and is now ready to be used; I've decided to change the terminology after
   reading in various places how the term "finalise" is usually strongly
   connected with end-of-life operations on an object and/or class, not a
   ready-for-use operation.  This note is a reminder to myself and a hint to
   others about this change, in case vestiges of the earlier naming scheme
   slip through my edits and/or appear in the code.  */

extern charclass_t *
charclass_alloc (void);

extern charclass_index_t
charclass_completed (charclass_t *ccl);

extern void
charclass_abandon (charclass_t *ccl);

/* Functions to map between pointer references and index references for
   a charclass.  As explained above, the index is convenient as it is
   typically an array reference, and is usually not much larger than the
   number of classes that have been allocated.  */

extern charclass_t * _GL_ATTRIBUTE_PURE
charclass_get_pointer (charclass_index_t const index);

extern charclass_index_t _GL_ATTRIBUTE_PURE
charclass_get_index (charclass_t const *ccl);

/* Return a static string describing a class (Note: not reentrant).  */
extern char *
charclass_describe (charclass_t const *ccl);

#endif /* CHARCLASS_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="dfa-prl.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="dfa-prl.c"

/* dfa.c - deterministic extended regexp routines for GNU
   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

#include <config.h>

#include "dfa.h"

#include <assert.h>
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <string.h>
#include <locale.h>

/* HOOK: Hack in interfaces to new charclass and fsa* modules.  */
#include "charclass.h"
#include "fsatoken.h"
#include "fsalex.h"
#include "fsamusts.h"
#include "fsaparse.h"
#include "mbcsets.h"
#include "proto-lexparse.h"

/* HOOK: File handle for hook/parallel lex/parse debug/log messages */
FILE *pll_log = NULL;

#define HOOK_LOG_FILENAME   "/tmp/parallel.log"

/* HOOK: Static variables to hold opaque parser and lexer contexts.  */
static fsaparse_ctxt_t *parser = NULL;
static fsalex_ctxt_t *lexer = NULL;

/* HOOK: Store musts list as a static variable so we can free it correctly.  */
static fsamusts_list_element_t *musts = NULL;


#define STREQ(a, b) (strcmp (a, b) == 0)

/* ISASCIIDIGIT differs from isdigit, as follows:
   - Its arg may be any int or unsigned int; it need not be an unsigned char.
   - It's guaranteed to evaluate its argument exactly once.
   - It's typically faster.
   Posix 1003.2-1992 section 2.5.2.1 page 50 lines 1556-1558 says that
   only '0' through '9' are digits.  Prefer ISASCIIDIGIT to isdigit unless
   it's important to use the locale's definition of "digit" even when the
   host does not conform to Posix.  */
#define ISASCIIDIGIT(c) ((unsigned) (c) - '0' <= 9)

#include "gettext.h"
#define _(str) gettext (str)

#include <wchar.h>
#include <wctype.h>

#include "xalloc.h"

/* HPUX defines these as macros in sys/param.h.  */
#ifdef setbit
# undef setbit
#endif
#ifdef clrbit
# undef clrbit
#endif

/* First integer value that is greater than any character code.  */
enum { NOTCHAR = 1 << CHAR_BIT };

/* This represents part of a character class.  It must be unsigned and
   at least CHARCLASS_WORD_BITS wide.  Any excess bits are zero.  */
typedef unsigned int charclass_word;

/* The number of bits used in a charclass word.  utf8_classes assumes
   this is exactly 32.  */
enum { CHARCLASS_WORD_BITS = 32 };

/* The maximum useful value of a charclass_word; all used bits are 1.  */
#define CHARCLASS_WORD_MASK \
  (((charclass_word) 1 << (CHARCLASS_WORD_BITS - 1) << 1) - 1)

/* Number of words required to hold a bit for every character.  */
enum
{
  CHARCLASS_WORDS = (NOTCHAR + CHARCLASS_WORD_BITS - 1) / CHARCLASS_WORD_BITS
};

/* Sets of unsigned characters are stored as bit vectors in arrays of ints.  */
typedef charclass_word charclass[CHARCLASS_WORDS];

/* Convert a possibly-signed character to an unsigned character.  This is
   a bit safer than casting to unsigned char, since it catches some type
   errors that the cast doesn't.  */
static unsigned char
to_uchar (char ch)
{
  return ch;
}

/* Contexts tell us whether a character is a newline or a word constituent.
   Word-constituent characters are those that satisfy iswalnum, plus '_'.
   Each character has a single CTX_* value; bitmasks of CTX_* values denote
   a particular character class.

   A state also stores a context value, which is a bitmask of CTX_* values.
   A state's context represents a set of characters that the state's
   predecessors must match.  For example, a state whose context does not
   include CTX_LETTER will never have transitions where the previous
   character is a word constituent.  A state whose context is CTX_ANY
   might have transitions from any character.  */

#define CTX_NONE	1
#define CTX_LETTER	2
#define CTX_NEWLINE	4
#define CTX_ANY		7

/* Sometimes characters can only be matched depending on the surrounding
   context.  Such context decisions depend on what the previous character
   was, and the value of the current (lookahead) character.  Context
   dependent constraints are encoded as 8 bit integers.  Each bit that
   is set indicates that the constraint succeeds in the corresponding
   context.

   bit 8-11 - valid contexts when next character is CTX_NEWLINE
   bit 4-7  - valid contexts when next character is CTX_LETTER
   bit 0-3  - valid contexts when next character is CTX_NONE

   The macro SUCCEEDS_IN_CONTEXT determines whether a given constraint
   succeeds in a particular context.  Prev is a bitmask of possible
   context values for the previous character, curr is the (single-bit)
   context value for the lookahead character.  */
#define NEWLINE_CONSTRAINT(constraint) (((constraint) >> 8) & 0xf)
#define LETTER_CONSTRAINT(constraint)  (((constraint) >> 4) & 0xf)
#define OTHER_CONSTRAINT(constraint)    ((constraint)       & 0xf)

#define SUCCEEDS_IN_CONTEXT(constraint, prev, curr) \
  ((((curr) & CTX_NONE      ? OTHER_CONSTRAINT (constraint) : 0) \
    | ((curr) & CTX_LETTER  ? LETTER_CONSTRAINT (constraint) : 0) \
    | ((curr) & CTX_NEWLINE ? NEWLINE_CONSTRAINT (constraint) : 0)) & (prev))

/* The following macros describe what a constraint depends on.  */
#define PREV_NEWLINE_CONSTRAINT(constraint) (((constraint) >> 2) & 0x111)
#define PREV_LETTER_CONSTRAINT(constraint)  (((constraint) >> 1) & 0x111)
#define PREV_OTHER_CONSTRAINT(constraint)    ((constraint)       & 0x111)

#define PREV_NEWLINE_DEPENDENT(constraint) \
  (PREV_NEWLINE_CONSTRAINT (constraint) != PREV_OTHER_CONSTRAINT (constraint))
#define PREV_LETTER_DEPENDENT(constraint) \
  (PREV_LETTER_CONSTRAINT (constraint) != PREV_OTHER_CONSTRAINT (constraint))

/* Tokens that match the empty string subject to some constraint actually
   work by applying that constraint to determine what may follow them,
   taking into account what has gone before.  The following values are
   the constraints corresponding to the special tokens previously defined.  */
#define NO_CONSTRAINT         0x777
#define BEGLINE_CONSTRAINT    0x444
#define ENDLINE_CONSTRAINT    0x700
#define BEGWORD_CONSTRAINT    0x050
#define ENDWORD_CONSTRAINT    0x202
#define LIMWORD_CONSTRAINT    0x252
#define NOTLIMWORD_CONSTRAINT 0x525

/* The regexp is parsed into an array of tokens in postfix form.  Some tokens
   are operators and others are terminal symbols.  Most (but not all) of these
   codes are returned by the lexical analyzer.  */

typedef ptrdiff_t token;

/* Predefined token values.  */
enum
{
  END = -1,                     /* END is a terminal symbol that matches the
                                   end of input; any value of END or less in
                                   the parse tree is such a symbol.  Accepting
                                   states of the DFA are those that would have
                                   a transition on END.  */

  /* Ordinary character values are terminal symbols that match themselves.  */

  EMPTY = NOTCHAR,              /* EMPTY is a terminal symbol that matches
                                   the empty string.  */

  BACKREF,                      /* BACKREF is generated by \<digit>
                                   or by any other construct that
                                   is not completely handled.  If the scanner
                                   detects a transition on backref, it returns
                                   a kind of "semi-success" indicating that
                                   the match will have to be verified with
                                   a backtracking matcher.  */

  BEGLINE,                      /* BEGLINE is a terminal symbol that matches
                                   the empty string at the beginning of a
                                   line.  */

  ENDLINE,                      /* ENDLINE is a terminal symbol that matches
                                   the empty string at the end of a line.  */

  BEGWORD,                      /* BEGWORD is a terminal symbol that matches
                                   the empty string at the beginning of a
                                   word.  */

  ENDWORD,                      /* ENDWORD is a terminal symbol that matches
                                   the empty string at the end of a word.  */

  LIMWORD,                      /* LIMWORD is a terminal symbol that matches
                                   the empty string at the beginning or the
                                   end of a word.  */

  NOTLIMWORD,                   /* NOTLIMWORD is a terminal symbol that
                                   matches the empty string not at
                                   the beginning or end of a word.  */

  QMARK,                        /* QMARK is an operator of one argument that
                                   matches zero or one occurrences of its
                                   argument.  */

  STAR,                         /* STAR is an operator of one argument that
                                   matches the Kleene closure (zero or more
                                   occurrences) of its argument.  */

  PLUS,                         /* PLUS is an operator of one argument that
                                   matches the positive closure (one or more
                                   occurrences) of its argument.  */

  REPMN,                        /* REPMN is a lexical token corresponding
                                   to the {m,n} construct.  REPMN never
                                   appears in the compiled token vector.  */

  CAT,                          /* CAT is an operator of two arguments that
                                   matches the concatenation of its
                                   arguments.  CAT is never returned by the
                                   lexical analyzer.  */

  OR,                           /* OR is an operator of two arguments that
                                   matches either of its arguments.  */

  LPAREN,                       /* LPAREN never appears in the parse tree,
                                   it is only a lexeme.  */

  RPAREN,                       /* RPAREN never appears in the parse tree.  */

  ANYCHAR,                      /* ANYCHAR is a terminal symbol that matches
                                   a valid multibyte (or single byte) character.
                                   It is used only if MB_CUR_MAX > 1.  */

  MBCSET,                       /* MBCSET is similar to CSET, but for
                                   multibyte characters.  */

  WCHAR,                        /* Only returned by lex.  wctok contains
                                   the wide character representation.  */

  CSET                          /* CSET and (and any value greater) is a
                                   terminal symbol that matches any of a
                                   class of characters.  */
};


/* States of the recognizer correspond to sets of positions in the parse
   tree, together with the constraints under which they may be matched.
   So a position is encoded as an index into the parse tree together with
   a constraint.  */
typedef struct
{
  size_t index;                 /* Index into the parse array.  */
  unsigned int constraint;      /* Constraint for matching this position.  */
} position;

/* Sets of positions are stored as arrays.  */
typedef struct
{
  position *elems;              /* Elements of this position set.  */
  size_t nelem;                 /* Number of elements in this set.  */
  size_t alloc;                 /* Number of elements allocated in ELEMS.  */
} position_set;

/* Sets of leaves are also stored as arrays.  */
typedef struct
{
  size_t *elems;                /* Elements of this position set.  */
  size_t nelem;                 /* Number of elements in this set.  */
} leaf_set;

/* A state of the dfa consists of a set of positions, some flags,
   and the token value of the lowest-numbered position of the state that
   contains an END token.  */
typedef struct
{
  size_t hash;                  /* Hash of the positions of this state.  */
  position_set elems;           /* Positions this state could match.  */
  unsigned char context;        /* Context from previous state.  */
  bool has_backref;		/* This state matches a \<digit>.  */
  bool has_mbcset;		/* This state matches a MBCSET.  */
  unsigned short constraint;    /* Constraint for this state to accept.  */
  token first_end;              /* Token value of the first END in elems.  */
  position_set mbps;            /* Positions which can match multibyte
                                   characters, e.g., period.
                                   Used only if MB_CUR_MAX > 1.  */
} dfa_state;

/* States are indexed by state_num values.  These are normally
   nonnegative but -1 is used as a special value.  */
typedef ptrdiff_t state_num;

/* A bracket operator.
   e.g., [a-c], [[:alpha:]], etc.  */
struct mb_char_classes
{
  ptrdiff_t cset;
  bool invert;
  wchar_t *chars;               /* Normal characters.  */
  size_t nchars;
  wctype_t *ch_classes;         /* Character classes.  */
  size_t nch_classes;
  struct			/* Range characters.  */
  {
    wchar_t beg;		/* Range start.  */
    wchar_t end;		/* Range end.  */
  } *ranges;
  size_t nranges;
  char **equivs;                /* Equivalence classes.  */
  size_t nequivs;
  char **coll_elems;
  size_t ncoll_elems;           /* Collating elements.  */
};

/* A compiled regular expression.  */
struct dfa
{
  /* Fields filled by the scanner.  */
  charclass *charclasses;       /* Array of character sets for CSET tokens.  */
  size_t cindex;                /* Index for adding new charclasses.  */
  size_t calloc;                /* Number of charclasses allocated.  */

  /* Fields filled by the parser.  */
  token *tokens;                /* Postfix parse array.  */
  size_t tindex;                /* Index for adding new tokens.  */
  size_t talloc;                /* Number of tokens currently allocated.  */
  size_t depth;                 /* Depth required of an evaluation stack
                                   used for depth-first traversal of the
                                   parse tree.  */
  size_t nleaves;               /* Number of leaves on the parse tree.  */
  size_t nregexps;              /* Count of parallel regexps being built
                                   with dfaparse.  */
  bool fast;			/* The DFA is fast.  */
  bool multibyte;		/* MB_CUR_MAX > 1.  */
  token utf8_anychar_classes[5]; /* To lower ANYCHAR in UTF-8 locales.  */
  mbstate_t mbs;		/* Multibyte conversion state.  */

  /* The following are valid only if MB_CUR_MAX > 1.  */

  /* The value of multibyte_prop[i] is defined by following rule.
     if tokens[i] < NOTCHAR
     bit 0 : tokens[i] is the first byte of a character, including
     single-byte characters.
     bit 1 : tokens[i] is the last byte of a character, including
     single-byte characters.

     if tokens[i] = MBCSET
     ("the index of mbcsets corresponding to this operator" << 2) + 3

     e.g.
     tokens
     = 'single_byte_a', 'multi_byte_A', single_byte_b'
     = 'sb_a', 'mb_A(1st byte)', 'mb_A(2nd byte)', 'mb_A(3rd byte)', 'sb_b'
     multibyte_prop
     = 3     , 1               ,  0              ,  2              , 3
   */
  int *multibyte_prop;

  /* A table indexed by byte values that contains the corresponding wide
     character (if any) for that byte.  WEOF means the byte is not a
     valid single-byte character.  */
  wint_t mbrtowc_cache[NOTCHAR];

  /* Array of the bracket expression in the DFA.  */
  struct mb_char_classes *mbcsets;
  size_t nmbcsets;
  size_t mbcsets_alloc;

  /* Fields filled by the superset.  */
  struct dfa *superset;             /* Hint of the dfa.  */

  /* Fields filled by the state builder.  */
  dfa_state *states;            /* States of the dfa.  */
  state_num sindex;             /* Index for adding new states.  */
  size_t salloc;		/* Number of states currently allocated.  */

  /* Fields filled by the parse tree->NFA conversion.  */
  position_set *follows;        /* Array of follow sets, indexed by position
                                   index.  The follow of a position is the set
                                   of positions containing characters that
                                   could conceivably follow a character
                                   matching the given position in a string
                                   matching the regexp.  Allocated to the
                                   maximum possible position index.  */
  bool searchflag;		/* We are supposed to build a searching
                                   as opposed to an exact matcher.  A searching
                                   matcher finds the first and shortest string
                                   matching a regexp anywhere in the buffer,
                                   whereas an exact matcher finds the longest
                                   string matching, but anchored to the
                                   beginning of the buffer.  */

  /* Fields filled by dfaexec.  */
  state_num tralloc;            /* Number of transition tables that have
                                   slots so far, not counting trans[-1].  */
  int trcount;                  /* Number of transition tables that have
                                   actually been built.  */
  state_num **trans;            /* Transition tables for states that can
                                   never accept.  If the transitions for a
                                   state have not yet been computed, or the
                                   state could possibly accept, its entry in
                                   this table is NULL.  This points to one
                                   past the start of the allocated array,
                                   and trans[-1] is always NULL.  */
  state_num **fails;            /* Transition tables after failing to accept
                                   on a state that potentially could do so.  */
  int *success;                 /* Table of acceptance conditions used in
                                   dfaexec and computed in build_state.  */
  state_num *newlines;          /* Transitions on newlines.  The entry for a
                                   newline in any transition table is always
                                   -1 so we can count lines without wasting
                                   too many cycles.  The transition for a
                                   newline is stored separately and handled
                                   as a special case.  Newline is also used
                                   as a sentinel at the end of the buffer.  */
  struct dfamust *musts;        /* List of strings, at least one of which
                                   is known to appear in any r.e. matching
                                   the dfa.  */
  position_set mb_follows;	/* Follow set added by ANYCHAR and/or MBCSET
                                   on demand.  */
  int *mb_match_lens;           /* Array of length reduced by ANYCHAR and/or
                                   MBCSET.  Null if mb_follows.elems has not
                                   been allocated.  */
};

/* Some macros for user access to dfa internals.  */

/* S could possibly be an accepting state of R.  */
#define ACCEPTING(s, r) ((r).states[s].constraint)

/* STATE accepts in the specified context.  */
#define ACCEPTS_IN_CONTEXT(prev, curr, state, dfa) \
  SUCCEEDS_IN_CONTEXT ((dfa).states[state].constraint, prev, curr)

static void dfamust (struct dfa *dfa);
static void regexp (void);

static void
dfambcache (struct dfa *d)
{
  int i;
  for (i = CHAR_MIN; i <= CHAR_MAX; ++i)
    {
      char c = i;
      unsigned char uc = i;
      mbstate_t s = { 0 };
      wchar_t wc;
      d->mbrtowc_cache[uc] = mbrtowc (&wc, &c, 1, &s) <= 1 ? wc : WEOF;
    }
}

/* Store into *PWC the result of converting the leading bytes of the
   multibyte buffer S of length N bytes, using the mbrtowc_cache in *D
   and updating the conversion state in *D.  On conversion error,
   convert just a single byte, to WEOF.  Return the number of bytes
   converted.

   This differs from mbrtowc (PWC, S, N, &D->mbs) as follows:

   * PWC points to wint_t, not to wchar_t.
   * The last arg is a dfa *D instead of merely a multibyte conversion
     state D->mbs.  D also contains an mbrtowc_cache for speed.
   * N must be at least 1.
   * S[N - 1] must be a sentinel byte.
   * Shift encodings are not supported.
   * The return value is always in the range 1..N.
   * D->mbs is always valid afterwards.
   * *PWC is always set to something.  */
static size_t
mbs_to_wchar (wint_t *pwc, char const *s, size_t n, struct dfa *d)
{
  unsigned char uc = s[0];
  wint_t wc = d->mbrtowc_cache[uc];

  if (wc == WEOF)
    {
      wchar_t wch;
      size_t nbytes = mbrtowc (&wch, s, n, &d->mbs);
      if (0 < nbytes && nbytes < (size_t) -2)
        {
          *pwc = wch;
          return nbytes;
        }
      memset (&d->mbs, 0, sizeof d->mbs);
    }

  *pwc = wc;
  return 1;
}

#ifdef DEBUG

static void
prtok (token t)
{
  char const *s;

  if (t < 0)
    fprintf (stderr, "END");
  else if (t < NOTCHAR)
    {
      int ch = t;
      fprintf (stderr, "%c", ch);
    }
  else
    {
      switch (t)
        {
        case EMPTY:
          s = "EMPTY";
          break;
        case BACKREF:
          s = "BACKREF";
          break;
        case BEGLINE:
          s = "BEGLINE";
          break;
        case ENDLINE:
          s = "ENDLINE";
          break;
        case BEGWORD:
          s = "BEGWORD";
          break;
        case ENDWORD:
          s = "ENDWORD";
          break;
        case LIMWORD:
          s = "LIMWORD";
          break;
        case NOTLIMWORD:
          s = "NOTLIMWORD";
          break;
        case QMARK:
          s = "QMARK";
          break;
        case STAR:
          s = "STAR";
          break;
        case PLUS:
          s = "PLUS";
          break;
        case CAT:
          s = "CAT";
          break;
        case OR:
          s = "OR";
          break;
        case LPAREN:
          s = "LPAREN";
          break;
        case RPAREN:
          s = "RPAREN";
          break;
        case ANYCHAR:
          s = "ANYCHAR";
          break;
        case MBCSET:
          s = "MBCSET";
          break;
        default:
          s = "CSET";
          break;
        }
      fprintf (stderr, "%s", s);
    }
}
#endif /* DEBUG */

/* Stuff pertaining to charclasses.  */

static bool
tstbit (unsigned int b, charclass const c)
{
  return c[b / CHARCLASS_WORD_BITS] >> b % CHARCLASS_WORD_BITS & 1;
}

static void
setbit (unsigned int b, charclass c)
{
  c[b / CHARCLASS_WORD_BITS] |= (charclass_word) 1 << b % CHARCLASS_WORD_BITS;
}

static void
clrbit (unsigned int b, charclass c)
{
  c[b / CHARCLASS_WORD_BITS] &= ~((charclass_word) 1
                                  << b % CHARCLASS_WORD_BITS);
}

static void
copyset (charclass const src, charclass dst)
{
  memcpy (dst, src, sizeof (charclass));
}

static void
zeroset (charclass s)
{
  memset (s, 0, sizeof (charclass));
}

static void
notset (charclass s)
{
  int i;

  for (i = 0; i < CHARCLASS_WORDS; ++i)
    s[i] = CHARCLASS_WORD_MASK & ~s[i];
}

static bool
equal (charclass const s1, charclass const s2)
{
  return memcmp (s1, s2, sizeof (charclass)) == 0;
}

/* Ensure that the array addressed by PTR holds at least NITEMS +
   (PTR || !NITEMS) items.  Either return PTR, or reallocate the array
   and return its new address.  Although PTR may be null, the returned
   value is never null.

   The array holds *NALLOC items; *NALLOC is updated on reallocation.
   ITEMSIZE is the size of one item.  Avoid O(N**2) behavior on arrays
   growing linearly.  */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
  if (nitems < *nalloc)
    return ptr;
  *nalloc = nitems;
  return x2nrealloc (ptr, nalloc, itemsize);
}

/* In DFA D, find the index of charclass S, or allocate a new one.  */
static size_t
dfa_charclass_index (struct dfa *d, charclass const s)
{
  size_t i;

  for (i = 0; i < d->cindex; ++i)
    if (equal (s, d->charclasses[i]))
      return i;
  d->charclasses = maybe_realloc (d->charclasses, d->cindex, &d->calloc,
                                  sizeof *d->charclasses);
  ++d->cindex;
  copyset (s, d->charclasses[i]);
  return i;
}

/* A pointer to the current dfa is kept here during parsing.  */
static struct dfa *dfa;

/* Find the index of charclass S in the current DFA, or allocate a new one.  */
static size_t
charclass_index (charclass const s)
{
  return dfa_charclass_index (dfa, s);
}

/* Syntax bits controlling the behavior of the lexical analyzer.  */
static reg_syntax_t syntax_bits, syntax_bits_set;

/* Flag for case-folding letters into sets.  */
static bool case_fold;

/* End-of-line byte in data.  */
static unsigned char eolbyte;

/* Cache of char-context values.  */
static int sbit[NOTCHAR];

/* Set of characters considered letters.  */
static charclass letters;

/* Set of characters that are newline.  */
static charclass newline;

/* Add this to the test for whether a byte is word-constituent, since on
   BSD-based systems, many values in the 128..255 range are classified as
   alphabetic, while on glibc-based systems, they are not.  */
#ifdef __GLIBC__
# define is_valid_unibyte_character(c) 1
#else
# define is_valid_unibyte_character(c) (btowc (c) != WEOF)
#endif

/* C is a "word-constituent" byte.  */
#define IS_WORD_CONSTITUENT(C) \
  (is_valid_unibyte_character (C) && (isalnum (C) || (C) == '_'))

static int
char_context (unsigned char c)
{
  if (c == eolbyte)
    return CTX_NEWLINE;
  if (IS_WORD_CONSTITUENT (c))
    return CTX_LETTER;
  return CTX_NONE;
}

static int
wchar_context (wint_t wc)
{
  if (wc == (wchar_t) eolbyte || wc == 0)
    return CTX_NEWLINE;
  if (wc == L'_' || iswalnum (wc))
    return CTX_LETTER;
  return CTX_NONE;
}

typedef struct regex_name_mapping_struct
{
  reg_syntax_t flag;
  const char *name;
} regex_name_mapping_t;

static regex_name_mapping_t regex_names[] = {
  {RE_BACKSLASH_ESCAPE_IN_LISTS, "backslash_escape_in_lists"},
  {RE_BK_PLUS_QM,                "bk_plus_qm"},
  {RE_CHAR_CLASSES,              "char_classes"},
  {RE_CONTEXT_INDEP_ANCHORS,     "context_indep_anchors"},
  {RE_CONTEXT_INDEP_OPS,         "context_indep_ops"},
  {RE_CONTEXT_INVALID_OPS,       "context_invalid_ops"},
  {RE_DOT_NEWLINE,               "dot_newline"},
  {RE_DOT_NOT_NULL,              "dot_not_null"},
  {RE_HAT_LISTS_NOT_NEWLINE,     "hat_lists_not_newline"},
  {RE_INTERVALS,                 "intervals"},
  {RE_LIMITED_OPS,               "limited_ops"},
  {RE_NEWLINE_ALT,               "newline_alt"},
  {RE_NO_BK_BRACES,              "no_bk_braces"},
  {RE_NO_BK_PARENS,              "no_bk_parens"},
  {RE_NO_BK_REFS,                "no_bk_refs"},
  {RE_NO_BK_VBAR,                "no_bk_vbar"},
  {RE_NO_EMPTY_RANGES,           "no_empty_ranges"},
  {RE_UNMATCHED_RIGHT_PAREN_ORD, "unmatched_right_paren_ord"},
  {RE_NO_POSIX_BACKTRACKING,     "no_posix_backtracking"},
  {RE_NO_GNU_OPS,                "no_gnu_ops"},
  {RE_DEBUG,                     "debug"},
  {RE_INVALID_INTERVAL_ORD,      "invalid_interval_ord"},
  {RE_ICASE,                     "icase"},
  {RE_CARET_ANCHORS_HERE,        "caret_anchors_here"},
  {RE_CONTEXT_INVALID_DUP,       "context_invalid_dup"},
  {RE_NO_SUB,                    "no_sub"},
  {0, NULL}
};

/* HOOK: When we abort, get rid of resources that we've acquired, before
   finally handing control to the abort mechanism.  We do this to minimise
   resource allocation error reported by Valgrind; I believe that it is not
   vital to do so otherwise (but I may be wrong).  */
static void
HOOK_tear_down_additional_infrastructure (void)
{
/* Modules destroy themselves via a module-wide atexit () registration.  */
#if 0
  fsalex_destroy_module ();
  fsaparse_destroy_module ();
  charclass_destroy_module ();
#endif /* 0 */

  parser = NULL;
  lexer = NULL;
  /* fsatoken, proto-lexparse and dfa-prl itself do not need
     attention (yet?).  */

  /* Release the "musts" list we compiled earlier.  */
  {
    fsamusts_list_element_t *mlist;
    fsamusts_list_element_t *mnext;

    for (mlist = musts; mlist; mlist = mnext)
      {
       mnext = mlist->next;
       free (mlist->must);
       free (mlist);
      }

    musts = NULL;
  }
}

/* HOOK: Provide error/warning functions that include logging.  */
static void
lexer_warn(const char *warning)
{
  if (! pll_log)
    pll_log = fopen (HOOK_LOG_FILENAME, "a");
  fprintf (pll_log, "lexer warning: %s\n", warning);
  fflush (NULL);
  dfawarn (warning);
}
static _Noreturn void
lexer_error(const char *reason)
{
  if (! pll_log)
    pll_log = fopen (HOOK_LOG_FILENAME, "a");
  fprintf (pll_log, "lexer error: %s\n", reason);
  HOOK_tear_down_additional_infrastructure ();
  fflush (NULL);
  dfaerror (reason);
}

static void
parser_warn(const char *warning)
{
  if (! pll_log)
    pll_log = fopen (HOOK_LOG_FILENAME, "a");
  fprintf (pll_log, "parser warning: %s\n", warning);
  fflush (NULL);
  dfawarn (warning);
}
static _Noreturn void
parser_error(const char *reason)
{
  if (! pll_log)
    pll_log = fopen (HOOK_LOG_FILENAME, "a");
  fprintf (pll_log, "parser error: %s\n", reason);
  HOOK_tear_down_additional_infrastructure ();
  fflush (NULL);
  dfaerror (reason);
}

static void
HOOK_set_up_fsa_stuff_if_not_done_already (void)
{
  /* If lexer context is present, this function has been run previously.  */
  if (lexer != NULL)
    return;

  /* Initialise modules before creating instances.  */
  fsalex_initialise ();
  fsaparse_initialise ();

  /* Create a new lexer instance, and give it error/warning fns  */
  lexer = fsalex_new ();
  fsalex_exception_fns (lexer, lexer_warn, lexer_error);

  /* Start with a pool of 10 charclasses.  */
  charclass_initialise (10);

  /* Iniialise multibyte-char-sets module.  */
  mbcsets_initialise ();

  /* Create a new parser instance, and give it error/warning functions.
     Defer connecting up the lexer until lexer syntax has been set up.  */
  parser = fsaparse_new ();
  fsaparse_exception_fns (parser, parser_warn, parser_error);
}

/* Entry point to set syntax options.  */
void
dfasyntax (reg_syntax_t bits, int fold, unsigned char eol)
{
  unsigned int i;

  /* Hook: Debug buffer to record search syntax specifications.  */
  static char buf[256];
  char *p_buf;
  char *locale;

  syntax_bits_set = 1;
  syntax_bits = bits;
  case_fold = fold != 0;
  eolbyte = eol;

  HOOK_set_up_fsa_stuff_if_not_done_already ();

  /* HOOK: Tell fsalex module about syntax selections.  */
  fsalex_syntax (lexer, bits, fold, eol);

  /* HOOK: Record syntax selections in debug logfile.  */
  if (! pll_log)
    pll_log = fopen (HOOK_LOG_FILENAME, "a");
  locale = setlocale (LC_ALL, NULL);
  fprintf (pll_log, "\nSyntax: Case fold: %d; eol char: %02x; locale: %s",
          fold, (int) eol, locale);
  p_buf = buf;
  *p_buf++ = '\n';
  *p_buf++ = ' ';
  *p_buf   = '\0';
  for (i = 0; regex_names[i].name; i++)
    {
      char flag_char = (bits & regex_names[i].flag) ? '+' : '-';
      p_buf += sprintf(p_buf, " %c%s", flag_char, regex_names[i].name);
      if (strlen (buf) >= 82)
        {
          fprintf (pll_log, "%s", buf);
          p_buf = &buf[2];
          *p_buf = '\0';
        }
    }
  fprintf (pll_log, "%s\n", buf);

  /* Okay, now the parser can talk to a lexer that knows locale and syntax
     configuration details.  */
  fsaparse_lexer (parser, lexer,
                  (proto_lexparse_lex_fn_t *) fsalex_lex,
                  (proto_lexparse_exchange_fn_t *) fsalex_exchange);

  for (i = 0; i < NOTCHAR; ++i)
    {
      sbit[i] = char_context (i);
      switch (sbit[i])
        {
        case CTX_LETTER:
          setbit (i, letters);
          break;
        case CTX_NEWLINE:
          setbit (i, newline);
          break;
        }
    }
}

/* Set a bit in the charclass for the given wchar_t.  Do nothing if WC
   is represented by a multi-byte sequence.  Even for MB_CUR_MAX == 1,
   this may happen when folding case in weird Turkish locales where
   dotless i/dotted I are not included in the chosen character set.
   Return whether a bit was set in the charclass.  */
static bool
setbit_wc (wint_t wc, charclass c)
{
  int b = wctob (wc);
  if (b == EOF)
    return false;

  setbit (b, c);
  return true;
}

/* Set a bit for B and its case variants in the charclass C.
   MB_CUR_MAX must be 1.  */
static void
setbit_case_fold_c (int b, charclass c)
{
  int ub = toupper (b);
  int i;
  for (i = 0; i < NOTCHAR; i++)
    if (toupper (i) == ub)
      setbit (i, c);
}



/* UTF-8 encoding allows some optimizations that we can't otherwise
   assume in a multibyte encoding.  */
int
using_utf8 (void)
{
  static int utf8 = -1;
  if (utf8 < 0)
    {
      wchar_t wc;
      mbstate_t mbs = { 0 };
      utf8 = mbrtowc (&wc, "\xc4\x80", 2, &mbs) == 2 && wc == 0x100;
    }
  return utf8;
}

/* The current locale is known to be a unibyte locale
   without multicharacter collating sequences and where range
   comparisons simply use the native encoding.  These locales can be
   processed more efficiently.  */

static bool
using_simple_locale (void)
{
  /* The native character set is known to be compatible with
     the C locale.  The following test isn't perfect, but it's good
     enough in practice, as only ASCII and EBCDIC are in common use
     and this test correctly accepts ASCII and rejects EBCDIC.  */
  enum { native_c_charset =
    ('\b' == 8 && '\t' == 9 && '\n' == 10 && '\v' == 11 && '\f' == 12
     && '\r' == 13 && ' ' == 32 && '!' == 33 && '"' == 34 && '#' == 35
     && '%' == 37 && '&' == 38 && '\'' == 39 && '(' == 40 && ')' == 41
     && '*' == 42 && '+' == 43 && ',' == 44 && '-' == 45 && '.' == 46
     && '/' == 47 && '0' == 48 && '9' == 57 && ':' == 58 && ';' == 59
     && '<' == 60 && '=' == 61 && '>' == 62 && '?' == 63 && 'A' == 65
     && 'Z' == 90 && '[' == 91 && '\\' == 92 && ']' == 93 && '^' == 94
     && '_' == 95 && 'a' == 97 && 'z' == 122 && '{' == 123 && '|' == 124
     && '}' == 125 && '~' == 126)
  };

  if (! native_c_charset || dfa->multibyte)
    return false;
  else
    {
      static int unibyte_c = -1;
      if (unibyte_c < 0)
        {
          char const *locale = setlocale (LC_ALL, NULL);
          unibyte_c = (!locale
                       || STREQ (locale, "C")
                       || STREQ (locale, "POSIX"));
        }
      return unibyte_c;
    }
}

/* Lexical analyzer.  All the dross that deals with the obnoxious
   GNU Regex syntax bits is located here.  The poor, suffering
   reader is referred to the GNU Regex documentation for the
   meaning of the @#%!@#%^!@ syntax bits.  */

static char const *lexptr;      /* Pointer to next input character.  */
static size_t lexleft;          /* Number of characters remaining.  */
static token lasttok;           /* Previous token returned; initially END.  */
static bool laststart;		/* We're separated from beginning or (,
                                   | only by zero-width characters.  */
static size_t parens;           /* Count of outstanding left parens.  */
static int minrep, maxrep;      /* Repeat counts for {m,n}.  */

static int cur_mb_len = 1;      /* Length of the multibyte representation of
                                   wctok.  */

static wint_t wctok;		/* Wide character representation of the current
                                   multibyte character, or WEOF if there was
                                   an encoding error.  Used only if
                                   MB_CUR_MAX > 1.  */


/* Fetch the next lexical input character.  Set C (of type int) to the
   next input byte, except set C to EOF if the input is a multibyte
   character of length greater than 1.  Set WC (of type wint_t) to the
   value of the input if it is a valid multibyte character (possibly
   of length 1); otherwise set WC to WEOF.  If there is no more input,
   report EOFERR if EOFERR is not null, and return lasttok = END
   otherwise.  */
# define FETCH_WC(c, wc, eoferr)		\
  do {						\
    if (! lexleft)				\
      {						\
        if ((eoferr) != 0)			\
          dfaerror (eoferr);			\
        else					\
          return lasttok = END;			\
      }						\
    else					\
      {						\
        wint_t _wc;				\
        size_t nbytes = mbs_to_wchar (&_wc, lexptr, lexleft, dfa); \
        cur_mb_len = nbytes;			\
        (wc) = _wc;				\
        (c) = nbytes == 1 ? to_uchar (*lexptr) : EOF;    \
        lexptr += nbytes;			\
        lexleft -= nbytes;			\
      }						\
  } while (0)

#ifndef MIN
# define MIN(a,b) ((a) < (b) ? (a) : (b))
#endif

/* The set of wchar_t values C such that there's a useful locale
   somewhere where C != towupper (C) && C != towlower (towupper (C)).
   For example, 0x00B5 (U+00B5 MICRO SIGN) is in this table, because
   towupper (0x00B5) == 0x039C (U+039C GREEK CAPITAL LETTER MU), and
   towlower (0x039C) == 0x03BC (U+03BC GREEK SMALL LETTER MU).  */
static short const lonesome_lower[] =
  {
    0x00B5, 0x0131, 0x017F, 0x01C5, 0x01C8, 0x01CB, 0x01F2, 0x0345,
    0x03C2, 0x03D0, 0x03D1, 0x03D5, 0x03D6, 0x03F0, 0x03F1,

    /* U+03F2 GREEK LUNATE SIGMA SYMBOL lacks a specific uppercase
       counterpart in locales predating Unicode 4.0.0 (April 2003).  */
    0x03F2,

    0x03F5, 0x1E9B, 0x1FBE,
  };

/* Maximum number of characters that can be the case-folded
   counterparts of a single character, not counting the character
   itself.  This is 1 for towupper, 1 for towlower, and 1 for each
   entry in LONESOME_LOWER.  */
enum
{ CASE_FOLDED_BUFSIZE = 2 + sizeof lonesome_lower / sizeof *lonesome_lower };

/* Find the characters equal to C after case-folding, other than C
   itself, and store them into FOLDED.  Return the number of characters
   stored.  */
static int
case_folded_counterparts (wchar_t c, wchar_t folded[CASE_FOLDED_BUFSIZE])
{
  int i;
  int n = 0;
  wint_t uc = towupper (c);
  wint_t lc = towlower (uc);
  if (uc != c)
    folded[n++] = uc;
  if (lc != uc && lc != c && towupper (lc) == uc)
    folded[n++] = lc;
  for (i = 0; i < sizeof lonesome_lower / sizeof *lonesome_lower; i++)
    {
      wint_t li = lonesome_lower[i];
      if (li != lc && li != uc && li != c && towupper (li) == uc)
        folded[n++] = li;
    }
  return n;
}

typedef int predicate (int);

/* The following list maps the names of the Posix named character classes
   to predicate functions that determine whether a given character is in
   the class.  The leading [ has already been eaten by the lexical
   analyzer.  */
struct dfa_ctype
{
  const char *name;
  predicate *func;
  bool single_byte_only;
};

static const struct dfa_ctype prednames[] = {
  {"alpha", isalpha, false},
  {"upper", isupper, false},
  {"lower", islower, false},
  {"digit", isdigit, true},
  {"xdigit", isxdigit, false},
  {"space", isspace, false},
  {"punct", ispunct, false},
  {"alnum", isalnum, false},
  {"print", isprint, false},
  {"graph", isgraph, false},
  {"cntrl", iscntrl, false},
  {"blank", isblank, false},
  {NULL, NULL, false}
};

static const struct dfa_ctype *_GL_ATTRIBUTE_PURE
find_pred (const char *str)
{
  unsigned int i;
  for (i = 0; prednames[i].name; ++i)
    if (STREQ (str, prednames[i].name))
      break;

  return &prednames[i];
}

/* Multibyte character handling sub-routine for lex.
   Parse a bracket expression and build a struct mb_char_classes.  */
static token
parse_bracket_exp (void)
{
  bool invert;
  int c, c1, c2;
  charclass ccl;

  /* This is a bracket expression that dfaexec is known to
     process correctly.  */
  bool known_bracket_exp = true;

  /* Used to warn about [:space:].
     Bit 0 = first character is a colon.
     Bit 1 = last character is a colon.
     Bit 2 = includes any other character but a colon.
     Bit 3 = includes ranges, char/equiv classes or collation elements.  */
  int colon_warning_state;

  wint_t wc;
  wint_t wc2;
  wint_t wc1 = 0;

  /* Work area to build a mb_char_classes.  */
  struct mb_char_classes *work_mbc;
  size_t chars_al, ranges_al, ch_classes_al, equivs_al, coll_elems_al;

  chars_al = ranges_al = ch_classes_al = equivs_al = coll_elems_al = 0;
  if (dfa->multibyte)
    {
      dfa->mbcsets = maybe_realloc (dfa->mbcsets, dfa->nmbcsets,
                                    &dfa->mbcsets_alloc,
                                    sizeof *dfa->mbcsets);

      /* dfa->multibyte_prop[] hold the index of dfa->mbcsets.
         We will update dfa->multibyte_prop[] in addtok, because we can't
         decide the index in dfa->tokens[].  */

      /* Initialize work area.  */
      work_mbc = &(dfa->mbcsets[dfa->nmbcsets++]);
      memset (work_mbc, 0, sizeof *work_mbc);
    }
  else
    work_mbc = NULL;

  memset (ccl, 0, sizeof ccl);
  FETCH_WC (c, wc, _("unbalanced ["));
  if (c == '^')
    {
      FETCH_WC (c, wc, _("unbalanced ["));
      invert = true;
      known_bracket_exp = using_simple_locale ();
    }
  else
    invert = false;

  colon_warning_state = (c == ':');
  do
    {
      c1 = NOTCHAR;	/* Mark c1 as not initialized.  */
      colon_warning_state &= ~2;

      /* Note that if we're looking at some other [:...:] construct,
         we just treat it as a bunch of ordinary characters.  We can do
         this because we assume regex has checked for syntax errors before
         dfa is ever called.  */
      if (c == '[')
        {
          FETCH_WC (c1, wc1, _("unbalanced ["));

          if ((c1 == ':' && (syntax_bits & RE_CHAR_CLASSES))
              || c1 == '.' || c1 == '=')
            {
              enum { MAX_BRACKET_STRING_LEN = 32 };
              char str[MAX_BRACKET_STRING_LEN + 1];
              size_t len = 0;
              for (;;)
                {
                  FETCH_WC (c, wc, _("unbalanced ["));
                  if ((c == c1 && *lexptr == ']') || lexleft == 0)
                    break;
                  if (len < MAX_BRACKET_STRING_LEN)
                    str[len++] = c;
                  else
                    /* This is in any case an invalid class name.  */
                    str[0] = '\0';
                }
              str[len] = '\0';

              /* Fetch bracket.  */
              FETCH_WC (c, wc, _("unbalanced ["));
              if (c1 == ':')
                /* Build character class.  POSIX allows character
                   classes to match multicharacter collating elements,
                   but the regex code does not support that, so do not
                   worry about that possibility.  */
                {
                  char const *class
                    = (case_fold && (STREQ (str, "upper")
                                     || STREQ (str, "lower")) ? "alpha" : str);
                  const struct dfa_ctype *pred = find_pred (class);
                  if (!pred)
                    dfaerror (_("invalid character class"));

                  if (dfa->multibyte && !pred->single_byte_only)
                    {
                      /* Store the character class as wctype_t.  */
                      wctype_t wt = wctype (class);

                      work_mbc->ch_classes
                        = maybe_realloc (work_mbc->ch_classes,
                                         work_mbc->nch_classes, &ch_classes_al,
                                         sizeof *work_mbc->ch_classes);
                      work_mbc->ch_classes[work_mbc->nch_classes++] = wt;
                    }

                  for (c2 = 0; c2 < NOTCHAR; ++c2)
                    if (pred->func (c2))
                      setbit (c2, ccl);
                }
              else
                known_bracket_exp = false;

              colon_warning_state |= 8;

              /* Fetch new lookahead character.  */
              FETCH_WC (c1, wc1, _("unbalanced ["));
              continue;
            }

          /* We treat '[' as a normal character here.  c/c1/wc/wc1
             are already set up.  */
        }

      if (c == '\\' && (syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
        FETCH_WC (c, wc, _("unbalanced ["));

      if (c1 == NOTCHAR)
        FETCH_WC (c1, wc1, _("unbalanced ["));

      if (c1 == '-')
        /* build range characters.  */
        {
          FETCH_WC (c2, wc2, _("unbalanced ["));

          /* A bracket expression like [a-[.aa.]] matches an unknown set.
             Treat it like [-a[.aa.]] while parsing it, and
             remember that the set is unknown.  */
          if (c2 == '[' && *lexptr == '.')
            {
              known_bracket_exp = false;
              c2 = ']';
            }

          if (c2 != ']')
            {
              if (c2 == '\\' && (syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
                FETCH_WC (c2, wc2, _("unbalanced ["));

              if (dfa->multibyte)
                {
                  /* When case folding map a range, say [m-z] (or even [M-z])
                     to the pair of ranges, [m-z] [M-Z].  Although this code
                     is wrong in multiple ways, it's never used in practice.
                     FIXME: Remove this (and related) unused code.  */
                  if (wc != WEOF && wc2 != WEOF)
                    {
                      work_mbc->ranges
                        = maybe_realloc (work_mbc->ranges, work_mbc->nranges + 2,
                                         &ranges_al, sizeof *work_mbc->ranges);
                      work_mbc->ranges[work_mbc->nranges].beg
                        = case_fold ? towlower (wc) : wc;
                      work_mbc->ranges[work_mbc->nranges++].end
                        = case_fold ? towlower (wc2) : wc2;

                      if (case_fold && (iswalpha (wc) || iswalpha (wc2)))
                        {
                          work_mbc->ranges[work_mbc->nranges].beg = towupper (wc);
                          work_mbc->ranges[work_mbc->nranges++].end
                            = towupper (wc2);
                        }
                    }
                }
              else if (using_simple_locale ())
                {
                  for (c1 = c; c1 <= c2; c1++)
                    setbit (c1, ccl);
                  if (case_fold)
                    {
                      int uc = toupper (c);
                      int uc2 = toupper (c2);
                      for (c1 = 0; c1 < NOTCHAR; c1++)
                        {
                          int uc1 = toupper (c1);
                          if (uc <= uc1 && uc1 <= uc2)
                            setbit (c1, ccl);
                        }
                    }
                }
              else
                known_bracket_exp = false;

              colon_warning_state |= 8;
              FETCH_WC (c1, wc1, _("unbalanced ["));
              continue;
            }

          /* In the case [x-], the - is an ordinary hyphen,
             which is left in c1, the lookahead character.  */
          lexptr -= cur_mb_len;
          lexleft += cur_mb_len;
        }

      colon_warning_state |= (c == ':') ? 2 : 4;

      if (!dfa->multibyte)
        {
          if (case_fold)
            setbit_case_fold_c (c, ccl);
          else
            setbit (c, ccl);
          continue;
        }

      if (wc == WEOF)
        known_bracket_exp = false;
      else
        {
          wchar_t folded[CASE_FOLDED_BUFSIZE + 1];
          int i;
          int n = (case_fold ? case_folded_counterparts (wc, folded + 1) + 1
                   : 1);
          folded[0] = wc;
          for (i = 0; i < n; i++)
            if (!setbit_wc (folded[i], ccl))
              {
                work_mbc->chars
                  = maybe_realloc (work_mbc->chars, work_mbc->nchars,
                                   &chars_al, sizeof *work_mbc->chars);
                work_mbc->chars[work_mbc->nchars++] = folded[i];
              }
        }
    }
  while ((wc = wc1, (c = c1) != ']'));

  if (colon_warning_state == 7)
    dfawarn (_("character class syntax is [[:space:]], not [:space:]"));

  if (! known_bracket_exp)
    return BACKREF;

  if (dfa->multibyte)
    {
      static charclass zeroclass;
      work_mbc->invert = invert;
      work_mbc->cset = equal (ccl, zeroclass) ? -1 : charclass_index (ccl);
      return MBCSET;
    }

  if (invert)
    {
      assert (!dfa->multibyte);
      notset (ccl);
      if (syntax_bits & RE_HAT_LISTS_NOT_NEWLINE)
        clrbit (eolbyte, ccl);
    }

  return CSET + charclass_index (ccl);
}

static token
original_lex (void)
{
  int c, c2;
  bool backslash = false;
  charclass ccl;
  int i;

  /* Basic plan: We fetch a character.  If it's a backslash,
     we set the backslash flag and go through the loop again.
     On the plus side, this avoids having a duplicate of the
     main switch inside the backslash case.  On the minus side,
     it means that just about every case begins with
     "if (backslash) ...".  */
  for (i = 0; i < 2; ++i)
    {
      FETCH_WC (c, wctok, NULL);

      switch (c)
        {
        case '\\':
          if (backslash)
            goto normal_char;
          if (lexleft == 0)
            dfaerror (_("unfinished \\ escape"));
          backslash = true;
          break;

        case '^':
          if (backslash)
            goto normal_char;
          if (syntax_bits & RE_CONTEXT_INDEP_ANCHORS
              || lasttok == END || lasttok == LPAREN || lasttok == OR)
            return lasttok = BEGLINE;
          goto normal_char;

        case '$':
          if (backslash)
            goto normal_char;
          if (syntax_bits & RE_CONTEXT_INDEP_ANCHORS
              || lexleft == 0
              || (syntax_bits & RE_NO_BK_PARENS
                  ? lexleft > 0 && *lexptr == ')'
                  : lexleft > 1 && lexptr[0] == '\\' && lexptr[1] == ')')
              || (syntax_bits & RE_NO_BK_VBAR
                  ? lexleft > 0 && *lexptr == '|'
                  : lexleft > 1 && lexptr[0] == '\\' && lexptr[1] == '|')
              || ((syntax_bits & RE_NEWLINE_ALT)
                  && lexleft > 0 && *lexptr == '\n'))
            return lasttok = ENDLINE;
          goto normal_char;

        case '1':
        case '2':
        case '3':
        case '4':
        case '5':
        case '6':
        case '7':
        case '8':
        case '9':
          if (backslash && !(syntax_bits & RE_NO_BK_REFS))
            {
              laststart = false;
              return lasttok = BACKREF;
            }
          goto normal_char;

        case '`':
          if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
            return lasttok = BEGLINE; /* FIXME: should be beginning of string */
          goto normal_char;

        case '\'':
          if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
            return lasttok = ENDLINE;   /* FIXME: should be end of string */
          goto normal_char;

        case '<':
          if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
            return lasttok = BEGWORD;
          goto normal_char;

        case '>':
          if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
            return lasttok = ENDWORD;
          goto normal_char;

        case 'b':
          if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
            return lasttok = LIMWORD;
          goto normal_char;

        case 'B':
          if (backslash && !(syntax_bits & RE_NO_GNU_OPS))
            return lasttok = NOTLIMWORD;
          goto normal_char;

        case '?':
          if (syntax_bits & RE_LIMITED_OPS)
            goto normal_char;
          if (backslash != ((syntax_bits & RE_BK_PLUS_QM) != 0))
            goto normal_char;
          if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
            goto normal_char;
          return lasttok = QMARK;

        case '*':
          if (backslash)
            goto normal_char;
          if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
            goto normal_char;
          return lasttok = STAR;

        case '+':
          if (syntax_bits & RE_LIMITED_OPS)
            goto normal_char;
          if (backslash != ((syntax_bits & RE_BK_PLUS_QM) != 0))
            goto normal_char;
          if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
            goto normal_char;
          return lasttok = PLUS;

        case '{':
          if (!(syntax_bits & RE_INTERVALS))
            goto normal_char;
          if (backslash != ((syntax_bits & RE_NO_BK_BRACES) == 0))
            goto normal_char;
          if (!(syntax_bits & RE_CONTEXT_INDEP_OPS) && laststart)
            goto normal_char;

          /* Cases:
             {M} - exact count
             {M,} - minimum count, maximum is infinity
             {,N} - 0 through N
             {,} - 0 to infinity (same as '*')
             {M,N} - M through N */
          {
            char const *p = lexptr;
            char const *lim = p + lexleft;
            minrep = maxrep = -1;
            for (; p != lim && ISASCIIDIGIT (*p); p++)
              {
                if (minrep < 0)
                  minrep = *p - '0';
                else
                  minrep = MIN (RE_DUP_MAX + 1, minrep * 10 + *p - '0');
              }
            if (p != lim)
              {
                if (*p != ',')
                  maxrep = minrep;
                else
                  {
                    if (minrep < 0)
                      minrep = 0;
                    while (++p != lim && ISASCIIDIGIT (*p))
                      {
                        if (maxrep < 0)
                          maxrep = *p - '0';
                        else
                          maxrep = MIN (RE_DUP_MAX + 1, maxrep * 10 + *p - '0');
                      }
                  }
              }
            if (! ((! backslash || (p != lim && *p++ == '\\'))
                   && p != lim && *p++ == '}'
                   && 0 <= minrep && (maxrep < 0 || minrep <= maxrep)))
              {
                if (syntax_bits & RE_INVALID_INTERVAL_ORD)
                  goto normal_char;
                dfaerror (_("invalid content of \\{\\}"));
              }
            if (RE_DUP_MAX < maxrep)
              dfaerror (_("regular expression too big"));
            lexptr = p;
            lexleft = lim - p;
          }
          laststart = false;
          return lasttok = REPMN;

        case '|':
          if (syntax_bits & RE_LIMITED_OPS)
            goto normal_char;
          if (backslash != ((syntax_bits & RE_NO_BK_VBAR) == 0))
            goto normal_char;
          laststart = true;
          return lasttok = OR;

        case '\n':
          if (syntax_bits & RE_LIMITED_OPS
              || backslash || !(syntax_bits & RE_NEWLINE_ALT))
            goto normal_char;
          laststart = true;
          return lasttok = OR;

        case '(':
          if (backslash != ((syntax_bits & RE_NO_BK_PARENS) == 0))
            goto normal_char;
          ++parens;
          laststart = true;
          return lasttok = LPAREN;

        case ')':
          if (backslash != ((syntax_bits & RE_NO_BK_PARENS) == 0))
            goto normal_char;
          if (parens == 0 && syntax_bits & RE_UNMATCHED_RIGHT_PAREN_ORD)
            goto normal_char;
          --parens;
          laststart = false;
          return lasttok = RPAREN;

        case '.':
          if (backslash)
            goto normal_char;
          if (dfa->multibyte)
            {
              /* In multibyte environment period must match with a single
                 character not a byte.  So we use ANYCHAR.  */
              laststart = false;
              return lasttok = ANYCHAR;
            }
          zeroset (ccl);
          notset (ccl);
          if (!(syntax_bits & RE_DOT_NEWLINE))
            clrbit (eolbyte, ccl);
          if (syntax_bits & RE_DOT_NOT_NULL)
            clrbit ('\0', ccl);
          laststart = false;
          return lasttok = CSET + charclass_index (ccl);

        case 's':
        case 'S':
          if (!backslash || (syntax_bits & RE_NO_GNU_OPS))
            goto normal_char;
          if (!dfa->multibyte)
            {
              zeroset (ccl);
              for (c2 = 0; c2 < NOTCHAR; ++c2)
                if (isspace (c2))
                  setbit (c2, ccl);
              if (c == 'S')
                notset (ccl);
              laststart = false;
              return lasttok = CSET + charclass_index (ccl);
            }

#define PUSH_LEX_STATE(s)			\
  do						\
    {						\
      char const *lexptr_saved = lexptr;	\
      size_t lexleft_saved = lexleft;		\
      lexptr = (s);				\
      lexleft = strlen (lexptr)

#define POP_LEX_STATE()				\
      lexptr = lexptr_saved;			\
      lexleft = lexleft_saved;			\
    }						\
  while (0)

          /* FIXME: see if optimizing this, as is done with ANYCHAR and
             add_utf8_anychar, makes sense.  */

          /* \s and \S are documented to be equivalent to [[:space:]] and
             [^[:space:]] respectively, so tell the lexer to process those
             strings, each minus its "already processed" '['.  */
          PUSH_LEX_STATE (c == 's' ? "[:space:]]" : "^[:space:]]");

          lasttok = parse_bracket_exp ();

          POP_LEX_STATE ();

          laststart = false;
          return lasttok;

        case 'w':
        case 'W':
          if (!backslash || (syntax_bits & RE_NO_GNU_OPS))
            goto normal_char;
          zeroset (ccl);
          for (c2 = 0; c2 < NOTCHAR; ++c2)
            if (IS_WORD_CONSTITUENT (c2))
              setbit (c2, ccl);
          if (c == 'W')
            notset (ccl);
          laststart = false;
          return lasttok = CSET + charclass_index (ccl);

        case '[':
          if (backslash)
            goto normal_char;
          laststart = false;
          return lasttok = parse_bracket_exp ();

        default:
        normal_char:
          laststart = false;
          /* For multibyte character sets, folding is done in atom.  Always
             return WCHAR.  */
          if (dfa->multibyte)
            return lasttok = WCHAR;

          if (case_fold && isalpha (c))
            {
              zeroset (ccl);
              setbit_case_fold_c (c, ccl);
              return lasttok = CSET + charclass_index (ccl);
            }

          return lasttok = c;
        }
    }

  /* The above loop should consume at most a backslash
     and some other character.  */
  abort ();
  return END;                   /* keeps pedantic compilers happy.  */
}

static token
lex (void)
{
  token            original_token;
  fsatoken_token_t fsalex_token;
  const char *alarm;

  original_token = original_lex ();
  fsalex_token   = fsalex_lex (lexer);
  alarm = "";
  if ((original_token != fsalex_token) && (original_token < CSET))
    alarm = " *** ERROR: Token mismatch";
  fprintf (pll_log, "Token debug: Original, fsalex: %08lx %08lx %s\n",
         original_token, fsalex_token, alarm);

  if (fsalex_token == FSATOKEN_TK_REPMN)
    {
      int x_minrep, x_maxrep;
      x_minrep = fsalex_exchange(lexer,
                                 PROTO_LEXPARSE_OP_GET_REPMN_MIN, NULL);
      x_maxrep = fsalex_exchange(lexer,
                                 PROTO_LEXPARSE_OP_GET_REPMN_MAX, NULL);
      fprintf (pll_log, "       Original REPMN{%d,%d};  ", minrep, maxrep);
      fprintf (pll_log, "  FSATOKEN_TK_REPMN{%d,%d}\n", x_minrep, x_maxrep);
      if ((x_minrep != minrep) || (x_maxrep != maxrep))
        fprintf (pll_log, " *** ERROR: Min, max rep doesn't match\n");
    }

  else if (fsalex_token >= FSATOKEN_TK_CSET)
    {
      size_t index;
      unsigned int * orig_ccl;
      int i;
      charclass_t *charset;
      char *description;
      static char buf[256];
      char *p_buf;

/* Nybble (4bit)-to-char conversion array for little-bit-endian nybbles.  */
static const char *disp_nybble = "084c2a6e195d3b7f";

      /* Report details of the original charclas produced by dfa.c.  */
      index = original_token - CSET;
      p_buf = buf;
      orig_ccl = dfa->charclasses[index];
      for (i = 0; i < CHARCLASS_WORDS; i += 2)
        {
          charclass_word j = orig_ccl[i];
          *p_buf++ = ' ';
          *p_buf++ = disp_nybble[(j >>  0) & 0x0f];
          *p_buf++ = disp_nybble[(j >>  4) & 0x0f];
          *p_buf++ = disp_nybble[(j >>  8) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 12) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 16) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 20) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 24) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 28) & 0x0f];

          j = orig_ccl[i + 1];
          *p_buf++ = disp_nybble[(j >>  0) & 0x0f];
          *p_buf++ = disp_nybble[(j >>  4) & 0x0f];
          *p_buf++ = disp_nybble[(j >>  8) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 12) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 16) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 20) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 24) & 0x0f];
          *p_buf++ = disp_nybble[(j >> 28) & 0x0f];
        }
      *p_buf++ = '\0';
      fprintf (pll_log, "              original [%3lu]:%s\n", index, buf);

      /* Also report the charclass member details from fsalex etc.  */
      index = fsalex_token - FSATOKEN_TK_CSET;
      charset = charclass_get_pointer (index);
      description = charclass_describe (charset);
      index = charclass_get_index (charset);
      fprintf (pll_log, "    fsalex: [%3lu] %s\n", index, description);
      if (! charclass_equal (charset, (charclass_t const *) orig_ccl))
        fprintf (pll_log, " *** ERROR: Charclass mismatch\n");
    }

  return original_token;
}

static void
show_musts (const char *title, fsamusts_list_element_t *list)
{
  fsamusts_list_element_t *elem;
  static char buf[256];
  char *p_buf;

  fprintf (pll_log, "\n%s:\n", title);

  p_buf = buf;
  for (elem = list; elem != NULL; elem = elem->next)
    {
      if (((p_buf - buf) + 4 + strlen (elem->must)) > 72)
        {
          fprintf(pll_log, " %s\n", buf);
          p_buf = buf;
        }
        p_buf += sprintf (p_buf, " (%s) >%s<",
                          elem->exact ? "Entire" : "partial",
                          elem->must);
    }
  fprintf (pll_log, "%s\n", buf);
}

/* Recursive descent parser for regular expressions.  */

static token tok;               /* Lookahead token.  */
static size_t depth;            /* Current depth of a hypothetical stack
                                   holding deferred productions.  This is
                                   used to determine the depth that will be
                                   required of the real stack later on in
                                   dfaanalyze.  */

static void
addtok_mb (token t, int mbprop)
{
  if (dfa->talloc == dfa->tindex)
    {
      dfa->tokens = x2nrealloc (dfa->tokens, &dfa->talloc,
                                sizeof *dfa->tokens);
      if (dfa->multibyte)
        dfa->multibyte_prop = xnrealloc (dfa->multibyte_prop, dfa->talloc,
                                         sizeof *dfa->multibyte_prop);
    }
  if (dfa->multibyte)
    dfa->multibyte_prop[dfa->tindex] = mbprop;
  dfa->tokens[dfa->tindex++] = t;

  switch (t)
    {
    case QMARK:
    case STAR:
    case PLUS:
      break;

    case CAT:
    case OR:
      --depth;
      break;

    case BACKREF:
      dfa->fast = false;
      /* fallthrough */
    default:
      ++dfa->nleaves;
      /* fallthrough */
    case EMPTY:
      ++depth;
      break;
    }
  if (depth > dfa->depth)
    dfa->depth = depth;
}

static void addtok_wc (wint_t wc);

/* Add the given token to the parse tree, maintaining the depth count and
   updating the maximum depth if necessary.  */
static void
addtok (token t)
{
  if (dfa->multibyte && t == MBCSET)
    {
      bool need_or = false;
      struct mb_char_classes *work_mbc = &dfa->mbcsets[dfa->nmbcsets - 1];

      /* Extract wide characters into alternations for better performance.
         This does not require UTF-8.  */
      if (!work_mbc->invert)
        {
          size_t i;
          for (i = 0; i < work_mbc->nchars; i++)
            {
              addtok_wc (work_mbc->chars[i]);
              if (need_or)
                addtok (OR);
              need_or = true;
            }
          work_mbc->nchars = 0;
        }

      /* If the MBCSET is non-inverted and doesn't include neither
         character classes including multibyte characters, range
         expressions, equivalence classes nor collating elements,
         it can be replaced to a simple CSET. */
      if (work_mbc->invert
          || work_mbc->nch_classes != 0
          || work_mbc->nranges != 0
          || work_mbc->nequivs != 0 || work_mbc->ncoll_elems != 0)
        {
          addtok_mb (MBCSET, ((dfa->nmbcsets - 1) << 2) + 3);
          if (need_or)
            addtok (OR);
        }
      else
        {
          /* Characters have been handled above, so it is possible
             that the mbcset is empty now.  Do nothing in that case.  */
          if (work_mbc->cset != -1)
            {
              addtok (CSET + work_mbc->cset);
              if (need_or)
                addtok (OR);
            }
        }
    }
  else
    {
      addtok_mb (t, 3);
    }
}

/* We treat a multibyte character as a single atom, so that DFA
   can treat a multibyte character as a single expression.

   e.g., we construct the following tree from "<mb1><mb2>".
   <mb1(1st-byte)><mb1(2nd-byte)><CAT><mb1(3rd-byte)><CAT>
   <mb2(1st-byte)><mb2(2nd-byte)><CAT><mb2(3rd-byte)><CAT><CAT> */
static void
addtok_wc (wint_t wc)
{
  unsigned char buf[MB_LEN_MAX];
  mbstate_t s = { 0 };
  int i;
  size_t stored_bytes = wcrtomb ((char *) buf, wc, &s);

  if (stored_bytes != (size_t) -1)
    cur_mb_len = stored_bytes;
  else
    {
      /* This is merely stop-gap.  buf[0] is undefined, yet skipping
         the addtok_mb call altogether can corrupt the heap.  */
      cur_mb_len = 1;
      buf[0] = 0;
    }

  addtok_mb (buf[0], cur_mb_len == 1 ? 3 : 1);
  for (i = 1; i < cur_mb_len; i++)
    {
      addtok_mb (buf[i], i == cur_mb_len - 1 ? 2 : 0);
      addtok (CAT);
    }
}

static void
add_utf8_anychar (void)
{
  static const charclass utf8_classes[5] = {
    /* 80-bf: non-leading bytes.  */
    {0, 0, 0, 0, CHARCLASS_WORD_MASK, CHARCLASS_WORD_MASK, 0, 0},

    /* 00-7f: 1-byte sequence.  */
    {CHARCLASS_WORD_MASK, CHARCLASS_WORD_MASK, CHARCLASS_WORD_MASK,
     CHARCLASS_WORD_MASK, 0, 0, 0, 0},

    /* c2-df: 2-byte sequence.  */
    {0, 0, 0, 0, 0, 0, ~3 & CHARCLASS_WORD_MASK, 0},

    /* e0-ef: 3-byte sequence.  */
    {0, 0, 0, 0, 0, 0, 0, 0xffff},

    /* f0-f7: 4-byte sequence.  */
    {0, 0, 0, 0, 0, 0, 0, 0xff0000}
  };
  const unsigned int n = sizeof (utf8_classes) / sizeof (utf8_classes[0]);
  unsigned int i;

  /* Define the five character classes that are needed below.  */
  if (dfa->utf8_anychar_classes[0] == 0)
    for (i = 0; i < n; i++)
      {
        charclass c;
        copyset (utf8_classes[i], c);
        if (i == 1)
          {
            if (!(syntax_bits & RE_DOT_NEWLINE))
              clrbit (eolbyte, c);
            if (syntax_bits & RE_DOT_NOT_NULL)
              clrbit ('\0', c);
          }
        dfa->utf8_anychar_classes[i] = CSET + charclass_index (c);
      }

  /* A valid UTF-8 character is

     ([0x00-0x7f]
     |[0xc2-0xdf][0x80-0xbf]
     |[0xe0-0xef[0x80-0xbf][0x80-0xbf]
     |[0xf0-f7][0x80-0xbf][0x80-0xbf][0x80-0xbf])

     which I'll write more concisely "B|CA|DAA|EAAA".  Factor the [0x00-0x7f]
     and you get "B|(C|(D|EA)A)A".  And since the token buffer is in reverse
     Polish notation, you get "B C D E A CAT OR A CAT OR A CAT OR".  */
  for (i = 1; i < n; i++)
    addtok (dfa->utf8_anychar_classes[i]);
  while (--i > 1)
    {
      addtok (dfa->utf8_anychar_classes[0]);
      addtok (CAT);
      addtok (OR);
    }
}

/* The grammar understood by the parser is as follows.

   regexp:
     regexp OR branch
     branch

   branch:
     branch closure
     closure

   closure:
     closure QMARK
     closure STAR
     closure PLUS
     closure REPMN
     atom

   atom:
     <normal character>
     <multibyte character>
     ANYCHAR
     MBCSET
     CSET
     BACKREF
     BEGLINE
     ENDLINE
     BEGWORD
     ENDWORD
     LIMWORD
     NOTLIMWORD
     LPAREN regexp RPAREN
     <empty>

   The parser builds a parse tree in postfix form in an array of tokens.  */

static void
atom (void)
{
  if (tok == WCHAR)
    {
      if (wctok == WEOF)
        addtok (BACKREF);
      else
        {
          addtok_wc (wctok);

          if (case_fold)
            {
              wchar_t folded[CASE_FOLDED_BUFSIZE];
              int i, n = case_folded_counterparts (wctok, folded);
              for (i = 0; i < n; i++)
                {
                  addtok_wc (folded[i]);
                  addtok (OR);
                }
            }
        }

      tok = lex ();
    }
  else if (tok == ANYCHAR && using_utf8 ())
    {
      /* For UTF-8 expand the period to a series of CSETs that define a valid
         UTF-8 character.  This avoids using the slow multibyte path.  I'm
         pretty sure it would be both profitable and correct to do it for
         any encoding; however, the optimization must be done manually as
         it is done above in add_utf8_anychar.  So, let's start with
         UTF-8: it is the most used, and the structure of the encoding
         makes the correctness more obvious.  */
      add_utf8_anychar ();
      tok = lex ();
    }
  else if ((tok >= 0 && tok < NOTCHAR) || tok >= CSET || tok == BACKREF
           || tok == BEGLINE || tok == ENDLINE || tok == BEGWORD
           || tok == ANYCHAR || tok == MBCSET
           || tok == ENDWORD || tok == LIMWORD || tok == NOTLIMWORD)
    {
      addtok (tok);
      tok = lex ();
    }
  else if (tok == LPAREN)
    {
      tok = lex ();
      regexp ();
      if (tok != RPAREN)
        dfaerror (_("unbalanced ("));
      tok = lex ();
    }
  else
    addtok (EMPTY);
}

/* Return the number of tokens in the given subexpression.  */
static size_t _GL_ATTRIBUTE_PURE
nsubtoks (size_t tindex)
{
  size_t ntoks1;

  switch (dfa->tokens[tindex - 1])
    {
    default:
      return 1;
    case QMARK:
    case STAR:
    case PLUS:
      return 1 + nsubtoks (tindex - 1);
    case CAT:
    case OR:
      ntoks1 = nsubtoks (tindex - 1);
      return 1 + ntoks1 + nsubtoks (tindex - 1 - ntoks1);
    }
}

/* Copy the given subexpression to the top of the tree.  */
static void
copytoks (size_t tindex, size_t ntokens)
{
  size_t i;

  if (dfa->multibyte)
    for (i = 0; i < ntokens; ++i)
      addtok_mb (dfa->tokens[tindex + i], dfa->multibyte_prop[tindex + i]);
  else
    for (i = 0; i < ntokens; ++i)
      addtok_mb (dfa->tokens[tindex + i], 3);
}

static void
closure (void)
{
  int i;
  size_t tindex, ntokens;

  atom ();
  while (tok == QMARK || tok == STAR || tok == PLUS || tok == REPMN)
    if (tok == REPMN && (minrep || maxrep))
      {
        ntokens = nsubtoks (dfa->tindex);
        tindex = dfa->tindex - ntokens;
        if (maxrep < 0)
          addtok (PLUS);
        if (minrep == 0)
          addtok (QMARK);
        for (i = 1; i < minrep; ++i)
          {
            copytoks (tindex, ntokens);
            addtok (CAT);
          }
        for (; i < maxrep; ++i)
          {
            copytoks (tindex, ntokens);
            addtok (QMARK);
            addtok (CAT);
          }
        tok = lex ();
      }
    else if (tok == REPMN)
      {
        dfa->tindex -= nsubtoks (dfa->tindex);
        tok = lex ();
        closure ();
      }
    else
      {
        addtok (tok);
        tok = lex ();
      }
}

static void
branch (void)
{
  closure ();
  while (tok != RPAREN && tok != OR && tok >= 0)
    {
      closure ();
      addtok (CAT);
    }
}

static void
regexp (void)
{
  branch ();
  while (tok == OR)
    {
      tok = lex ();
      branch ();
      addtok (OR);
    }
}

/* Main entry point for the parser.  S is a string to be parsed, len is the
   length of the string, so s can include NUL characters.  D is a pointer to
   the struct dfa to parse into.  */
void
dfaparse (char const *s, size_t len, struct dfa *d)
{
  dfa = d;
  lexptr = s;
  lexleft = len;
  lasttok = END;
  laststart = true;
  parens = 0;
  if (dfa->multibyte)
    {
      cur_mb_len = 0;
      memset (&d->mbs, 0, sizeof d->mbs);
    }

  if (!syntax_bits_set)
    dfaerror (_("no syntax specified"));

  tok = lex ();
  depth = d->depth;

  regexp ();

  if (tok != END)
    dfaerror (_("unbalanced )"));

  addtok (END - d->nregexps);
  addtok (CAT);

  if (d->nregexps)
    addtok (OR);

  ++d->nregexps;
}

/* Some primitives for operating on sets of positions.  */

/* Copy one set to another.  */
static void
copy (position_set const *src, position_set * dst)
{
  if (dst->alloc < src->nelem)
    {
      free (dst->elems);
      dst->alloc = src->nelem;
      dst->elems = x2nrealloc (NULL, &dst->alloc, sizeof *dst->elems);
    }
  memcpy (dst->elems, src->elems, src->nelem * sizeof *dst->elems);
  dst->nelem = src->nelem;
}

static void
alloc_position_set (position_set * s, size_t size)
{
  s->elems = xnmalloc (size, sizeof *s->elems);
  s->alloc = size;
  s->nelem = 0;
}

/* Insert position P in set S.  S is maintained in sorted order on
   decreasing index.  If there is already an entry in S with P.index
   then merge (logically-OR) P's constraints into the one in S.
   S->elems must point to an array large enough to hold the resulting set.  */
static void
insert (position p, position_set * s)
{
  size_t count = s->nelem;
  size_t lo = 0, hi = count;
  size_t i;
  while (lo < hi)
    {
      size_t mid = (lo + hi) >> 1;
      if (s->elems[mid].index > p.index)
        lo = mid + 1;
      else
        hi = mid;
    }

  if (lo < count && p.index == s->elems[lo].index)
    {
      s->elems[lo].constraint |= p.constraint;
      return;
    }

  s->elems = maybe_realloc (s->elems, count, &s->alloc, sizeof *s->elems);
  for (i = count; i > lo; i--)
    s->elems[i] = s->elems[i - 1];
  s->elems[lo] = p;
  ++s->nelem;
}

/* Merge two sets of positions into a third.  The result is exactly as if
   the positions of both sets were inserted into an initially empty set.  */
static void
merge (position_set const *s1, position_set const *s2, position_set * m)
{
  size_t i = 0, j = 0;

  if (m->alloc < s1->nelem + s2->nelem)
    {
      free (m->elems);
      m->elems = maybe_realloc (NULL, s1->nelem + s2->nelem, &m->alloc,
                                sizeof *m->elems);
    }
  m->nelem = 0;
  while (i < s1->nelem && j < s2->nelem)
    if (s1->elems[i].index > s2->elems[j].index)
      m->elems[m->nelem++] = s1->elems[i++];
    else if (s1->elems[i].index < s2->elems[j].index)
      m->elems[m->nelem++] = s2->elems[j++];
    else
      {
        m->elems[m->nelem] = s1->elems[i++];
        m->elems[m->nelem++].constraint |= s2->elems[j++].constraint;
      }
  while (i < s1->nelem)
    m->elems[m->nelem++] = s1->elems[i++];
  while (j < s2->nelem)
    m->elems[m->nelem++] = s2->elems[j++];
}

/* Delete a position from a set.  */
static void
delete (position p, position_set * s)
{
  size_t i;

  for (i = 0; i < s->nelem; ++i)
    if (p.index == s->elems[i].index)
      break;
  if (i < s->nelem)
    for (--s->nelem; i < s->nelem; ++i)
      s->elems[i] = s->elems[i + 1];
}

/* Find the index of the state corresponding to the given position set with
   the given preceding context, or create a new state if there is no such
   state.  Context tells whether we got here on a newline or letter.  */
static state_num
state_index (struct dfa *d, position_set const *s, int context)
{
  size_t hash = 0;
  int constraint;
  state_num i, j;

  for (i = 0; i < s->nelem; ++i)
    hash ^= s->elems[i].index + s->elems[i].constraint;

  /* Try to find a state that exactly matches the proposed one.  */
  for (i = 0; i < d->sindex; ++i)
    {
      if (hash != d->states[i].hash || s->nelem != d->states[i].elems.nelem
          || context != d->states[i].context)
        continue;
      for (j = 0; j < s->nelem; ++j)
        if (s->elems[j].constraint
            != d->states[i].elems.elems[j].constraint
            || s->elems[j].index != d->states[i].elems.elems[j].index)
          break;
      if (j == s->nelem)
        return i;
    }

  /* We'll have to create a new state.  */
  d->states = maybe_realloc (d->states, d->sindex, &d->salloc,
                             sizeof *d->states);
  d->states[i].hash = hash;
  alloc_position_set (&d->states[i].elems, s->nelem);
  copy (s, &d->states[i].elems);
  d->states[i].context = context;
  d->states[i].has_backref = false;
  d->states[i].has_mbcset = false;
  d->states[i].constraint = 0;
  d->states[i].first_end = 0;
  d->states[i].mbps.nelem = 0;
  d->states[i].mbps.elems = NULL;

  for (j = 0; j < s->nelem; ++j)
    if (d->tokens[s->elems[j].index] < 0)
      {
        constraint = s->elems[j].constraint;
        if (SUCCEEDS_IN_CONTEXT (constraint, context, CTX_ANY))
          d->states[i].constraint |= constraint;
        if (!d->states[i].first_end)
          d->states[i].first_end = d->tokens[s->elems[j].index];
      }
    else if (d->tokens[s->elems[j].index] == BACKREF)
      {
        d->states[i].constraint = NO_CONSTRAINT;
        d->states[i].has_backref = true;
      }

  ++d->sindex;

  return i;
}

/* Find the epsilon closure of a set of positions.  If any position of the set
   contains a symbol that matches the empty string in some context, replace
   that position with the elements of its follow labeled with an appropriate
   constraint.  Repeat exhaustively until no funny positions are left.
   S->elems must be large enough to hold the result.  */
static void
epsclosure (position_set *s, struct dfa const *d, char *visited)
{
  size_t i, j;
  position p, old;
  bool initialized = false;

  for (i = 0; i < s->nelem; ++i)
    if (d->tokens[s->elems[i].index] >= NOTCHAR
        && d->tokens[s->elems[i].index] != BACKREF
        && d->tokens[s->elems[i].index] != ANYCHAR
        && d->tokens[s->elems[i].index] != MBCSET
        && d->tokens[s->elems[i].index] < CSET)
      {
        if (!initialized)
          {
            memset (visited, 0, d->tindex * sizeof (*visited));
            initialized = true;
          }
        old = s->elems[i];
        p.constraint = old.constraint;
        delete (s->elems[i], s);
        if (visited[old.index])
          {
            --i;
            continue;
          }
        visited[old.index] = 1;
        switch (d->tokens[old.index])
          {
          case BEGLINE:
            p.constraint &= BEGLINE_CONSTRAINT;
            break;
          case ENDLINE:
            p.constraint &= ENDLINE_CONSTRAINT;
            break;
          case BEGWORD:
            p.constraint &= BEGWORD_CONSTRAINT;
            break;
          case ENDWORD:
            p.constraint &= ENDWORD_CONSTRAINT;
            break;
          case LIMWORD:
            p.constraint &= LIMWORD_CONSTRAINT;
            break;
          case NOTLIMWORD:
            p.constraint &= NOTLIMWORD_CONSTRAINT;
            break;
          default:
            break;
          }
        for (j = 0; j < d->follows[old.index].nelem; ++j)
          {
            p.index = d->follows[old.index].elems[j].index;
            insert (p, s);
          }
        /* Force rescan to start at the beginning.  */
        i = -1;
      }
}

/* Returns the set of contexts for which there is at least one
   character included in C.  */

static int
charclass_context (charclass c)
{
  int context = 0;
  unsigned int j;

  if (tstbit (eolbyte, c))
    context |= CTX_NEWLINE;

  for (j = 0; j < CHARCLASS_WORDS; ++j)
    {
      if (c[j] & letters[j])
        context |= CTX_LETTER;
      if (c[j] & ~(letters[j] | newline[j]))
        context |= CTX_NONE;
    }

  return context;
}

/* Returns the contexts on which the position set S depends.  Each context
   in the set of returned contexts (let's call it SC) may have a different
   follow set than other contexts in SC, and also different from the
   follow set of the complement set (sc ^ CTX_ANY).  However, all contexts
   in the complement set will have the same follow set.  */

static int _GL_ATTRIBUTE_PURE
state_separate_contexts (position_set const *s)
{
  int separate_contexts = 0;
  size_t j;

  for (j = 0; j < s->nelem; ++j)
    {
      if (PREV_NEWLINE_DEPENDENT (s->elems[j].constraint))
        separate_contexts |= CTX_NEWLINE;
      if (PREV_LETTER_DEPENDENT (s->elems[j].constraint))
        separate_contexts |= CTX_LETTER;
    }

  return separate_contexts;
}


/* Perform bottom-up analysis on the parse tree, computing various functions.
   Note that at this point, we're pretending constructs like \< are real
   characters rather than constraints on what can follow them.

   Nullable:  A node is nullable if it is at the root of a regexp that can
   match the empty string.
   *  EMPTY leaves are nullable.
   * No other leaf is nullable.
   * A QMARK or STAR node is nullable.
   * A PLUS node is nullable if its argument is nullable.
   * A CAT node is nullable if both its arguments are nullable.
   * An OR node is nullable if either argument is nullable.

   Firstpos:  The firstpos of a node is the set of positions (nonempty leaves)
   that could correspond to the first character of a string matching the
   regexp rooted at the given node.
   * EMPTY leaves have empty firstpos.
   * The firstpos of a nonempty leaf is that leaf itself.
   * The firstpos of a QMARK, STAR, or PLUS node is the firstpos of its
     argument.
   * The firstpos of a CAT node is the firstpos of the left argument, union
     the firstpos of the right if the left argument is nullable.
   * The firstpos of an OR node is the union of firstpos of each argument.

   Lastpos:  The lastpos of a node is the set of positions that could
   correspond to the last character of a string matching the regexp at
   the given node.
   * EMPTY leaves have empty lastpos.
   * The lastpos of a nonempty leaf is that leaf itself.
   * The lastpos of a QMARK, STAR, or PLUS node is the lastpos of its
     argument.
   * The lastpos of a CAT node is the lastpos of its right argument, union
     the lastpos of the left if the right argument is nullable.
   * The lastpos of an OR node is the union of the lastpos of each argument.

   Follow:  The follow of a position is the set of positions that could
   correspond to the character following a character matching the node in
   a string matching the regexp.  At this point we consider special symbols
   that match the empty string in some context to be just normal characters.
   Later, if we find that a special symbol is in a follow set, we will
   replace it with the elements of its follow, labeled with an appropriate
   constraint.
   * Every node in the firstpos of the argument of a STAR or PLUS node is in
     the follow of every node in the lastpos.
   * Every node in the firstpos of the second argument of a CAT node is in
     the follow of every node in the lastpos of the first argument.

   Because of the postfix representation of the parse tree, the depth-first
   analysis is conveniently done by a linear scan with the aid of a stack.
   Sets are stored as arrays of the elements, obeying a stack-like allocation
   scheme; the number of elements in each set deeper in the stack can be
   used to determine the address of a particular set's array.  */
void
dfaanalyze (struct dfa *d, int searchflag)
{
  /* Array allocated to hold position sets.  */
  position *posalloc = xnmalloc (d->nleaves, 2 * sizeof *posalloc);
  /* Firstpos and lastpos elements.  */
  position *firstpos = posalloc + d->nleaves;
  position *lastpos = firstpos + d->nleaves;

  /* Stack for element counts and nullable flags.  */
  struct
  {
    /* Whether the entry is nullable.  */
    bool nullable;

    /* Counts of firstpos and lastpos sets.  */
    size_t nfirstpos;
    size_t nlastpos;
  } *stkalloc = xnmalloc (d->depth, sizeof *stkalloc), *stk = stkalloc;

  position_set tmp;             /* Temporary set for merging sets.  */
  position_set merged;          /* Result of merging sets.  */
  int separate_contexts;        /* Context wanted by some position.  */
  size_t i, j;
  position *pos;
  char *visited = xnmalloc (d->tindex, sizeof *visited);

#ifdef DEBUG
  fprintf (stderr, "dfaanalyze:\n");
  for (i = 0; i < d->tindex; ++i)
    {
      fprintf (stderr, " %zd:", i);
      prtok (d->tokens[i]);
    }
  putc ('\n', stderr);
#endif

  d->searchflag = searchflag != 0;
  alloc_position_set (&merged, d->nleaves);
  d->follows = xcalloc (d->tindex, sizeof *d->follows);

  for (i = 0; i < d->tindex; ++i)
    {
      switch (d->tokens[i])
        {
        case EMPTY:
          /* The empty set is nullable.  */
          stk->nullable = true;

          /* The firstpos and lastpos of the empty leaf are both empty.  */
          stk->nfirstpos = stk->nlastpos = 0;
          stk++;
          break;

        case STAR:
        case PLUS:
          /* Every element in the firstpos of the argument is in the follow
             of every element in the lastpos.  */
          tmp.nelem = stk[-1].nfirstpos;
          tmp.elems = firstpos;
          pos = lastpos;
          for (j = 0; j < stk[-1].nlastpos; ++j)
            {
              merge (&tmp, &d->follows[pos[j].index], &merged);
              copy (&merged, &d->follows[pos[j].index]);
            }
          /* fallthrough */

        case QMARK:
          /* A QMARK or STAR node is automatically nullable.  */
          if (d->tokens[i] != PLUS)
            stk[-1].nullable = true;
          break;

        case CAT:
          /* Every element in the firstpos of the second argument is in the
             follow of every element in the lastpos of the first argument.  */
          tmp.nelem = stk[-1].nfirstpos;
          tmp.elems = firstpos;
          pos = lastpos + stk[-1].nlastpos;
          for (j = 0; j < stk[-2].nlastpos; ++j)
            {
              merge (&tmp, &d->follows[pos[j].index], &merged);
              copy (&merged, &d->follows[pos[j].index]);
            }

          /* The firstpos of a CAT node is the firstpos of the first argument,
             union that of the second argument if the first is nullable.  */
          if (stk[-2].nullable)
            stk[-2].nfirstpos += stk[-1].nfirstpos;
          else
            firstpos += stk[-1].nfirstpos;

          /* The lastpos of a CAT node is the lastpos of the second argument,
             union that of the first argument if the second is nullable.  */
          if (stk[-1].nullable)
            stk[-2].nlastpos += stk[-1].nlastpos;
          else
            {
              pos = lastpos + stk[-2].nlastpos;
              for (j = stk[-1].nlastpos; j-- > 0;)
                pos[j] = lastpos[j];
              lastpos += stk[-2].nlastpos;
              stk[-2].nlastpos = stk[-1].nlastpos;
            }

          /* A CAT node is nullable if both arguments are nullable.  */
          stk[-2].nullable &= stk[-1].nullable;
          stk--;
          break;

        case OR:
          /* The firstpos is the union of the firstpos of each argument.  */
          stk[-2].nfirstpos += stk[-1].nfirstpos;

          /* The lastpos is the union of the lastpos of each argument.  */
          stk[-2].nlastpos += stk[-1].nlastpos;

          /* An OR node is nullable if either argument is nullable.  */
          stk[-2].nullable |= stk[-1].nullable;
          stk--;
          break;

        default:
          /* Anything else is a nonempty position.  (Note that special
             constructs like \< are treated as nonempty strings here;
             an "epsilon closure" effectively makes them nullable later.
             Backreferences have to get a real position so we can detect
             transitions on them later.  But they are nullable.  */
          stk->nullable = d->tokens[i] == BACKREF;

          /* This position is in its own firstpos and lastpos.  */
          stk->nfirstpos = stk->nlastpos = 1;
          stk++;

          --firstpos, --lastpos;
          firstpos->index = lastpos->index = i;
          firstpos->constraint = lastpos->constraint = NO_CONSTRAINT;

          /* Allocate the follow set for this position.  */
          alloc_position_set (&d->follows[i], 1);
          break;
        }
#ifdef DEBUG
      /* ... balance the above nonsyntactic #ifdef goo...  */
      fprintf (stderr, "node %zd:", i);
      prtok (d->tokens[i]);
      putc ('\n', stderr);
      fprintf (stderr,
               stk[-1].nullable ? " nullable: yes\n" : " nullable: no\n");
      fprintf (stderr, " firstpos:");
      for (j = stk[-1].nfirstpos; j-- > 0;)
        {
          fprintf (stderr, " %zd:", firstpos[j].index);
          prtok (d->tokens[firstpos[j].index]);
        }
      fprintf (stderr, "\n lastpos:");
      for (j = stk[-1].nlastpos; j-- > 0;)
        {
          fprintf (stderr, " %zd:", lastpos[j].index);
          prtok (d->tokens[lastpos[j].index]);
        }
      putc ('\n', stderr);
#endif
    }

  /* For each follow set that is the follow set of a real position, replace
     it with its epsilon closure.  */
  for (i = 0; i < d->tindex; ++i)
    if (d->tokens[i] < NOTCHAR || d->tokens[i] == BACKREF
        || d->tokens[i] == ANYCHAR || d->tokens[i] == MBCSET
        || d->tokens[i] >= CSET)
      {
#ifdef DEBUG
        fprintf (stderr, "follows(%zd:", i);
        prtok (d->tokens[i]);
        fprintf (stderr, "):");
        for (j = d->follows[i].nelem; j-- > 0;)
          {
            fprintf (stderr, " %zd:", d->follows[i].elems[j].index);
            prtok (d->tokens[d->follows[i].elems[j].index]);
          }
        putc ('\n', stderr);
#endif
        copy (&d->follows[i], &merged);
        epsclosure (&merged, d, visited);
        copy (&merged, &d->follows[i]);
      }

  /* Get the epsilon closure of the firstpos of the regexp.  The result will
     be the set of positions of state 0.  */
  merged.nelem = 0;
  for (i = 0; i < stk[-1].nfirstpos; ++i)
    insert (firstpos[i], &merged);
  epsclosure (&merged, d, visited);

  /* Build the initial state.  */
  separate_contexts = state_separate_contexts (&merged);
  state_index (d, &merged,
               (separate_contexts & CTX_NEWLINE
                ? CTX_NEWLINE : separate_contexts ^ CTX_ANY));

  free (posalloc);
  free (stkalloc);
  free (merged.elems);
  free (visited);
}


/* Find, for each character, the transition out of state s of d, and store
   it in the appropriate slot of trans.

   We divide the positions of s into groups (positions can appear in more
   than one group).  Each group is labeled with a set of characters that
   every position in the group matches (taking into account, if necessary,
   preceding context information of s).  For each group, find the union
   of the its elements' follows.  This set is the set of positions of the
   new state.  For each character in the group's label, set the transition
   on this character to be to a state corresponding to the set's positions,
   and its associated backward context information, if necessary.

   If we are building a searching matcher, we include the positions of state
   0 in every state.

   The collection of groups is constructed by building an equivalence-class
   partition of the positions of s.

   For each position, find the set of characters C that it matches.  Eliminate
   any characters from C that fail on grounds of backward context.

   Search through the groups, looking for a group whose label L has nonempty
   intersection with C.  If L - C is nonempty, create a new group labeled
   L - C and having the same positions as the current group, and set L to
   the intersection of L and C.  Insert the position in this group, set
   C = C - L, and resume scanning.

   If after comparing with every group there are characters remaining in C,
   create a new group labeled with the characters of C and insert this
   position in that group.  */
void
dfastate (state_num s, struct dfa *d, state_num trans[])
{
  leaf_set grps[NOTCHAR];       /* As many as will ever be needed.  */
  charclass labels[NOTCHAR];    /* Labels corresponding to the groups.  */
  size_t ngrps = 0;             /* Number of groups actually used.  */
  position pos;                 /* Current position being considered.  */
  charclass matches;            /* Set of matching characters.  */
  charclass_word matchesf;	/* Nonzero if matches is nonempty.  */
  charclass intersect;          /* Intersection with some label set.  */
  charclass_word intersectf;	/* Nonzero if intersect is nonempty.  */
  charclass leftovers;          /* Stuff in the label that didn't match.  */
  charclass_word leftoversf;	/* Nonzero if leftovers is nonempty.  */
  position_set follows;         /* Union of the follows of some group.  */
  position_set tmp;             /* Temporary space for merging sets.  */
  int possible_contexts;        /* Contexts that this group can match.  */
  int separate_contexts;        /* Context that new state wants to know.  */
  state_num state;              /* New state.  */
  state_num state_newline;      /* New state on a newline transition.  */
  state_num state_letter;       /* New state on a letter transition.  */
  bool next_isnt_1st_byte = false; /* We can't add state0.  */
  size_t i, j, k;

  zeroset (matches);

  for (i = 0; i < d->states[s].elems.nelem; ++i)
    {
      pos = d->states[s].elems.elems[i];
      if (d->tokens[pos.index] >= 0 && d->tokens[pos.index] < NOTCHAR)
        setbit (d->tokens[pos.index], matches);
      else if (d->tokens[pos.index] >= CSET)
        copyset (d->charclasses[d->tokens[pos.index] - CSET], matches);
      else
        {
          if (d->tokens[pos.index] == MBCSET
              || d->tokens[pos.index] == ANYCHAR)
            {
              /* MB_CUR_MAX > 1 */
              if (d->tokens[pos.index] == MBCSET)
                d->states[s].has_mbcset = true;
              /* ANYCHAR and MBCSET must match with a single character, so we
                 must put it to d->states[s].mbps, which contains the positions
                 which can match with a single character not a byte.  */
              if (d->states[s].mbps.nelem == 0)
                alloc_position_set (&d->states[s].mbps, 1);
              insert (pos, &(d->states[s].mbps));
            }
          continue;
        }

      /* Some characters may need to be eliminated from matches because
         they fail in the current context.  */
      if (pos.constraint != NO_CONSTRAINT)
        {
          if (!SUCCEEDS_IN_CONTEXT (pos.constraint,
                                    d->states[s].context, CTX_NEWLINE))
            for (j = 0; j < CHARCLASS_WORDS; ++j)
              matches[j] &= ~newline[j];
          if (!SUCCEEDS_IN_CONTEXT (pos.constraint,
                                    d->states[s].context, CTX_LETTER))
            for (j = 0; j < CHARCLASS_WORDS; ++j)
              matches[j] &= ~letters[j];
          if (!SUCCEEDS_IN_CONTEXT (pos.constraint,
                                    d->states[s].context, CTX_NONE))
            for (j = 0; j < CHARCLASS_WORDS; ++j)
              matches[j] &= letters[j] | newline[j];

          /* If there are no characters left, there's no point in going on.  */
          for (j = 0; j < CHARCLASS_WORDS && !matches[j]; ++j)
            continue;
          if (j == CHARCLASS_WORDS)
            continue;
        }

      for (j = 0; j < ngrps; ++j)
        {
          /* If matches contains a single character only, and the current
             group's label doesn't contain that character, go on to the
             next group.  */
          if (d->tokens[pos.index] >= 0 && d->tokens[pos.index] < NOTCHAR
              && !tstbit (d->tokens[pos.index], labels[j]))
            continue;

          /* Check if this group's label has a nonempty intersection with
             matches.  */
          intersectf = 0;
          for (k = 0; k < CHARCLASS_WORDS; ++k)
            intersectf |= intersect[k] = matches[k] & labels[j][k];
          if (!intersectf)
            continue;

          /* It does; now find the set differences both ways.  */
          leftoversf = matchesf = 0;
          for (k = 0; k < CHARCLASS_WORDS; ++k)
            {
              /* Even an optimizing compiler can't know this for sure.  */
              charclass_word match = matches[k], label = labels[j][k];

              leftoversf |= leftovers[k] = ~match & label;
              matchesf |= matches[k] = match & ~label;
            }

          /* If there were leftovers, create a new group labeled with them.  */
          if (leftoversf)
            {
              copyset (leftovers, labels[ngrps]);
              copyset (intersect, labels[j]);
              grps[ngrps].elems = xnmalloc (d->nleaves,
                                            sizeof *grps[ngrps].elems);
              memcpy (grps[ngrps].elems, grps[j].elems,
                      sizeof (grps[j].elems[0]) * grps[j].nelem);
              grps[ngrps].nelem = grps[j].nelem;
              ++ngrps;
            }

          /* Put the position in the current group.  The constraint is
             irrelevant here.  */
          grps[j].elems[grps[j].nelem++] = pos.index;

          /* If every character matching the current position has been
             accounted for, we're done.  */
          if (!matchesf)
            break;
        }

      /* If we've passed the last group, and there are still characters
         unaccounted for, then we'll have to create a new group.  */
      if (j == ngrps)
        {
          copyset (matches, labels[ngrps]);
          zeroset (matches);
          grps[ngrps].elems = xnmalloc (d->nleaves, sizeof *grps[ngrps].elems);
          grps[ngrps].nelem = 1;
          grps[ngrps].elems[0] = pos.index;
          ++ngrps;
        }
    }

  alloc_position_set (&follows, d->nleaves);
  alloc_position_set (&tmp, d->nleaves);

  /* If we are a searching matcher, the default transition is to a state
     containing the positions of state 0, otherwise the default transition
     is to fail miserably.  */
  if (d->searchflag)
    {
      /* Find the state(s) corresponding to the positions of state 0.  */
      copy (&d->states[0].elems, &follows);
      separate_contexts = state_separate_contexts (&follows);
      state = state_index (d, &follows, separate_contexts ^ CTX_ANY);
      if (separate_contexts & CTX_NEWLINE)
        state_newline = state_index (d, &follows, CTX_NEWLINE);
      else
        state_newline = state;
      if (separate_contexts & CTX_LETTER)
        state_letter = state_index (d, &follows, CTX_LETTER);
      else
        state_letter = state;

      for (i = 0; i < NOTCHAR; ++i)
        trans[i] = (IS_WORD_CONSTITUENT (i)) ? state_letter : state;
      trans[eolbyte] = state_newline;
    }
  else
    for (i = 0; i < NOTCHAR; ++i)
      trans[i] = -1;

  for (i = 0; i < ngrps; ++i)
    {
      follows.nelem = 0;

      /* Find the union of the follows of the positions of the group.
         This is a hideously inefficient loop.  Fix it someday.  */
      for (j = 0; j < grps[i].nelem; ++j)
        for (k = 0; k < d->follows[grps[i].elems[j]].nelem; ++k)
          insert (d->follows[grps[i].elems[j]].elems[k], &follows);

      if (d->multibyte)
        {
          /* If a token in follows.elems is not 1st byte of a multibyte
             character, or the states of follows must accept the bytes
             which are not 1st byte of the multibyte character.
             Then, if a state of follows encounter a byte, it must not be
             a 1st byte of a multibyte character nor single byte character.
             We cansel to add state[0].follows to next state, because
             state[0] must accept 1st-byte

             For example, we assume <sb a> is a certain single byte
             character, <mb A> is a certain multibyte character, and the
             codepoint of <sb a> equals the 2nd byte of the codepoint of
             <mb A>.
             When state[0] accepts <sb a>, state[i] transit to state[i+1]
             by accepting accepts 1st byte of <mb A>, and state[i+1]
             accepts 2nd byte of <mb A>, if state[i+1] encounter the
             codepoint of <sb a>, it must not be <sb a> but 2nd byte of
             <mb A>, so we cannot add state[0].  */

          next_isnt_1st_byte = false;
          for (j = 0; j < follows.nelem; ++j)
            {
              if (!(d->multibyte_prop[follows.elems[j].index] & 1))
                {
                  next_isnt_1st_byte = true;
                  break;
                }
            }
        }

      /* If we are building a searching matcher, throw in the positions
         of state 0 as well.  */
      if (d->searchflag && (!d->multibyte || !next_isnt_1st_byte))
        {
          merge (&d->states[0].elems, &follows, &tmp);
          copy (&tmp, &follows);
        }

      /* Find out if the new state will want any context information.  */
      possible_contexts = charclass_context (labels[i]);
      separate_contexts = state_separate_contexts (&follows);

      /* Find the state(s) corresponding to the union of the follows.  */
      if ((separate_contexts & possible_contexts) != possible_contexts)
        state = state_index (d, &follows, separate_contexts ^ CTX_ANY);
      else
        state = -1;
      if (separate_contexts & possible_contexts & CTX_NEWLINE)
        state_newline = state_index (d, &follows, CTX_NEWLINE);
      else
        state_newline = state;
      if (separate_contexts & possible_contexts & CTX_LETTER)
        state_letter = state_index (d, &follows, CTX_LETTER);
      else
        state_letter = state;

      /* Set the transitions for each character in the current label.  */
      for (j = 0; j < CHARCLASS_WORDS; ++j)
        for (k = 0; k < CHARCLASS_WORD_BITS; ++k)
          if (labels[i][j] >> k & 1)
            {
              int c = j * CHARCLASS_WORD_BITS + k;

              if (c == eolbyte)
                trans[c] = state_newline;
              else if (IS_WORD_CONSTITUENT (c))
                trans[c] = state_letter;
              else if (c < NOTCHAR)
                trans[c] = state;
            }
    }

  for (i = 0; i < ngrps; ++i)
    free (grps[i].elems);
  free (follows.elems);
  free (tmp.elems);
}

/* Make sure D's state arrays are large enough to hold NEW_STATE.  */
static void
realloc_trans_if_necessary (struct dfa *d, state_num new_state)
{
  state_num oldalloc = d->tralloc;
  if (oldalloc <= new_state)
    {
      state_num **realtrans = d->trans ? d->trans - 1 : NULL;
      size_t newalloc, newalloc1;
      newalloc1 = new_state + 1;
      realtrans = x2nrealloc (realtrans, &newalloc1, sizeof *realtrans);
      realtrans[0] = NULL;
      d->trans = realtrans + 1;
      d->tralloc = newalloc = newalloc1 - 1;
      d->fails = xnrealloc (d->fails, newalloc, sizeof *d->fails);
      d->success = xnrealloc (d->success, newalloc, sizeof *d->success);
      d->newlines = xnrealloc (d->newlines, newalloc, sizeof *d->newlines);
      for (; oldalloc < newalloc; oldalloc++)
        {
          d->trans[oldalloc] = NULL;
          d->fails[oldalloc] = NULL;
        }
    }
}

/* Some routines for manipulating a compiled dfa's transition tables.
   Each state may or may not have a transition table; if it does, and it
   is a non-accepting state, then d->trans[state] points to its table.
   If it is an accepting state then d->fails[state] points to its table.
   If it has no table at all, then d->trans[state] is NULL.
   TODO: Improve this comment, get rid of the unnecessary redundancy.  */

static void
build_state (state_num s, struct dfa *d)
{
  state_num *trans;             /* The new transition table.  */
  state_num i, maxstate;

  /* Set an upper limit on the number of transition tables that will ever
     exist at once.  1024 is arbitrary.  The idea is that the frequently
     used transition tables will be quickly rebuilt, whereas the ones that
     were only needed once or twice will be cleared away.  However, do
     not clear the initial state, as it's always used.  */
  if (d->trcount >= 1024)
    {
      for (i = 1; i < d->tralloc; ++i)
        {
          free (d->trans[i]);
          free (d->fails[i]);
          d->trans[i] = d->fails[i] = NULL;
        }
      d->trcount = 1;
    }

  ++d->trcount;

  /* Set up the success bits for this state.  */
  d->success[s] = 0;
  if (ACCEPTS_IN_CONTEXT (d->states[s].context, CTX_NEWLINE, s, *d))
    d->success[s] |= CTX_NEWLINE;
  if (ACCEPTS_IN_CONTEXT (d->states[s].context, CTX_LETTER, s, *d))
    d->success[s] |= CTX_LETTER;
  if (ACCEPTS_IN_CONTEXT (d->states[s].context, CTX_NONE, s, *d))
    d->success[s] |= CTX_NONE;

  trans = xmalloc (NOTCHAR * sizeof *trans);
  dfastate (s, d, trans);

  /* Now go through the new transition table, and make sure that the trans
     and fail arrays are allocated large enough to hold a pointer for the
     largest state mentioned in the table.  */
  maxstate = -1;
  for (i = 0; i < NOTCHAR; ++i)
    if (maxstate < trans[i])
      maxstate = trans[i];
  realloc_trans_if_necessary (d, maxstate);

  /* Keep the newline transition in a special place so we can use it as
     a sentinel.  */
  d->newlines[s] = trans[eolbyte];
  trans[eolbyte] = -1;

  if (ACCEPTING (s, *d))
    d->fails[s] = trans;
  else
    d->trans[s] = trans;
}

/* Multibyte character handling sub-routines for dfaexec.  */

/* Return values of transit_state_singlebyte, and
   transit_state_consume_1char.  */
typedef enum
{
  TRANSIT_STATE_IN_PROGRESS,    /* State transition has not finished.  */
  TRANSIT_STATE_DONE,           /* State transition has finished.  */
  TRANSIT_STATE_END_BUFFER      /* Reach the end of the buffer.  */
} status_transit_state;

/* Consume a single byte and transit state from 's' to '*next_state'.
   This function is almost same as the state transition routin in dfaexec.
   But state transition is done just once, otherwise matching succeed or
   reach the end of the buffer.  */
static status_transit_state
transit_state_singlebyte (struct dfa *d, state_num s, unsigned char const *p,
                          state_num * next_state)
{
  state_num *t;
  state_num works = s;

  status_transit_state rval = TRANSIT_STATE_IN_PROGRESS;

  while (rval == TRANSIT_STATE_IN_PROGRESS)
    {
      if ((t = d->trans[works]) != NULL)
        {
          works = t[*p];
          rval = TRANSIT_STATE_DONE;
          if (works < 0)
            works = 0;
        }
      else if (works < 0)
        works = 0;
      else if (d->fails[works])
        {
          works = d->fails[works][*p];
          rval = TRANSIT_STATE_DONE;
        }
      else
        {
          build_state (works, d);
        }
    }
  *next_state = works;
  return rval;
}

/* Match a "." against the current context.  Return the length of the
   match, in bytes.  POS is the position of the ".".  */
static int
match_anychar (struct dfa *d, state_num s, position pos,
               wint_t wc, size_t mbclen)
{
  int context;

  /* Check syntax bits.  */
  if (wc == (wchar_t) eolbyte)
    {
      if (!(syntax_bits & RE_DOT_NEWLINE))
        return 0;
    }
  else if (wc == (wchar_t) '\0')
    {
      if (syntax_bits & RE_DOT_NOT_NULL)
        return 0;
    }
  else if (wc == WEOF)
    return 0;

  context = wchar_context (wc);
  if (!SUCCEEDS_IN_CONTEXT (pos.constraint, d->states[s].context, context))
    return 0;

  return mbclen;
}

/* Match a bracket expression against the current context.
   Return the length of the match, in bytes.
   POS is the position of the bracket expression.  */
static int
match_mb_charset (struct dfa *d, state_num s, position pos,
                  char const *p, wint_t wc, size_t match_len)
{
  size_t i;
  bool match;              /* Matching succeeded.  */
  int op_len;              /* Length of the operator.  */
  char buffer[128];

  /* Pointer to the structure to which we are currently referring.  */
  struct mb_char_classes *work_mbc;

  int context;

  /* Check syntax bits.  */
  if (wc == (wchar_t) eolbyte)
    {
      if (!(syntax_bits & RE_DOT_NEWLINE))
        return 0;
    }
  else if (wc == (wchar_t) '\0')
    {
      if (syntax_bits & RE_DOT_NOT_NULL)
        return 0;
    }
  else if (wc == WEOF)
    return 0;

  context = wchar_context (wc);
  if (!SUCCEEDS_IN_CONTEXT (pos.constraint, d->states[s].context, context))
    return 0;

  /* Assign the current referring operator to work_mbc.  */
  work_mbc = &(d->mbcsets[(d->multibyte_prop[pos.index]) >> 2]);
  match = !work_mbc->invert;

  /* Match in range 0-255?  */
  if (wc < NOTCHAR && work_mbc->cset != -1
      && tstbit (to_uchar (wc), d->charclasses[work_mbc->cset]))
    goto charset_matched;

  /* match with a character class?  */
  for (i = 0; i < work_mbc->nch_classes; i++)
    {
      if (iswctype ((wint_t) wc, work_mbc->ch_classes[i]))
        goto charset_matched;
    }

  strncpy (buffer, p, match_len);
  buffer[match_len] = '\0';

  /* match with an equivalence class?  */
  for (i = 0; i < work_mbc->nequivs; i++)
    {
      op_len = strlen (work_mbc->equivs[i]);
      strncpy (buffer, p, op_len);
      buffer[op_len] = '\0';
      if (strcoll (work_mbc->equivs[i], buffer) == 0)
        {
          match_len = op_len;
          goto charset_matched;
        }
    }

  /* match with a collating element?  */
  for (i = 0; i < work_mbc->ncoll_elems; i++)
    {
      op_len = strlen (work_mbc->coll_elems[i]);
      strncpy (buffer, p, op_len);
      buffer[op_len] = '\0';

      if (strcoll (work_mbc->coll_elems[i], buffer) == 0)
        {
          match_len = op_len;
          goto charset_matched;
        }
    }

  /* match with a range?  */
  for (i = 0; i < work_mbc->nranges; i++)
    {
      if (work_mbc->ranges[i].beg <= wc && wc <= work_mbc->ranges[i].end)
        goto charset_matched;
    }

  /* match with a character?  */
  for (i = 0; i < work_mbc->nchars; i++)
    {
      if (wc == work_mbc->chars[i])
        goto charset_matched;
    }

  match = !match;

charset_matched:
  return match ? match_len : 0;
}

/* Check whether each of 'd->states[s].mbps.elem' can match.  Then return the
   array which corresponds to 'd->states[s].mbps.elem'; each element of the
   array contains the number of bytes with which the element can match.

   The caller MUST free the array which this function return.  */
static int *
check_matching_with_multibyte_ops (struct dfa *d, state_num s,
                                   char const *p, wint_t wc, size_t mbclen)
{
  size_t i;
  int *rarray;

  rarray = d->mb_match_lens;
  for (i = 0; i < d->states[s].mbps.nelem; ++i)
    {
      position pos = d->states[s].mbps.elems[i];
      switch (d->tokens[pos.index])
        {
        case ANYCHAR:
          rarray[i] = match_anychar (d, s, pos, wc, mbclen);
          break;
        case MBCSET:
          rarray[i] = match_mb_charset (d, s, pos, p, wc, mbclen);
          break;
        default:
          break;                /* cannot happen.  */
        }
    }
  return rarray;
}

/* Consume a single character and enumerate all of the positions which can
   be the next position from the state 's'.

   'match_lens' is the input.  It can be NULL, but it can also be the output
   of check_matching_with_multibyte_ops for optimization.

   'mbclen' and 'pps' are the output.  'mbclen' is the length of the
   character consumed, and 'pps' is the set this function enumerates.  */
static status_transit_state
transit_state_consume_1char (struct dfa *d, state_num s,
                             unsigned char const **pp,
                             wint_t wc, size_t mbclen,
                             int *match_lens)
{
  size_t i, j;
  int k;
  state_num s1, s2;
  status_transit_state rs = TRANSIT_STATE_DONE;

  if (! match_lens && d->states[s].mbps.nelem != 0)
    match_lens = check_matching_with_multibyte_ops (d, s, (char const *) *pp,
                                                    wc, mbclen);

  /* Calculate the state which can be reached from the state 's' by
     consuming 'mbclen' single bytes from the buffer.  */
  s1 = s;
  for (k = 0; k < mbclen; k++)
    {
      s2 = s1;
      rs = transit_state_singlebyte (d, s2, (*pp)++, &s1);
    }
  copy (&d->states[s1].elems, &d->mb_follows);

  /* Add all of the positions which can be reached from 's' by consuming
     a single character.  */
  for (i = 0; i < d->states[s].mbps.nelem; i++)
    {
      if (match_lens[i] == mbclen)
        for (j = 0; j < d->follows[d->states[s].mbps.elems[i].index].nelem;
             j++)
          insert (d->follows[d->states[s].mbps.elems[i].index].elems[j],
                  &d->mb_follows);
    }

  /* FIXME: this return value is always ignored.  */
  return rs;
}

/* Transit state from s, then return new state and update the pointer of the
   buffer.  This function is for some operator which can match with a multi-
   byte character or a collating element (which may be multi characters).  */
static state_num
transit_state (struct dfa *d, state_num s, unsigned char const **pp,
               unsigned char const *end)
{
  state_num s1;
  int mbclen;  /* The length of current input multibyte character.  */
  int maxlen = 0;
  size_t i, j;
  int *match_lens = NULL;
  size_t nelem = d->states[s].mbps.nelem;       /* Just a alias.  */
  unsigned char const *p1 = *pp;
  wint_t wc;

  if (nelem > 0)
    /* This state has (a) multibyte operator(s).
       We check whether each of them can match or not.  */
    {
      /* Note: caller must free the return value of this function.  */
      mbclen = mbs_to_wchar (&wc, (char const *) *pp, end - *pp, d);
      match_lens = check_matching_with_multibyte_ops (d, s, (char const *) *pp,
                                                      wc, mbclen);

      for (i = 0; i < nelem; i++)
        /* Search the operator which match the longest string,
           in this state.  */
        {
          if (match_lens[i] > maxlen)
            maxlen = match_lens[i];
        }
    }

  if (nelem == 0 || maxlen == 0)
    /* This state has no multibyte operator which can match.
       We need to check only one single byte character.  */
    {
      status_transit_state rs;
      rs = transit_state_singlebyte (d, s, *pp, &s1);

      /* We must update the pointer if state transition succeeded.  */
      if (rs == TRANSIT_STATE_DONE)
        ++*pp;

      return s1;
    }

  /* This state has some operators which can match a multibyte character.  */
  d->mb_follows.nelem = 0;

  /* 'maxlen' may be longer than the length of a character, because it may
     not be a character but a (multi character) collating element.
     We enumerate all of the positions which 's' can reach by consuming
     'maxlen' bytes.  */
  transit_state_consume_1char (d, s, pp, wc, mbclen, match_lens);

  s1 = state_index (d, &d->mb_follows, wchar_context (wc));
  realloc_trans_if_necessary (d, s1);

  while (*pp - p1 < maxlen)
    {
      mbclen = mbs_to_wchar (&wc, (char const *) *pp, end - *pp, d);
      transit_state_consume_1char (d, s1, pp, wc, mbclen, NULL);

      for (i = 0; i < nelem; i++)
        {
          if (match_lens[i] == *pp - p1)
            for (j = 0;
                 j < d->follows[d->states[s1].mbps.elems[i].index].nelem; j++)
              insert (d->follows[d->states[s1].mbps.elems[i].index].elems[j],
                      &d->mb_follows);
        }

      s1 = state_index (d, &d->mb_follows, wchar_context (wc));
      realloc_trans_if_necessary (d, s1);
    }
  return s1;
}

/* Search through a buffer looking for a match to the given struct dfa.
   Find the first occurrence of a string matching the regexp in the
   buffer, and the shortest possible version thereof.  Return a pointer to
   the first character after the match, or NULL if none is found.  BEGIN
   points to the beginning of the buffer, and END points to the first byte
   after its end.  Note however that we store a sentinel byte (usually
   newline) in *END, so the actual buffer must be one byte longer.
   When ALLOW_NL is nonzero, newlines may appear in the matching string.
   If COUNT is non-NULL, increment *COUNT once for each newline processed.
   Finally, if BACKREF is non-NULL set *BACKREF to indicate whether we
   encountered a back-reference (1) or not (0).  The caller may use this
   to decide whether to fall back on a backtracking matcher.  */
char *
dfaexec (struct dfa *d, char const *begin, char *end,
         int allow_nl, size_t *count, int *backref)
{
  state_num s, s1;              /* Current state.  */
  unsigned char const *p, *mbp; /* Current input character.  */
  state_num **trans, *t;        /* Copy of d->trans so it can be optimized
                                   into a register.  */
  unsigned char eol = eolbyte;  /* Likewise for eolbyte.  */
  unsigned char saved_end;
  size_t nlcount = 0;

  if (!d->tralloc)
    {
      realloc_trans_if_necessary (d, 1);
      build_state (0, d);
    }

  s = s1 = 0;
  p = mbp = (unsigned char const *) begin;
  trans = d->trans;
  saved_end = *(unsigned char *) end;
  *end = eol;

  if (d->multibyte)
    {
      memset (&d->mbs, 0, sizeof d->mbs);
      if (! d->mb_match_lens)
        {
          d->mb_match_lens = xnmalloc (d->nleaves, sizeof *d->mb_match_lens);
          alloc_position_set (&d->mb_follows, d->nleaves);
        }
    }

  for (;;)
    {
      if (d->multibyte)
        {
          while ((t = trans[s]) != NULL)
            {
              s1 = s;

              if (s == 0)
                {
                  /* The initial state may encounter a byte which is not
                     a single byte character nor the first byte of a
                     multibyte character.  But it is incorrect for the
                     initial state to accept such a byte.  For example,
                     in Shift JIS the regular expression "\\" accepts
                     the codepoint 0x5c, but should not accept the second
                     byte of the codepoint 0x815c.  Then the initial
                     state must skip the bytes that are not a single
                     byte character nor the first byte of a multibyte
                     character.  */
                  wint_t wc;
                  while (mbp < p)
                    mbp += mbs_to_wchar (&wc, (char const *) mbp,
                                         end - (char const *) mbp, d);
                  p = mbp;

                  if ((char *) p > end)
                    {
                      p = NULL;
                      goto done;
                    }
                }

              if (d->states[s].mbps.nelem == 0)
                {
                  s = t[*p++];
                  continue;
                }

              /* Falling back to the glibc matcher in this case gives
                 better performance (up to 25% better on [a-z], for
                 example) and enables support for collating symbols and
                 equivalence classes.  */
              if (d->states[s].has_mbcset && backref)
                {
                  *backref = 1;
                  goto done;
                }

              /* Can match with a multibyte character (and multi character
                 collating element).  Transition table might be updated.  */
              s = transit_state (d, s, &p, (unsigned char *) end);
              mbp = p;
              trans = d->trans;
            }
        }
      else
        {
          while ((t = trans[s]) != NULL)
            {
              s1 = t[*p++];
              if ((t = trans[s1]) == NULL)
                {
                  state_num tmp = s;
                  s = s1;
                  s1 = tmp;     /* swap */
                  break;
                }
              s = t[*p++];
            }
        }

      if ((char *) p > end)
        {
          p = NULL;
          goto done;
        }

      if (s >= 0 && d->fails[s])
        {
          if (d->success[s] & sbit[*p])
            {
              if (backref)
                *backref = d->states[s].has_backref;
              goto done;
            }

          s1 = s;
          if (d->multibyte)
            {
              /* Can match with a multibyte character (and multicharacter
                 collating element).  Transition table might be updated.  */
              s = transit_state (d, s, &p, (unsigned char *) end);
              mbp = p;
              trans = d->trans;
            }
          else
            s = d->fails[s][*p++];
          continue;
        }

      /* If the previous character was a newline, count it, and skip
         checking of multibyte character boundary until here.  */
      if (p[-1] == eol)
        {
          nlcount++;
          mbp = p;
        }

      if (s >= 0)
        {
          if (!d->trans[s])
            build_state (s, d);
          trans = d->trans;
          continue;
        }

      if (p[-1] == eol && allow_nl)
        {
          s = d->newlines[s1];
          continue;
        }

      s = 0;
    }

 done:
  if (count)
    *count += nlcount;
  *end = saved_end;
  return (char *) p;
}

struct dfa *
dfasuperset (struct dfa const *d)
{
  return d->superset;
}

bool
dfaisfast (struct dfa const *d)
{
  return d->fast;
}

static void
free_mbdata (struct dfa *d)
{
  size_t i;

  free (d->multibyte_prop);

  for (i = 0; i < d->nmbcsets; ++i)
    {
      size_t j;
      struct mb_char_classes *p = &(d->mbcsets[i]);
      free (p->chars);
      free (p->ch_classes);
      free (p->ranges);

      for (j = 0; j < p->nequivs; ++j)
        free (p->equivs[j]);
      free (p->equivs);

      for (j = 0; j < p->ncoll_elems; ++j)
        free (p->coll_elems[j]);
      free (p->coll_elems);
    }

  free (d->mbcsets);
  free (d->mb_follows.elems);
  free (d->mb_match_lens);
  d->mb_match_lens = NULL;
}

/* Initialize the components of a dfa that the other routines don't
   initialize for themselves.  */
void
dfainit (struct dfa *d)
{
  memset (d, 0, sizeof *d);
  d->multibyte = MB_CUR_MAX > 1;
  d->fast = !d->multibyte;
}

static void
dfaoptimize (struct dfa *d)
{
  size_t i;
  bool have_backref = false;

  if (!using_utf8 ())
    return;

  for (i = 0; i < d->tindex; ++i)
    {
      switch (d->tokens[i])
        {
        case ANYCHAR:
          /* Lowered.  */
          abort ();
        case BACKREF:
          have_backref = true;
          break;
        case MBCSET:
          /* Requires multi-byte algorithm.  */
          return;
        default:
          break;
        }
    }

  if (!have_backref && d->superset)
    {
      /* The superset DFA is not likely to be much faster, so remove it.  */
      dfafree (d->superset);
      free (d->superset);
      d->superset = NULL;
    }

  free_mbdata (d);
  d->multibyte = false;
}

static void
dfassbuild (struct dfa *d)
{
  size_t i, j;
  charclass ccl;
  bool have_achar = false;
  bool have_nchar = false;
  struct dfa *sup = dfaalloc ();

  *sup = *d;
  sup->multibyte = false;
  sup->multibyte_prop = NULL;
  sup->mbcsets = NULL;
  sup->superset = NULL;
  sup->states = NULL;
  sup->sindex = 0;
  sup->follows = NULL;
  sup->tralloc = 0;
  sup->trans = NULL;
  sup->fails = NULL;
  sup->success = NULL;
  sup->newlines = NULL;
  sup->musts = NULL;

  sup->charclasses = xnmalloc (sup->calloc, sizeof *sup->charclasses);
  memcpy (sup->charclasses, d->charclasses,
          d->cindex * sizeof *sup->charclasses);

  sup->tokens = xnmalloc (d->tindex, 2 * sizeof *sup->tokens);
  sup->talloc = d->tindex * 2;

  for (i = j = 0; i < d->tindex; i++)
    {
      switch (d->tokens[i])
        {
        case ANYCHAR:
        case MBCSET:
        case BACKREF:
          zeroset (ccl);
          notset (ccl);
          sup->tokens[j++] = CSET + dfa_charclass_index (sup, ccl);
          sup->tokens[j++] = STAR;
          if (d->tokens[i + 1] == QMARK || d->tokens[i + 1] == STAR
              || d->tokens[i + 1] == PLUS)
            i++;
          have_achar = true;
          break;
        case BEGWORD:
        case ENDWORD:
        case LIMWORD:
        case NOTLIMWORD:
          if (d->multibyte)
            {
              /* These constraints aren't supported in a multibyte locale.
                 Ignore them in the superset DFA, and treat them as
                 backreferences in the main DFA.  */
              sup->tokens[j++] = EMPTY;
              d->tokens[i] = BACKREF;
              break;
            }
        default:
          sup->tokens[j++] = d->tokens[i];
          if ((0 <= d->tokens[i] && d->tokens[i] < NOTCHAR)
              || d->tokens[i] >= CSET)
            have_nchar = true;
          break;
        }
    }
  sup->tindex = j;

  if (have_nchar && (have_achar || d->multibyte))
    d->superset = sup;
  else
    {
      dfafree (sup);
      free (sup);
    }
}

/* Run the fsa* etc modules (parsing/lexing/tokens/charclass) first, then
   run the original dfa code, with a hook added that the parser/lexer
   interface in the original code calls the new code in parallel, so that
   the veracity of the lexical token streams can be inspected (they should
   be identical).  After these runs, the parser token list from each side
   is extracted and compared, and should also be identical.   */

/* Parse and analyze a single string of the given length.  */
void
dfacomp (char const *s, size_t len, struct dfa *d, int searchflag)
{
  size_t i;

  HOOK_set_up_fsa_stuff_if_not_done_already ();

  /* HOOK: Write the search pattern to the logfile..  */
  if (! pll_log)
    pll_log = fopen(HOOK_LOG_FILENAME, "a");
  fprintf (pll_log, "Pattern:");
  for (i = 0; i < len; i++)
      fprintf (pll_log, "  %c", isprint (s[i]) ? s[i] : ' ');
  fprintf (pll_log, "\n        ");
  for (i = 0; i < len; i++)
    fprintf (pll_log, " %02x", ((unsigned) s[i]) & 0xff);
  fprintf (pll_log, "\n");

  /* HOOK: Run the new code first so any dependencies on old code are more
     likely to bring up errors.  */
  fsalex_pattern (lexer, s, len);
  fsaparse_parse (parser);

  /* Re-prime fsalex () to analyse the pattern a second time, so that
     lexical tokens from both systems can be compared side-by-side.
     This lexical-token-check watches the activity at the parser/lexer
     boundary; the code below compares the outputs of the original parser
     and fsaparse, and they should also be functionally identical (some
     differences can occur in the indices added to CSET tokens, but
     the referenced set should be identical).  */
  fsalex_pattern (lexer, s, len);

  dfainit (d);
  dfambcache (d);
  dfaparse (s, len, d);
  dfamust (d);

  /* YET ANOTHER HACK, 16 April 2014 (was it related to the lunar eclipse
     last night?? !!?? )
     Compare, side-by-side, the list of tokens generated by dfa.c and by
     fsaparse, and write these to the debug log file.  As elsewhere, these
     should be identical, as the modularised code starts as a functional
     clone of the original code.  (Later, if/when tokens are reworked to
     maintain abstractions at a higher level, the token lists will
     differ.)  */
  {
    size_t nr_tokens;
    fsatoken_token_t *token_list;
    size_t j;
    size_t max_index;

    fsaparse_get_token_list (parser, &nr_tokens, &token_list);
    fprintf (pll_log, "\ntokens:  original  fsaparse\n");
    max_index = d->tindex > nr_tokens ? d->tindex : nr_tokens;
    for (j = 0; j < max_index; ++j)
      {
        static char buf[256];
        if (j < d->tindex)
          {
            sprintf (buf, "%02lx", d->tokens[j]);
            fprintf (pll_log, "%17s ", buf);
          }
          else
            fprintf (pll_log, "%17s", "");
        if (j < nr_tokens)
          {
            sprintf (buf, "%02lx", token_list[j]);
            fprintf (pll_log, "%9s", buf);
          }
        fprintf (pll_log, "\n");
      }

    /* And finally, see how extracting musts from dfa.c compares to extracting
       musts via the fsa/charclass family of functions; again, these should
       be identical.  */
    musts = (fsamusts_list_element_t *) d->musts;
    show_musts ("original dfa.c", musts);

    /* ANOTHER UGLY HACK: Rely on dfa.c's case_fold and unibyte locale when
       instructing dfamust how to operate; an "Exchange" function might be
       more appropriate in the short-to-mid-term, but in the longer term,
       the token vocabluary should get more expressive, so that information
       can be conveyed directly.  */
    musts = fsamusts_must (NULL, nr_tokens, token_list,
                           /* dfa.c copy: */ case_fold,
                           /* current (dfa.c) locale: */ MB_CUR_MAX == 1);
    show_musts ("fsa* et al functions", musts);
  }

  dfassbuild (d);
  dfaoptimize (d);
  dfaanalyze (d, searchflag);
  if (d->superset)
    {
      d->fast = true;
      dfaanalyze (d->superset, searchflag);
    }
}

/* Free the storage held by the components of a dfa.  */
void
dfafree (struct dfa *d)
{
  size_t i;
  struct dfamust *dm, *ndm;

  free (d->charclasses);
  free (d->tokens);

  if (d->multibyte)
    free_mbdata (d);

  for (i = 0; i < d->sindex; ++i)
    {
      free (d->states[i].elems.elems);
      free (d->states[i].mbps.elems);
    }
  free (d->states);

  if (d->follows)
    {
      for (i = 0; i < d->tindex; ++i)
        free (d->follows[i].elems);
      free (d->follows);
    }

  if (d->trans)
    {
      for (i = 0; i < d->tralloc; ++i)
        {
          free (d->trans[i]);
          free (d->fails[i]);
        }

      free (d->trans - 1);
      free (d->fails);
      free (d->newlines);
      free (d->success);
    }

  for (dm = d->musts; dm; dm = ndm)
    {
      ndm = dm->next;
      free (dm->must);
      free (dm);
    }

  if (d->superset)
    dfafree (d->superset);

  /* HOOK: Tear down all the infrastructure added for parallel operation.  */
  HOOK_tear_down_additional_infrastructure ();
}

/* Having found the postfix representation of the regular expression,
   try to find a long sequence of characters that must appear in any line
   containing the r.e.
   Finding a "longest" sequence is beyond the scope here;
   we take an easy way out and hope for the best.
   (Take "(ab|a)b"--please.)

   We do a bottom-up calculation of sequences of characters that must appear
   in matches of r.e.'s represented by trees rooted at the nodes of the postfix
   representation:
        sequences that must appear at the left of the match ("left")
        sequences that must appear at the right of the match ("right")
        lists of sequences that must appear somewhere in the match ("in")
        sequences that must constitute the match ("is")

   When we get to the root of the tree, we use one of the longest of its
   calculated "in" sequences as our answer.  The sequence we find is returned in
   d->must (where "d" is the single argument passed to "dfamust");
   the length of the sequence is returned in d->mustn.

   The sequences calculated for the various types of node (in pseudo ANSI c)
   are shown below.  "p" is the operand of unary operators (and the left-hand
   operand of binary operators); "q" is the right-hand operand of binary
   operators.

   "ZERO" means "a zero-length sequence" below.

        Type	left		right		is		in
        ----	----		-----		--		--
        char c	# c		# c		# c		# c

        ANYCHAR	ZERO		ZERO		ZERO		ZERO

        MBCSET	ZERO		ZERO		ZERO		ZERO

        CSET	ZERO		ZERO		ZERO		ZERO

        STAR	ZERO		ZERO		ZERO		ZERO

        QMARK	ZERO		ZERO		ZERO		ZERO

        PLUS	p->left		p->right	ZERO		p->in

        CAT	(p->is==ZERO)?	(q->is==ZERO)?	(p->is!=ZERO &&	p->in plus
                p->left :	q->right :	q->is!=ZERO) ?	q->in plus
                p->is##q->left	p->right##q->is	p->is##q->is :	p->right##q->left
                                                ZERO

        OR	longest common	longest common	(do p->is and	substrings common to
                leading		trailing	q->is have same	p->in and q->in
                (sub)sequence	(sub)sequence	length and
                of p->left	of p->right	content) ?
                and q->left	and q->right	p->is : NULL

   If there's anything else we recognize in the tree, all four sequences get set
   to zero-length sequences.  If there's something we don't recognize in the
   tree, we just return a zero-length sequence.

   Break ties in favor of infrequent letters (choosing 'zzz' in preference to
   'aaa')?

   And ... is it here or someplace that we might ponder "optimizations" such as
        egrep 'psi|epsilon'	->	egrep 'psi'
        egrep 'pepsi|epsilon'	->	egrep 'epsi'
                                        (Yes, we now find "epsi" as a "string
                                        that must occur", but we might also
                                        simplify the *entire* r.e. being sought)
        grep '[c]'		->	grep 'c'
        grep '(ab|a)b'		->	grep 'ab'
        grep 'ab*'		->	grep 'a'
        grep 'a*b'		->	grep 'b'

   There are several issues:

   Is optimization easy (enough)?

   Does optimization actually accomplish anything,
   or is the automaton you get from "psi|epsilon" (for example)
   the same as the one you get from "psi" (for example)?

   Are optimizable r.e.'s likely to be used in real-life situations
   (something like 'ab*' is probably unlikely; something like is
   'psi|epsilon' is likelier)?  */

static char *
icatalloc (char *old, char const *new)
{
  char *result;
  size_t oldsize;
  size_t newsize = strlen (new);
  if (newsize == 0)
    return old;
  oldsize = strlen (old);
  result = xrealloc (old, oldsize + newsize + 1);
  memcpy (result + oldsize, new, newsize + 1);
  return result;
}

static char *_GL_ATTRIBUTE_PURE
istrstr (char const *lookin, char const *lookfor)
{
  char const *cp;
  size_t len;

  len = strlen (lookfor);
  for (cp = lookin; *cp != '\0'; ++cp)
    if (strncmp (cp, lookfor, len) == 0)
      return (char *) cp;
  return NULL;
}

static void
freelist (char **cpp)
{
  while (*cpp)
    free (*cpp++);
}

static char **
enlist (char **cpp, char *new, size_t len)
{
  size_t i, j;
  new = memcpy (xmalloc (len + 1), new, len);
  new[len] = '\0';
  /* Is there already something in the list that's new (or longer)?  */
  for (i = 0; cpp[i] != NULL; ++i)
    if (istrstr (cpp[i], new) != NULL)
      {
        free (new);
        return cpp;
      }
  /* Eliminate any obsoleted strings.  */
  j = 0;
  while (cpp[j] != NULL)
    if (istrstr (new, cpp[j]) == NULL)
      ++j;
    else
      {
        free (cpp[j]);
        if (--i == j)
          break;
        cpp[j] = cpp[i];
        cpp[i] = NULL;
      }
  /* Add the new string.  */
  cpp = xnrealloc (cpp, i + 2, sizeof *cpp);
  cpp[i] = new;
  cpp[i + 1] = NULL;
  return cpp;
}

/* Given pointers to two strings, return a pointer to an allocated
   list of their distinct common substrings.  */
static char **
comsubs (char *left, char const *right)
{
  char **cpp = xzalloc (sizeof *cpp);
  char *lcp;

  for (lcp = left; *lcp != '\0'; ++lcp)
    {
      size_t len = 0;
      char *rcp = strchr (right, *lcp);
      while (rcp != NULL)
        {
          size_t i;
          for (i = 1; lcp[i] != '\0' && lcp[i] == rcp[i]; ++i)
            continue;
          if (i > len)
            len = i;
          rcp = strchr (rcp + 1, *lcp);
        }
      if (len != 0)
        cpp = enlist (cpp, lcp, len);
    }
  return cpp;
}

static char **
addlists (char **old, char **new)
{
  for (; *new; new++)
    old = enlist (old, *new, strlen (*new));
  return old;
}

/* Given two lists of substrings, return a new list giving substrings
   common to both.  */
static char **
inboth (char **left, char **right)
{
  char **both = xzalloc (sizeof *both);
  size_t lnum, rnum;

  for (lnum = 0; left[lnum] != NULL; ++lnum)
    {
      for (rnum = 0; right[rnum] != NULL; ++rnum)
        {
          char **temp = comsubs (left[lnum], right[rnum]);
          both = addlists (both, temp);
          freelist (temp);
          free (temp);
        }
    }
  return both;
}

typedef struct must must;

struct must
{
  char **in;
  char *left;
  char *right;
  char *is;
  bool begline;
  bool endline;
  must *prev;
};

static must *
allocmust (must *mp)
{
  must *new_mp = xmalloc (sizeof *new_mp);
  new_mp->in = xzalloc (sizeof *new_mp->in);
  new_mp->left = xzalloc (2);
  new_mp->right = xzalloc (2);
  new_mp->is = xzalloc (2);
  new_mp->begline = false;
  new_mp->endline = false;
  new_mp->prev = mp;
  return new_mp;
}

static void
resetmust (must *mp)
{
  freelist (mp->in);
  mp->in[0] = NULL;
  mp->left[0] = mp->right[0] = mp->is[0] = '\0';
  mp->begline = false;
  mp->endline = false;
}

static void
freemust (must *mp)
{
  freelist (mp->in);
  free (mp->in);
  free (mp->left);
  free (mp->right);
  free (mp->is);
  free (mp);
}

static void
dfamust (struct dfa *d)
{
  must *mp = NULL;
  char const *result = "";
  size_t ri;
  size_t i;
  bool exact = false;
  bool begline = false;
  bool endline = false;
  struct dfamust *dm;

  for (ri = 0; ri < d->tindex; ++ri)
    {
      token t = d->tokens[ri];
      switch (t)
        {
        case BEGLINE:
          mp = allocmust (mp);
          mp->begline = true;
          break;
        case ENDLINE:
          mp = allocmust (mp);
          mp->endline = true;
          break;
        case LPAREN:
        case RPAREN:
          assert (!"neither LPAREN nor RPAREN may appear here");

        case EMPTY:
        case BEGWORD:
        case ENDWORD:
        case LIMWORD:
        case NOTLIMWORD:
        case BACKREF:
        case ANYCHAR:
        case MBCSET:
          mp = allocmust (mp);
          break;

        case STAR:
        case QMARK:
          resetmust (mp);
          break;

        case OR:
          {
            char **new;
            must *rmp = mp;
            must *lmp = mp = mp->prev;
            size_t j, ln, rn, n;

            /* Guaranteed to be.  Unlikely, but ...  */
            if (STREQ (lmp->is, rmp->is))
              {
                lmp->begline &= rmp->begline;
                lmp->endline &= rmp->endline;
              }
            else
              {
                lmp->is[0] = '\0';
                lmp->begline = false;
                lmp->endline = false;
              }
            /* Left side--easy */
            i = 0;
            while (lmp->left[i] != '\0' && lmp->left[i] == rmp->left[i])
              ++i;
            lmp->left[i] = '\0';
            /* Right side */
            ln = strlen (lmp->right);
            rn = strlen (rmp->right);
            n = ln;
            if (n > rn)
              n = rn;
            for (i = 0; i < n; ++i)
              if (lmp->right[ln - i - 1] != rmp->right[rn - i - 1])
                break;
            for (j = 0; j < i; ++j)
              lmp->right[j] = lmp->right[(ln - i) + j];
            lmp->right[j] = '\0';
            new = inboth (lmp->in, rmp->in);
            freelist (lmp->in);
            free (lmp->in);
            lmp->in = new;
            freemust (rmp);
          }
          break;

        case PLUS:
          mp->is[0] = '\0';
          break;

        case END:
          assert (!mp->prev);
          for (i = 0; mp->in[i] != NULL; ++i)
            if (strlen (mp->in[i]) > strlen (result))
              result = mp->in[i];
          if (STREQ (result, mp->is))
            {
              exact = true;
              begline = mp->begline;
              endline = mp->endline;
            }
          goto done;

        case CAT:
          {
            must *rmp = mp;
            must *lmp = mp = mp->prev;

            /* In.  Everything in left, plus everything in
               right, plus concatenation of
               left's right and right's left.  */
            lmp->in = addlists (lmp->in, rmp->in);
            if (lmp->right[0] != '\0' && rmp->left[0] != '\0')
              {
                size_t lrlen = strlen (lmp->right);
                size_t rllen = strlen (rmp->left);
                char *tp = xmalloc (lrlen + rllen);
                memcpy (tp, lmp->right, lrlen);
                memcpy (tp + lrlen, rmp->left, rllen);
                lmp->in = enlist (lmp->in, tp, lrlen + rllen);
                free (tp);
              }
            /* Left-hand */
            if (lmp->is[0] != '\0')
              lmp->left = icatalloc (lmp->left, rmp->left);
            /* Right-hand */
            if (rmp->is[0] == '\0')
              lmp->right[0] = '\0';
            lmp->right = icatalloc (lmp->right, rmp->right);
            /* Guaranteed to be */
            if ((lmp->is[0] != '\0' || lmp->begline)
                && (rmp->is[0] != '\0' || rmp->endline))
              {
                lmp->is = icatalloc (lmp->is, rmp->is);
                lmp->endline = rmp->endline;
              }
            else
              {
                lmp->is[0] = '\0';
                lmp->begline = false;
                lmp->endline = false;
              }
            freemust (rmp);
          }
          break;

        case '\0':
          /* Not on *my* shift.  */
          goto done;

        default:
          mp = allocmust (mp);
          if (CSET <= t)
            {
              /* If T is a singleton, or if case-folding in a unibyte
                 locale and T's members all case-fold to the same char,
                 convert T to one of its members.  Otherwise, do
                 nothing further with T.  */
              charclass *ccl = &d->charclasses[t - CSET];
              int j;
              for (j = 0; j < NOTCHAR; j++)
                if (tstbit (j, *ccl))
                  break;
              if (! (j < NOTCHAR))
                break;
              t = j;
              while (++j < NOTCHAR)
                if (tstbit (j, *ccl)
                    && ! (case_fold && !d->multibyte
                          && toupper (j) == toupper (t)))
                  break;
              if (j < NOTCHAR)
                break;
            }
          mp->is[0] = mp->left[0] = mp->right[0]
            = case_fold && !d->multibyte ? toupper (t) : t;
          mp->is[1] = mp->left[1] = mp->right[1] = '\0';
          mp->in = enlist (mp->in, mp->is, 1);
          break;
        }
    }
done:
  if (*result)
    {
      dm = xmalloc (sizeof *dm);
      dm->exact = exact;
      dm->begline = begline;
      dm->endline = endline;
      dm->must = xstrdup (result);
      dm->next = d->musts;
      d->musts = dm;
    }

  while (mp)
    {
      must *prev = mp->prev;
      freemust (mp);
      mp = prev;
    }
}

struct dfa *
dfaalloc (void)
{
  return xmalloc (sizeof (struct dfa));
}

struct dfamust *_GL_ATTRIBUTE_PURE
dfamusts (struct dfa const *d)
{
  return d->musts;
}

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="fsalex.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsalex.c"

/* fsalex - Repackage pattern text as compact, expressive tokens

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* Always import environment-specific configuration items first.  */
#include <config.h>    /* define _GNU_SOURCE for regex extensions.  */

#include <assert.h>
#include "charclass.h"
#include <ctype.h>
#include "fsalex.h"
#include "fsatoken.h"
#include <limits.h>
#include <locale.h>
#include "mbcsets.h"
#include "proto-lexparse.h"
#include <regex.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <wctype.h>
#include "xalloc.h"

#include "gettext.h"
#define _(str) gettext (str)

/* ISASCIIDIGIT differs from isdigit, as follows:
   - Its arg may be any int or unsigned int; it need not be an unsigned char.
   - It's guaranteed to evaluate its argument exactly once.
   - It's typically faster.
   Posix 1003.2-1992 section 2.5.2.1 page 50 lines 1556-1558 says that
   only '0' through '9' are digits.  Prefer ISASCIIDIGIT to isdigit unless
   it's important to use the locale's definition of "digit" even when the
   host does not conform to Posix.  */
#define ISASCIIDIGIT(c) ((unsigned) (c) - '0' <= 9)

#define STREQ(a, b) (strcmp (a, b) == 0)

#ifndef MIN
# define MIN(a,b) ((a) < (b) ? (a) : (b))
#endif

/* The following list maps the names of the Posix named character
   classes to predicate functions that determine whether a given
   character is in the class.  The leading [ has already been eaten by
   the lexical analyzer.  Additional objects are provided to assist
   the client: wchar_desc for multibyte matching, and class for octet
   matching.  Lazy evaluation and caching are used to minimise
   processing costs, so these additional items are only valid after a
   class has been located using find_pred ().  */
typedef int predicate_t (wint_t, wctype_t);
typedef struct predicate_entry_struct
{
  const char *name;
  wctype_t wchar_desc;
  charclass_t *class;
  bool may_be_multibyte;
} predicate_entry_t;

/* This list is a template, copied into each lexer's state, and
   interrogated and updated from there.  The membership of a class can
   vary due to locale and other settings, so each lexer must maintain
   its own list.  Duplicate class sharing across different lexer
   instances is facilitated by checks in charclass_completed.  */
/* Locale portability note: We may need to record the entire explicit
   locale when syntax () is called, and then use isalpha_l () etc.
   here, as otherwise the wchar_desc may be interpreted at runtime in
   the context of the current locale.  If a complete copy of the locale
   needs to be stored locally, see uselocale(3) and duplocale(3) (and
   free the copy with freelocale(3)).  */
static predicate_entry_t template_predicate_list[] = {
  {"alpha",  0, NULL, true},
  {"alnum",  0, NULL, true},
  {"blank",  0, NULL, true},
  {"cntrl",  0, NULL, true},
  {"digit",  0, NULL, false},
  {"graph",  0, NULL, true},
  {"lower",  0, NULL, true},
  {"print",  0, NULL, true},
  {"punct",  0, NULL, true},
  {"space",  0, NULL, true},
  {"upper",  0, NULL, true},
  {"xdigit", 0, NULL, true},
  {NULL, 0, NULL}
};

#define PREDICATE_TEMPLATE_ITEMS \
    (sizeof template_predicate_list / sizeof *template_predicate_list)

/* Flesh out the opaque instance context type given in the header.  */
struct fsalex_ctxt_struct
{
  /* Singly-linked list of all lexer instances, so destroy_module can
     release all resources by traversing the list.  */
  fsalex_ctxt_t *next_instance;

  /* Using the lexer without setting the syntax is a fatal error, so
     use a flag so we can report such errors in a direct fashion.  */
  bool syntax_initialised;

  /* Syntax flags/characters directing how to interpret the pattern.  */
  reg_syntax_t syntax_bits;
  bool case_fold;
  unsigned char eolbyte;

  /* Exception handling is done by explicit callbacks.  */
  fsalex_warn_callback_fn *warn_client;
  fsalex_error_callback_fn *abandon_with_error;

  /* Pattern pointer/length, updated as pattern is consumed.  */
  char const *lexptr;
  size_t lexleft;

  /* Break out some regex syntax bits into boolean vars.  Do this for
     the ones that are heavily used, and/or where the nature of the
     bitmask flag test tends to clutter the lexer code.  */
  bool re_gnu_ops;             /* GNU regex operators are allowed.  */

  /* Carry dotclass here, as it's easier for clients (UTF-8) to perform
     class operations with this class, rather than to know intimate
     details of the regex syntax configuration bits and items such as
     eolbyte.  */
  charclass_t *dotclass;         /* ".": All chars except eolbyte and/or
                                        NUL, depending on syntax flags.  */
  charclass_index_t dotclass_index;

  /* Work variables to help organise lexer operation.  */
  fsatoken_token_t lasttok;
  bool laststart;
  size_t parens;

  /* Character class predicate mapping/caching table.  */
  predicate_entry_t predicates[PREDICATE_TEMPLATE_ITEMS];

  /* Minrep and maxrep are actually associated with the REPMN token,
     and need to be accessible outside this module (by the parser),
     perhaps by an explicit interface call.  In the far, far future,
     a completely-reworked token list may see these values properly
     become integrated into the token stream (perhaps by a pair of
     "Parameter" tokens?  Perhaps by a MINREP token with 1 parameter,
     followed by a MAXREP token with a corresponding parameter?)  */
  int minrep, maxrep;

  /* Booleans to simplify unibyte/multibyte code selection paths.  In
     addition, other flags are placed here to summarise properties of
     the locale in a concise fashion that can useful when
     implementing optimisations.  There may be some overlap and/or
     redundancy here; flag names are chosen to let the user be direct
     and concise, but we only provide names for the more popular cases,
     not all.  */
  char *locale_name;
  bool unibyte_locale;
  bool multibyte_locale;
  bool ascii_7bit_encoding;
  bool using_simple_locale;

  /* REVIEWME: Wide-character support variables.  */
  int cur_mb_len;       /* Length (in bytes) of the last character
                           fetched; this is needed when backing up
                           during lexing.  In a non-multibyte situation
                           (locale?), this variable remains at 1;
                           otherwise, it is updated as required by
                           FETCH_WC.  */

  /* These variables are used only if in a multibyte locale.  */
  wchar_t wctok;         /* Storage for a single multibyte character,
                            used both during lexing, and as the implied
                            parameter of a WCHAR token returned by the
                            lexer.  */
  mbstate_t mbrtowc_state; /* State management area for mbrtowc to
                              use. */

  /* A table indexed by byte values that contains the corresponding wide
     character (if any) for that byte.  WEOF means the byte is not a
     valid single-byte character.  */
  wint_t mbrtowc_cache[FSATOKEN_NOTCHAR];

  mbcsets_set_t **mbcsets;
  size_t nmbcsets;
  size_t mbcsets_alloc;
};

/* Linked list of all instances created by this module.  */
static fsalex_ctxt_t *fsalex_instance_list_head = NULL;

/* Ensure that the array addressed by PTR holds at least NITEMS +
   (PTR || !NITEMS) items.  Either return PTR, or reallocate the array
   and return its new address.  Although PTR may be null, the returned
   value is never null.

   The array holds *NALLOC items; *NALLOC is updated on reallocation.
   ITEMSIZE is the size of one item.  Avoid O(N**2) behavior on arrays
   growing linearly.  */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
  if (nitems < *nalloc)
    return ptr;
  *nalloc = nitems;
  return x2nrealloc (ptr, nalloc, itemsize);
}

/* Set a bit in the charclass for the given wchar_t.  Do nothing if WC
   is represented by a multi-byte sequence.  Even in unibyte locales,
   this may happen when folding case in weird Turkish locales where
   dotless i/dotted I are not included in the chosen character set.
   Return whether a bit was set in the charclass.  */
/* Set a bit for B and its case variants in the charclass C.
  We must be in an unibyte locale.  */
static void
setbit_case_fold_c (int b, charclass_t *c)
{
  int ub = toupper (b);
  int i;
  for (i = 0; i < FSATOKEN_NOTCHAR; i++)
    if (toupper (i) == ub)
      charclass_setbit (i, c);
}

/* Convert a possibly-signed character to an unsigned character.  This is
   a bit safer than casting to unsigned char, since it catches some type
   errors that the cast doesn't.  */
static unsigned char
to_uchar (char ch)
{
  return ch;
}

static void
initialise_uchar_to_wc_cache (fsalex_ctxt_t *lexer)
{
  int i;
  for (i = CHAR_MIN; i <= CHAR_MAX; ++i)
    {
      char c = i;
      unsigned char uc = i;
      mbstate_t s = { 0 };
      wchar_t wc;
      lexer->mbrtowc_cache[uc] = mbrtowc (&wc, &c, 1, &s) <= 1 ? wc : WEOF;
    }
}

/* This function is intimately connected with multibyte (wide-char)
   handling in the macro FETCH_WC below, in the case where
   FETCH_SINGLE_CHAR has run but the result has been found to be
   inconclusive.  It works by unwinding the FETCH_SINGLE_CHAR
   side-effects (lexptr/lexleft), then calling mbrtowc on the pattern
   space, and communicates mbrtowc's understanding of the octet
   stream back to the caller:
     - If a valid multibyte octet sequence is next, then the wide
       character associated with this sequence is written back to
       *p_wchar, and the number of octets consumed is returned; or
     - If the sequence is invalid for any reason, the mbrtowc working
       state is reset (zeroed), *p_wchar is not modified, and 1 is
       returned.
   Lexer state variables, including cur_mb_len, mbs, lexleft and
   lexptr, are updated as appropriate by this function (mainly if
   mbrtowc succeeds).  The wide char NUL is unusual as it is a
   1-octet sequence, the length returned is 0, we report it as length
   1, but write the converted wide character in temp_wchar to the
   caller.  */
/* Additional notes:  This code, in partnership with the macro
   FETCH_WC, is closely related to mbs_to_wchar in dfa.c.  There is
   documentation there (e.g. pattern must end in a sentinel, shift
   encodings not supported, plus other comments/guarantees) that is
   important, but I'm deferring writing up anything at present until
   I see how this code is received.  */
static size_t
fetch_offset_wide_char (fsalex_ctxt_t *lexer, wchar_t *p_wchar)
{
  size_t nbytes;
  wchar_t temp_wchar;

  nbytes = mbrtowc (&temp_wchar,
                    lexer->lexptr - 1, lexer->lexleft + 1,
                    &lexer->mbrtowc_state);
  switch (nbytes)
    {
    case (size_t) -2:
    case (size_t) -1:
      /* Conversion failed: Incomplete (-2) or invalid (-1)
         sequence.  */
      memset (&lexer->mbrtowc_state, 0, sizeof (lexer->mbrtowc_state));
      return 1;

    case (size_t) 0:
      /* This is the wide NUL character, actually 1 byte long. */
      nbytes = 1;
      break;

    default:
      /* Converted character is in temp_wchar, and nbytes is a byte
         count.  */
      break;
    }
  /* We converted 1 or more bytes, tell result to caller.  */
  *p_wchar = temp_wchar;

  /* Update the number of bytes consumed (offset by 1 since
     FETCH_SINGLE_CHAR grabbed one earlier).  */
  lexer->lexptr  += nbytes - 1;
  lexer->lexleft -= nbytes - 1;

  return nbytes;
}

/* Single-character input fetch, with EOF/error handling.  Note that
   characters become unsigned here.  If no characters are available,
   the macro either returns END or reports an error, depending on
   eoferr.  Otherwise, one character is consumed (lexptr/lexleft),
   the char is converted into an unsigned char, and is written into
   the parameter c.  */
#define FETCH_SINGLE_CHAR(lexer, c, eoferr)                  \
  do {                                                       \
    if (! (lexer)->lexleft)                                  \
      {                                                      \
        if ((eoferr) != 0)                                   \
          (lexer)->abandon_with_error (eoferr);              \
        else                                                 \
          return FSATOKEN_TK_END;                            \
      }                                                      \
    (c) = to_uchar (*(lexer)->lexptr++);                     \
    (lexer)->lexleft--;                                      \
  } while (0)

/* Do the fetch in stages: Single char, octet+multibyte cache
   check, and possible wide char fetch if the cache result indicates
   that the input sequence is longer than a single octet.  The first
   fetch handles end-of-input cases (if this happens, control never
   reaches the rest of the macro); otherwise, it returns temp_uchar
   which is used in the cache lookup, and may be the single-octet
   result.  A cache result of WEOF means that the octet is not a
   complete sequence by itself, so a second fetch tweaks lexptr/lexleft
   to undo the single-char-fetch side-effects, and, depending on
   mbrtowc valid/invalid result, propagates either the multichar fetch
   or the single-char fetch back to the caller.  */
# define FETCH_WC(lexer, c, wc, eoferr)                      \
  do {                                                       \
    wchar_t temp_wc;                                         \
    unsigned char temp_uchar;                                \
    (lexer)->cur_mb_len = 1;                                 \
    FETCH_SINGLE_CHAR ((lexer), temp_uchar, (eoferr));       \
    temp_wc = (lexer)->mbrtowc_cache[temp_uchar];            \
    if (temp_wc != WEOF)                                     \
      {                                                      \
        (c)  = temp_uchar;                                   \
        (wc) = temp_wc;                                      \
      }                                                      \
    else                                                     \
      {                                                      \
        size_t nbytes;                                       \
        temp_wc = temp_uchar;                                \
        nbytes = fetch_offset_wide_char ((lexer), &temp_wc); \
        (wc) = temp_wc;                                      \
        (c) = nbytes == 1 ? temp_uchar : EOF;                \
        (lexer)->cur_mb_len = nbytes;                        \
      }                                                      \
  } while (0)

/* Given a predicate name, find it in a list, and report the list
   entry to the caller.  If the name is not recognised, the function
   returns NULL.  The list entry includes a charclass set and (if
   relevant) a wide-char descriptor for testing for the predicate.
   Lazy evaluation and caching are used to keep processing costs
   down.  */
static predicate_entry_t *
find_pred (fsalex_ctxt_t *lexer, const char *str)
{
  predicate_entry_t *p_entry;
  charclass_t *work_class;

  for (p_entry = lexer->predicates; p_entry->name; p_entry++)
    {
      if (STREQ (str, p_entry->name))
        break;
    }

  /* If there was no matching predicate name found, return NULL.  */
  if (! p_entry->name)
    return NULL;

  /* Is the charclass pointer NULL for this entry? */
  if (p_entry->class == NULL)
    {
      /* Yes, allocate, set up and cache a charclass for this
         predicate.  Note that the wchar_desc entries were set up in
         fsalex_syntax ().  */
      int i;
      charclass_index_t index;
      wctype_t wctype_desc;

      wctype_desc = p_entry->wchar_desc;
      work_class = charclass_alloc ();
      for (i = 0; i < FSATOKEN_NOTCHAR; i++)
        {
          wchar_t wc;

          /* Try integer->unsigned char->wide char using lexer's
             mbrtowc_cache array, and, if successful, test for class
             membership, and set the bit in the class if the value is
             a member.  */
          /* FIXME: iswctype depends on the *current* locale
             (LC_CTYPE); this breaks the long-term goal of having each
             lexer instance use only the locale in force when
             fsalex_syntax was called.  We can fix this by using
             isalnum_l () etc here, but carrying around a full copy of
             the locale in the lexer instance may be expensive or
             possibly non-portable, so it's being avoided while this
             code is still in an experimental/proof-of-concept
             form.  */
          wc = lexer->mbrtowc_cache[i];
          if (iswctype (wc, wctype_desc))
            charclass_setbit (i, work_class);
        }

      /* Mark the class as completed, and obtain a persistent class
         pointer.  */
      index = charclass_completed (work_class);
      p_entry->class = charclass_get_pointer (index);

    }

  /* Return predicate entry to the caller.  */
  return p_entry;
}

/* The set of wchar_t values C such that there's a useful locale
   somewhere where C != towupper (C) && C != towlower (towupper (C)).
   For example, 0x00B5 (U+00B5 MICRO SIGN) is in this table, because
   towupper (0x00B5) == 0x039C (U+039C GREEK CAPITAL LETTER MU), and
   towlower (0x039C) == 0x03BC (U+03BC GREEK SMALL LETTER MU).  */
static short const lonesome_lower[] =
  {
    0x00B5, 0x0131, 0x017F, 0x01C5, 0x01C8, 0x01CB, 0x01F2, 0x0345,
    0x03C2, 0x03D0, 0x03D1, 0x03D5, 0x03D6, 0x03F0, 0x03F1,

    /* U+03F2 GREEK LUNATE SIGMA SYMBOL lacks a specific uppercase
       counterpart in locales predating Unicode 4.0.0 (April 2003).  */
    0x03F2,

    0x03F5, 0x1E9B, 0x1FBE,
  };

/* Maximum number of wide characters that can be the case-folded
   counterparts of a single wide character, not counting the character
   itself.  This is 1 for towupper, 1 for towlower, and 1 for each
   entry in lonesome_lower[].  */
enum { CASE_FOLDED_BUFSIZE = 1 + 1 + 19 };

/* Given a wide character c, and working to a search specification
   that demands that case-folded variants of the character must match
   wherever that character could match, generate a list of all
   possible match counterparts to the specified character, and return
   that list, the characters via a parameter pointer, and the list
   length (possibly 0) via the function return value.  */
/* ?? CHECKME: Should parameter "c" in, and/or "folded" out, be wint_t?  */
static int
case_folded_counterparts (fsalex_ctxt_t *lexer,
                          wchar_t c,
                          wchar_t folded[CASE_FOLDED_BUFSIZE])
{
  int i;
  int n = 0;
  wint_t uc = towupper (c);
  wint_t lc = towlower (uc);
  if (uc != c)
    folded[n++] = uc;
  if (lc != uc && lc != c && towupper (lc) == uc)
    folded[n++] = lc;
  for (i = 0; i < sizeof lonesome_lower / sizeof *lonesome_lower; i++)
    {
      wint_t li = lonesome_lower[i];
      if (li != lc && li != uc && li != c && towupper (li) == uc)
        folded[n++] = li;
    }
  return n;
}

/* Multibyte character handling sub-routine for lex.
   Parse a bracket expression and build a struct mb_char_classes.  */
static fsatoken_token_t
parse_bracket_exp (fsalex_ctxt_t *lexer)
{
  bool invert;
  int c, c1, c2;
  charclass_t *ccl;
  mbcsets_set_t *work_mbc = NULL;


  /* This is a bracket expression that dfaexec is known to
     process correctly.  */
  bool known_bracket_exp = true;

  /* Used to warn about [:space:].
     Bit 0 = first character is a colon.
     Bit 1 = last character is a colon.
     Bit 2 = includes any other character but a colon.
     Bit 3 = includes ranges, char/equiv classes or collation elements.  */
  int colon_warning_state;

  wint_t wc;
  wint_t wc2;
  wint_t wc1 = 0;

  if (lexer->multibyte_locale)
    {
      /* Ensure we have space to store the new set.  */
      lexer->mbcsets = maybe_realloc (lexer->mbcsets,
                                      lexer->nmbcsets,
                                      &lexer->mbcsets_alloc,
                                      sizeof work_mbc);

      /* Initialize work area.  */
      work_mbc = mbcsets_new ();
      lexer->mbcsets[lexer->nmbcsets++] = work_mbc;
    }

  ccl = charclass_alloc ();
  FETCH_WC (lexer, c, wc, _("unbalanced ["));
  if (c == '^')
    {
      FETCH_WC (lexer, c, wc, _("unbalanced ["));
      invert = true;
      known_bracket_exp = lexer->using_simple_locale;
    }
  else
    invert = false;

  colon_warning_state = (c == ':');
  do
    {
      c1 = FSATOKEN_NOTCHAR;	/* Mark c1 as not initialized.  */
      colon_warning_state &= ~2;

      /* Note that if we're looking at some other [:...:] construct,
         we just treat it as a bunch of ordinary characters.  We can do
         this because we assume regex has checked for syntax errors before
         dfa is ever called.  */
      if (c == '[')
        {
          FETCH_WC (lexer, c1, wc1, _("unbalanced ["));

          if ((c1 == ':' && (lexer->syntax_bits & RE_CHAR_CLASSES))
              || c1 == '.' || c1 == '=')
            {
              enum { MAX_BRACKET_STRING_LEN = 32 };
              char str[MAX_BRACKET_STRING_LEN + 1];
              size_t len = 0;
              for (;;)
                {
                  FETCH_WC (lexer, c, wc, _("unbalanced ["));
                  if ((c == c1 && *lexer->lexptr == ']')
                          || lexer->lexleft == 0)
                    break;
                  if (len < MAX_BRACKET_STRING_LEN)
                    str[len++] = c;
                  else
                    /* This is in any case an invalid class name.  */
                    str[0] = '\0';
                }
              str[len] = '\0';

              /* Fetch bracket.  */
              FETCH_WC (lexer, c, wc, _("unbalanced ["));
              if (c1 == ':')
                /* Find and merge named character class.  POSIX allows
                   character classes to match multicharacter collating
                   elements, but the regex code does not support that,
                   so do not worry about that possibility.  */
                {
                  char const *class;
                  predicate_entry_t *pred;

                  class = str;
                  if (lexer->case_fold && (STREQ (class, "upper")
                                      || STREQ (class, "lower")))
                    class = "alpha";
                  pred = find_pred (lexer, class);
                  if (! pred)
                    lexer->abandon_with_error (_("invalid character class"));
                  charclass_unionset (pred->class, ccl);

                  /* Does this class have a wide-char type descriptor? */
                  if (lexer->multibyte_locale && pred->may_be_multibyte)
                    {
                      mbcsets_add_class(work_mbc, pred->wchar_desc);
                    }
                }
              else
                known_bracket_exp = false;

              colon_warning_state |= 8;

              /* Fetch new lookahead character.  */
              FETCH_WC (lexer, c1, wc1, _("unbalanced ["));
              continue;
            }

          /* We treat '[' as a normal character here.  c/c1/wc/wc1
             are already set up.  */
        }

      if (c == '\\' && (lexer->syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
        FETCH_WC (lexer, c, wc, _("unbalanced ["));

      if (c1 == FSATOKEN_NOTCHAR)
        FETCH_WC (lexer, c1, wc1, _("unbalanced ["));

      if (c1 == '-')
        /* build range characters.  */
        {
          FETCH_WC (lexer, c2, wc2, _("unbalanced ["));

          /* A bracket expression like [a-[.aa.]] matches an unknown
             set.  Treat it like [-a[.aa.]] while parsing it, and
             remember that the set is unknown.  */
          if (c2 == '[' && *lexer->lexptr == '.')
            {
              known_bracket_exp = false;
              c2 = ']';
            }

          if (c2 != ']')
            {
              if (c2 == '\\' && (lexer->syntax_bits & RE_BACKSLASH_ESCAPE_IN_LISTS))
                FETCH_WC (lexer, c2, wc2, _("unbalanced ["));

              if (lexer->multibyte_locale)
                {
                  /* Merely record range as given; don't try to handle the
                     case-folding variants here for now.  */
                  if (wc != WEOF && wc2 != WEOF)
                    {
                      mbcsets_add_range(work_mbc, wc, wc2);

                      /* FIXME, part 2:  Since the canonical measure of case
                         equivalence is uppercase(A) == uppercase(B), perhaps
                         we could do this transformation above if lexer->case_fold
                         is true... but need to be careful not to throw away
                         information.  */
                    }
                }
              else if (lexer->using_simple_locale)
                {
                  charclass_setbit_range (c, c2, ccl);
                  if (lexer->case_fold)
                    {
                      int uc = toupper (c);
                      int uc2 = toupper (c2);
                      for (c1 = 0; c1 < FSATOKEN_NOTCHAR; c1++)
                        {
                          int uc1 = toupper (c1);
                          if (uc <= uc1 && uc1 <= uc2)
                            charclass_setbit (c1, ccl);
                        }
                    }
                }
              else
                known_bracket_exp = false;

              colon_warning_state |= 8;
              FETCH_WC (lexer, c1, wc1, _("unbalanced ["));
              continue;
            }

          /* In the case [x-], the - is an ordinary hyphen,
             which is left in c1, the lookahead character.  */
          lexer->lexptr  -= lexer->cur_mb_len;
          lexer->lexleft += lexer->cur_mb_len;
        }

      colon_warning_state |= (c == ':') ? 2 : 4;

      if (lexer->unibyte_locale)
        {
          if (lexer->case_fold)
            setbit_case_fold_c (c, ccl);
          else
            charclass_setbit (c, ccl);
          continue;
        }

      if (wc == WEOF)
        known_bracket_exp = false;
      else
        {
          wchar_t folded[CASE_FOLDED_BUFSIZE + 1];
          int n = 1;

          folded[0] = wc;
          if (lexer->case_fold)
            n += case_folded_counterparts (lexer, wc, &folded[1]);
          mbcsets_add_wchar_list (work_mbc, n, folded);
        }
    }
  while ((wc = wc1, (c = c1) != ']'));

  if (colon_warning_state == 7)
    lexer->warn_client (_("character class syntax is [[:space:]], not [:space:]"));

  if (! known_bracket_exp)
    return FSATOKEN_TK_BACKREF;

  if (lexer->multibyte_locale)
    {
      mbcsets_set_match_sense (work_mbc, invert);
      mbcsets_receive_incomplete_charclass (work_mbc, ccl);
      mbcsets_completed (work_mbc);
      return FSATOKEN_TK_MBCSET;
    }

  if (invert)
    {
      charclass_notset (ccl);
      if (lexer->syntax_bits & RE_HAT_LISTS_NOT_NEWLINE)
        charclass_clrbit (lexer->eolbyte, ccl);
    }

  return FSATOKEN_TK_CSET + charclass_completed (ccl);
}

fsatoken_token_t
fsalex_lex (fsalex_ctxt_t *lexer)
{
  int c;
  bool backslash = false;
  int i;
  predicate_entry_t *predicate;
  charclass_t *work_class;
  charclass_index_t class_index;
  mbcsets_set_t *work_mbc;
  bool invert;

  /* Ensure that syntax () has been called on this lexer instance; many
     things will fail if this isn't done.  */
  assert (lexer->syntax_initialised);

  /* Basic plan: We fetch a character.  If it's a backslash,
     we set the backslash flag and go through the loop again.
     On the plus side, this avoids having a duplicate of the
     main switch inside the backslash case.  On the minus side,
     it means that just about every case begins with
     "if (backslash) ...".  */
  for (i = 0; i < 2; ++i)
    {
      FETCH_WC (lexer, c, lexer->wctok, NULL);

      switch (c)
        {
        case '\\':
          if (backslash)
            goto normal_char;
          if (lexer->lexleft == 0)
            lexer->abandon_with_error (_("unfinished \\ escape"));
          backslash = true;
          break;

        case '^':
          if (backslash)
            goto normal_char;
          if (lexer->syntax_bits & RE_CONTEXT_INDEP_ANCHORS
              || lexer->lasttok == FSATOKEN_TK_END || lexer->lasttok == FSATOKEN_TK_LPAREN || lexer->lasttok == FSATOKEN_TK_OR)
            return lexer->lasttok = FSATOKEN_TK_BEGLINE;
          goto normal_char;

        case '$':
          if (backslash)
            goto normal_char;
          if (lexer->syntax_bits & RE_CONTEXT_INDEP_ANCHORS
              || lexer->lexleft == 0
              || (lexer->syntax_bits & RE_NO_BK_PARENS
                  ? lexer->lexleft > 0 && *lexer->lexptr == ')'
                  : lexer->lexleft > 1 && lexer->lexptr[0] == '\\' && lexer->lexptr[1] == ')')
              || (lexer->syntax_bits & RE_NO_BK_VBAR
                  ? lexer->lexleft > 0 && *lexer->lexptr == '|'
                  : lexer->lexleft > 1 && lexer->lexptr[0] == '\\' && lexer->lexptr[1] == '|')
              || ((lexer->syntax_bits & RE_NEWLINE_ALT)
                  && lexer->lexleft > 0 && *lexer->lexptr == '\n'))
            return lexer->lasttok = FSATOKEN_TK_ENDLINE;
          goto normal_char;

        case '1':
        case '2':
        case '3':
        case '4':
        case '5':
        case '6':
        case '7':
        case '8':
        case '9':
          if (backslash && !(lexer->syntax_bits & RE_NO_BK_REFS))
            {
              lexer->laststart = false;
              return lexer->lasttok = FSATOKEN_TK_BACKREF;
            }
          goto normal_char;

        case '`':
          if (backslash && lexer->re_gnu_ops)
            return lexer->lasttok = FSATOKEN_TK_BEGLINE; /* FIXME: should be beginning of string */
          goto normal_char;

        case '\'':
          if (backslash && lexer->re_gnu_ops)
            return lexer->lasttok = FSATOKEN_TK_ENDLINE;   /* FIXME: should be end of string */
          goto normal_char;

        case '<':
          if (backslash && lexer->re_gnu_ops)
            return lexer->lasttok = FSATOKEN_TK_BEGWORD;
          goto normal_char;

        case '>':
          if (backslash && lexer->re_gnu_ops)
            return lexer->lasttok = FSATOKEN_TK_ENDWORD;
          goto normal_char;

        case 'b':
          if (backslash && lexer->re_gnu_ops)
            return lexer->lasttok = FSATOKEN_TK_LIMWORD;
          goto normal_char;

        case 'B':
          if (backslash && lexer->re_gnu_ops)
            return lexer->lasttok = FSATOKEN_TK_NOTLIMWORD;
          goto normal_char;

        case '?':
          if (lexer->syntax_bits & RE_LIMITED_OPS)
            goto normal_char;
          if (backslash != ((lexer->syntax_bits & RE_BK_PLUS_QM) != 0))
            goto normal_char;
          if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
            goto normal_char;
          return lexer->lasttok = FSATOKEN_TK_QMARK;

        case '*':
          if (backslash)
            goto normal_char;
          if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
            goto normal_char;
          return lexer->lasttok = FSATOKEN_TK_STAR;

        case '+':
          if (lexer->syntax_bits & RE_LIMITED_OPS)
            goto normal_char;
          if (backslash != ((lexer->syntax_bits & RE_BK_PLUS_QM) != 0))
            goto normal_char;
          if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
            goto normal_char;
          return lexer->lasttok = FSATOKEN_TK_PLUS;

        case '{':
          if (!(lexer->syntax_bits & RE_INTERVALS))
            goto normal_char;
          if (backslash != ((lexer->syntax_bits & RE_NO_BK_BRACES) == 0))
            goto normal_char;
          if (!(lexer->syntax_bits & RE_CONTEXT_INDEP_OPS) && lexer->laststart)
            goto normal_char;

          /* Cases:
             {M} - exact count
             {M,} - minimum count, maximum is infinity
             {,N} - 0 through N
             {,} - 0 to infinity (same as '*')
             {M,N} - M through N */
          {
            char const *p = lexer->lexptr;
            char const *lim = p + lexer->lexleft;
            int minrep = -1;
            int maxrep = -1;
            for (; p != lim && ISASCIIDIGIT (*p); p++)
              {
                if (minrep < 0)
                  minrep = *p - '0';
                else
                  minrep = MIN (RE_DUP_MAX + 1, minrep * 10 + *p - '0');
              }
            if (p != lim)
              {
                if (*p != ',')
                  maxrep = minrep;
                else
                  {
                    if (minrep < 0)
                      minrep = 0;
                    while (++p != lim && ISASCIIDIGIT (*p))
                      {
                        if (maxrep < 0)
                          maxrep = *p - '0';
                        else
                          maxrep = MIN (RE_DUP_MAX + 1, maxrep * 10 + *p - '0');
                      }
                  }
              }
            if (! ((! backslash || (p != lim && *p++ == '\\'))
                   && p != lim && *p++ == '}'
                   && 0 <= minrep && (maxrep < 0 || minrep <= maxrep)))
              {
                if (lexer->syntax_bits & RE_INVALID_INTERVAL_ORD)
                  goto normal_char;
                lexer->abandon_with_error (_("invalid content of \\{\\}"));
              }
            if (RE_DUP_MAX < maxrep)
              lexer->abandon_with_error (_("regular expression too big"));
            lexer->lexptr = p;
            lexer->lexleft = lim - p;
            lexer->minrep = minrep;
            lexer->maxrep = maxrep;
          }
          lexer->laststart = false;
          return lexer->lasttok = FSATOKEN_TK_REPMN;

        case '|':
          if (lexer->syntax_bits & RE_LIMITED_OPS)
            goto normal_char;
          if (backslash != ((lexer->syntax_bits & RE_NO_BK_VBAR) == 0))
            goto normal_char;
          lexer->laststart = true;
          return lexer->lasttok = FSATOKEN_TK_OR;

        case '\n':
          if (lexer->syntax_bits & RE_LIMITED_OPS
              || backslash || !(lexer->syntax_bits & RE_NEWLINE_ALT))
            goto normal_char;
          lexer->laststart = true;
          return lexer->lasttok = FSATOKEN_TK_OR;

        case '(':
          if (backslash != ((lexer->syntax_bits & RE_NO_BK_PARENS) == 0))
            goto normal_char;
          ++lexer->parens;
          lexer->laststart = true;
          return lexer->lasttok = FSATOKEN_TK_LPAREN;

        case ')':
          if (backslash != ((lexer->syntax_bits & RE_NO_BK_PARENS) == 0))
            goto normal_char;
          if (lexer->parens == 0 && lexer->syntax_bits & RE_UNMATCHED_RIGHT_PAREN_ORD)
            goto normal_char;
          --lexer->parens;
          lexer->laststart = false;
          return lexer->lasttok = FSATOKEN_TK_RPAREN;

        case '.':
          if (backslash)
            goto normal_char;
          lexer->laststart = false;
          if (lexer->multibyte_locale)
            {
              /* In multibyte environment period must match with a single
                 character not a byte.  So we use FSATOKEN_TK_ANYCHAR.  */
              return lexer->lasttok = FSATOKEN_TK_ANYCHAR;
            }
          return lexer->lasttok = FSATOKEN_TK_CSET + lexer->dotclass_index;

        case 's':
        case 'S':
           /* "\s" == "[[:space:]]"; "\S" == "[^[:space:]]".  */
          if (! (backslash && lexer->re_gnu_ops))
            goto normal_char;
          lexer->laststart = false;
          invert = c == 'S';
          predicate = find_pred (lexer, "space");
          if (c == 's')
            class_index = charclass_get_index (predicate->class);
          else
            {
              /* Invert the predicate class in a work class.  */
              work_class = charclass_alloc ();
              charclass_copyset (predicate->class, work_class);
              charclass_notset (work_class);
              class_index = charclass_completed (work_class);
            }
          if (lexer->unibyte_locale)
            return lexer->lasttok = FSATOKEN_TK_CSET + class_index;

          /* FIXME: see if optimizing this, as is done with FSATOKEN_TK_ANYCHAR and
             add_utf8_anychar, makes sense.  */

          /* Multibyte locale:  Fill out an entire set description.  */
          lexer->mbcsets = maybe_realloc (lexer->mbcsets, lexer->nmbcsets,
                                          &lexer->mbcsets_alloc,
                                          sizeof work_mbc);
          work_mbc = mbcsets_new ();
          lexer->mbcsets[lexer->nmbcsets++] = work_mbc;
          mbcsets_set_match_sense (work_mbc, invert);
          mbcsets_add_class (work_mbc, predicate->wchar_desc);

          /* ?? REVIEWME: "invert" in parse_bracket_exp leads to FSATOKEN_TK_BACKREF
             (== "this is too hard for me") token via known_bracket_exp
             flag. */
          if ((c == 'S') && !lexer->using_simple_locale)
            return lexer->lasttok = FSATOKEN_TK_BACKREF;
          else
            return lexer->lasttok = FSATOKEN_TK_MBCSET;

         case 'w':
         case 'W':
           /* Can mean "[_[:alnum:]]" (\w) or its inverse (\W).  */
           if (! (backslash && lexer->re_gnu_ops))
             goto normal_char;
           lexer->laststart = false;
           predicate = find_pred (lexer, "alnum");
           work_class = charclass_alloc ();
           charclass_copyset (predicate->class, work_class);
           charclass_setbit ('_', work_class);
           if (c == 'W')
             charclass_notset (work_class);
           return lexer->lasttok = FSATOKEN_TK_CSET
                       + charclass_completed (work_class);

        case '[':
          if (backslash)
            goto normal_char;
          lexer->laststart = false;
          return lexer->lasttok = parse_bracket_exp (lexer);

        default:
        normal_char:
          lexer->laststart = false;
          /* For multibyte character sets, folding is done in atom, so
             always return FSATOKEN_TK_WCHAR.  */
          if (lexer->multibyte_locale)
            return lexer->lasttok = FSATOKEN_TK_WCHAR;

          if (lexer->case_fold && isalpha (c))
            {
              charclass_t *ccl = charclass_alloc ();
              setbit_case_fold_c (c, ccl);
              return lexer->lasttok = FSATOKEN_TK_CSET
                          + charclass_completed (ccl);
            }

          return lexer->lasttok = c;
        }
    }

  /* The above loop should consume at most a backslash
     and some other character.  */
  abort ();
  return FSATOKEN_TK_END;                   /* keeps pedantic compilers happy.  */
}

/* Receive the pattern, and reset the lexical analyser state.  The
   interpretation of the chars (octets?) in the pattern (ASCII chars?
   variable-length UTF-8 sequences?  Simplified Chinese?  etc.) depends on
   the locale that was in force when fsalex_syntax () was called.  NULs may
   be present amongst the codes, which is why the length is given
   explicitly, rather than relying on strlen(3).  */
void
fsalex_pattern (fsalex_ctxt_t *lexer,
                char const *pattern, size_t const pattern_len)

{
  /* Copy parameters to internal state variables.  */
  lexer->lexptr = pattern;
  lexer->lexleft = pattern_len;

  /* Reset lexical scanner state.  */
  lexer->lasttok = FSATOKEN_TK_END;
  lexer->laststart = 1;
  lexer->parens = 0;

  /* Reset multibyte parsing state.  */
  lexer->cur_mb_len = 1;
  memset (&lexer->mbrtowc_state, 0, sizeof (lexer->mbrtowc_state));
}

/* Report whether codes 0-127 conform to ASCII encoding.  This is handy
   for optimisation, particularly lower-case and upper-case characters,
   as the codes are contiguous, unlike EBCDIC.  ASCII is also a common
   denominator in many other unibyte code pages (locales), and also in
   the multibyte UTF-8 locale.  */
static bool _GL_ATTRIBUTE_PURE
char_encoding_7bits_is_ascii (void)
{
  /* True if the native character set is known to be compatible with
     the C locale.  The following test isn't perfect, but it's good
     enough in practice, as only ASCII and EBCDIC are in common use
     and this test correctly accepts ASCII and rejects EBCDIC.  */
  const bool is_ascii =
        '\b' ==   8 && '\t' ==   9 && '\n' ==  10 && '\v' ==  11
     && '\f' ==  12 && '\r' ==  13 && ' '  ==  32 && '!'  ==  33
     && '"'  ==  34 && '#'  ==  35 && '%'  ==  37 && '&'  ==  38
     && '\'' ==  39 && '('  ==  40 && ')'  ==  41 && '*'  ==  42
     && '+'  ==  43 && ','  ==  44 && '-'  ==  45 && '.'  ==  46
     && '/'  ==  47 && '0'  ==  48 && '9'  ==  57 && ':'  ==  58
     && ';'  ==  59 && '<'  ==  60 && '='  ==  61 && '>'  ==  62
     && '?'  ==  63 && 'A'  ==  65 && 'Z'  ==  90 && '['  ==  91
     && '\\' ==  92 && ']'  ==  93 && '^'  ==  94 && '_'  ==  95
     && 'a'  ==  97 && 'z'  == 122 && '{'  == 123 && '|'  == 124
     && '}'  == 125 && '~'  == 126;
  return is_ascii;
}

/* Receive syntax directives, and other pattern interpretation
   instructions such as case folding and end-of-line character.
   In addition, this function configures various internal structures
   based on the locale in force.  */
void
fsalex_syntax (fsalex_ctxt_t *lexer,
               reg_syntax_t bits, int fold, unsigned char eol)
{
  charclass_t *work_class;
  predicate_entry_t *pred;
  char const *locale_name;

  /* Set a flag noting that this lexer has had its syntax params
     set.  */
  lexer->syntax_initialised = true;

  /* Record the function parameters in our local context.  */
  lexer->syntax_bits = bits;
  lexer->case_fold = fold;
  lexer->eolbyte = eol;

  /* Set up unibyte/multibyte flags, based on MB_CUR_MAX, which
     depends on the current locale.  We capture this information here
     as the locale may change later.  At present, we don't capture
     MB_CUR_MAX itself.  */
  if (MB_CUR_MAX > 1)
    {
      /* Multibyte locale: Prepare booleans to make code easier to
         read */
      lexer->unibyte_locale = false;
      lexer->multibyte_locale = true;

      /* Discard any earlier storage we may have acquired.  */
      free (lexer->mbcsets);
      lexer->mbcsets = NULL;

      /* Set up an array of structures to hold multibyte character
         sets.  */
      lexer->nmbcsets = 0;
      lexer->mbcsets_alloc = 2;
      lexer->mbcsets = xzalloc (sizeof (*lexer->mbcsets)
                                      * lexer->mbcsets_alloc);
    }
    else
    {
      /* Unibyte locale: Prepare booleans to make code easier to
         read */
      lexer->unibyte_locale = true;
      lexer->multibyte_locale = false;
    }

  /* Charclass guarantees that class index 0 is zeroclass, so we don't
     need to set it up here.  */

 /* Set up a character class to match anychar ('.'), tailored to
    accommodate options from the regex syntax.  */
  work_class = charclass_alloc ();
  charclass_notset (work_class);
  if (! (lexer->syntax_bits & RE_DOT_NEWLINE))
    {
      charclass_clrbit (lexer->eolbyte, work_class);
    }
  if (lexer->syntax_bits & RE_DOT_NOT_NULL)
    {
      charclass_clrbit (0, work_class);
    }
  lexer->dotclass_index = charclass_completed (work_class);
  lexer->dotclass = charclass_get_pointer (lexer->dotclass_index);

  /* Testing for the absence of RE_NO_GNU_OPS in syntax_bits happens
     often, so set a direct flag variable:  This makes code more
     readable.  */
  lexer->re_gnu_ops = ! (lexer->syntax_bits & RE_NO_GNU_OPS);

  /* Initialise cache and other tables that have syntax and/or locale
     influences.  */

  /* Set up the wchar_desc fields of the predicate table.  */
  for (pred = lexer->predicates; pred->name != NULL; pred++)
    pred->wchar_desc = wctype (pred->name);

  /* Record the name of the current locale.  Note that setlocale can
     return NULL for the name.  */
  lexer->locale_name = NULL;
  locale_name = setlocale (LC_ALL, NULL);
  if (locale_name != NULL)
    lexer->locale_name = xstrdup (locale_name);

  /* Determine if the locale's treatment of codes 0-127 matches ASCII,
     as some operations (e.g. "[c-v]") become easier to handle.  */
  lexer->ascii_7bit_encoding = char_encoding_7bits_is_ascii ();

  /* Further optimisations are possible if the locale is an unibyte
     ASCII 7-bit C/POSIX environment, and this is a common case, so
     spend some effort looking for this situation.  */
  lexer->using_simple_locale = false;
  if (lexer->unibyte_locale && lexer->ascii_7bit_encoding)
    {
      lexer->using_simple_locale = ! lexer->locale_name
                                   || STREQ (lexer->locale_name, "C")
                                   || STREQ (lexer->locale_name, "POSIX");
    }

  /* Initialise cache that distinguishes between unibyte and multibyte
     (wide) characters based on the leading octet.  */
  initialise_uchar_to_wc_cache (lexer);

  /* ?? If a complete copy of the locale needs to be stored within
     this instance, see  uselocale(3) and duplocale(3) (and free the
     copy with freelocale(3)).  */

}

/* Define external function to do non-core data exchanges between the lexer
   and the parser.  This function must conform to
   proto_lexparse_exchange_fn_t.  This interface lets two instances
   communicate without requiring help from an outside party.  */
int
fsalex_exchange (fsalex_ctxt_t *lexer,
                 proto_lexparse_opcode_t opcode,
                 void *param)
{
  switch (opcode)
    {
    case PROTO_LEXPARSE_OP_GET_IS_MULTIBYTE_LOCALE:
      return (int) lexer->multibyte_locale;
    case PROTO_LEXPARSE_OP_GET_REPMN_MIN:
      return lexer->minrep;
    case PROTO_LEXPARSE_OP_GET_REPMN_MAX:
      return lexer->maxrep;
    case PROTO_LEXPARSE_OP_GET_WIDE_CHAR_LIST_MAX:
      /* Original token, plus upper/lowercase, plus extended variants.  */
      return 1 + CASE_FOLDED_BUFSIZE;
    case PROTO_LEXPARSE_OP_GET_WIDE_CHARS:
      {
        wchar_t *p_wc = (wchar_t *) param;

        /* Record the original token (wctok) implicitly associated with
           WCHAR.  */
        *p_wc++ = lexer->wctok;

        /* Add in direct and/or indirect case equivalents, as appropriate.  */
        if (lexer->case_fold)
          p_wc += case_folded_counterparts (lexer, lexer->wctok, p_wc);

        /* Return how many wide chars (at least 1) are in the list.  */
        return p_wc - (wchar_t *) param;
      }
      break;
    case PROTO_LEXPARSE_OP_GET_DOTCLASS:
      *((charclass_t **) param) = lexer->dotclass;
      break;
    case PROTO_LEXPARSE_OP_GET_MBCSET:
      /* Return the set associated with the most recent MBCSET token.  */
      *((mbcsets_set_t **) param) = lexer->mbcsets[lexer->nmbcsets - 1];
      break;
    default:
      /* ?? Not sure if we should complain/assert or merely ignore an
         opcode that we don't recognise here.  */
      assert (!"unrecognised PROTO_LEXPARSE opcode");
      /* NOTREACHED */
      break;
    }

    /* If we reach here, return value is unimportant, so just say 0.  */
    return 0;
}

/* Receive functions to deal with exceptions detected by the lexer:
   Warnings and errors.  Internally, we add the _Noreturn attribute
   to the error callback, to help the compiler with code flow
   analysis.  */
void
fsalex_exception_fns (fsalex_ctxt_t *lexer,
                      fsalex_warn_callback_fn *warningfn,
                      fsalex_error_callback_fn *errorfn)
{
  /* Record the provided functions in the lexer's context.  */
  lexer->warn_client        = warningfn;
  lexer->abandon_with_error = errorfn;
}

/* Add "not provided!" stub function that gets called if the client
   fails to provide proper resources.  This is a hack, merely to get
   the module started; better treatment needs to be added later.  */
static void
no_function_provided (void *unused)
{
 assert (!"fsalex: Plug-in function required, but not provided.");
}

/* Generate a new instance of an FSA lexer.  */
fsalex_ctxt_t *
fsalex_new (void)
{
  fsalex_ctxt_t *new_context;

  /* Acquire zeroed memory for new lexer context.  */
  new_context = XZALLOC (fsalex_ctxt_t);

  /* ?? Point warning and error functions to a "you need to tell me
     these first!" function? */
  new_context->warn_client        = (fsalex_warn_callback_fn *)
                                    no_function_provided;
  new_context->abandon_with_error = (fsalex_error_callback_fn *)
                                    no_function_provided;

  /* Default to working in a non-multibyte locale.  In some cases,
     FETCH_WC never sets this variable (as it's assumed to be 1), so
     fulfil this expectation here.  */
  new_context->cur_mb_len = 1;

  /* Copy the template predicate list into this context, so that we
     can have lexer-specific named predicate classes.  */
  memcpy (new_context->predicates, template_predicate_list,
         sizeof (new_context->predicates));

  /* Default to unibyte locale at first; the final locale setting is
     made according to what's in force when fsalex_syntax () is
     called.  */
  new_context->unibyte_locale = true;
  new_context->multibyte_locale = false;

  /* Many things depend on decisions made in fsalex_syntax (), so note
     here that it hasn't been called yet, and fail gracefully later if
     the client hasn't called the function before commencing work.  */
  new_context->syntax_initialised = false;

  /* Add this instance at the head of the module's list.  */
  new_context->next_instance = fsalex_instance_list_head;
  fsalex_instance_list_head = new_context;

  return new_context;
}

/* Internal function to free all resources directly or indirectly used
   by an instance.  The pointer is no longer valid after this call.  */
static void
free_instance (fsalex_ctxt_t *lexer)
{
  /* Free top-level structures.  */
  free (lexer->locale_name);
  free (lexer->mbcsets);
  free (lexer);

  /* Note that multibyte sets are no longer stored explicitly in
     the instance; apart from the pointer list (freed above), the
     set contents are freed separately when mbcsets module is
     destroyed.  */
}

/* Destroy all lexer instances, plus any associated resources owned by the
   module.  */
void
fsalex_destroy_module (void)
{
  fsalex_ctxt_t *p_list;
  fsalex_ctxt_t *p_next;

  /* Move the global list head into a local variable, and immediately
     clear the global.  This is a half-hearted attempt to avoid race
     conditions; to do things properly, a system-wide atomic
     operation (locked, including multi-CPU cache coherency) operation
     should be used.  */
  p_list = fsalex_instance_list_head;
  fsalex_instance_list_head = NULL;

  /* Traverse the list of instances, releasing all resources
     associated with each one.  */
  while (p_list)
    {
      p_next = p_list->next_instance;
      free_instance (p_list);
      p_list = p_next;
    }
}

/* Prepare module for operation.  */
void
fsalex_initialise (void)
{
  /* Initialise the linked list of instances created by this module.  */
  fsalex_instance_list_head = NULL;

  atexit (fsalex_destroy_module);
}

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="fsalex.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsalex.h"

/* fsalex - Repackage pattern text as compact, expressive tokens

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */


#ifndef FSALEX_H
#define FSALEX_H 1

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include "fsatoken.h"
#include "proto-lexparse.h"
#include <regex.h>

/* Multiple lexer instances can exist in parallel, so define an
   opaque type to collect together all the context relating to each
   instance.  */
typedef struct fsalex_ctxt_struct fsalex_ctxt_t;

/* Prepare module for operation.  */
extern void
fsalex_initialise (void);

/* Destroy all lexer instances, plus any associated resources owned by the
   module.  */
extern void
fsalex_destroy_module (void);

/* Generate a new instance of an FSA lexer.  */
extern fsalex_ctxt_t *
fsalex_new (void);

/* ?? Reserve "fsalex_discard ()" for getting rid of a lexer
   instance.  */

/* Receive syntax directives, and other pattern interpretation
   instructions such as case folding and end-of-line character.
   In addition, this function configures various internal structures
   based on the locale in force.  */
extern void
fsalex_syntax (fsalex_ctxt_t *lexer,
               reg_syntax_t bits, int fold, unsigned char eol);

/* Receive the pattern, and reset the lexical analyser state.  The
   interpretation of the chars (octets?) in the pattern (ASCII chars?
   variable-length UTF-8 sequences?  Simplified Chinese?  etc.) depends on
   the locale that was in force when fsalex_syntax () was called.  NULs may
   be present amongst the codes, which is why the length is given
   explicitly, rather than relying on strlen(3).  */
extern void
fsalex_pattern (fsalex_ctxt_t *lexer,
                char const *pattern, size_t const pattern_len);

/* Define function prototypes for warning and error callbacks.  */
typedef void
fsalex_warn_callback_fn (const char *);
typedef void /* ?? _Noreturn? */
fsalex_error_callback_fn (const char *);

/* Receive functions to deal with exceptions detected by the lexer:
   Warnings and errors.  Internally, we add the _Noreturn attribute
   to the error callback, to help the compiler with code flow
   analysis.  */
extern void
fsalex_exception_fns (fsalex_ctxt_t *lexer,
                      fsalex_warn_callback_fn *warningfn,
                      fsalex_error_callback_fn *errorfn);

/* Main function to incrementally consume and interpret the pattern
   text, and return a token describing a single lexical element as a
   token, perhaps with implied parameters such as character classes
   for CSET tokens, and {min,max} values for each REPMN token.  The
   user should call this function repeatedly, receiving one token each
   time, until the lexer detects a fatal error, or returns the END
   token.  */
/* This function must conform to proto_lexparse_lex_fn_t.  */
extern fsatoken_token_t
fsalex_lex (fsalex_ctxt_t *lexer);

/* Define external function to do non-core data exchanges between the lexer
   and the parser.  This function must conform to
   proto_lexparse_exchange_fn_t.  This interface lets two instances
   communicate without requiring help from an outside party.  */
extern int
fsalex_exchange (fsalex_ctxt_t *lexer,
                 proto_lexparse_opcode_t opcode,
                 void *param);


#endif /* FSALEX_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="fsamusts.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsamusts.c"

/* fsamusts -- Report a list of must-have simple strings in the pattern

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* (?? Long description/discussion goes here...) */

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include <assert.h>
#include "charclass.h"
#include <ctype.h>
#include "fsamusts.h"
#include "fsatoken.h"
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include "xalloc.h"

#if DEBUG
#include <stdio.h>
#endif /* DEBUG */

#define STREQ(a, b) (strcmp (a, b) == 0)

/* Having found the postfix representation of the regular expression,
   try to find a long sequence of characters that must appear in any line
   containing the r.e.
   Finding a "longest" sequence is beyond the scope here;
   we take an easy way out and hope for the best.
   (Take "(ab|a)b"--please.)

   We do a bottom-up calculation of sequences of characters that must appear
   in matches of r.e.'s represented by trees rooted at the nodes of the postfix
   representation:
        sequences that must appear at the left of the match ("left")
        sequences that must appear at the right of the match ("right")
        lists of sequences that must appear somewhere in the match ("in")
        sequences that must constitute the match ("is")

   When we get to the root of the tree, we use one of the longest of its
   calculated "in" sequences as our answer.  The sequence we find is returned in
   d->must (where "d" is the single argument passed to "dfamust");
   the length of the sequence is returned in d->mustn.

   The sequences calculated for the various types of node (in pseudo ANSI c)
   are shown below.  "p" is the operand of unary operators (and the left-hand
   operand of binary operators); "q" is the right-hand operand of binary
   operators.

   "ZERO" means "a zero-length sequence" below.

        Type	left		right		is		in
        ----	----		-----		--		--
        char c	# c		# c		# c		# c

        ANYCHAR	ZERO		ZERO		ZERO		ZERO

        MBCSET	ZERO		ZERO		ZERO		ZERO

        CSET	ZERO		ZERO		ZERO		ZERO

        STAR	ZERO		ZERO		ZERO		ZERO

        QMARK	ZERO		ZERO		ZERO		ZERO

        PLUS	p->left		p->right	ZERO		p->in

        CAT	(p->is==ZERO)?	(q->is==ZERO)?	(p->is!=ZERO &&	p->in plus
                p->left :	q->right :	q->is!=ZERO) ?	q->in plus
                p->is##q->left	p->right##q->is	p->is##q->is :	p->right##q->left
                                                ZERO

        OR	longest common	longest common	(do p->is and	substrings common to
                leading		trailing	q->is have same	p->in and q->in
                (sub)sequence	(sub)sequence	length and
                of p->left	of p->right	content) ?
                and q->left	and q->right	p->is : NULL

   If there's anything else we recognize in the tree, all four sequences get set
   to zero-length sequences.  If there's something we don't recognize in the
   tree, we just return a zero-length sequence.

   Break ties in favor of infrequent letters (choosing 'zzz' in preference to
   'aaa')?

   And ... is it here or someplace that we might ponder "optimizations" such as
        egrep 'psi|epsilon'	->	egrep 'psi'
        egrep 'pepsi|epsilon'	->	egrep 'epsi'
                                        (Yes, we now find "epsi" as a "string
                                        that must occur", but we might also
                                        simplify the *entire* r.e. being sought)
        grep '[c]'		->	grep 'c'
        grep '(ab|a)b'		->	grep 'ab'
        grep 'ab*'		->	grep 'a'
        grep 'a*b'		->	grep 'b'

   There are several issues:

   Is optimization easy (enough)?

   Does optimization actually accomplish anything,
   or is the automaton you get from "psi|epsilon" (for example)
   the same as the one you get from "psi" (for example)?

   Are optimizable r.e.'s likely to be used in real-life situations
   (something like 'ab*' is probably unlikely; something like is
   'psi|epsilon' is likelier)?  */

static char *
icatalloc (char *old, char const *new)
{
  char *result;
  size_t oldsize;
  size_t newsize = strlen (new);
  if (newsize == 0)
    return old;
  oldsize = strlen (old);
  result = xrealloc (old, oldsize + newsize + 1);
  memcpy (result + oldsize, new, newsize + 1);
  return result;
}

static char *_GL_ATTRIBUTE_PURE
istrstr (char const *lookin, char const *lookfor)
{
  char const *cp;
  size_t len;

  len = strlen (lookfor);
  for (cp = lookin; *cp != '\0'; ++cp)
    if (strncmp (cp, lookfor, len) == 0)
      return (char *) cp;
  return NULL;
}

static void
freelist (char **cpp)
{
  while (*cpp)
    free (*cpp++);
}

static char **
enlist (char **cpp, char *new, size_t len)
{
  size_t i, j;
  new = memcpy (xmalloc (len + 1), new, len);
  new[len] = '\0';
  /* Is there already something in the list that's new (or longer)?  */
  for (i = 0; cpp[i] != NULL; ++i)
    if (istrstr (cpp[i], new) != NULL)
      {
        free (new);
        return cpp;
      }
  /* Eliminate any obsoleted strings.  */
  j = 0;
  while (cpp[j] != NULL)
    if (istrstr (new, cpp[j]) == NULL)
      ++j;
    else
      {
        free (cpp[j]);
        if (--i == j)
          break;
        cpp[j] = cpp[i];
        cpp[i] = NULL;
      }
  /* Add the new string.  */
  cpp = xnrealloc (cpp, i + 2, sizeof *cpp);
  cpp[i] = new;
  cpp[i + 1] = NULL;
  return cpp;
}

/* Given pointers to two strings, return a pointer to an allocated
   list of their distinct common substrings.  */
static char **
comsubs (char *left, char const *right)
{
  char **cpp = xzalloc (sizeof *cpp);
  char *lcp;

  for (lcp = left; *lcp != '\0'; ++lcp)
    {
      size_t len = 0;
      char *rcp = strchr (right, *lcp);
      while (rcp != NULL)
        {
          size_t i;
          for (i = 1; lcp[i] != '\0' && lcp[i] == rcp[i]; ++i)
            continue;
          if (i > len)
            len = i;
          rcp = strchr (rcp + 1, *lcp);
        }
      if (len != 0)
        cpp = enlist (cpp, lcp, len);
    }
  return cpp;
}

static char **
addlists (char **old, char **new)
{
  for (; *new; new++)
    old = enlist (old, *new, strlen (*new));
  return old;
}

/* Given two lists of substrings, return a new list giving substrings
   common to both.  */
static char **
inboth (char **left, char **right)
{
  char **both = xzalloc (sizeof *both);
  size_t lnum, rnum;

  for (lnum = 0; left[lnum] != NULL; ++lnum)
    {
      for (rnum = 0; right[rnum] != NULL; ++rnum)
        {
          char **temp = comsubs (left[lnum], right[rnum]);
          both = addlists (both, temp);
          freelist (temp);
          free (temp);
        }
    }
  return both;
}

typedef struct must must;

struct must
{
  char **in;
  char *left;
  char *right;
  char *is;
  bool begline;
  bool endline;
  must *prev;
};

static must *
allocmust (must *mp)
{
  must *new_mp = xmalloc (sizeof *new_mp);
  new_mp->in = xzalloc (sizeof *new_mp->in);
  new_mp->left = xzalloc (2);
  new_mp->right = xzalloc (2);
  new_mp->is = xzalloc (2);
  new_mp->begline = false;
  new_mp->endline = false;
  new_mp->prev = mp;
  return new_mp;
}

static void
resetmust (must *mp)
{
  freelist (mp->in);
  mp->in[0] = NULL;
  mp->left[0] = mp->right[0] = mp->is[0] = '\0';
  mp->begline = false;
  mp->endline = false;
}

static void
freemust (must *mp)
{
  freelist (mp->in);
  free (mp->in);
  free (mp->left);
  free (mp->right);
  free (mp->is);
  free (mp);
}

/* Receive an existing list (possibly empty) of must-have strings, together
   with a list of the tokens for the current FSA (postfix tree order), and
   if there are any more must-have strings in the token list, add them to
   the must-have list.  Returns the possibly-modified list to the caller.
   Locale and syntax items are partially covered here by the case_fold and
   unibyte_locale flags, but this is incomplete, and should be addressed by
   Stage 2 (improving the expressiveness of tokens).  */
fsamusts_list_element_t *
fsamusts_must (fsamusts_list_element_t *must_list,
              size_t nr_tokens, fsatoken_token_t *token_list,
              bool case_fold, bool unibyte_locale)
{
  must *mp = NULL;
  char const *result = "";
  size_t ri;
  size_t i;
  bool exact = false;
  bool begline = false;
  bool endline = false;

  for (ri = 0; ri < nr_tokens; ++ri)
    {
      fsatoken_token_t t = token_list[ri];
      switch (t)
        {
        case FSATOKEN_TK_BEGLINE:
          mp = allocmust (mp);
          mp->begline = true;
          break;
        case FSATOKEN_TK_ENDLINE:
          mp = allocmust (mp);
          mp->endline = true;
          break;
        case FSATOKEN_TK_LPAREN:
        case FSATOKEN_TK_RPAREN:
          assert (!"neither FSATOKEN_TK_LPAREN nor FSATOKEN_TK_RPAREN may appear here");

        case FSATOKEN_TK_EMPTY:
        case FSATOKEN_TK_BEGWORD:
        case FSATOKEN_TK_ENDWORD:
        case FSATOKEN_TK_LIMWORD:
        case FSATOKEN_TK_NOTLIMWORD:
        case FSATOKEN_TK_BACKREF:
        case FSATOKEN_TK_ANYCHAR:
        case FSATOKEN_TK_MBCSET:
          mp = allocmust (mp);
          break;

        case FSATOKEN_TK_STAR:
        case FSATOKEN_TK_QMARK:
          resetmust (mp);
          break;

        case FSATOKEN_TK_OR:
          {
            char **new;
            must *rmp = mp;
            must *lmp = mp = mp->prev;
            size_t j, ln, rn, n;

            /* Guaranteed to be.  Unlikely, but ...  */
            if (STREQ (lmp->is, rmp->is))
              {
                lmp->begline &= rmp->begline;
                lmp->endline &= rmp->endline;
              }
            else
              {
                lmp->is[0] = '\0';
                lmp->begline = false;
                lmp->endline = false;
              }
            /* Left side--easy */
            i = 0;
            while (lmp->left[i] != '\0' && lmp->left[i] == rmp->left[i])
              ++i;
            lmp->left[i] = '\0';
            /* Right side */
            ln = strlen (lmp->right);
            rn = strlen (rmp->right);
            n = ln;
            if (n > rn)
              n = rn;
            for (i = 0; i < n; ++i)
              if (lmp->right[ln - i - 1] != rmp->right[rn - i - 1])
                break;
            for (j = 0; j < i; ++j)
              lmp->right[j] = lmp->right[(ln - i) + j];
            lmp->right[j] = '\0';
            new = inboth (lmp->in, rmp->in);
            freelist (lmp->in);
            free (lmp->in);
            lmp->in = new;
            freemust (rmp);
          }
          break;

        case FSATOKEN_TK_PLUS:
          mp->is[0] = '\0';
          break;

        case FSATOKEN_TK_END:
          assert (!mp->prev);
          for (i = 0; mp->in[i] != NULL; ++i)
            if (strlen (mp->in[i]) > strlen (result))
              result = mp->in[i];
          if (STREQ (result, mp->is))
            {
              exact = true;
              begline = mp->begline;
              endline = mp->endline;
            }
          goto done;

        case FSATOKEN_TK_CAT:
          {
            must *rmp = mp;
            must *lmp = mp = mp->prev;

            /* In.  Everything in left, plus everything in
               right, plus concatenation of
               left's right and right's left.  */
            lmp->in = addlists (lmp->in, rmp->in);
            if (lmp->right[0] != '\0' && rmp->left[0] != '\0')
              {
                size_t lrlen = strlen (lmp->right);
                size_t rllen = strlen (rmp->left);
                char *tp = xmalloc (lrlen + rllen);
                memcpy (tp, lmp->right, lrlen);
                memcpy (tp + lrlen, rmp->left, rllen);
                lmp->in = enlist (lmp->in, tp, lrlen + rllen);
                free (tp);
              }
            /* Left-hand */
            if (lmp->is[0] != '\0')
              lmp->left = icatalloc (lmp->left, rmp->left);
            /* Right-hand */
            if (rmp->is[0] == '\0')
              lmp->right[0] = '\0';
            lmp->right = icatalloc (lmp->right, rmp->right);
            /* Guaranteed to be */
            if ((lmp->is[0] != '\0' || lmp->begline)
                && (rmp->is[0] != '\0' || rmp->endline))
              {
                lmp->is = icatalloc (lmp->is, rmp->is);
                lmp->endline = rmp->endline;
              }
            else
              {
                lmp->is[0] = '\0';
                lmp->begline = false;
                lmp->endline = false;
              }
            freemust (rmp);
          }
          break;

        case '\0':
          /* Not on *my* shift.  */
          goto done;

        default:
          mp = allocmust (mp);
          if (FSATOKEN_TK_CSET <= t)
            {
              /* If T is a singleton, or if case-folding in a unibyte
                 locale and T's members all case-fold to the same char,
                 convert T to one of its members.  Otherwise, do
                 nothing further with T.  */
              charclass_t *ccl = charclass_get_pointer (t - FSATOKEN_TK_CSET);
              int j;
              for (j = 0; j < FSATOKEN_NOTCHAR; j++)
                if (charclass_tstbit (j, ccl))
                  break;
              if (! (j < FSATOKEN_NOTCHAR))
                break;
              t = j;
              while (++j < FSATOKEN_NOTCHAR)
                if (charclass_tstbit (j, ccl)
                    && ! (case_fold && unibyte_locale
                          && toupper (j) == toupper (t)))
                  break;
              if (j < FSATOKEN_NOTCHAR)
                break;
            }
          mp->is[0] = mp->left[0] = mp->right[0]
            = case_fold && unibyte_locale ? toupper (t) : t;
          mp->is[1] = mp->left[1] = mp->right[1] = '\0';
          mp->in = enlist (mp->in, mp->is, 1);
          break;
        }
    }
done:
  if (strlen (result))
    {
      fsamusts_list_element_t *dm;

      dm = xmalloc (sizeof *dm);
      dm->exact = exact;
      dm->begline = begline;
      dm->endline = endline;
      dm->must = xstrdup (result);
      dm->next = must_list;
      must_list = dm;
    }

  while (mp)
    {
      must *prev = mp->prev;
      freemust (mp);
      mp = prev;
    }

  return must_list;
}

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="fsamusts.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsamusts.h"

/* fsamusts -- Report a list of must-have simple strings in the pattern

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* ?? Insert long description/discussion here.  */

#ifndef FSAMUSTS_H
#define FSAMUSTS_H 1

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include "fsatoken.h"

/* Element of a list of strings, at least one of which is known to
   appear in any R.E. matching the DFA. */
typedef struct fsamusts_list_element
{
  bool exact;
  bool begline;
  bool endline;
  char *must;
  struct fsamusts_list_element *next;
} fsamusts_list_element_t;

/* Receive an existing list (possibly empty) of must-have strings, together
   with a list of the tokens for the current FSA (postfix tree order), and
   if there are any more must-have strings in the token list, add them to
   the must-have list.  Returns the possibly-modified list to the caller.
   Locale and syntax items are partially covered here by the case_fold and
   unibyte_locale flags, but this is incomplete, and should be addressed by
   Stage 2 (improving the expressiveness of tokens).  */
extern fsamusts_list_element_t *
fsamusts_must (fsamusts_list_element_t *must_list,
              size_t nr_tokens, fsatoken_token_t *token_list,
              bool case_fold, bool unibyte_locale);

#endif /* FSAMUSTS_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="fsaparse.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsaparse.c"

/* fsaparse -- Build a structure naming relationships (sequences,
               alternatives, backreferences, options and precedence)
               of tokens

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* This function receives a stream of tokens from fsalex, and processes
   them to impose precedence rules and to describe complex pattern
   elements that are beyond the capability of the simple lexer.  In
   addition to the cases explicit in the syntax (e.g."(ab|c)",
   variable-length multibyte encodings (UTF-8; codesets including
   modifiers and/or shift items) also require these enhanced
   facilities.  */

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include <assert.h>
#include "charclass.h"
#include "fsaparse.h"
#include "fsalex.h"
#include "fsatoken.h"
#include "mbcsets.h"
#include "proto-lexparse.h"
#include <stdbool.h>
#include <stdlib.h>
#include "xalloc.h"

/* gettext.h ensures that we don't use gettext if ENABLE_NLS is not defined */
#include "gettext.h"
#define _(str) gettext (str)

#include <wchar.h>
#include <wctype.h>

#if HAVE_LANGINFO_CODESET
# include <langinfo.h>
#endif

/* fsaparse_ctxt: Gather all the context to do with the parser into a
   single struct.  We do this mainly because it make it easier to
   contemplate having multiple instances of this module running in
   parallel, but also because it makes translating from "dfa->" easier.
   This definition fleshes out the opaque type given in the module
   header.  */
struct fsaparse_ctxt_struct
{
  /* Singly-linked list of all parser instances, so destroy_module can
     release all resources by traversing the list.  */
  fsaparse_ctxt_t *next_instance;

  /* Warning and abort functions provided by client.  */
  fsalex_warn_callback_fn *warn_client;
  fsalex_error_callback_fn *abandon_with_error;

  /* Plug-in functions and context to deal with lexer at arm's
     length.  */
  proto_lexparse_lex_fn_t *lexer;
  proto_lexparse_exchange_fn_t *lex_exchange;
  void *lex_context;

  /* Information about locale (needs to sync with lexer...?) */
  bool multibyte_locale;
  bool unibyte_locale;

  /* Variable to store a dynamically-allocated list of wide characters.
     The parser needs to fetch this list whenever a WCHAR token is
     received.  */
  int wide_chars_max;
  wchar_t *wide_char_list;

  fsatoken_token_t lookahead_token;

  size_t current_depth;      /* Current depth of a hypothetical stack
                             holding deferred productions.  This is
                             used to determine the depth that will be
                             required of the real stack later on in
                             dfaanalyze.  */
  /* Fields filled by the parser.  */
  fsatoken_token_t *tokens;                /* Postfix parse array.  */
  size_t tindex;                /* Index for adding new tokens.  */
  size_t talloc;                /* Number of tokens currently allocated.  */
  size_t depth;                 /* Depth required of an evaluation stack
                                   used for depth-first traversal of the
                                   parse tree.  */
  size_t nleaves;               /* Number of leaves on the parse tree.  */
  size_t nregexps;              /* Count of parallel regexps being built
                                   with dfaparse.  */
  bool fast;			/* The DFA is fast.  */
  bool multibyte;		/* MB_CUR_MAX > 1.  */
  fsatoken_token_t utf8_anychar_classes[5]; /* To lower ANYCHAR in UTF-8 locales.  */
  mbstate_t mbs;		/* Multibyte conversion state.  */

  /* The following are valid only if MB_CUR_MAX > 1.  */

  /* The value of multibyte_prop[i] is defined by following rule.
     if tokens[i] < NOTCHAR
     bit 0 : tokens[i] is the first byte of a character, including
     single-byte characters.
     bit 1 : tokens[i] is the last byte of a character, including
     single-byte characters.

     if tokens[i] = MBCSET
     ("the index of mbcsets corresponding to this operator" << 2) + 3

     e.g.
     tokens
     = 'single_byte_a', 'multi_byte_A', single_byte_b'
     = 'sb_a', 'mb_A(1st byte)', 'mb_A(2nd byte)', 'mb_A(3rd byte)', 'sb_b'
     multibyte_prop
     = 3     , 1               ,  0              ,  2              , 3
   */
  int *multibyte_prop;

  mbcsets_set_t **mbcsets;
  size_t nmbcsets;
  size_t mbcsets_alloc;
};

/* Linked list of all instances created by this module.  */
static fsaparse_ctxt_t *fsaparse_instance_list_head = NULL;

/* Ensure that the array addressed by PTR holds at least NITEMS +
   (PTR || !NITEMS) items.  Either return PTR, or reallocate the array
   and return its new address.  Although PTR may be null, the returned
   value is never null.

   The array holds *NALLOC items; *NALLOC is updated on reallocation.
   ITEMSIZE is the size of one item.  Avoid O(N**2) behavior on arrays
   growing linearly.  */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
  if (nitems < *nalloc)
    return ptr;
  *nalloc = nitems;
  return x2nrealloc (ptr, nalloc, itemsize);
}

/* UTF-8 encoding allows some optimizations that we can't otherwise
   assume in a multibyte encoding.  */
static int
using_utf8 (void)
{
  static int utf8 = -1;
  if (utf8 < 0)
    {
      wchar_t wc;
      mbstate_t mbs = { 0 };
      utf8 = mbrtowc (&wc, "\xc4\x80", 2, &mbs) == 2 && wc == 0x100;
    }
  return utf8;
}

/* Recursive descent parser for regular expressions.  */

static void
addtok_mb (fsaparse_ctxt_t *parser, fsatoken_token_t t, int mbprop)
{
  if (parser->talloc == parser->tindex)
    {
      parser->tokens = x2nrealloc (parser->tokens, &parser->talloc,
                                sizeof *parser->tokens);
      if (parser->multibyte)
        parser->multibyte_prop = xnrealloc (parser->multibyte_prop, parser->talloc,
                                         sizeof *parser->multibyte_prop);
    }
  if (parser->multibyte)
    parser->multibyte_prop[parser->tindex] = mbprop;
  parser->tokens[parser->tindex++] = t;

  switch (t)
    {
    case FSATOKEN_TK_QMARK:
    case FSATOKEN_TK_STAR:
    case FSATOKEN_TK_PLUS:
      break;

    case FSATOKEN_TK_CAT:
    case FSATOKEN_TK_OR:
      --parser->current_depth;
      break;

    case FSATOKEN_TK_BACKREF:
      parser->fast = false;
      /* fallthrough */
    default:
      ++parser->nleaves;
      /* fallthrough */
    case FSATOKEN_TK_EMPTY:
      ++parser->current_depth;
      break;
    }
  if (parser->depth < parser->current_depth)
    parser->depth = parser->current_depth;
}

static void addtok_wc (fsaparse_ctxt_t *parser, wint_t wc);

/* Sigh.  Mbcsets is so disruptive (intimate details hidden, by
   design), that we are writing addtok here from scratch.  */
static void
addtok (fsaparse_ctxt_t *parser, fsatoken_token_t t)
{
  bool need_or = false;
  mbcsets_set_t *work_mbc;
  bool invert;
  charclass_t *charclass;
  size_t nchars;
  size_t nch_classes;
  size_t nranges;
  size_t nequivs;
  size_t ncoll_elems;

  if (t != FSATOKEN_TK_MBCSET || parser->unibyte_locale)
    {
      addtok_mb (parser, t, 3);
      return;
    }

  work_mbc = parser->mbcsets[parser->nmbcsets - 1];
  mbcsets_get_characteristics (work_mbc,
                               &invert,
                               &charclass,
                               &nchars,
                               &nch_classes,
                               &nranges,
                               &nequivs,
                               &ncoll_elems);

  if (nchars && !invert)
    {
      size_t i;
      wchar_t *char_list;

      char_list = xnmalloc (nchars, sizeof *char_list);
      mbcsets_get_chars (work_mbc, char_list);
      addtok_wc (parser, char_list[0]);
      for (i = 1; i < nchars; i++)
        {
          addtok_wc (parser, char_list[i]);
          addtok (parser, FSATOKEN_TK_OR);
        }
      free (char_list);
      need_or = true;
    }

  /* If the set contains non-trivial components, it's too hard to deal
     with here, so hand it on to higher-up machinery.  */
  if (invert || nch_classes || nranges || ncoll_elems)
    {
      addtok_mb (parser, FSATOKEN_TK_MBCSET,
                 ((parser->nmbcsets - 1) << 2) + 3);
      if (need_or)
        addtok (parser, FSATOKEN_TK_OR);
    }

  /* Individual characters have been handled above, so the only case
     remaining is where codes have been simplified into charclass
     members.  */
  else if (charclass)
    {
      addtok (parser, FSATOKEN_TK_CSET + charclass_get_index (charclass));
      if (need_or)
        addtok (parser, FSATOKEN_TK_OR);
    }
}

/* We treat a multibyte character as a single atom, so that DFA
   can treat a multibyte character as a single expression.

   e.g., we construct the following tree from "<mb1><mb2>".
   <mb1(1st-byte)><mb1(2nd-byte)><FSATOKEN_TK_CAT><mb1(3rd-byte)><FSATOKEN_TK_CAT>
   <mb2(1st-byte)><mb2(2nd-byte)><FSATOKEN_TK_CAT><mb2(3rd-byte)><FSATOKEN_TK_CAT><FSATOKEN_TK_CAT> */
static void
addtok_wc (fsaparse_ctxt_t *parser, wint_t wc)
{
  unsigned char buf[MB_LEN_MAX];
  mbstate_t s = { 0 };
  int i;
  int cur_mb_len;
  size_t stored_bytes = wcrtomb ((char *) buf, wc, &s);

  if (stored_bytes != (size_t) -1)
    cur_mb_len = stored_bytes;
  else
    {
      /* This is merely stop-gap.  buf[0] is undefined, yet skipping
         the addtok_mb call altogether can corrupt the heap.  */
      cur_mb_len = 1;
      buf[0] = 0;
    }

  addtok_mb (parser, buf[0], cur_mb_len == 1 ? 3 : 1);
  for (i = 1; i < cur_mb_len; i++)
    {
      addtok_mb (parser, buf[i], i == cur_mb_len - 1 ? 2 : 0);
      addtok (parser, FSATOKEN_TK_CAT);
    }
}
static void
add_utf8_anychar (fsaparse_ctxt_t *parser)
{
  unsigned int i;

  /* Have we set up the classes for the 1-byte to 4-byte sequence types?  */
  if (parser->utf8_anychar_classes[0] == 0)
    {
      /* No, first time we've been called, so set them up now.  */
      charclass_t *ccl;
      const charclass_t *dotclass;

      /* Index 0: 80-bf -- Non-leading bytes.  */
      ccl = charclass_alloc ();
      charclass_setbit_range (0x80, 0xbf, ccl);
      parser->utf8_anychar_classes[0] = charclass_completed (ccl);

      /* Index 1: 00-7f -- 1-byte leading seq, minus dotclass exceptions.  */
      ccl = charclass_alloc ();
      charclass_setbit_range (0x00, 0x7f, ccl);
      fsalex_exchange (parser->lex_context, PROTO_LEXPARSE_OP_GET_DOTCLASS,
                       &dotclass);
      charclass_intersectset (dotclass, ccl);
      parser->utf8_anychar_classes[1] = charclass_completed (ccl);

      /* Index 2: c2-df -- 2-byte sequence.  */
      ccl = charclass_alloc ();
      charclass_setbit_range (0xc2, 0xdf, ccl);
      parser->utf8_anychar_classes[2] = charclass_completed (ccl);

      /* Index 2: e0-ef -- 3-byte sequence.  */
      ccl = charclass_alloc ();
      charclass_setbit_range (0xe0, 0xef, ccl);
      parser->utf8_anychar_classes[3] = charclass_completed (ccl);

      /* Index 2: f0-f7 -- 4-byte sequence.  */
      ccl = charclass_alloc ();
      charclass_setbit_range (0xf0, 0xf7, ccl);
      parser->utf8_anychar_classes[4] = charclass_completed (ccl);
    }

  /* A valid UTF-8 character is

     ([0x00-0x7f]
     |[0xe0-0xef][0x80-0xbf][0x80-0xbf]xc2-0xdf][0x80-0xbf]
     |[0xe0-0xef][0x80-0xbf][0x80-0xbf]xe0-0xef[0x80-0xbf][0x80-0xbf]
     |[0xe0-0xef][0x80-0xbf][0x80-0xbf]xf0-f7][0x80-0xbf][0x80-0xbf][0x80-0xbf])

     which I'll write more concisely "B|CA|DAA|EAAA".  Factor the [0x00-0x7f]
     and you get "B|(C|(D|EA)A)A".  And since the token buffer is in reverse
     Polish notation, you get "B C D E A CAT OR A CAT OR A CAT OR".  */
  /* Write out leaf tokens for each of the four possible starting bytes.  */
  for (i = 1; i < 5; i++)
    addtok (parser, FSATOKEN_TK_CSET + parser->utf8_anychar_classes[i]);
  /* Add follow-on classes, plus tokens to build a postfix tree
     covering all four alternatives of valid UTF-8 sequences.  */
  for (i = 1; i <= 3; i++)
    {
      addtok (parser, FSATOKEN_TK_CSET + parser->utf8_anychar_classes[0]);
      addtok (parser, FSATOKEN_TK_CAT);
      addtok (parser, FSATOKEN_TK_OR);
    }
}

/* The grammar understood by the parser is as follows.

   regexp:
     regexp FSATOKEN_TK_OR branch
     branch

   branch:
     branch closure
     closure

   closure:
     closure FSATOKEN_TK_QMARK
     closure FSATOKEN_TK_STAR
     closure FSATOKEN_TK_PLUS
     closure FSATOKEN_TK_REPMN
     atom

   atom:
     <normal character>
     <multibyte character>
     FSATOKEN_TK_ANYCHAR
     FSATOKEN_TK_MBCSET
     FSATOKEN_TK_CSET
     FSATOKEN_TK_BACKREF
     FSATOKEN_TK_BEGLINE
     FSATOKEN_TK_ENDLINE
     FSATOKEN_TK_BEGWORD
     FSATOKEN_TK_ENDWORD
     FSATOKEN_TK_LIMWORD
     FSATOKEN_TK_NOTLIMWORD
     FSATOKEN_TK_LPAREN regexp FSATOKEN_TK_RPAREN
     <empty>

   The parser builds a parse tree in postfix form in an array of tokens.  */

/* Provide a forward declaration for regexp, as it is at the top of
   the parse tree, but is referenced by atom, at the bottom of the
   tree.  */
static void regexp (fsaparse_ctxt_t *parser);

/* 3 June 2014: I must be a sucker for punishment, and/or going crazy;
    whatever.  I've looked at the partially-edited code for "atom",
    looking at it in the (incomplete) light of introducing the mbcsets
    change... and have decided, just as I previously did with
    closure () below, to rewrite it from scratch.  So, here goes...  */
static void
atom (fsaparse_ctxt_t *parser)
{
  fsatoken_token_t tok = parser->lookahead_token;

  /* For a unibyte character, it is its own token.  */
  if (tok >= 0 && tok < FSATOKEN_NOTCHAR)
    {
      addtok (parser, tok);
      parser->lookahead_token = parser->lexer (parser->lex_context);
      return;
    }

  /* For a unibyte character set (CSET + index), it is its own token.  */
  if (tok >= FSATOKEN_TK_CSET)
    {
      addtok (parser, tok);
      parser->lookahead_token = parser->lexer (parser->lex_context);
      return;
    }

  /* For ANYCHAR in an UTF-8 setting, use a hierarchy of CSETs.  */
  if (tok == FSATOKEN_TK_ANYCHAR && using_utf8 ())
    {
      /* For UTF-8, expand the period to a series of CSETs that define
         a valid UTF-8 character.  This avoids using the slow multibyte
         path.  I'm pretty sure it would be both profitable and correct
         to do it for any encoding; however, the optimization must be
         done manually as it is done above in add_utf8_anychar.  So,
         let's start with UTF-8: it is the most used, and the structure
         of the encoding makes the correctness more obvious.  */
      add_utf8_anychar (parser);
      parser->lookahead_token = parser->lexer (parser->lex_context);
      return;
    }

  /* Use switch to carve up the remaining cases into groups.  */
  switch (tok)
    {
    case FSATOKEN_TK_WCHAR:
      {
        /* Wide character, possibly including equivalent characters
           (probably due to selection of case folding).  */
        int nr_wide_chars;

        nr_wide_chars = fsalex_exchange (parser->lex_context,
                                         PROTO_LEXPARSE_OP_GET_WIDE_CHARS,
                                         parser->wide_char_list);
        assert(nr_wide_chars >= 1);
        if (parser->wide_char_list[0] == WEOF)
          addtok (parser, FSATOKEN_TK_BACKREF);
        else
          {
            int i;

            addtok_wc (parser, parser->wide_char_list[0]);
            for (i = 1; i < nr_wide_chars; i++)
              {
                addtok_wc (parser, parser->wide_char_list[i]);
                addtok (parser, FSATOKEN_TK_OR);
              }
          }
        parser->lookahead_token = parser->lexer (parser->lex_context);
        return;
      }

    case FSATOKEN_TK_MBCSET:
      {
        mbcsets_set_t *work_mbc;

        /* Acquire space to store set pointer.  Sadly (?), fsaparse
           maintains a duplicate list to fsalex at present.  */
        parser->mbcsets = maybe_realloc (parser->mbcsets,
                                         parser->nmbcsets,
                                         &parser->mbcsets_alloc,
                                         sizeof work_mbc);

        /* Get set pointer from fsalex, and add it to the list.  */
        fsalex_exchange (parser->lex_context,
                         PROTO_LEXPARSE_OP_GET_MBCSET,
                         &work_mbc);
        parser->mbcsets[parser->nmbcsets++] = work_mbc;

        /* We could fall-through case labels to the addtok code below,
           but am keeping code sections separate due to an overabundance
           of caution.  */
        addtok (parser, tok);
        parser->lookahead_token = parser->lexer (parser->lex_context);
        return;
      }

    case FSATOKEN_TK_ANYCHAR:
    case FSATOKEN_TK_BACKREF:
    case FSATOKEN_TK_BEGLINE:
    case FSATOKEN_TK_ENDLINE:
    case FSATOKEN_TK_BEGWORD:
    case FSATOKEN_TK_ENDWORD:
    case FSATOKEN_TK_LIMWORD:
    case FSATOKEN_TK_NOTLIMWORD:
      addtok (parser, tok);
      parser->lookahead_token = parser->lexer (parser->lex_context);
      return;

    case FSATOKEN_TK_LPAREN:
      parser->lookahead_token = parser->lexer (parser->lex_context);
      regexp (parser);
      if (parser->lookahead_token != FSATOKEN_TK_RPAREN)
        parser->abandon_with_error (_("unbalanced ("));
      parser->lookahead_token = parser->lexer (parser->lex_context);
      return;

    default:
      addtok (parser, FSATOKEN_TK_EMPTY);
      return;
  }

  /* NOTREACHED */
}

/* Return the number of tokens in the given subexpression.  */
static size_t _GL_ATTRIBUTE_PURE
nsubtoks (fsaparse_ctxt_t *parser, size_t tindex)
{
  size_t ntoks1;

  switch (parser->tokens[tindex - 1])
    {
    default:
      return 1;
    case FSATOKEN_TK_QMARK:
    case FSATOKEN_TK_STAR:
    case FSATOKEN_TK_PLUS:
      return 1 + nsubtoks (parser, tindex - 1);
    case FSATOKEN_TK_CAT:
    case FSATOKEN_TK_OR:
      ntoks1 = nsubtoks (parser, tindex - 1);
      return 1 + ntoks1 + nsubtoks (parser, tindex - 1 - ntoks1);
    }
}

/* Copy the given subexpression to the top of the tree.  */
static void
copytoks (fsaparse_ctxt_t *parser, size_t tindex, size_t ntokens)
{
  size_t i;

  if (parser->multibyte)
    for (i = 0; i < ntokens; ++i)
      addtok_mb (parser, parser->tokens[tindex + i], parser->multibyte_prop[tindex + i]);
  else
    for (i = 0; i < ntokens; ++i)
      addtok_mb (parser, parser->tokens[tindex + i], 3);
}

/* Rewriting fsaparse:closure () from scratch; original is clever but a
   little tricky to follow, so I'm trying to break up a while + compound-if
   loop into a simpler construct (more like a finite-state machine).  Also,
   edits such as replacing "dfa->" with "parser->" are done here, adding
   "parser" as a parameter in lots of places, as well as the long-winded
   FSATOKEN_TK_" prefix.

   I'm not sure if this version is an improvement over the original; the
   need to use "parser->lookahead_token" instead of "tok" influenced my
   decision to try this... but the jury is still out.  */
static void
closure (fsaparse_ctxt_t *parser)
{
restart_closure:
  atom (parser);
  for (;;)
    {
      switch (parser->lookahead_token)
        {
          case FSATOKEN_TK_QMARK:
          case FSATOKEN_TK_STAR:
          case FSATOKEN_TK_PLUS:
            addtok (parser, parser->lookahead_token);
            parser->lookahead_token = parser->lexer (parser->lex_context);
            continue;

          case FSATOKEN_TK_REPMN:
            /* REPMN needs extra work; move outside the switch statement.  */
            break;

          default:
            /* Merely let the intial atom call stand as our return result.  */
            return;
        }

      /* Deal with REPMN{min, max} cases in a separate block.  */
      {
        int i;
        size_t prev_sub_index, ntokens;
        int minrep, maxrep;

        /* Get the {min, max} pair decoded by the lexer.  */
        minrep = parser->lex_exchange (parser->lex_context,
                                       PROTO_LEXPARSE_OP_GET_REPMN_MIN,
                                       NULL);
        maxrep = parser->lex_exchange (parser->lex_context,
                                       PROTO_LEXPARSE_OP_GET_REPMN_MAX,
                                       NULL);

        /* Find out how many tokens are in the peceding token list that are
           covered by this REPMN directive.  This involves carefully working
           backwards through the linear, postfix token ordering.  */
        ntokens = nsubtoks (parser, parser->tindex);

        /* If min and max are both zero, merely remove preceding
           subexpression, get a new token, and restart the atom/closure
           processing from the top of the function.  Not sure if people will
           like this goto statement, but we'll give it a whirl.   */
        if (minrep == 0 && maxrep == 0)
          {
            parser->tindex -= ntokens;
            parser->lookahead_token = parser->lexer (parser->lex_context);
            goto restart_closure;
          }

        /* Non-zero min or max, defined as follows:
             {n}   The preceding item is matched exactly n times.
             {n,}  The preceding item is matched n or more times.
             {,m}  The preceding item is matched at most m times (GNU ext.)
             {n,m} The preceding item is matched at least n, but not more
                   than m times.
           For {n,} and {,m} cases, the omitted parameter is reported here
           as a negative value.  */
        prev_sub_index = parser->tindex - ntokens;
        if (maxrep < 0)
          addtok (parser, FSATOKEN_TK_PLUS);
        if (minrep == 0)
          addtok (parser, FSATOKEN_TK_QMARK);
        for (i = 1; i < minrep; ++i)
          {
            copytoks (parser, prev_sub_index, ntokens);
            addtok (parser, FSATOKEN_TK_CAT);
          }
        for (; i < maxrep; ++i)
          {
            copytoks (parser, prev_sub_index, ntokens);
            addtok (parser, FSATOKEN_TK_QMARK);
            addtok (parser, FSATOKEN_TK_CAT);
          }
        /* Prime the parser with the next token after REPMN and loop.  */
        parser->lookahead_token = parser->lexer (parser->lex_context);
      }
    }
}

static void
branch (fsaparse_ctxt_t *parser)
{
  fsatoken_token_t tok;

  closure (parser);
  tok = parser->lookahead_token;
  while (tok != FSATOKEN_TK_RPAREN && tok != FSATOKEN_TK_OR && tok >= 0)
    {
      closure (parser);
      tok = parser->lookahead_token;
      addtok (parser, FSATOKEN_TK_CAT);
    }
}

static void
regexp (fsaparse_ctxt_t *parser)
{
  branch (parser);
  while (parser->lookahead_token == FSATOKEN_TK_OR)
    {
      parser->lookahead_token = parser->lexer (parser->lex_context);
      branch (parser);
      addtok (parser, FSATOKEN_TK_OR);
    }
}

/* Main entry point for the parser.  Parser is a pointer to a parser
   context struct created by fsaparse_new.  Before calling this function,
   the parser instance must be supplied with a lexer (fsaparse_lexer), and
   also with callback functions to receive warning and error reports
   (fsaparse_esception_fns).  */
void
fsaparse_parse (fsaparse_ctxt_t *parser)
{
  /* Obtain an initial token for lookahead, and keep tracking tree depth.  */
  parser->lookahead_token = parser->lexer (parser->lex_context);
  parser->current_depth = parser->depth;

  /* Run regexp to manage the next level of parsing.  */
  regexp (parser);
  if (parser->lookahead_token != FSATOKEN_TK_END)
    parser->abandon_with_error (_("unbalanced )"));

  /* If multiple expressions are parsed, second and subsequent patters are
     presented as alternatives to preceding patterns.  */
  addtok (parser, FSATOKEN_TK_END - parser->nregexps);
  addtok (parser, FSATOKEN_TK_CAT);
  if (parser->nregexps)
    addtok (parser, FSATOKEN_TK_OR);

  ++parser->nregexps;
}

/* Receive functions to deal with exceptions detected by the parser:
   Warnings and errors.  Internally, we add the _Noreturn attribute
   to the error callback, to help the compiler with code flow
   analysis.  */
extern void
fsaparse_exception_fns (fsaparse_ctxt_t *parser,
                      fsaparse_warn_callback_fn *warningfn,
                      fsaparse_error_callback_fn *errorfn)
{
  /* Exception handling is done by explicit callbacks.  */
  parser->warn_client = warningfn;
  parser->abandon_with_error = errorfn;
}

/* Add "not provided!" stub function that gets called if the client
   fails to provide proper resources.  This is a hack, merely to get the
   module started; better treatment needs to be added later.  */
static void
no_function_provided (void *unused)
{
  assert (!"fsaparse: Plug-in function required, but not provided.");
}

/* Receiver a lexer function, plus lexer instance context pointer, for use by
   the parser.  Although not needed initially, this plug-in architecture may
   be useful in the future, and it breaks up some of the intricate
   connections that made the original dfa.c code so daunting.  */
void
fsaparse_lexer (fsaparse_ctxt_t *parser,
                void *lexer_context,
                proto_lexparse_lex_fn_t *lex_fn,
                proto_lexparse_exchange_fn_t *lex_exchange_fn)
{
  bool is_multibyte;
  int wchars_max;

  /* Record supplied lexer function and context for use later.  */
  parser->lex_context  = lexer_context;
  parser->lexer        = lex_fn;
  parser->lex_exchange = lex_exchange_fn;

  /* Query lexer to get multibyte nature of this locale.  */
  is_multibyte
      = lex_exchange_fn (lexer_context,
                         PROTO_LEXPARSE_OP_GET_IS_MULTIBYTE_LOCALE,
                         NULL);
  parser->multibyte_locale = is_multibyte;
  parser->unibyte_locale = ! is_multibyte;

  /* Set up WCHAR token infrastructure:  Wide-char list/array/whatever.  We
     free the wide_char_list first, in case someone calls this function
     multiple times, so that earlier allocations are correctly freed up, and
     we use the value advised by the most recent lexer presented to us.  */
  wchars_max = lex_exchange_fn (lexer_context,
                                PROTO_LEXPARSE_OP_GET_WIDE_CHAR_LIST_MAX,
                                NULL);
  parser->wide_chars_max = wchars_max;
  free (parser->wide_char_list);
  parser->wide_char_list = NULL;
  if (wchars_max)
    parser->wide_char_list = xnmalloc (wchars_max, sizeof (wchar_t));
}

/* Generate a new instance of an FSA parser.  */
fsaparse_ctxt_t *
fsaparse_new (void)
{
  fsaparse_ctxt_t *new_context;

  /* Acquire zeroed memory for new parser context.  */
  new_context = XZALLOC (fsaparse_ctxt_t);

  /* ?? Point warning, error and lexer functions to a "you need to tell me
     these first!" function? */
  new_context->warn_client        = (fsaparse_warn_callback_fn *)
                                    no_function_provided;
  new_context->abandon_with_error = (fsaparse_error_callback_fn *)
                                    no_function_provided;
  new_context->lexer              = (fsaparse_lexer_fn_t  *)
                                    no_function_provided;

  /* Default to unibyte locale... but we synchronise with the lexer later. */
  new_context->multibyte_locale = false;
  new_context->unibyte_locale = true;

  /* Add this instance at the head of the module's list.  */
  new_context->next_instance = fsaparse_instance_list_head;
  fsaparse_instance_list_head = new_context;

  return new_context;
}

/* After parsing, report a list of tokens describing the pattern.  Complex
   structures such as alternation, backreferences, and locale-induced
   complexity such as variable-length utf8 sequences are described here by
   appending operators that apply to the preceding item(s) (postfix
   notation).  */
void
fsaparse_get_token_list (fsaparse_ctxt_t *parser,
                         size_t *nr_tokens,
                         fsatoken_token_t **token_list)
{
  *nr_tokens  = parser->tindex;
  *token_list = parser->tokens;
}

/* Internal function to free all resources directly or indirectly used by an
   instance.  The pointer is no longer valid after this call.  */
static void
free_instance (fsaparse_ctxt_t *parser)
{
  free (parser->wide_char_list);
  free (parser->tokens);
  free (parser->multibyte_prop);
  free (parser);
}

/* Destroy all parser instances, plus any associated resources owned by the
   module.  */
void
fsaparse_destroy_module (void)
{
  fsaparse_ctxt_t *p_list;
  fsaparse_ctxt_t *p_next;

  /* Move the global list head into a local variable, and immediately clear
     the global.  This is a half-hearted attempt to avoid race conditions;
     to do things properly, a system-wide atomic operation (locked,
     including multi-CPU cache coherency) operation should be used.  */
  p_list = fsaparse_instance_list_head;
  fsaparse_instance_list_head = NULL;

  /* Traverse the list of instances, releasing all resources associated
     with each one.  */
  while (p_list)
    {
      p_next = p_list->next_instance;
      free_instance (p_list);
      p_list = p_next;
    }
}

/* Prepare module for operation.  */
void
fsaparse_initialise (void)
{
  /* Initialise the linked list of instances created by this module.  */
  fsaparse_instance_list_head = NULL;

  /* Clean up resources upon exit.  */
  atexit (fsaparse_destroy_module);
}

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="fsaparse.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsaparse.h"

/* fsaparse -- Build a structure naming relationships (sequences,
               alternatives, options and precedence) of tokens

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* This function receives a stream of tokens from fsalex, and
   processes them to impose precedence rules and to describe complex
   pattern elements that are beyond the capability of the simple
   lexer.  In addition to the cases explicit in the syntax (e.g.
   "(ab|c)", variable-length multibyte encodings (UTF-8; codesets
   including modifiers and/or shift items) also require these enhanced
   facilities.  */

#ifndef FSAPARSE_H
#define FSAPARSE_H 1

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include "fsatoken.h"
#include "proto-lexparse.h"

/* Multiple parser instances can exist in parallel, so define an opaque
   type to collect together all the context relating to each instance.  */
typedef struct fsaparse_ctxt_struct fsaparse_ctxt_t;

/* Allow configurable parser/lexer combinations by using a plugin interface
   for lexer invocation.  */
typedef fsatoken_token_t
fsaparse_lexer_fn_t (void *lexer_context);

/* Functions to initialise and tear down entire module and all associated
   instances.  */

/* Prepare module for operation.  */
extern void
fsaparse_initialise (void);

/* Destroy all parser instances, plus any associated resources owned by the
   module.  */
extern void
fsaparse_destroy_module (void);

/* Functions to create a new parser instance, and to connect it with
   a compatible lexer instance.  */

/* Generate a new instance of an FSA parser.  */
extern fsaparse_ctxt_t *
fsaparse_new (void);

/* Receiver a lexer function, plus lexer instance context pointer, for use by
   the parser.  Although not needed initially, this plug-in architecture may
   be useful in the future, and it breaks up some of the intricate
   connections that made the original dfa.c code so daunting.  */
extern void
fsaparse_lexer (fsaparse_ctxt_t *parser,
                void *lexer_context,
                proto_lexparse_lex_fn_t *lex_fn,
                proto_lexparse_exchange_fn_t *lex_exchange_fn);

/* Define function prototypes for warning and error callbacks.  */
typedef void
fsaparse_warn_callback_fn (const char *);
typedef void /* ?? _Noreturn? */
fsaparse_error_callback_fn (const char *);

/* Receive functions to deal with exceptions detected by the parser:
   Warnings and errors.  Internally, we add the _Noreturn attribute
   to the error callback, to help the compiler with code flow
   analysis.  */
extern void
fsaparse_exception_fns (fsaparse_ctxt_t *parser,
                      fsaparse_warn_callback_fn *warningfn,
                      fsaparse_error_callback_fn *errorfn);

/* Main entry point for the parser.  Parser is a pointer to a parser
   context struct created by fsaparse_new.  Before calling this function,
   the parser instance must be supplied with a lexer (fsaparse_lexer), and
   also with callback functions to receive warning and error reports
   (fsaparse_esception_fns).  */
extern void
fsaparse_parse (fsaparse_ctxt_t *parser);

/* After parsing, report a list of tokens describing the pattern.  Complex
   structures such as alternation, backreferences, and locale-induced
   complexity such as variable-length utf8 sequences are described here by
   appending operators that apply to the preceding item(s) (postfix
   notation).  */
extern void
fsaparse_get_token_list (fsaparse_ctxt_t *parser,
                         size_t *nr_tokens,
                         fsatoken_token_t **token_list);

#endif /* FSAPARSE_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="fsatoken.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsatoken.c"

/* fsatoken - Support routines specific to token definitions

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* The majority of the fsatoken[ch] module is in fsatoken.h, as it is
   shared by other modules.  This file provides token-specific support
   functions, such as functions to print tokens (for debugging).

   Although there is a relationship between some generic constructs
   such as character classes and the CSET token defined here, the
   generic items are defined in a separate support library, not in this
   module.  This is because these tokens are very FSA/grep-specific,
   whereas the generic consructs are potentially widely useable, and may
   even be amenable to hardware-specific optimisations (such as
   superscalar opcodes such as:
         - and/or/set/clear/test-and-set/test-and-clear; and/or
         - bit counting
   operations).  */

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include "fsatoken.h"
#include <stdio.h>

#ifdef DEBUG

void
fsatoken_prtok (fsatoken_token_t t)
{
  char const *s;

  if (t < 0)
    fprintf (stderr, "FSATOKEN_TK_END");
  else if (t < FSATOKEN_NOTCHAR)
    {
      int ch = t;
      fprintf (stderr, "%c", ch);
    }
  else
    {
      switch (t)
        {
        case FSATOKEN_TK_EMPTY:
          s = "FSATOKEN_TK_EMPTY";
          break;
        case FSATOKEN_TK_BACKREF:
          s = "FSATOKEN_TK_BACKREF";
          break;
        case FSATOKEN_TK_BEGLINE:
          s = "FSATOKEN_TK_BEGLINE";
          break;
        case FSATOKEN_TK_ENDLINE:
          s = "FSATOKEN_TK_ENDLINE";
          break;
        case FSATOKEN_TK_BEGWORD:
          s = "FSATOKEN_TK_BEGWORD";
          break;
        case FSATOKEN_TK_ENDWORD:
          s = "FSATOKEN_TK_ENDWORD";
          break;
        case FSATOKEN_TK_LIMWORD:
          s = "FSATOKEN_TK_LIMWORD";
          break;
        case FSATOKEN_TK_NOTLIMWORD:
          s = "FSATOKEN_TK_NOTLIMWORD";
          break;
        case FSATOKEN_TK_QMARK:
          s = "FSATOKEN_TK_QMARK";
          break;
        case FSATOKEN_TK_STAR:
          s = "FSATOKEN_TK_STAR";
          break;
        case FSATOKEN_TK_PLUS:
          s = "FSATOKEN_TK_PLUS";
          break;
        case FSATOKEN_TK_CAT:
          s = "FSATOKEN_TK_CAT";
          break;
        case FSATOKEN_TK_OR:
          s = "FSATOKEN_TK_OR";
          break;
        case FSATOKEN_TK_LPAREN:
          s = "FSATOKEN_TK_LPAREN";
          break;
        case FSATOKEN_TK_RPAREN:
          s = "FSATOKEN_TK_RPAREN";
          break;
        case FSATOKEN_TK_ANYCHAR:
          s = "FSATOKEN_TK_ANYCHAR";
          break;
        case FSATOKEN_TK_MBCSET:
          s = "FSATOKEN_TK_MBCSET";
          break;
        default:
          s = "FSATOKEN_TK_CSET";
          break;
        }
      fprintf (stderr, "%s", s);
    }
}

#endif /* DEBUG */
/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="fsatoken.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fsatoken.h"

/* fsatoken - Create tokens for a compact, coherent regular expression language

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* Regular expression patterns are presented as text, possibly ASCII;
   the format is very expressive, but this comes at the cost of being
   somewhat expensive to interpret (including identifying invalid
   patterns).  By tokenising the pattern, we make life much easier for
   the parser and other search machinery that follows.

   This file defines the tokens that we use, both for the benefit of
   the lexer/parser/dfa analyser that share this information, and for
   other machinery (such as the C compiler) that may need to store
   and/or manipulate these items.  */


#ifndef FSATOKEN_H
#define FSATOKEN_H 1

/* Always import environment-specific configuration items first.  */
#include <config.h>

/* Obtain defn. of ptrdiff_t from stddef.h, and CHAR_BIT from limits.h  */
#include <limits.h>
#include <stddef.h>

/* C stream octets, and non-stream EOF, are self-representing tokens.
   We need to include stdio.h to obtain the definition of EOF.  */
#include <stdio.h>

/* First integer value that is greater than any character code.  */
enum { FSATOKEN_NOTCHAR = 1 << CHAR_BIT };

/* The regexp is parsed into an array of tokens in postfix form.  Some tokens
   are operators and others are terminal symbols.  Most (but not all) of these
   codes are returned by the lexical analyzer.  */

typedef ptrdiff_t fsatoken_token_t;

/* Predefined token values.  */
enum
{
  FSATOKEN_TK_END = -1,                     /* FSATOKEN_TK_END is a terminal symbol that matches the
                                   end of input; any value of FSATOKEN_TK_END or less in
                                   the parse tree is such a symbol.  Accepting
                                   states of the DFA are those that would have
                                   a transition on FSATOKEN_TK_END.  */

  /* Ordinary character values are terminal symbols that match themselves.  */

  FSATOKEN_TK_EMPTY = FSATOKEN_NOTCHAR,              /* FSATOKEN_TK_EMPTY is a terminal symbol that matches
                                   the empty string.  */

  FSATOKEN_TK_BACKREF,                      /* FSATOKEN_TK_BACKREF is generated by \<digit>
                                   or by any other construct that
                                   is not completely handled.  If the scanner
                                   detects a transition on backref, it returns
                                   a kind of "semi-success" indicating that
                                   the match will have to be verified with
                                   a backtracking matcher.  */

  FSATOKEN_TK_BEGLINE,                      /* FSATOKEN_TK_BEGLINE is a terminal symbol that matches
                                   the empty string at the beginning of a
                                   line.  */

  FSATOKEN_TK_ENDLINE,                      /* FSATOKEN_TK_ENDLINE is a terminal symbol that matches
                                   the empty string at the end of a line.  */

  FSATOKEN_TK_BEGWORD,                      /* FSATOKEN_TK_BEGWORD is a terminal symbol that matches
                                   the empty string at the beginning of a
                                   word.  */

  FSATOKEN_TK_ENDWORD,                      /* FSATOKEN_TK_ENDWORD is a terminal symbol that matches
                                   the empty string at the end of a word.  */

  FSATOKEN_TK_LIMWORD,                      /* FSATOKEN_TK_LIMWORD is a terminal symbol that matches
                                   the empty string at the beginning or the
                                   end of a word.  */

  FSATOKEN_TK_NOTLIMWORD,                   /* FSATOKEN_TK_NOTLIMWORD is a terminal symbol that
                                   matches the empty string not at
                                   the beginning or end of a word.  */

  FSATOKEN_TK_QMARK,                        /* FSATOKEN_TK_QMARK is an operator of one argument that
                                   matches zero or one occurrences of its
                                   argument.  */

  FSATOKEN_TK_STAR,                         /* FSATOKEN_TK_STAR is an operator of one argument that
                                   matches the Kleene closure (zero or more
                                   occurrences) of its argument.  */

  FSATOKEN_TK_PLUS,                         /* FSATOKEN_TK_PLUS is an operator of one argument that
                                   matches the positive closure (one or more
                                   occurrences) of its argument.  */

  FSATOKEN_TK_REPMN,                        /* FSATOKEN_TK_REPMN is a lexical token corresponding
                                   to the {m,n} construct.  FSATOKEN_TK_REPMN never
                                   appears in the compiled token vector.  */

  FSATOKEN_TK_CAT,                          /* FSATOKEN_TK_CAT is an operator of two arguments that
                                   matches the concatenation of its
                                   arguments.  FSATOKEN_TK_CAT is never returned by the
                                   lexical analyzer.  */

  FSATOKEN_TK_OR,                           /* FSATOKEN_TK_OR is an operator of two arguments that
                                   matches either of its arguments.  */

  FSATOKEN_TK_LPAREN,                       /* FSATOKEN_TK_LPAREN never appears in the parse tree,
                                   it is only a lexeme.  */

  FSATOKEN_TK_RPAREN,                       /* FSATOKEN_TK_RPAREN never appears in the parse tree.  */

  FSATOKEN_TK_ANYCHAR,                      /* FSATOKEN_TK_ANYCHAR is a terminal symbol that matches
                                   a valid multibyte (or single byte) character.
                                   It is used only if MB_CUR_MAX > 1.  */

  FSATOKEN_TK_MBCSET,                       /* FSATOKEN_TK_MBCSET is similar to FSATOKEN_TK_CSET, but for
                                   multibyte characters.  */

  FSATOKEN_TK_WCHAR,                        /* Only returned by lex.  wctok contains
                                   the wide character representation.  */

  FSATOKEN_TK_CSET                          /* FSATOKEN_TK_CSET and (and any value greater) is a
                                   terminal symbol that matches any of a
                                   class of characters.  */
};


/* prtok - Display token name (for debugging) */
#ifdef DEBUG
extern void
fsatoken_prtok (fsatoken_token_t t);

#endif /* DEBUG */

#endif /* FSATOKEN_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-csrc;
 name="mbcsets.c"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="mbcsets.c"

/* mbcsets -- Handle multi-byte and/or locale-dependent sets of chars.

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Repackaged by "untangle" script, written by behoffski.  */

/* ?? Need to document things here...
 */

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include "charclass.h"
#include "mbcsets.h"
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <wctype.h>
#include "xalloc.h"

/* Flesh out opaque type given in the header.  */
struct mbcsets_set_struct
{
  /* Singly-linked list of all instances, so destroy_module can release
     all resources by traversing the list.  */
  mbcsets_set_t *next_instance;

  charclass_t *charclass;
  bool invert;
  wchar_t *chars;               /* Normal characters.  */
  size_t nchars;
  size_t chars_alloc;
  wctype_t *ch_classes;         /* Character classes.  */
  size_t nch_classes;
  size_t ch_classes_alloc;
  struct                        /* Range characters.  */
  {
    wchar_t beg;                /* Range start.  */
    wchar_t end;                /* Range end.  */
  } *ranges;
  size_t nranges;
  size_t ranges_alloc;
  char **equivs;                /* Equivalence classes.  */
  size_t nequivs;
  size_t equivs_alloc;
  char **coll_elems;
  size_t ncoll_elems;           /* Collating elements.  */
  size_t coll_alloc;
};

/* Linked list of all instances created by this module.  */
static mbcsets_set_t *mbcsets_instances_list_head = NULL;

/* Ensure that the array addressed by PTR holds at least NITEMS +
   (PTR || !NITEMS) items.  Either return PTR, or reallocate the array
   and return its new address.  Although PTR may be null, the returned
   value is never null.

   The array holds *NALLOC items; *NALLOC is updated on reallocation.
   ITEMSIZE is the size of one item.  Avoid O(N**2) behavior on arrays
   growing linearly.  */
static void *
maybe_realloc (void *ptr, size_t nitems, size_t *nalloc, size_t itemsize)
{
  if (nitems < *nalloc)
    return ptr;
  *nalloc = nitems;
  return x2nrealloc (ptr, nalloc, itemsize);
}

/* By default, classes match the specified characters.  Regular
   expressions allow this sense to be inverted, usually by the
   convention of "^" being the first character of a bracketed class.
   By default, positive sense is selected; this function lets the
   user specify the sense, probably to specify inverted matching.   */
void
mbcsets_set_match_sense (mbcsets_set_t *mbc, bool invert)
{
  mbc->invert = invert;
}

/* Individual wide characters.  */
void
mbcsets_add_wchar (mbcsets_set_t *mbc, wint_t wc)
{
  /* ?? Quietly ignore WEOF... is this reasonable?  */
  if (wc == WEOF)
    return;

  /* Could this character fit into a charclass set? */
  /* ?? We don't know directly the signedness of wint_t; in gnulib, it
     is unsigned, and so testing for "wc >= 0" is flagged as an error
     as it is redundant.  Have removed the test here for now, but not
     confident that this is the best treatment.  */
  if (/* wc >= 0 && */ wc < CHARCLASS_NOTCHAR)
    {
     /* Yes, does this wide char have a valid unichar representation? */
     /* ?? The documentation states that wctob should not be used;
        perhaps the unibyte cache mbrtowc_cache in fsalex, built by
        initialise_uchar_to_wc_cache (), might be relevant here?  */
     int b = wctob (wc);
     if (b != EOF)
       {
         /* Yes, add the char (byte?) (octet?) to the charclass set.  */
         charclass_setbit (b, mbc->charclass);
         return;
       }
    }

  /* Ensure we have space to store another character.  */
  mbc->chars = maybe_realloc(mbc->chars, mbc->nchars,
                             &mbc->chars_alloc,
                             sizeof *mbc->chars);

  /* Add the character to the list.  */
  mbc->chars[mbc->nchars++] = wc;
}

/* Add a list of wide characters (note: not wide integers).  */
void
mbcsets_add_wchar_list (mbcsets_set_t *mbc,
                        size_t len, wchar_t *wc_list)
{
  size_t i;

  /* Ensure we have space to store the incoming list element(s).  */
  mbc->chars = maybe_realloc(mbc->chars, mbc->nchars + len,
                             &mbc->chars_alloc,
                             sizeof *mbc->chars);

  /* Add all the characters to the list.  (?? use memcpy here?)  */
  for (i = 0; i < len; i++)
    mbc->chars[mbc->nchars++] = wc_list[i];
}

/* Common character classes, e.g. alpha, digit, punct etc.  */
void
mbcsets_add_class (mbcsets_set_t *mbc, wctype_t wchar_class)
{
  /* ?? We don't check that the descriptor is valid.  */

  /* Ensure we have space to store another class descriptor.  */
  mbc->ch_classes = maybe_realloc(mbc->ch_classes, mbc->nch_classes,
                                  &mbc->ch_classes_alloc,
                                  sizeof *mbc->ch_classes);

  /* Add the class descriptor to the list.  */
  mbc->ch_classes[mbc->nch_classes++] = wchar_class;
}

/* Explicit character ranges.  */
void
mbcsets_add_range (mbcsets_set_t *mbc, wint_t beg, wint_t end)
{
  /* ?? We don't check that the begin/end chars are valid.  */

  /* Ensure we have space to store another begin/end char pair.  */
  mbc->ranges = maybe_realloc(mbc->ranges, mbc->nranges + 1,
                              &mbc->ranges_alloc,
                              sizeof *mbc->ranges);

  /* Add the range to the list.  */
  mbc->ranges[mbc->nranges].beg   = beg;
  mbc->ranges[mbc->nranges++].end = end;
}

/* Receive an "in-work" character class, which may or may not have
   members.  Mbcset takes ownership of this set, and, depending on
   the circumstances, either maintains it internally, or else copies
   its contents (if any) to its internals, and releases (abandons)
   the supplied set.  This function must not applied to a set that
   has been completed.  */
void
mbcsets_receive_incomplete_charclass (mbcsets_set_t *mbc,
                                      charclass_t *ccl)
{
  charclass_unionset (ccl, mbc->charclass);
  charclass_abandon (ccl);
}

/* Mark a set as completed; the implementation may also analyse and
   optimise the set at this point (e.g. use charclasses to represent
   unibyte characters; merge overlapping ranges; remove the
   individual listing of a character if it is covered by a range,
   etc.)

   In addition, note that no further changes (e.g. receive another
   incomplete charclass) are allowed for this set, once "completed"
   is called.  */
void
mbcsets_completed (mbcsets_set_t *mbc)
{
  charclass_t *zeroset;

  /* Did we end up putting anything into the charclass?  */
  zeroset = charclass_get_pointer (0);
  if (charclass_equal (mbc->charclass, zeroset))
    {
      /* No, abandon the class, and use NULL as our sentinel.  */
      charclass_abandon (mbc->charclass);
      mbc->charclass = NULL;
    }
    else
    {
      /* Yes, complete the class, and obtain a persistent pointer.  */
      charclass_index_t index;
      index = charclass_completed (mbc->charclass);
      mbc->charclass = charclass_get_pointer (index);
    }
}

/* Retrieve high-level information about the class, which is useful
   (in fsaparse) for deciding on how to deal with it.  We are forced
   to provide significant query resources since we demand that the
   type internal remain opaque (even though the initial
   implementation may do a poor job of this effort).  */
void
mbcsets_get_characteristics (mbcsets_set_t *mbc,
                             bool *p_invert,
                             charclass_t **pp_charclass,
                             size_t *p_nchars,
                             size_t *p_nch_classes,
                             size_t *p_nranges,
                             size_t *p_nequivs,
                             size_t *p_ncoll_elems)
{
  *p_invert      = mbc->invert;
  *pp_charclass  = mbc->charclass;
  *p_nchars      = mbc->nchars;
  *p_nch_classes = mbc->nch_classes;
  *p_nranges     = mbc->nranges;
  *p_nequivs     = mbc->nequivs;
  *p_ncoll_elems = mbc->ncoll_elems;
}

/* Copy wide char list to caller's work area.  */
void
mbcsets_get_chars (mbcsets_set_t *mbc, wchar_t *char_list)
{
  memcpy (char_list, mbc->chars, mbc->nchars * sizeof(*char_list));
}

/* Prepare module for operation.  */
void
mbcsets_initialise (void)
{
  /* Initialise the linked list of instances created by this module.  */
  mbcsets_instances_list_head = NULL;

  atexit (mbcsets_destroy_module);
}

/* Internal function to free all resources directly or indirectly used by
   an instance.  The pointer is no longer valid after this call.  */
static void
free_instance (mbcsets_set_t *mbc)
{
  size_t i;

  free (mbc->chars);
  free (mbc->ch_classes);
  free (mbc->ranges);

  for (i = 0; i < mbc->nequivs; ++i)
    free (mbc->equivs[i]);
  free (mbc->equivs);

  for (i = 0; i < mbc->ncoll_elems; ++i)
    free (mbc->coll_elems[i]);
  free (mbc->coll_elems);

  free (mbc);
}

/* Destroy all classes, plus any associated resources owned by the
   module.  */
void
mbcsets_destroy_module (void)
{
  mbcsets_set_t *p_list;
  mbcsets_set_t *p_next;

  /* Move the global list head into a local variable, and immediately
     clear the global.  This is a half-hearted attempt to avoid race
     conditions; to do things properly, a system-wide atomic operation
     (locked, including multi-CPU cache coherency) operation should be
     used.  */
  p_list = mbcsets_instances_list_head;
  mbcsets_instances_list_head = NULL;

  /* Traverse the list of instances, releasing all resources associated
     with each one.  */
  while (p_list)
    {
      p_next = p_list->next_instance;
      free_instance (p_list);
      p_list = p_next;
    }
}

/* Generate a new instance of a multibyte-character set descriptor.  */
mbcsets_set_t *
mbcsets_new (void)
{
  mbcsets_set_t *new_set;

  /* Allocate memory for new instance.  */
  new_set = xzalloc (sizeof (*new_set));

  /* Lint new instance into list of instances created by this module.  */
  new_set->next_instance = mbcsets_instances_list_head;
  mbcsets_instances_list_head = new_set;

  /* Allocate a charclass set to the instance, so we can easily push
     codes into that representation if possible.  */
  new_set->charclass = charclass_alloc ();

  /* Report created instance to the caller.  */
  return new_set;
}

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="mbcsets.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="mbcsets.h"

/* mbcsets -- Handle multi-byte and/or locale-dependent sets of chars.

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Created by "untangle" script, written by behoffski.

?? more stuff here

*/

#ifndef MBCSETS_H
#define MBCSETS_H 1

/* Always import environment-specific configuration items first.  */
#include <config.h>

#include "charclass.h"
#include <stddef.h>
#include <wctype.h>

/* Define the multibyte-set descriptor as an opaque type.  */
typedef struct mbcsets_set_struct mbcsets_set_t;

/* Prepare module for operation.  */
extern void
mbcsets_initialise (void);

/* Destroy all classes, plus any associated resources owned by the
   module.  */
extern void
mbcsets_destroy_module (void);

/* Generate a new instance of a multibyte-character set descriptor.  */
extern mbcsets_set_t *
mbcsets_new (void);

/* By default, classes match the specified characters.  Regular
   expressions allow this sense to be inverted, usually by the
   convention of "^" being the first character of a bracketed class.
   By default, positive sense is selected; this function lets the
   user specify the sense, probably to specify inverted matching.   */
extern void
mbcsets_set_match_sense (mbcsets_set_t *mbc, bool invert);

/* Individual wide characters.  */
extern void
mbcsets_add_wchar (mbcsets_set_t *mbc, wint_t wc);

/* Add a list of wide characters (note: not wide integers).  */
extern void
mbcsets_add_wchar_list (mbcsets_set_t *mbc,
                        size_t len, wchar_t *wc_list);

/* Common character classes, e.g. alpha, digit, punct etc.  */
extern void
mbcsets_add_class (mbcsets_set_t *mbc, wctype_t wchar_class);

/* Explicit character ranges.  */
extern void
mbcsets_add_range (mbcsets_set_t *mbc, wint_t beg, wint_t end);

/* Receive an "in-work" character class, which may or may not have
   members.  Mbcset takes ownership of this set, and, depending on
   the circumstances, either maintains it internally, or else copies
   its contents (if any) to its internals, and releases (abandons)
   the supplied set.  This function must not applied to a set that
   has been completed.  */
extern void
mbcsets_receive_incomplete_charclass (mbcsets_set_t *mbc,
                                      charclass_t *ccl);

/* Retrieve high-level information about the class, which is useful
   (in fsaparse) for deciding on how to deal with it.  We are forced
   to provide significant query resources since we demand that the
   type internal remain opaque (even though the initial
   implementation may do a poor job of this effort).  */
extern void
mbcsets_get_characteristics (mbcsets_set_t *mbc,
                             bool *p_invert,
                             charclass_t **pp_charclass,
                             size_t *p_nchars,
                             size_t *p_nch_classes,
                             size_t *p_nranges,
                             size_t *p_nequivs,
                             size_t *p_ncoll_elems);

/* Copy wide char list to caller's work area.  */
extern void
mbcsets_get_chars (mbcsets_set_t *mbc, wchar_t *char_list);

/* Mark a set as completed; the implementation may also analyse and
   optimise the set at this point (e.g. use charclasses to represent
   unibyte characters; merge overlapping ranges; remove the
   individual listing of a character if it is covered by a range,
   etc.)

   In addition, note that no further changes (e.g. receive another
   incomplete charclass) are allowed for this set, once "completed"
   is called.  */
extern void
mbcsets_completed (mbcsets_set_t *mbc);

#endif /* MBCSETS_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/x-chdr;
 name="proto-lexparse.h"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="proto-lexparse.h"

/* proto-lexparse -- Define how lexer and parser can interact.

   Copyright (C) 1988, 1998, 2000, 2002, 2004-2005, 2007-2014 Free Software
   Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 3, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc.,
   51 Franklin Street - Fifth Floor, Boston, MA  02110-1301, USA */

/* Written June, 1988 by Mike Haertel
   Modified July, 1988 by Arthur David Olson to assist BMG speedups  */

/* 2014: Created by "untangle" script, written by behoffski.

(EVENTUALLYREWRITEME: A very lengthy editorial/discussion of this module
follows, partially interspersed with a description of its intended
function.  Comments and criticism are welcome; if adopted, I expect that
this comment block will be rewritten to be much more focussed and much
shorter.)

This file has been added very, very late in the non-linear process of
developing this code.  It addresses a feature that was not anticipated early
on, but which I believe is worth exploring in the future:  The presence of
pluggable lexers to supply to the parser, or, equivalently, breaking up the
master/slave relationship between the parser and the lexer, and replacing it
with a peer-to-peer conversation (perhaps the software equivalent of a
shared bus).

Quite a bit of the need for this protocol/interface module is that I've
demanded that the individual modules strictly hide their internals, and only
share information and/or control of resources (such as the parser calling
lex ()) in an explicit fashion.  Early on, the nature of connections between
the lexer and the parser appeared with the need to communicate {min,max}
values as part of the REPMN token; then the need to communicate the wide
character implied by the WCHAR token.

The crisis came arose from an early decision that at the time seemed fairly
mild, but gradually grew and grew over time: when I decided to extend the
meaning of the fsalex_syntax () call to not only include the parameters
named in the call (reg_syntax_t, case folding, eol char), but to also
capture the locale in force at that point, and make the lexer's behaviour
obey that locale, even if the locale was subsequently changed.

Another example of data interchange is that the lexer builds structures to
model multibyte character sets, but (at the time or writing) does not provide
an interface for clients to access this information.  The need for many and
varied interface function has been steadily growing, and this is just another
example.

The breaking point came when I saw that the parser needed to know if it was
in a unibyte or multibyte locale (currently done by testing MB_CUR_MAX > 1).
If the lexer was the authority on the locale, then the parser needed to
query the lexer, since, because of the  pluggable-lexer architecture I'd
created, the lexer was already the authority, but needed to share its
information with others.  The number of specialised get/set functions needed
was out of control, and, if a pluggable architecture was to be maintained,the
number had to be drastically reduced.

Hence, this module, that defines the interface/protocol that any lexer must
serve, and which allows a lexer and its client to exchange information and
negotiate in a flexible fashion.  This comes at the extent of type safety in
many cases, but makes data exchange more formal.

This module only models essential exchanges between the parser and lexer
instances, and encourages the user to negotiate directly with the modules
for unrelated topics.  For example, the parser does not need to know what the
original plain-text pattern was; it receives a full description of the
pattern via the lexer tokens and associated parameters/structures.  So, there
is no "pattern" interchange described here:  The client can work directly
with the lexer.

The design is to split interactions between modules into two sets:  For code
that is so central to the design of the modules on each side, where an
indirect approach would tend to significantly complicate the exchange, and/or
where efficiency is a central concern, then a direct function is provided to
facilitate each such possible exchange.

The other set consists of the majority of cases (by type, but usually not by
runtime volume):  For these cases, a generic "exchange" function is provided
by the lexer, and is plugged into the parser.  This function has parameters
and return values that permit the parties to exchange information, although
some strict type safety is sacrificed where memory pointers can cross the
boundary.  A slightly simplified definition of the exchange function is:

    int lexer_provided_exchange (void *lexer_instance_context,
                                 exchange_opcode_enum opcode,
                                 void *generic_parameter);

The opcode defines what meaning (if any) is assigned to the parameter and/or
to the return value.  The exchange function, these opcodes, and the required
meaning the values involved are formally defined by this module, and both the
lexer and the parser must conform to this protocol/interface.  By having this
module as an independent entity, it reinforces the independence of each party
in the arrangement, and can ease the creation of alternatives (e.g. perhaps
use a much simpler lexer for "grep -F" in a unibyte locale?).

On the downside, some of the initial opcodes/exchanges I'm writing here are
obviously directly lifted from the existing dfa.c lexer/parser interactions
(e.g. the multibyte sets struct(s)).  I'm hoping that this will not be a
fatal flaw, but in the final analysis this module is a side-effect of
requiring a pluggable lexer architecture, and this may not be acceptable to
others.  */


#ifndef PROTO_LEXPARSE_H
#define PROTO_LEXPARSE_H 1

/* Always import environment-specific configuration items first.  */
#include <config.h>

/* The lexer returns a token, defined by fsatoken.  */
#include "fsatoken.h"

/* Define opcodes for lexer/parser exchanges.  */
typedef enum proto_lexparse_opcode_enum
{
/*  Possible future opcode: PROTO_LEXPARSE_OP_GET_LOCALE,  entire
    locale from uselocale/duplocale */
  PROTO_LEXPARSE_OP_GET_IS_MULTIBYTE_LOCALE,
  PROTO_LEXPARSE_OP_GET_REPMN_MIN,
  PROTO_LEXPARSE_OP_GET_REPMN_MAX,
  PROTO_LEXPARSE_OP_GET_WIDE_CHAR_LIST_MAX,
  PROTO_LEXPARSE_OP_GET_WIDE_CHARS,
  PROTO_LEXPARSE_OP_GET_DOTCLASS,
  PROTO_LEXPARSE_OP_GET_MBCSET,
} proto_lexparse_opcode_t;

/* Most opcodes above are self-documenting; the "WIDE_CHAR(S)"
   items require a little more explanation:
   1. When the lexer returns a WCHAR token, there is at least one
      explicit character (specified in the pattern), plus a list of
      zero or more alternate characters that also match (usually
      because case folding is selected).  The parser should use
      GET_WIDE_CHARS in response to a WCHAR token to fetch this list
      (including the explicit character as the first element of the
      list); and
   2. The GET_WIDE_CHAR_LIST_MAX is provided so that the parser can
      allocate storage for the longest possible list, either as a
      once-off allocation during startup, or on a case-by-case
      basis.  */
/* Declare prototypes for main lex function (lex (), fetch a token),
   and the exchange function.  */
typedef fsatoken_token_t proto_lexparse_lex_fn_t (void *lexer_context);

typedef int proto_lexparse_exchange_fn_t (void *lexer_context,
                                          proto_lexparse_opcode_t opcode,
                                          void *parameter);


#endif /* PROTO_LEXPARSE_H */

/* vim:set shiftwidth=2: */

--------------050008000707080900010407
Content-Type: text/plain; charset=us-ascii;
 name="Makefile.am"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="Makefile.am"

IyMgUHJvY2VzcyB0aGlzIGZpbGUgd2l0aCBhdXRvbWFrZSB0byBjcmVhdGUgTWFrZWZpbGUu
aW4KIyBDb3B5cmlnaHQgMTk5Ny0xOTk4LCAyMDA1LTIwMTQgRnJlZSBTb2Z0d2FyZSBGb3Vu
ZGF0aW9uLCBJbmMuCiMKIyBUaGlzIHByb2dyYW0gaXMgZnJlZSBzb2Z0d2FyZTsgeW91IGNh
biByZWRpc3RyaWJ1dGUgaXQgYW5kL29yIG1vZGlmeQojIGl0IHVuZGVyIHRoZSB0ZXJtcyBv
ZiB0aGUgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgYXMgcHVibGlzaGVkIGJ5CiMgdGhl
IEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbjsgZWl0aGVyIHZlcnNpb24gMywgb3IgKGF0IHlv
dXIgb3B0aW9uKQojIGFueSBsYXRlciB2ZXJzaW9uLgojCiMgVGhpcyBwcm9ncmFtIGlzIGRp
c3RyaWJ1dGVkIGluIHRoZSBob3BlIHRoYXQgaXQgd2lsbCBiZSB1c2VmdWwsCiMgYnV0IFdJ
VEhPVVQgQU5ZIFdBUlJBTlRZOyB3aXRob3V0IGV2ZW4gdGhlIGltcGxpZWQgd2FycmFudHkg
b2YKIyBNRVJDSEFOVEFCSUxJVFkgb3IgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFSIFBVUlBP
U0UuICBTZWUgdGhlCiMgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgZm9yIG1vcmUgZGV0
YWlscy4KIwojIFlvdSBzaG91bGQgaGF2ZSByZWNlaXZlZCBhIGNvcHkgb2YgdGhlIEdOVSBH
ZW5lcmFsIFB1YmxpYyBMaWNlbnNlCiMgYWxvbmcgd2l0aCB0aGlzIHByb2dyYW0uICBJZiBu
b3QsIHNlZSA8aHR0cDovL3d3dy5nbnUub3JnL2xpY2Vuc2VzLz4uCgpMTiA9IGxuCgpBTV9D
RkxBR1MgPSAkKFdBUk5fQ0ZMQUdTKSAkKFdFUlJPUl9DRkxBR1MpCgojIFRlbGwgdGhlIGxp
bmtlciB0byBvbWl0IHJlZmVyZW5jZXMgdG8gdW51c2VkIHNoYXJlZCBsaWJyYXJpZXMuCkFN
X0xERkxBR1MgPSAkKElHTk9SRV9VTlVTRURfTElCUkFSSUVTX0NGTEFHUykKCmJpbl9QUk9H
UkFNUyA9IGdyZXAKYmluX1NDUklQVFMgPSBlZ3JlcCBmZ3JlcApncmVwX1NPVVJDRVMgPSBn
cmVwLmMgc2VhcmNodXRpbHMuYyBcCiAgICAgICAgICBjaGFyY2xhc3MuYyBmc2F0b2tlbi5j
IGZzYWxleC5jIGZzYXBhcnNlLmMgZnNhbXVzdHMuYyBtYmNzZXRzLmMgZGZhLXBybC5jIGRm
YXNlYXJjaC5jIFwKICAgICAgICAgIGt3c2V0LmMga3dzZWFyY2guYyBcCiAgICAgICAgICBw
Y3Jlc2VhcmNoLmMKbm9pbnN0X0hFQURFUlMgPSBncmVwLmggZGZhLmgga3dzZXQuaCBzZWFy
Y2guaCBzeXN0ZW0uaAoKIyBTb21ldGltZXMsIHRoZSBleHBhbnNpb24gb2YgJChMSUJJTlRM
KSBpbmNsdWRlcyAtbGMgd2hpY2ggbWF5CiMgaW5jbHVkZSBtb2R1bGVzIGRlZmluaW5nIHZh
cmlhYmxlcyBsaWtlICdvcHRpbmQnLCBzbyBsaWJncmVwdXRpbHMuYQojIG11c3QgcHJlY2Vk
ZSAkKExJQklOVEwpIGluIG9yZGVyIHRvIGVuc3VyZSB3ZSB1c2UgR05VIGdldG9wdC4KIyBC
dXQgbGliZ3JlcHV0aWxzLmEgbXVzdCBhbHNvIGZvbGxvdyAkKExJQklOVEwpLCBzaW5jZSBs
aWJpbnRsIHVzZXMKIyByZXBsYWNlbWVudCBmdW5jdGlvbnMgZGVmaW5lZCBpbiBsaWJncmVw
dXRpbHMuYS4KTERBREQgPSBcCiAgLi4vbGliL2xpYmdyZXB1dGlscy5hICQoTElCSU5UTCkg
Li4vbGliL2xpYmdyZXB1dGlscy5hICQoTElCSUNPTlYpIFwKICAkKExJQlRIUkVBRCkKCmdy
ZXBfTERBREQgPSAkKExEQUREKSAkKExJQl9QQ1JFKQpsb2NhbGVkaXIgPSAkKGRhdGFkaXIp
L2xvY2FsZQpBTV9DUFBGTEFHUyA9IC1JJCh0b3BfYnVpbGRkaXIpL2xpYiAtSSQodG9wX3Ny
Y2RpcikvbGliCgpFWFRSQV9ESVNUID0gZG9zYnVmLmMgZWdyZXAuc2gKCmVncmVwIGZncmVw
OiBlZ3JlcC5zaCBNYWtlZmlsZQoJJChBTV9WX0dFTilncmVwPWBlY2hvIGdyZXAgfCBzZWQg
LWUgJyQodHJhbnNmb3JtKSdgCSYmIFwKCWNhc2UgJEAgaW4gZWdyZXApIG9wdGlvbj0tRTs7
IGZncmVwKSBvcHRpb249LUY7OyBlc2FjCSYmIFwKCXNoZWxsX2RvZXNfc3Vic3RyaW5ncz0n
c2V0IHgveSAmJiBkPSQkezElLyp9ICYmIHRlc3QgIiQkZCIgPSB4JyAmJiBcCglpZiAkKFNI
RUxMKSAtYyAiJCRzaGVsbF9kb2VzX3N1YnN0cmluZ3MiIDI+L2Rldi9udWxsOyB0aGVuIFwK
CSAgZWRpdF9zdWJzdHJpbmc9J3MsWCxYLCc7IFwKCWVsc2UgXAoJICBlZGl0X3N1YnN0cmlu
Zz0ncyxcJCR7MCUvXCp9LGBleHByICJYJCQwIiA6ICdcJydYXFwoLipcXCkvJ1wnJ2AsZyc7
IFwKCWZpICYmIFwKCXNlZCAtZSAnc3xbQF1TSEVMTEB8JChTSEVMTCl8ZycgXAoJICAgIC1l
ICIkJGVkaXRfc3Vic3RyaW5nIiBcCgkgICAgLWUgInN8W0BdZ3JlcEB8JCRncmVwfGciIFwK
CSAgICAtZSAic3xbQF1vcHRpb25AfCQkb3B0aW9ufGciIDwkKHNyY2RpcikvZWdyZXAuc2gg
PiRALXQKCSQoQU1fVl9hdCljaG1vZCAreCAkQC10CgkkKEFNX1ZfYXQpbXYgJEAtdCAkQAoK
Q0xFQU5GSUxFUyA9IGVncmVwIGZncmVwICotdAo=
--------------050008000707080900010407--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: behoffski <behoffski@HIDDEN>
Subject: bug#17712: Acknowledgement (Files created/modified by "untangle"
 script)
Message-ID: <handler.17712.B.140202733014848.ack <at> debbugs.gnu.org>
References: <53913CF2.2000404@HIDDEN>
X-Gnu-PR-Message: ack 17712
X-Gnu-PR-Package: grep
Reply-To: 17712 <at> debbugs.gnu.org
Date: Fri, 06 Jun 2014 04:03:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-grep@HIDDEN

If you wish to submit further information on this problem, please
send it to 17712 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
17712: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D17712
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 7 Jun 2014 22:29:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 07 18:29:58 2014
Received: from localhost ([127.0.0.1]:40816 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1WtP7d-0004Cu-NN
	for submit <at> debbugs.gnu.org; Sat, 07 Jun 2014 18:29:58 -0400
Received: from smtp.cs.ucla.edu ([131.179.128.62]:45820)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eggert@HIDDEN>) id 1WtP7b-0004CY-6g
 for control <at> debbugs.gnu.org; Sat, 07 Jun 2014 18:29:55 -0400
Received: from localhost (localhost.localdomain [127.0.0.1])
 by smtp.cs.ucla.edu (Postfix) with ESMTP id BA628A60005
 for <control <at> debbugs.gnu.org>; Sat,  7 Jun 2014 15:29:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu
Received: from smtp.cs.ucla.edu ([127.0.0.1])
 by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id WXqXI42zsHwU for <control <at> debbugs.gnu.org>;
 Sat,  7 Jun 2014 15:29:41 -0700 (PDT)
Received: from [192.168.1.9] (pool-108-0-233-62.lsanca.fios.verizon.net
 [108.0.233.62])
 by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 45BD039E8013
 for <control <at> debbugs.gnu.org>; Sat,  7 Jun 2014 15:29:41 -0700 (PDT)
Message-ID: <53939254.9010602@HIDDEN>
Date: Sat, 07 Jun 2014 15:29:40 -0700
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: control <at> debbugs.gnu.org
Subject: severity adjustment
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -3.0 (---)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.0 (---)

severity 17712 wishlist
severity 17711 wishlist

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.