GNU bug report logs - #28666
sed 4.2.2 testsuite binary3 test never returns on powerpc64-unknown-linux-gnu

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28666 in the body.
You can then email your comments to 28666 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dennis Clarke <dclarke <at> blastwave.org>
To: bug-sed <at> gnu.org
Subject: sed 4.2.2 testsuite binary3 test never returns on
 powerpc64-unknown-linux-gnu
Date: Sun, 1 Oct 2017 22:06:49 -0400

System is a Debian 8.8 powerpc64 type system on a PowerMac G5 and the
test phase of sed 4.2.2 becomes "wedged" in test binary3 with 100% cpu
usage on a single core. Waited for 2 hours however it never returns.

Is there a way to run this as a single isolated test?

I see this ....

.
.
.
XFAIL: utf8-4
PASS: badenc
PASS: inplace-hold
PASS: brackets
PASS: amp-escape
PASS: help
PASS: version
PASS: file
PASS: quiet
PASS: factor
^C


ppc64_deb8_$ ps -ef | grep "sed"
root       595     1  0 Aug27 ?        00:00:00 /usr/sbin/cups-browsed
dclarke   8794  1766  0 19:30 pts/1    00:00:00 tee 
../sed-4.2.2_linux_3.16.0-4-
dclarke  10796 10792  0 19:30 pts/1    00:00:00 /usr/local/bin/gmake 
SED=../sed/
dclarke  10797 10796  0 19:30 pts/1    00:00:00 /bin/sh -c LC_ALL=C 
../sed/sed
dclarke  10798 10797 99 19:30 pts/1    00:04:46 ../sed/sed -n -f 
./binary3.sed
dclarke  10803  1671  0 19:35 pts/0    00:00:00 grep sed
ppc64_deb8_$


Waited and waited.  Never returns.


Dennis Clarke

Message #8 received at 28666 <at> debbugs.gnu.org (full text, mbox):

From: Dennis Clarke <dclarke <at> blastwave.org>
To: 28666 <at> debbugs.gnu.org
Subject: More info regarding binary3 sed test which hangs
Date: Sun, 1 Oct 2017 22:28:22 -0400

I looked into the binary3 test a bit and tried this manually :

deb8_ppc64$
deb8_ppc64$ LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp | 
LC_ALL=C tr -d \\r > binary3.out_manual


It just runs endlessly and never returns.

Not sure what data to add in here.

Message #11 received at 28666 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Dennis Clarke <dclarke <at> blastwave.org>
Cc: 28666 <at> debbugs.gnu.org
Subject: Re: bug#28666: sed 4.2.2 testsuite binary3 test never returns on
 powerpc64-unknown-linux-gnu
Date: Mon, 2 Oct 2017 12:59:04 -0700

On Sun, Oct 1, 2017 at 7:06 PM, Dennis Clarke <dclarke <at> blastwave.org> wrote:
>
> System is a Debian 8.8 powerpc64 type system on a PowerMac G5 and the
> test phase of sed 4.2.2 becomes "wedged" in test binary3 with 100% cpu
> usage on a single core. Waited for 2 hours however it never returns.
>
> Is there a way to run this as a single isolated test?
>
> I see this ....
>
> .
> .
> .
> XFAIL: utf8-4
> PASS: badenc
> PASS: inplace-hold
> PASS: brackets
> PASS: amp-escape
> PASS: help
> PASS: version
> PASS: file
> PASS: quiet
> PASS: factor
> ^C
>
>
> ppc64_deb8_$ ps -ef | grep "sed"
> root       595     1  0 Aug27 ?        00:00:00 /usr/sbin/cups-browsed
> dclarke   8794  1766  0 19:30 pts/1    00:00:00 tee
> ../sed-4.2.2_linux_3.16.0-4-
> dclarke  10796 10792  0 19:30 pts/1    00:00:00 /usr/local/bin/gmake
> SED=../sed/
> dclarke  10797 10796  0 19:30 pts/1    00:00:00 /bin/sh -c LC_ALL=C
> ../sed/sed
> dclarke  10798 10797 99 19:30 pts/1    00:04:46 ../sed/sed -n -f
> ./binary3.sed
> dclarke  10803  1671  0 19:35 pts/0    00:00:00 grep sed
> ppc64_deb8_$
>
>
> Waited and waited.  Never returns.

Thank you for the report, and especially for testing on a less-common
architecture.

Would you please try again but with the latest release: sed-4.4?
There were a lot of fixes between the version you used and 4.4.
Here's the announcement for sed-4.4:
  https://savannah.gnu.org/forum/forum.php?forum_id=8793

Message #14 received at 28666 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Dennis Clarke <dclarke <at> blastwave.org>
Cc: 28666 <at> debbugs.gnu.org
Subject: Re: bug#28666: More info regarding binary3 sed test which hangs
Date: Wed, 4 Oct 2017 08:15:26 -0700

On Sun, Oct 1, 2017 at 7:28 PM, Dennis Clarke <dclarke <at> blastwave.org> wrote:
> I looked into the binary3 test a bit and tried this manually :
>
> deb8_ppc64$
> deb8_ppc64$ LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp | LC_ALL=C
> tr -d \\r > binary3.out_manual
>
> It just runs endlessly and never returns.
>
> Not sure what data to add in here.

Thank you for the details. I presume this is with the latest?
Does it also infloop if you remove the output pipe:

  LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp > out

Can you run it via gdb and hit ^C then see where/why it's looping?

  LC_ALL=C gdb --args ../sed/sed

to run it under gdb, do this:

  (gdb) run -n -f ./binary3.sed < binary.inp

Message #17 received at 28666 <at> debbugs.gnu.org (full text, mbox):

From: Dennis Clarke <dclarke <at> blastwave.org>
To: Jim Meyering <jim <at> meyering.net>
Cc: 28666 <at> debbugs.gnu.org
Subject: Re: bug#28666: More info regarding binary3 sed test which hangs
Date: Wed, 4 Oct 2017 14:26:26 -0400

On 10/04/2017 11:15 AM, Jim Meyering wrote:
> On Sun, Oct 1, 2017 at 7:28 PM, Dennis Clarke <dclarke <at> blastwave.org> wrote:
>> I looked into the binary3 test a bit and tried this manually :
>>
>> deb8_ppc64$
>> deb8_ppc64$ LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp | LC_ALL=C
>> tr -d \\r > binary3.out_manual
>>
>> It just runs endlessly and never returns.
>>
>> Not sure what data to add in here.
> 
> Thank you for the details. I presume this is with the latest?

Direct from the release tarball. I also did try a git pull but that
 resulted in other problems.

> Does it also infloop if you remove the output pipe:
> 
>    LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp > out
> 
> Can you run it via gdb and hit ^C then see where/why it's looping?
> 
>    LC_ALL=C gdb --args ../sed/sed
> 
> to run it under gdb, do this:
> 
>    (gdb) run -n -f ./binary3.sed < binary.inp
> 

Well, here comes the frustrating problem : it all works now.  :-\

Why?

Your guess is as good as mine but there seems to have been a need for a
lot of locales to be installed as well as an old old dusty configure
option to be removed.   This is really long :

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28665



Assaf Gordon says that "--enable-regex-tests" should not be used.

quote:
    These 'regex' tests are very old, and have not been maintained
    since about 2004. We simply haven't gotten around to either fix
    them or remove them.

Also :

> FAIL: testsuite/utf8-ru.sh

Assaf Gordon says :
    This is one test that will certainly remain even after removing
    the "regex tests".

    As a first step to understand if this is a false positive or
    a sed failure, can you tell us which locales are installed
    on your computer ("locale -a") ?

To this I can tell you that I only had the essentials. A few utf8 and
not many of them.

Assaf Gordon also says :

    I see the test uses "ru_RU.UTF-8", and it's possible it uses the
    locale without first verifying it is available on the system.


I believe that must be true.

After I installed ALL the locale options that I could get as well as
removed the configure option for regex tests we get a clean configure
and a clean compile and testsuite.

So here I am baffled as to what infinite loops were running. I can tell
you that I was running "htop" in an xterm while the system tried and
tried ( vainly ) to perform a gcc bootstrap.  The htop showed me that
there were sed binary tests that had been running for 36 hours plus. I
have done a kill -9 on them two days ago but what result I got from that 
was that the ppid went away to be replaced by pid 1 and the binary tests 
were still running. I was amazed. I ran kill -9 on them again and again 
and was eventually able to make them go away.

Now I can not reproduce them.

I really hate problems like this that seem to be based on some minor
issue with locale or a bad configure option or it is a Tuesday and your
mother's birthday.  Makes no sense whatsoever to try to reproduce now
because it simply runs and run fine.

However let's see what happens anyways :

I will look into my list of attempts over and over for the past 3 days.

deb8_ppc64$ ls -loptr | grep "sed" | grep "^drw" | cut -c31-96
Oct  1 22:55 sed-4.4_linux_3.16.0-4-powerpc64.001/
Oct  1 23:24 sed-4.3_linux_3.16.0-4-powerpc64.001/
Oct  2 11:21 sed-4.4.84-a59fb_linux_3.16.0-4-powerpc64_ppc64.001/
Oct  2 12:06 sed-4.2.2_linux_3.16.0-4-powerpc64.001/
Oct  3 17:27 sed-4.4_linux_3.16.0-4-powerpc64.002/
Oct  3 17:35 sed-4.4_linux_3.16.0-4-powerpc64.002.pkg/
Oct  3 17:40 sed-4.4_linux_3.16.0-4-powerpc64.003/
Oct  3 17:43 sed-4.4_linux_3.16.0-4-powerpc64.003.pkg/

The last two directories that have the suffix ".pkg" are fully clean
and successful builds and tests.  I was so astonished that I did it
twice. Also, installed into /usr/local which is what I really wanted.

Let's go back in time a few days and try the test from within the
first point of failure :

deb8_ppc64$ cd sed-4.4_linux_3.16.0-4-powerpc64.001/testsuite
deb8_ppc64$ LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp > out

gone ... never to return.

deb8_ppc64$ file out
out: empty

It seems to produce nothing.  Let's reach for gdb ( which I also had to
build from sources myself ) :


deb8_ppc64$ LC_ALL=C gdb --args ../sed/sed
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../sed/sed...done.
(gdb) run -n -f ./binary3.sed < binary.inp
Starting program: 
/usr/local/build/sed-4.4_linux_3.16.0-4-powerpc64.001/sed/sed -n -f 
./binary3.sed < binary.inp

Full 100% CPU usage on a single core and into the infinite depths we go.

Wait a full minute and hit CTRL-C :

^C
Program received signal SIGINT, Interrupt.
match_regex (regex=0x10071b20, buf=0x10040080 
"aa-\nb;9876543210aaaaaaaaaR| D\n\n192\n168\n18", buflen=25, 
buf_start_offset=0,
    regarray=0x0, regsize=0) at sed/regexp.c:356
356       if ((regex->flags & REG_NEWLINE) && buffer_delimiter != '\n')

(gdb) print regex
$1 = (struct regex *) 0x10071b20

(gdb) print *regex
$3 = {pattern = {buffer = 0x10074930, allocated = 224, used = 224, 
syntax = 50790982, fastmap = 0x10071b90 "", translate = 0x0,
    re_nsub = 0, can_be_null = 0, regs_allocated = 1, fastmap_accurate 
= 1, no_sub = 1, not_bol = 0, not_eol = 0,
    newline_anchor = 0}, flags = 0, sz = 2, dfa = 0x10075440, begline = 
false, endline = false, re = "^"}
(gdb)

OKay ... this is at least something to look at. Still makes me wonder
what loop we are in.

Dennis

Message #20 received at 28666 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Dennis Clarke <dclarke <at> blastwave.org>, Jim Meyering <jim <at> meyering.net>
Cc: 28666 <at> debbugs.gnu.org
Subject: Re: bug#28666: More info regarding binary3 sed test which hangs
Date: Sat, 28 Oct 2017 22:03:25 -0600

Hello Dennis,

On 2017-10-04 12:26 PM, Dennis Clarke wrote:
> On 10/04/2017 11:15 AM, Jim Meyering wrote:
> 
>> Does it also infloop if you remove the output pipe:
>>
>>    LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp > out
>>
> Well, here comes the frustrating problem : it all works now.  :-\

I've tested the same on a Linux PowerPC64 (power7) machine and I'm not 
able to reproduce it, either with sed-4.4 or the latest git version of sed.

With both versions I see the following:

  $ ./sed/sed -n -f ./testsuite/binary3.sed < ./testsuite/binary.inp
  192
  168
  1
  0
  192
  168
  1
  255

With the latest sed, the file 'binary.inp' does not exist (it is 
generated on-the fly), and I've tested it both with 'binary.inp' from 
the previous release, and by running the new tests with:

  make check TESTS=testsuite/binary.sh SUBDIRS=.


Are you still able to reproduce the infloop on the previous release of 
sed (sed version 4.4) ?


regards,
 - assaf

Message #25 received at 28666-done <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Dennis Clarke <dclarke <at> blastwave.org>, Jim Meyering <jim <at> meyering.net>
Cc: 28666-done <at> debbugs.gnu.org
Subject: Re: bug#28666: More info regarding binary3 sed test which hangs
Date: Mon, 8 Oct 2018 14:11:53 -0600

With no further response, I'm closing this bug.

If the issue persists, we'll revisit it.

Message #28 received at 28666-done <at> debbugs.gnu.org (full text, mbox):

From: Dennis Clarke <dclarke <at> blastwave.org>
To: Assaf Gordon <assafgordon <at> gmail.com>, Jim Meyering <jim <at> meyering.net>
Cc: 28666-done <at> debbugs.gnu.org
Subject: Re: bug#28666: More info regarding binary3 sed test which hangs
Date: Mon, 8 Oct 2018 17:12:24 -0400

On 10/08/2018 04:11 PM, Assaf Gordon wrote:
> With no further response, I'm closing this bug.
> 
> If the issue persists, we'll revisit it.
> 

Go ahead. I have long since managed to get a clean test and build on
a Debian ppc64 sid machine with gcc 8.1.0.

Dennis

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.