GNU bug report logs - #17512
Valgrind and regression tests?

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Package: grep; Severity: wishlist; Reported by: behoffski <behoffski@HIDDEN>; Keywords: notabug; dated Fri, 16 May 2014 21:04:02 UTC; Maintainer for grep is bug-grep@HIDDEN.

Message received at 17512 <at> debbugs.gnu.org:


Received: (at 17512) by debbugs.gnu.org; 23 May 2014 08:28:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 23 04:28:41 2014
Received: from localhost ([127.0.0.1]:57253 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1WnkqG-0001JC-NT
	for submit <at> debbugs.gnu.org; Fri, 23 May 2014 04:28:41 -0400
Received: from ipmail05.adl6.internode.on.net ([150.101.137.143]:60387)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <behoffski@HIDDEN>) id 1WnkqD-0001Iw-5C
 for 17512 <at> debbugs.gnu.org; Fri, 23 May 2014 04:28:38 -0400
Received: from ppp14-2-51-95.lns21.adl2.internode.on.net (HELO [192.168.1.1])
 ([14.2.51.95])
 by ipmail05.adl6.internode.on.net with ESMTP; 23 May 2014 17:58:32 +0930
Message-ID: <537F06AE.3000105@HIDDEN>
Date: Fri, 23 May 2014 17:58:30 +0930
From: behoffski <behoffski@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: Jim Meyering <jim@HIDDEN>, Paul Eggert <eggert@HIDDEN>
Subject: Re: bug#17512: Valgrind/regression tests: Symlink test hang
References: <53767195.4070108@HIDDEN> <5376AEE2.6000801@HIDDEN>
 <CA+8g5KE7Sg3FFS=GCm0MrDOpvmYSpASzEkxj_fc2jEKu-cr1eA@HIDDEN>
In-Reply-To: <CA+8g5KE7Sg3FFS=GCm0MrDOpvmYSpASzEkxj_fc2jEKu-cr1eA@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 17512
Cc: 17512 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)

On 05/21/14 14:40, Jim Meyering wrote:
> On Fri, May 16, 2014 at 5:35 PM, Paul Eggert <eggert@HIDDEN> wrote:
>>
>> [...] Perhaps Jim has a better way to do it, though -- he's more of a valgrind
>> wizard than I.
>
> I use a script to replace each binary with this generated script:
>
> #!/bin/sh
> saved_binaries_dir="${0%/*}/.vg-tmp"
> export PATH=$saved_binaries_dir:$PATH
> prog=${0##*/}
> # Disable this: http://udrepper.livejournal.com/11429.html
> MALLOC_PERTURB_=0
> export MALLOC_PERTURB_
> exec /usr/bin/valgrind -v --suppressions=/dev/null \
>    --log-file=/tmp/vg-log-2014-05-20.22:05:28-%q{GREP_TEST_NAME} \
>    --leak-check=full --track-fds=yes --num-callers=15 -- "$prog" "$@"
>
> which refers to the original binary via a subdirectory named .vg-tmp
> relative to the original location of the binary. That script also moves
> each binary into the .vg-tmp subdir.
>
> Then, running "make check" creates many /tmp/vg-log-* files,
> each related to the name of a test.
>

G'day Jim and Paul,

Thanks for the feedback.  I found that Jim's script worked in a similar
fashion to the home-brew script I tried, although I didn't know that
the test name was available as an environment variable, and so my
script ended up creating one massive log file.

One glitch that appears in both my script and in the one given by Jim:
The "symlink" test hangs, consuming 0% CPU, when run under Valgrind.
I ended up replacing the test with an "exit 0" script just for the sake
of getting a fast sweep across the majority of the test rig.

I haven't tried to hunt down the reason for this hang; I'm currently
on a Gentoo Link machine, with GCC and Valgrind versions:

         gcc (Gentoo 4.7.3-r1 p1.4, pie-0.5.5) 4.7.3
         Copyright (C) 2012 Free Software Foundation, Inc.
         This is free software; see the source for copying conditions.  There is NO
         warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

         valgrind-3.8.1

While there's some 4.8-series GCC versions available (but masked
for my architecture at present), that include valgrind-related
goodies such as "-Og" and "-fsanitize=address", I'm choosing a more
conservative version at present.  There's also masked versions
of Valgrind available (3.8.1-r1, 3.9.0), but again I'm being conservative
at present.

I'll wait for grep-2.20 to appear, and then will investigate whether
upgrading various tools can resolve this hang, and will report
back to the list.

cheers,

behoffski (Brenton Hoff)
Programmer, Grouse Software

ps. Congratulations to everybody on the completion of grep-2.19.
Information forwarded to bug-grep@HIDDEN:
bug#17512; Package grep. Full text available.

Message received at 17512 <at> debbugs.gnu.org:


Received: (at 17512) by debbugs.gnu.org; 21 May 2014 05:10:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed May 21 01:10:35 2014
Received: from localhost ([127.0.0.1]:54828 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1WmynS-0001zP-9H
	for submit <at> debbugs.gnu.org; Wed, 21 May 2014 01:10:35 -0400
Received: from mail-wi0-f180.google.com ([209.85.212.180]:56823)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <meyering@HIDDEN>) id 1WmynM-0001z1-VX
 for 17512 <at> debbugs.gnu.org; Wed, 21 May 2014 01:10:29 -0400
Received: by mail-wi0-f180.google.com with SMTP id hi2so2091151wib.7
 for <17512 <at> debbugs.gnu.org>; Tue, 20 May 2014 22:10:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:sender:in-reply-to:references:from:date:message-id
 :subject:to:cc:content-type;
 bh=RxITGb0HhgjcP4M5D2svBiQFbs1LA6y1rhbE4zrn/Z8=;
 b=q8ZYo/OqAfJSEq5HieSch+DiRksaarix6cn4hrbSGf+R1JcZNN0ZLhKPPMBKC6kWHX
 B3ahwQ4LkSKkVjkYuJbvkERSzqvE279eEJx6JLulU2/eEsBw3W9alRSt6oeP1AXmouka
 ksZH4tAAGSZi0g4Ug0vfcWSoHUwJtzoyui0/ExerE9je+8xCaFQtSSvUS47tWYQisRT3
 HXgSYVzG/CGEi/OuL0iswLo2zDdyeQToBF3F1eBIAEJguW2+8O54RuQjRrqJyojKZA66
 /4FNzg600hHwsfs0mt+ikeXQrDXjJPBgz1deUymjrAX2agCGEwsvYTf/MVLO4vx6uUfV
 eA2g==
X-Received: by 10.180.109.69 with SMTP id hq5mr8128272wib.30.1400649023145;
 Tue, 20 May 2014 22:10:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.194.81.102 with HTTP; Tue, 20 May 2014 22:10:03 -0700 (PDT)
In-Reply-To: <5376AEE2.6000801@HIDDEN>
References: <53767195.4070108@HIDDEN> <5376AEE2.6000801@HIDDEN>
From: Jim Meyering <jim@HIDDEN>
Date: Tue, 20 May 2014 22:10:03 -0700
X-Google-Sender-Auth: tPqCNjHPFzgaO-U3bQnFKch8FSA
Message-ID: <CA+8g5KE7Sg3FFS=GCm0MrDOpvmYSpASzEkxj_fc2jEKu-cr1eA@HIDDEN>
Subject: Re: bug#17512: Valgrind and regression tests?
To: Paul Eggert <eggert@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 17512
Cc: 17512 <at> debbugs.gnu.org, behoffski <behoffski@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

On Fri, May 16, 2014 at 5:35 PM, Paul Eggert <eggert@HIDDEN> wrote:
> behoffski wrote:
>>
>>    1. Use a shell alias:
>>
>>         grep="valgrind --log-file=foo grep" make check
>
>
> That won't work, as too many parts of the system ignore the aliases.
>
>
>>    2. Replace grep/egrep/fgrep as appropriate to execute the program
>>       within a shell, where valgrind plus any options is incorporated
>>       into the program invocation and the logfile is treated
>>       appropriately in the program invocation aftermath.
>
>
> Yes, something like that should work.  You can set up your own
> implementation of grep, like this:
>
> #! /bin/sh
> exec valgrind --log-file=foo /full/path/to/actual/grep "$@"
>
> in a shell script called 'grep', in a different directory, and prepand that
> directory to your PATH.
>
> Perhaps Jim has a better way to do it, though -- he's more of a valgrind
> wizard than I.

I use a script to replace each binary with this generated script:

#!/bin/sh
saved_binaries_dir="${0%/*}/.vg-tmp"
export PATH=$saved_binaries_dir:$PATH
prog=${0##*/}
# Disable this: http://udrepper.livejournal.com/11429.html
MALLOC_PERTURB_=0
export MALLOC_PERTURB_
exec /usr/bin/valgrind -v --suppressions=/dev/null \
  --log-file=/tmp/vg-log-2014-05-20.22:05:28-%q{GREP_TEST_NAME} \
  --leak-check=full --track-fds=yes --num-callers=15 -- "$prog" "$@"

which refers to the original binary via a subdirectory named .vg-tmp
relative to the original location of the binary. That script also moves
each binary into the .vg-tmp subdir.

Then, running "make check" creates many /tmp/vg-log-* files,
each related to the name of a test.
Information forwarded to bug-grep@HIDDEN:
bug#17512; Package grep. Full text available.
Severity set to 'wishlist' from 'normal' Request was from Paul Eggert <eggert@HIDDEN> to control <at> debbugs.gnu.org. Full text available.
Added tag(s) notabug. Request was from Paul Eggert <eggert@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 17512 <at> debbugs.gnu.org:


Received: (at 17512) by debbugs.gnu.org; 17 May 2014 00:36:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 16 20:36:03 2014
Received: from localhost ([127.0.0.1]:50936 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1WlSba-0004gj-W6
	for submit <at> debbugs.gnu.org; Fri, 16 May 2014 20:36:03 -0400
Received: from smtp.cs.ucla.edu ([131.179.128.62]:53531)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eggert@HIDDEN>) id 1WlSbY-0004gA-NC
 for 17512 <at> debbugs.gnu.org; Fri, 16 May 2014 20:36:01 -0400
Received: from localhost (localhost.localdomain [127.0.0.1])
 by smtp.cs.ucla.edu (Postfix) with ESMTP id DE73AA60096;
 Fri, 16 May 2014 17:35:54 -0700 (PDT)
X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu
Received: from smtp.cs.ucla.edu ([127.0.0.1])
 by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 2wKviwnw4cnX; Fri, 16 May 2014 17:35:46 -0700 (PDT)
Received: from [192.168.1.9] (pool-108-0-233-62.lsanca.fios.verizon.net
 [108.0.233.62])
 by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 4B8A3A60001;
 Fri, 16 May 2014 17:35:46 -0700 (PDT)
Message-ID: <5376AEE2.6000801@HIDDEN>
Date: Fri, 16 May 2014 17:35:46 -0700
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: behoffski <behoffski@HIDDEN>, 17512 <at> debbugs.gnu.org
Subject: Re: bug#17512: Valgrind and regression tests?
References: <53767195.4070108@HIDDEN>
In-Reply-To: <53767195.4070108@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -3.0 (---)
X-Debbugs-Envelope-To: 17512
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.0 (---)

behoffski wrote:
>    1. Use a shell alias:
>
>         grep="valgrind --log-file=foo grep" make check

That won't work, as too many parts of the system ignore the aliases.

>    2. Replace grep/egrep/fgrep as appropriate to execute the program
>       within a shell, where valgrind plus any options is incorporated
>       into the program invocation and the logfile is treated
>       appropriately in the program invocation aftermath.

Yes, something like that should work.  You can set up your own 
implementation of grep, like this:

#! /bin/sh
exec valgrind --log-file=foo /full/path/to/actual/grep "$@"

in a shell script called 'grep', in a different directory, and prepand 
that directory to your PATH.

Perhaps Jim has a better way to do it, though -- he's more of a valgrind 
wizard than I.
Information forwarded to bug-grep@HIDDEN:
bug#17512; Package grep. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 16 May 2014 21:03:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 16 17:03:09 2014
Received: from localhost ([127.0.0.1]:37711 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1WlPHY-0007hG-KR
	for submit <at> debbugs.gnu.org; Fri, 16 May 2014 17:03:09 -0400
Received: from eggs.gnu.org ([208.118.235.92]:47845)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <behoffski@HIDDEN>) id 1WlPHV-0007gf-OD
 for submit <at> debbugs.gnu.org; Fri, 16 May 2014 17:03:06 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <behoffski@HIDDEN>) id 1WlPHP-0001Hk-Ux
 for submit <at> debbugs.gnu.org; Fri, 16 May 2014 17:03:00 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:36820)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <behoffski@HIDDEN>) id 1WlPHP-0001Hg-Sy
 for submit <at> debbugs.gnu.org; Fri, 16 May 2014 17:02:59 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:49562)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <behoffski@HIDDEN>) id 1WlPHO-0006KV-RG
 for bug-grep@HIDDEN; Fri, 16 May 2014 17:02:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <behoffski@HIDDEN>) id 1WlPHK-0001HF-DM
 for bug-grep@HIDDEN; Fri, 16 May 2014 17:02:58 -0400
Received: from ipmail07.adl2.internode.on.net
 ([2001:44b8:8060:ff02:300:1:2:7]:36911)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <behoffski@HIDDEN>) id 1WlPHK-0001GM-1q
 for bug-grep@HIDDEN; Fri, 16 May 2014 17:02:54 -0400
Received: from ppp14-2-51-95.lns21.adl2.internode.on.net (HELO [192.168.1.1])
 ([14.2.51.95])
 by ipmail07.adl2.internode.on.net with ESMTP; 17 May 2014 05:44:26 +0930
Message-ID: <53767195.4070108@HIDDEN>
Date: Sat, 17 May 2014 05:44:13 +0930
From: behoffski <behoffski@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: bug-grep@HIDDEN
Subject: Valgrind and regression tests?
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address
 (bad octet value).
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

G'day,

I would like to run the entire regression test for Grep (make check),
with Valgrind monitoring each invocation of grep etc, but with the
output going to a separate logfile.  (I have run valgrind for a few
cases, but have not covered the entire spectrum of the regression
test rig.)

The file grep/README-hacking recommends using valgrind, and also
suggests that hackers look at the file "README-valgrind", but no such
file exists.

I note that Valgrind sends its output to stderr by default, and that
the option --log-file=<filename> exists to redirect Valgrind output
elsewhere.  This would be needed in the case of grep et al, as stderr
is used in various ways by the programs.

Is there a simple way of incorporating valgrind into the entire
regression test run?  Two simple ideas come to mind, but I'm not
well versed in the POSIX/shell/makefile/Autoconf/regression rig setup
to know how realistic these are:

   1. Use a shell alias:

        grep="valgrind --log-file=foo grep" make check

   2. Replace grep/egrep/fgrep as appropriate to execute the program
      within a shell, where valgrind plus any options is incorporated
      into the program invocation and the logfile is treated
      appropriately in the program invocation aftermath.

Any hints, help, suggestions etc would be welcome.

cheers,

behoffski (Brenton Hoff)
Programmer, Grouse Software
Acknowledgement sent to behoffski <behoffski@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-grep@HIDDEN. Full text available.
Report forwarded to bug-grep@HIDDEN:
bug#17512; Package grep. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.