GNU logs - #20693, boring messages

Message sent to bug-automake@HIDDEN:

Subject: bug#20693: SKIP not counted in TAP
Resent-From: "Arthur Schwarz" <aschwarz1309@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 29 May 2015 15:16:01 +0000
Resent-Message-ID: <handler.20693.B.14329125463073 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 20693 <at> debbugs.gnu.org
From: "Arthur Schwarz" <aschwarz1309@HIDDEN>
Date: Fri, 29 May 2015 08:15:15 -0700
Message-ID: <88E8A9325FEB4AF482785AD592E22EA3@2600kPC>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Thread-Index: AdCaIkQaF9P3B5iNTc+fjpidDW8Prw==
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>


I think this is an error.

skip causes the following issues:
1: It is not reported.
2: It is not summarized.
3: It must be subtracted from the plan (tests run), and
4: The .log file has TAP generated output and the test data intermixed.

I think this is against the TAP definition?

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =
output =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

PASS: test2.tap 1
FAIL: test2.tap 2
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Testsuite summary for test 0.5
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
See src/test-suite.log
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =
test-suite.log =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D
   test 0.5: src/test-suite.log
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D

# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test2
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

skip
ok
PASS: test2.tap 1
not ok
FAIL: test2.tap 2
1..2

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =
program =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=A0
## Process this file with automake to produce Makefile.in

# select testing format
AUTOMAKE_OPTIONS =3D parallel-tests=20
EXTRA_DIST       =3D tap-driver.sh


# test options
TEST_EXTENSIONS    =3D .tap
TAP_LOG_DRIVER     =3D env AM_TAP_AWK=3D'$(AWK)' $(SHELL) =
./tap-driver.sh

# Testing=20
check_SCRIPTS      =3D test2.tap
TESTS              =3D test2.tap

test2.tap:
	echo '#!/bin/bash'                                  > test2.tap
	echo 'echo test2.tap "$$# [ $$@ ]"   >> test2.log' >> test2.tap
	echo "echo I am a tap script         >> test2.log" >> test2.tap
	echo echo '1..2'                                   >> test2.tap
	echo echo 'ok'                                     >> test2.tap
	echo echo 'not ok'                                 >> test2.tap
	echo echo 'skip'                                   >> test2.tap
	chmod +x test2.tap

CLEANFILES         =3D $(test_SOURCES)

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: "Arthur Schwarz" <aschwarz1309@HIDDEN>
Subject: bug#20693: Acknowledgement (SKIP not counted in TAP)
Message-ID: <handler.20693.B.14329125463073.ack <at> debbugs.gnu.org>
References: <88E8A9325FEB4AF482785AD592E22EA3@2600kPC>
X-Gnu-PR-Message: ack 20693
X-Gnu-PR-Package: automake
Reply-To: 20693 <at> debbugs.gnu.org
Date: Fri, 29 May 2015 15:16:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-automake@HIDDEN

If you wish to submit further information on this problem, please
send it to 20693 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
20693: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D20693
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-automake@HIDDEN:

Subject: bug#20693: SKIP not counted in TAP
Resent-From: Nick Bowler <nbowler@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 29 May 2015 17:13:03 +0000
Resent-Message-ID: <handler.20693.B20693.143291954820132 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Arthur Schwarz <aschwarz1309@HIDDEN>
Cc: 20693 <at> debbugs.gnu.org
Date: Fri, 29 May 2015 13:11:27 -0400
From: Nick Bowler <nbowler@HIDDEN>
Message-ID: <20150529171127.GA27194@HIDDEN>
References: <88E8A9325FEB4AF482785AD592E22EA3@2600kPC>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <88E8A9325FEB4AF482785AD592E22EA3@2600kPC>
Organization: Elliptic Technologies Inc.
User-Agent: Mutt/1.5.23 (2014-03-12)
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On 2015-05-29 08:15 -0700, Arthur Schwarz wrote:
> skip causes the following issues:
> 1: It is not reported.
> 2: It is not summarized.
> 3: It must be subtracted from the plan (tests run), and
> 4: The .log file has TAP generated output and the test data intermixed.
> 
> I think this is against the TAP definition?

In a way, because...

> skip
> ok
> PASS: test2.tap 1
> not ok
> FAIL: test2.tap 2
> 1..2

... it is your program's output that does not follow the specification.

There are two basic styles of skipping in TAP.  You can skip a single
test result:

  ok 1 # skip reason

or you can use the plan line to indicate that the entire test program is
to be skipped:

  1..0 # skip reason

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message sent to bug-automake@HIDDEN:

Subject: bug#20693: SKIP not counted in TAP
Resent-From: "Arthur Schwarz" <aschwarz1309@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 29 May 2015 21:50:03 +0000
Resent-Message-ID: <handler.20693.B20693.143293619013323 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: "'Nick Bowler'" <nbowler@HIDDEN>
Cc: 20693 <at> debbugs.gnu.org
From: "Arthur Schwarz" <aschwarz1309@HIDDEN>
References: <88E8A9325FEB4AF482785AD592E22EA3@2600kPC>
 <20150529171127.GA27194@HIDDEN>
In-Reply-To: <20150529171127.GA27194@HIDDEN>
Date: Fri, 29 May 2015 14:49:37 -0700
Message-ID: <CAD7DDAC9D25416EBEF8105AC5C9876A@HPDV7TNotebook>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AdCaMqH1uS9ziMH5SJijZa6IB48RugAHXHkQ
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Thanks Nick;

But doesn't this mean that tap-driver.sh didn't report a violation to the
TAP standard? And why is my output intermixed in the .trs file? If 'skip'
below is illegal, why isn't it tagged? (I think that multiline comments need
leading whitespace?)

As another issue, I looked at the awk code and have some potential revisions
below. I know very little about awk but I think the resultant code is faster
in that the 'is_first_read = 1' is not necessary and consequent checks for
each line input are unnecessary. 

As to your (sly and underhanded) comment about awk being defined (and I
should read it), well, in this case I agree. But in general where something
is important to the issue at hand obligating the user to read every relevant
manual bearing on the issue I feel is burdensome and unnecessary. I
understand the problems with following 'other peoples manuals' and burden
that that places on the maintenance staff, but requiring that readers of the
Automake User's Manual read the Autoconf, Automake, make, TAP standards,
bash for 'env' and etc. is needless. My opinion is that some of the material
in these (and other manuals) which have a direct bearing on Automake should
have more than a reference to Yet Another Manual. This issue directly bears
on, for example, TAP, where a few wise words from the Master's Of The
Universe will go a long way being productive. But, this is an opinion. Oh, I
am doing some writing for Section 15 of the Automake Manual to say that I am
putting some effort is providing an alternative viewpoint.

===================== tap-driver.sh =====================

   ## --------- ##
   ##  PARSING  ##
   ## --------- ##

   st = getline

   nextline = $0;  # I have no idea what $0 does.

   if (st < 0) # I/O error.
     abort("in input loop: only one input line")
   }

   do
     {
       # Involutions required so that we are able to read the exit status
       # from the last input line.
       # Copy any input line verbatim into the log file.

       o o o

       st = getline
       curline = nextline
       nextline = $0
       $0 = curline

   while(st >= 0)

   if (st < 0) # I/O error.
      fatal("I/O error while reading from input stream")
   }

On 2015-05-29 08:15 -0700, Arthur Schwarz wrote:
> skip causes the following issues:
> 1: It is not reported.
> 2: It is not summarized.
> 3: It must be subtracted from the plan (tests run), and
> 4: The .log file has TAP generated output and the test data intermixed.
> 
> I think this is against the TAP definition?

In a way, because...

> skip
> ok
> PASS: test2.tap 1
> not ok
> FAIL: test2.tap 2
> 1..2

... it is your program's output that does not follow the specification.

There are two basic styles of skipping in TAP.  You can skip a single
test result:

  ok 1 # skip reason

or you can use the plan line to indicate that the entire test program is
to be skipped:

  1..0 # skip reason

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#20693: SKIP not counted in TAP
Resent-From: "Arthur Schwarz" <aschwarz1309@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 29 May 2015 22:16:02 +0000
Resent-Message-ID: <handler.20693.B20693.143293772615640 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 20693
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: "'Nick Bowler'" <nbowler@HIDDEN>
Cc: 20693 <at> debbugs.gnu.org
Received: via spool by 20693-submit <at> debbugs.gnu.org id=B20693.143293772615640
          (code B ref 20693); Fri, 29 May 2015 22:16:02 +0000
Received: (at 20693) by debbugs.gnu.org; 29 May 2015 22:15:26 +0000
Received: from localhost ([127.0.0.1]:60905 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1YySYo-00044B-CS
	for submit <at> debbugs.gnu.org; Fri, 29 May 2015 18:15:26 -0400
Received: from nm8-vm2.access.bullet.mail.bf1.yahoo.com
 ([216.109.114.177]:57453) by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <aschwarz1309@HIDDEN>) id 1YySYm-00043y-Bh
 for 20693 <at> debbugs.gnu.org; Fri, 29 May 2015 18:15:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=att.net; s=s1024;
 t=1432937719; bh=DQ+QAVqw0Jmrl0EsOlWm4L2Z8GFD9/i2wXbn5+n/8Aw=;
 h=From:To:Cc:References:In-Reply-To:Subject:Date:From:Subject;
 b=SnwUdeDt17PDQfV5bserdFs4XaHc6j62bYqpkHYr0azojPfBdTCJruAp5lmTPsL5Jokaau1lZI10mveSAWlye6syioM9QVbwjtdW/4BvbiJSMjAqAfzUWuYM9fpRi/u910Eo1MpmcoTOHviDvs9DYX4KSXUrkrS2rCvdBGRQeM8=
Received: from [66.196.81.162] by nm8.access.bullet.mail.bf1.yahoo.com with
 NNFMP; 29 May 2015 22:15:19 -0000
Received: from [98.138.226.241] by tm8.access.bullet.mail.bf1.yahoo.com with
 NNFMP; 29 May 2015 22:15:19 -0000
Received: from [127.0.0.1] by smtp112.sbc.mail.ne1.yahoo.com with NNFMP;
 29 May 2015 22:15:18 -0000
X-Yahoo-Newman-Id: 876895.32270.bm@HIDDEN
X-Yahoo-Newman-Property: ymail-3
X-YMail-OSG: Ch_2RT0VM1neXmldjBeGrsOYO7TLbbWJiyEd2AifwRcpvbU
 3Ey8h1k5xuFV7BmNQrgT9iS8vR2jnlj5X_OEs0oxBJ6UhIz7BwMpSilWWtKh
 1Ih7dOdoP65dtq3QVMTnIbkm7AItG3Sk8S8C7mVwgj4RF48MLTueHOyUAchn
 t6rgjrgUNUx0Kq6skKweyP9mRhvGi7fSZTpjQuiLYk_AZ.XNcoWK0cGPOHhD
 PKoxrTrHKjGMtFXjQNtoOt9GmB8p1lDlJhkwXVXOtqFgdrHgao_rJuVXLtlB
 yPwC5ORgNO1qS_D8EtWOTRbziucgHR9pTqVq5OZ6oI7Iq81HaXVNDAet.Smy
 asNCOELN911q8m.3jM.d9SxLTajf6E3w3a588qkAgaK_Khq2v1tqwHqXSv40
 dcuJP2Lm82XLJWxjrE1Cr5qiCPhV9WbH8u8WWQ4vHB0fntYv0LJBt59HCn4J
 xb8I8Twq8ixyRYIz3jgrkpRgSvT2SmAdSF4dZZCgMwFk0DiEjY2hFK3Sv9.c
 y370CF1T0W.J9.QgIPvUrRdAvkuT3nTVoNbywH0tHVA--
X-Yahoo-SMTP: 0z5TJLmswBDohOc2QznCIeoaEiEalQiTSMLFNdKdTt4t
From: "Arthur Schwarz" <aschwarz1309@HIDDEN>
References: <88E8A9325FEB4AF482785AD592E22EA3@2600kPC>
 <20150529171127.GA27194@HIDDEN>
In-Reply-To: <20150529171127.GA27194@HIDDEN>
Date: Fri, 29 May 2015 15:15:14 -0700
Message-ID: <C9B2480AC4E04A96AD88D3E88EA263AB@HPDV7TNotebook>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Office Outlook 11
Thread-Index: AdCaMqH1uS9ziMH5SJijZa6IB48RugAKGqBw
X-MimeOLE: Produced By Microsoft MimeOLE V6.1.7601.17609
X-Spam-Score: 0.3 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)



-----Original Message-----In a way, because...

> skip
> ok
> PASS: test2.tap 1
> not ok
> FAIL: test2.tap 2
> 1..2

... it is your program's output that does not follow the specification.

There are two basic styles of skipping in TAP.  You can skip a single
test result:

  ok 1 # skip reason

or you can use the plan line to indicate that the entire test program is
to be skipped:

  1..0 # skip reason


Sorry to be such a pimple, but I just tried:

========== script ==========

#!/bin/bash
echo test2.tap "$# [ $@ ]"   >> test2.log
echo I am a tap script         >> test2.log
echo 1..3
echo ok
echo not ok
echo ok 3 # skip reason 


========== which yielded ==========

PASS: test2.tap 1
FAIL: test2.tap 2
PASS: test2.tap 3
=================================
Testsuite summary for test 0.5
=================================
# TOTAL: 3
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
================================
See src/test-suite.log
================================

========== and  ==========

I would have expected a 'skip' not a PASS! And the comment isn't output w/wo
TAP_LOG_DRIVER_FLAGS=--comments. However, 1..0 works if there is no test
output, as in the output only containing 1..0 and nothing else. More than
one test yields errors (# UNPLANNED).

If this is an error on my part (which it may well be) then I am confused
about your comment above.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.