GNU logs - #23521, boring messages

Message sent to bug-automake@HIDDEN:

Subject: bug#23521: XFAIL
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 12 May 2016 10:02:02 +0000
Resent-Message-ID: <handler.23521.B.146304729313589 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 23521 <at> debbugs.gnu.org
MIME-Version: 1.0
Date: Thu, 12 May 2016 11:01:11 +0100
Message-ID: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: multipart/alternative; boundary=94eb2c06fad6e1575c0532a23cc6
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--94eb2c06fad6e1575c0532a23cc6
Content-Type: text/plain; charset=UTF-8

The documentation says: "It's not uncommon, especially during early
development stages, that some tests fail for known reasons, and that the
developer doesn't want
to tackle these failures immediately (this is especially true when the
failing tests deal with corner cases)."

Another common use for "expected failure" is to write tests to check that
error conditions arise as expected, for example, by checking that a program
raises an error when given invalid input.

If that's a reasonable use of automake's test harness, perhaps the
documentation could reflect that, e.g. by adding:

"Another use for XFAIL is to mark tests that are supposed to fail, for
example, to check that a program raises an error when given invalid input."

It is often easier to write expected-to-fail tests this way (so that they
can all look the same), rather than have to have, for example, an extra
driver that converts expected errors into success codes for the automake
test harness.

-- 
http://rrt.sc3d.org

--94eb2c06fad6e1575c0532a23cc6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">The=
 documentation says: &quot;It&#39;s not uncommon, especially during early d=
evelopment stages, that some tests fail for known reasons, and that the dev=
eloper doesn&#39;t want</div><div class=3D"gmail_default">to tackle these f=
ailures immediately (this is especially true when the</div><div class=3D"gm=
ail_default">failing tests deal with corner cases).&quot;</div><div class=
=3D"gmail_default"><br></div><div class=3D"gmail_default">Another common us=
e for &quot;expected failure&quot; is to write tests to check that error co=
nditions arise as expected, for example, by checking that a program raises =
an error when given invalid input.</div><div class=3D"gmail_default"><br></=
div><div class=3D"gmail_default">If that&#39;s a reasonable use of automake=
&#39;s test harness, perhaps the documentation could reflect that, e.g. by =
adding:</div><div class=3D"gmail_default"><br></div><div class=3D"gmail_def=
ault">&quot;Another use for XFAIL is to mark tests that are supposed to fai=
l, for example, to check that a program raises an error when given invalid =
input.&quot;</div><div class=3D"gmail_default"><br></div><div class=3D"gmai=
l_default">It is often easier to write expected-to-fail tests this way (so =
that they can all look the same), rather than have to have, for example, an=
 extra driver that converts expected errors into success codes for the auto=
make test harness.</div><div><br></div>-- <br><div class=3D"gmail_signature=
"><a href=3D"http://rrt.sc3d.org" target=3D"_blank">http://rrt.sc3d.org</a>=
</div>
</div>

--94eb2c06fad6e1575c0532a23cc6--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Reuben Thomas <rrt@HIDDEN>
Subject: bug#23521: Acknowledgement (XFAIL)
Message-ID: <handler.23521.B.146304729313589.ack <at> debbugs.gnu.org>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
X-Gnu-PR-Message: ack 23521
X-Gnu-PR-Package: automake
Reply-To: 23521 <at> debbugs.gnu.org
Date: Thu, 12 May 2016 10:02:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-automake@HIDDEN

If you wish to submit further information on this problem, please
send it to 23521 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
23521: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D23521
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-automake@HIDDEN:

Subject: bug#23521: XFAIL
Resent-From: Mathieu Lirzin <mthl@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Wed, 18 May 2016 23:05:02 +0000
Resent-Message-ID: <handler.23521.B23521.14636126797673 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Reuben Thomas <rrt@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org
From: Mathieu Lirzin <mthl@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
Date: Thu, 19 May 2016 01:04:28 +0200
In-Reply-To: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 (Reuben Thomas's message of "Thu, 12 May 2016 11:01:11 +0100")
Message-ID: <87futfq877.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi,

Reuben Thomas <rrt@HIDDEN> writes:

> The documentation says: "It's not uncommon, especially during early
> development stages, that some tests fail for known reasons, and that
> the developer doesn't want
> to tackle these failures immediately (this is especially true when the
> failing tests deal with corner cases)."
>
> Another common use for "expected failure" is to write tests to check
> that error conditions arise as expected, for example, by checking that
> a program raises an error when given invalid input.

I agree that XFAIL can be ambiguous, however I think this usage is not
desirable.  It gives an additional opposite meaning to XFAIL symbol
which makes it even more confusing.

> If that's a reasonable use of automake's test harness, perhaps the
> documentation could reflect that, e.g. by adding:
>
> "Another use for XFAIL is to mark tests that are supposed to fail, for
> example, to check that a program raises an error when given invalid
> input."
>
> It is often easier to write expected-to-fail tests this way (so that
> they can all look the same), rather than have to have, for example, an
> extra driver that converts expected errors into success codes for the
> automake test harness.

What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?

Thanks

--=20
Mathieu Lirzin

Message sent to bug-automake@HIDDEN:

Subject: bug#23521: XFAIL
Resent-From: Peter Johansson <trojkan@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Wed, 18 May 2016 23:57:02 +0000
Resent-Message-ID: <handler.23521.B23521.146361576212104 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Mathieu Lirzin <mthl@HIDDEN>, Reuben Thomas <rrt@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
From: Peter Johansson <trojkan@HIDDEN>
Message-ID: <573D0107.1@HIDDEN>
Date: Thu, 19 May 2016 09:55:51 +1000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <87futfq877.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>



On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>> Another common use for "expected failure" is to write tests to check
>> >that error conditions arise as expected, for example, by checking that
>> >a program raises an error when given invalid input.
> I agree that XFAIL can be ambiguous, however I think this usage is not
> desirable.  It gives an additional opposite meaning to XFAIL symbol
> which makes it even more confusing.
>
I agree. When I wanna tests that a program fails with incorrect input, I 
prefer writing a tests that calls the program, check that it fails (exit 
1 or whatever is expected), and perhaps even parse the error message, 
and if it looks like I expect exit 0 aka PASS.

Cheers,
Peter

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#23521: XFAIL
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 19 May 2016 09:16:02 +0000
Resent-Message-ID: <handler.23521.B23521.146364934130756 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23521
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Mathieu Lirzin <mthl@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org
Received: via spool by 23521-submit <at> debbugs.gnu.org id=B23521.146364934130756
          (code B ref 23521); Thu, 19 May 2016 09:16:02 +0000
Received: (at 23521) by debbugs.gnu.org; 19 May 2016 09:15:41 +0000
Received: from localhost ([127.0.0.1]:57614 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3K3Q-000800-NE
	for submit <at> debbugs.gnu.org; Thu, 19 May 2016 05:15:40 -0400
Received: from mail-qg0-f43.google.com ([209.85.192.43]:35912)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3K3O-0007zn-M0
 for 23521 <at> debbugs.gnu.org; Thu, 19 May 2016 05:15:39 -0400
Received: by mail-qg0-f43.google.com with SMTP id w36so39853253qge.3
 for <23521 <at> debbugs.gnu.org>; Thu, 19 May 2016 02:15:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=H9b5kaCa5pxk3kedrvv5TqWWKXI2dITHPikbDJqB7Ao=;
 b=wI75O6ZJuuk8v0IFz3LlUpxJwKXPObirRLxvWGQm4yvRM+pqW3lit7ZFpst3AiLxZY
 ED1pteQWQPgeEAVEZzddfSG0Ihi4cqkMqCWHtF8UjOdScGrQLdT6yoTuGjGNI+6Qa/ZA
 ICWS3oaj+cjefxnri6DIaZKmLYYkkAGgtKEwY=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=H9b5kaCa5pxk3kedrvv5TqWWKXI2dITHPikbDJqB7Ao=;
 b=gbp/d8JY1oE8PaxIK/BAkpkN0aIlLiaPGtVgaX2HAtNvDGM82dQ4onpQDq94cQFc2a
 /yvNMLK/2yUychyR7UPXEoFWNlXIRNqnUqUn3yei/17jmaCucz1QWmuqN/mfgzX6dumU
 5vvTTg9cPRXLMNRXGk0wH3qb//6Z9sAxvavgNuQ4NbVJ/XrwEOVNQD4NPiK6sAFGe/2C
 TTouYompzXY1fDMNsuwvMVNZEe0pR7OBagCjW5Z2i7yfSv2dKx1MhbZfySmjDlxGJwXi
 ZdmSdHHAAOE7maWHr9n89W+ILDYHMpJelEQpudiVBIOHm650CqUt6KMC3XJKx+y1vTPe
 KbVQ==
X-Gm-Message-State: AOPr4FWm439gkdevKAD5okdYG1LUPkQeJu+heM+ngyzMIbg/0PEmDTkJQf7TA9C1BwjIGWHeNwYosTUqn4k3T81l
MIME-Version: 1.0
X-Received: by 10.140.102.117 with SMTP id v108mr12318743qge.103.1463649333173; 
 Thu, 19 May 2016 02:15:33 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Thu, 19 May 2016 02:15:33 -0700 (PDT)
In-Reply-To: <87futfq877.fsf@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
Date: Thu, 19 May 2016 10:15:33 +0100
Message-ID: <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: multipart/alternative; boundary=001a11c16ec494811405332e6a61
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--001a11c16ec494811405332e6a61
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:

> > It is often easier to write expected-to-fail tests this way (so that
> > they can all look the same), rather than have to have, for example, an
> > extra driver that converts expected errors into success codes for the
> > automake test harness.
>
> What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
>

=E2=80=8BA custom test driver.=E2=80=8B

--=20
http://rrt.sc3d.org

--001a11c16ec494811405332e6a61
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
19 May 2016 at 00:04, Mathieu Lirzin <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:mthl@HIDDEN" target=3D"_blank">mthl@HIDDEN</a>&gt;</span> wrote:<br></d=
iv><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">&gt; It is often easier to write expected-to-fail tests this =
way (so that<br>
&gt; they can all look the same), rather than have to have, for example, an=
<br>
&gt; extra driver that converts expected errors into success codes for the<=
br>
&gt; automake test harness.<br>
<br>
What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?<br></block=
quote><div><br></div><div><div class=3D"gmail_default" style=3D"font-size:s=
mall">=E2=80=8BA custom test driver.=E2=80=8B</div></div></div><div><br></d=
iv>-- <br><div class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" ta=
rget=3D"_blank">http://rrt.sc3d.org</a></div>
</div></div>

--001a11c16ec494811405332e6a61--

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#23521: XFAIL
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 19 May 2016 09:17:01 +0000
Resent-Message-ID: <handler.23521.B23521.146364939630870 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23521
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Peter Johansson <trojkan@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl@HIDDEN>
Received: via spool by 23521-submit <at> debbugs.gnu.org id=B23521.146364939630870
          (code B ref 23521); Thu, 19 May 2016 09:17:01 +0000
Received: (at 23521) by debbugs.gnu.org; 19 May 2016 09:16:36 +0000
Received: from localhost ([127.0.0.1]:57618 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3K4K-00081p-0L
	for submit <at> debbugs.gnu.org; Thu, 19 May 2016 05:16:36 -0400
Received: from mail-qk0-f170.google.com ([209.85.220.170]:34808)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3K4H-00081b-U0
 for 23521 <at> debbugs.gnu.org; Thu, 19 May 2016 05:16:34 -0400
Received: by mail-qk0-f170.google.com with SMTP id y126so9000590qke.1
 for <23521 <at> debbugs.gnu.org>; Thu, 19 May 2016 02:16:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=9QXDUEyA69fDc+y4Y4AAdyEw/TDMoqQ9rIDYYLoEF6s=;
 b=DyRWAHsNaxZAzhGe89ux4j8ZCCPL2cgFShz082YeIk25LJL6GZFJT8C47Led+w843T
 //ub2ERyf6NoTn1PmnvIbLx54fERN3shu7FMEwlEPFc4HEDIyv24HEYMaxh0jvao12G0
 IGOJj6aDg4bfR10w2EKFHBuIBJtW6g5uZDdKs=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=9QXDUEyA69fDc+y4Y4AAdyEw/TDMoqQ9rIDYYLoEF6s=;
 b=W4Lns2lWuUedfx3YiObbEFSiiMWICBmM/ZFEFg1SL49O1GopAVc9XBWyXzVAbgQSRQ
 rkHnniPBXzbAFVEPfFaqnBlWxbwAajY2H2n5Bdf//MzxEtUWCg3PBrTINTHSIu3tTqut
 7kFUZTksXko0QhACG20u8gpzUzroQcZfqyS/QIJuZYl7xyWXBkDZU7YDTBVOsdk1O0HT
 4lCHgI+FuPfFk2wffcf/S0k90v2Dtn3doob9KA4YjVnVz4ZzTEVURiNPac6EYKvTYdyN
 b9+ZCuPEZb0l4CElNpefgJqpfTFG3DRu64ReklV/Rx3d4+tl79vqzi9n2hQwyd3Vs1nO
 v7jA==
X-Gm-Message-State: AOPr4FXg+819s0MD2VdN3i4HupmlXyqUspk4/ZlwOqP3HE5Z9GoxOgZlO/Mz/BSBETThG/ohUWAnE3CPs7PnIrwq
MIME-Version: 1.0
X-Received: by 10.55.186.132 with SMTP id k126mr12880567qkf.176.1463649388628; 
 Thu, 19 May 2016 02:16:28 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Thu, 19 May 2016 02:16:28 -0700 (PDT)
In-Reply-To: <573D0107.1@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN> <573D0107.1@HIDDEN>
Date: Thu, 19 May 2016 10:16:28 +0100
Message-ID: <CAOnWdogbgeiS2YiLYXupMsibk6c5bueHPirm-gfKT0znufEAUw@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: multipart/alternative; boundary=94eb2c043626e2b45c05332e6d82
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--94eb2c043626e2b45c05332e6d82
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 19 May 2016 at 00:55, Peter Johansson <trojkan@HIDDEN> wrote:

>
>
> On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>
>> Another common use for "expected failure" is to write tests to check
>>> >that error conditions arise as expected, for example, by checking that
>>> >a program raises an error when given invalid input.
>>>
>> I agree that XFAIL can be ambiguous, however I think this usage is not
>> desirable.  It gives an additional opposite meaning to XFAIL symbol
>> which makes it even more confusing.
>>
>> I agree. When I wanna tests that a program fails with incorrect input, I
> prefer writing a tests that calls the program, check that it fails (exit =
1
> or whatever is expected), and perhaps even parse the error message, and i=
f
> it looks like I expect exit 0 aka PASS.
>

=E2=80=8BThanks. I shall continue with my "deviant" usage for now, but if t=
hat is
not considered normal, I understand that you won't want to document it.

--=20
http://rrt.sc3d.org

--94eb2c043626e2b45c05332e6d82
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
19 May 2016 at 00:55, Peter Johansson <span dir=3D"ltr">&lt;<a href=3D"mail=
to:trojkan@HIDDEN" target=3D"_blank">trojkan@HIDDEN</a>&gt;</span> wr=
ote:<br></div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockq=
uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><span class=3D""><br>
<br>
On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Another common use for &quot;expected failure&quot; is to write tests to ch=
eck<br>
&gt;that error conditions arise as expected, for example, by checking that<=
br>
&gt;a program raises an error when given invalid input.<br>
</blockquote>
I agree that XFAIL can be ambiguous, however I think this usage is not<br>
desirable.=C2=A0 It gives an additional opposite meaning to XFAIL symbol<br=
>
which makes it even more confusing.<br>
<br>
</blockquote></span>
I agree. When I wanna tests that a program fails with incorrect input, I pr=
efer writing a tests that calls the program, check that it fails (exit 1 or=
 whatever is expected), and perhaps even parse the error message, and if it=
 looks like I expect exit 0 aka PASS.<br></blockquote><div><br></div><div><=
div class=3D"gmail_default" style=3D"font-size:small;display:inline">=E2=80=
=8BThanks. I shall continue with my &quot;deviant&quot; usage for now, but =
if that is not considered normal, I understand that you won&#39;t want to d=
ocument it.</div></div></div><div><br></div>-- <br><div class=3D"gmail_sign=
ature"><a href=3D"http://rrt.sc3d.org" target=3D"_blank">http://rrt.sc3d.or=
g</a></div>
</div></div>

--94eb2c043626e2b45c05332e6d82--

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#23521: XFAIL
Resent-From: Gavin Smith <gavinsmith0123@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 20 May 2016 14:59:01 +0000
Resent-Message-ID: <handler.23521.B23521.146375628929030 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23521
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Mathieu Lirzin <mthl@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org, Reuben Thomas <rrt@HIDDEN>
Received: via spool by 23521-submit <at> debbugs.gnu.org id=B23521.146375628929030
          (code B ref 23521); Fri, 20 May 2016 14:59:01 +0000
Received: (at 23521) by debbugs.gnu.org; 20 May 2016 14:58:09 +0000
Received: from localhost ([127.0.0.1]:59195 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3lsO-0007YA-Te
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 10:58:09 -0400
Received: from mail-yw0-f174.google.com ([209.85.161.174]:33491)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gavinsmith0123@HIDDEN>) id 1b3lsO-0007We-2v
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 10:58:08 -0400
Received: by mail-yw0-f174.google.com with SMTP id x194so111436017ywd.0
 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 07:58:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc:content-transfer-encoding;
 bh=eYDZM4mJzWTHm+BsbEhjb3IIzcvoafekfERuphxc8pM=;
 b=WF/GRNCPsBvjDIkoKWNUhmofQQ23n4Ky/KTI21kyeyNNDeUaS8P7Yt/LKrAnzkzCUW
 zPyXTKqBhhdzT3OsWqK8zQnoEyZHCRiNMdwOOAda+O1BTrYtBdWy9iPouJ/gFit2Gu5x
 +clC2ktiNfQJAuIckjseekrBMP55cWIes9NdZ/hcSHD7373xMDBGCiv7wAHYG+93VVQy
 XBOxM9meprz/s/f3Tb6yivmII5nFAdXz5jpUvzRNKs1vJG9PViyfnrVMwAgvfsvdh0Ba
 xiX342rZVyPbOO3MWiwT5ZqKr1X8yX+A+QWpnYeefg4BCyJENYiwstDPxydBk1ldjwKu
 HUTg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc:content-transfer-encoding;
 bh=eYDZM4mJzWTHm+BsbEhjb3IIzcvoafekfERuphxc8pM=;
 b=l1gZ1KGXPfZcto80NHoFCRiv1sZUGuuU21lLaruBJ0iP0+ulrZ1tsWqcAsiun6mWjr
 qOd/ksqILTEOQmmR+HXT9dgEEGVOkJcGovdk6b55mMun9IfYlDUKCiViBQVXI+QvM905
 Oem3lGJkJBmH/jHgdRH4rgisfINans/cOpkrJKeILKX866b3m6wus+Qp7LVpWUJ5Rv6Q
 G/KAKaoahP73w5SEdwgYx8Mn2pCp3E4NiMcNE2EA/z84e33KxALKFE/Uk6zYk451Jxuk
 EKAe0aJ3iIC6MC5h7JLuuEm+mHFHw46LBLQiT9cDtnYiZQxtff8bnKhj3h17Aq4UQeeb
 2UeQ==
X-Gm-Message-State: AOPr4FVLa+fvq1m86tKIsvfcvEIEm+KWiMuA7dkKqAxVxX3w1BMK2Bvh7WfHJNdELaklyxAtCNCkp+BDAunZeA==
MIME-Version: 1.0
X-Received: by 10.129.106.9 with SMTP id f9mr2229650ywc.90.1463756282398; Fri,
 20 May 2016 07:58:02 -0700 (PDT)
Received: by 10.37.215.200 with HTTP; Fri, 20 May 2016 07:58:02 -0700 (PDT)
In-Reply-To: <87futfq877.fsf@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
Date: Fri, 20 May 2016 15:58:02 +0100
Message-ID: <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN>
From: Gavin Smith <gavinsmith0123@HIDDEN>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.2 (/)

On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
>> It is often easier to write expected-to-fail tests this way (so that
>> they can all look the same), rather than have to have, for example, an
>> extra driver that converts expected errors into success codes for the
>> automake test harness.
>
> What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?

This would be a reference to a "custom test driver".
https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-T=
est-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#23521: XFAIL
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 20 May 2016 15:05:02 +0000
Resent-Message-ID: <handler.23521.B23521.146375664629892 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23521
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Gavin Smith <gavinsmith0123@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl@HIDDEN>
Received: via spool by 23521-submit <at> debbugs.gnu.org id=B23521.146375664629892
          (code B ref 23521); Fri, 20 May 2016 15:05:02 +0000
Received: (at 23521) by debbugs.gnu.org; 20 May 2016 15:04:06 +0000
Received: from localhost ([127.0.0.1]:59200 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3lyA-0007m4-JN
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 11:04:06 -0400
Received: from mail-qg0-f48.google.com ([209.85.192.48]:35135)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3ly8-0007lW-VJ
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:04:05 -0400
Received: by mail-qg0-f48.google.com with SMTP id j18so13490944qge.2
 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 08:04:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=o3CNYIWm9AKJW/hosEoYm0gjbXfwG22+x1Ji7S2gU6M=;
 b=VkcSWvRCQ1xcefygHGodgm+m8AwpDki7lpL66ZHr8UV8J9CgzmViK9myczfK9xfJew
 kNc5K6X4+nsDxQ7R8Esn8FFjU8J1rEFntxD8q8F3wO2jgTDJDoMs+yWoGso7dGhrQKyB
 oxSKzdjnpU9dK5kf1QcKnXxXp2NnSNCEJ8e0E=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=o3CNYIWm9AKJW/hosEoYm0gjbXfwG22+x1Ji7S2gU6M=;
 b=XC+qsQX82iJm1A92RkuPp7IEWyL/TH/Hm+kUtwK/a5w3MU+LPzIRIUhoTvEkgZVYD5
 jLyuMmG6/Snlwz8Cz7JmILLS9VxoY3JUfIr8T5IZIDBdGb0NvRC7PmG5KIlYRidkIHum
 5K1ZRXi1upTsnuWkFOA/7K4KyBiE9ZDiL9db4mdS7Uf2MgzGJcmvNo8VX3IFocZG5Bdu
 866EBFtqMfSrIY5Oxg7jPXVABM6ocmzaEa83FyJr94nILtixir+jGJuUlRSow+dT367a
 hR4uV9zwSd5hdTY0gyVBpZaZkvu2wJ2P8Z13B0BmGb+Ie9kq3rsSKmEC4A5aB8DukUtr
 xpPg==
X-Gm-Message-State: AOPr4FWZ2S/6IpUEhqhVTHnVRE/1Q6cYRFnRQycX8L5uNBs44Hm0Wl887NZq++YyLlAIJ2DpFWvWDReqDr4kSV4h
MIME-Version: 1.0
X-Received: by 10.140.21.164 with SMTP id 33mr3786018qgl.34.1463756639432;
 Fri, 20 May 2016 08:03:59 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Fri, 20 May 2016 08:03:59 -0700 (PDT)
In-Reply-To: <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
 <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN>
Date: Fri, 20 May 2016 16:03:59 +0100
Message-ID: <CAOnWdoh3FQNeOu3Cvahn6Ggo_esxW91mxzUK4ADV9VP2J32Bfg@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: multipart/alternative; boundary=001a11c13aca8829fa053347669f
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--001a11c13aca8829fa053347669f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 20 May 2016 at 15:58, Gavin Smith <gavinsmith0123@HIDDEN> wrote:

> On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
> >> It is often easier to write expected-to-fail tests this way (so that
> >> they can all look the same), rather than have to have, for example, an
> >> extra driver that converts expected errors into success codes for the
> >> automake test harness.
> >
> > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
>
> This would be a reference to a "custom test driver".
>
> https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom=
-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support
>

=E2=80=8BThanks, that's what I meant.

--=20
http://rrt.sc3d.org

--001a11c13aca8829fa053347669f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
20 May 2016 at 15:58, Gavin Smith <span dir=3D"ltr">&lt;<a href=3D"mailto:g=
avinsmith0123@HIDDEN" target=3D"_blank">gavinsmith0123@HIDDEN</a>&gt;=
</span> wrote:<br></div><div class=3D"gmail_extra"><div class=3D"gmail_quot=
e"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex"><span class=3D"">On 19 May 2016 at 00:04,=
 Mathieu Lirzin &lt;<a href=3D"mailto:mthl@HIDDEN">mthl@HIDDEN</a>&gt; wr=
ote:<br>
</span><span class=3D"">&gt;&gt; It is often easier to write expected-to-fa=
il tests this way (so that<br>
&gt;&gt; they can all look the same), rather than have to have, for example=
, an<br>
&gt;&gt; extra driver that converts expected errors into success codes for =
the<br>
&gt;&gt; automake test harness.<br>
&gt;<br>
&gt; What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?<br>
<br>
</span>This would be a reference to a &quot;custom test driver&quot;.<br>
<a href=3D"https://www.gnu.org/software/automake/manual/html_node/Overview-=
of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support=
" rel=3D"noreferrer" target=3D"_blank">https://www.gnu.org/software/automak=
e/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of=
-Custom-Test-Drivers-Support</a><br>
</blockquote></div><br><div class=3D"gmail_default" style=3D"font-size:smal=
l">=E2=80=8BThanks, that&#39;s what I meant.</div><div><br></div>-- <br><di=
v class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" target=3D"_blan=
k">http://rrt.sc3d.org</a></div>
</div></div>

--001a11c13aca8829fa053347669f--

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#23521: XFAIL
Resent-From: Mathieu Lirzin <mthl@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 20 May 2016 15:51:02 +0000
Resent-Message-ID: <handler.23521.B23521.14637594098192 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23521
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Reuben Thomas <rrt@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org
Received: via spool by 23521-submit <at> debbugs.gnu.org id=B23521.14637594098192
          (code B ref 23521); Fri, 20 May 2016 15:51:02 +0000
Received: (at 23521) by debbugs.gnu.org; 20 May 2016 15:50:09 +0000
Received: from localhost ([127.0.0.1]:59232 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3mgj-000284-Ei
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:09 -0400
Received: from eggs.gnu.org ([208.118.235.92]:39715)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mthl@HIDDEN>) id 1b3mgi-00027s-Gd
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:08 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mthl@HIDDEN>) id 1b3mgZ-0006Uj-De
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:03 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_40,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50949)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <mthl@HIDDEN>)
 id 1b3mgZ-0006Ue-A1; Fri, 20 May 2016 11:49:59 -0400
Received: from mek33-4-82-236-46-88.fbx.proxad.net ([82.236.46.88]:36512
 helo=godel)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <mthl@HIDDEN>)
 id 1b3mgX-0007B1-IK; Fri, 20 May 2016 11:49:57 -0400
From: Mathieu Lirzin <mthl@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
 <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
Date: Fri, 20 May 2016 17:49:54 +0200
In-Reply-To: <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
 (Reuben Thomas's message of "Thu, 19 May 2016 10:15:33 +0100")
Message-ID: <8737pclof1.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.4 (------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.4 (------)

Reuben Thomas <rrt@HIDDEN> writes:

> On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
>
>     > It is often easier to write expected-to-fail tests this way (so
>     that
>     > they can all look the same), rather than have to have, for
>     example, an
>     > extra driver that converts expected errors into success codes
>     for the
>     > automake test harness.
>=20=20=20=20=20
>     What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
>=20=20=20=20=20
>
> =E2=80=8BA custom test driver.=E2=80=8B

OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
checking failures.  IMO the solution Peter proposed is nice and simple.

--=20
Mathieu Lirzin

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#23521: XFAIL
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 20 May 2016 16:01:01 +0000
Resent-Message-ID: <handler.23521.B23521.14637600399137 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 23521
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Mathieu Lirzin <mthl@HIDDEN>
Cc: 23521 <at> debbugs.gnu.org
Received: via spool by 23521-submit <at> debbugs.gnu.org id=B23521.14637600399137
          (code B ref 23521); Fri, 20 May 2016 16:01:01 +0000
Received: (at 23521) by debbugs.gnu.org; 20 May 2016 16:00:39 +0000
Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3mqs-0002NJ-OQ
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 12:00:38 -0400
Received: from mail-qk0-f180.google.com ([209.85.220.180]:35209)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3mqq-0002N3-5s
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 12:00:37 -0400
Received: by mail-qk0-f180.google.com with SMTP id n62so61136393qkc.2
 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 09:00:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=Onr1MyN7x6Lwx8jzFGhgjCtI73LIXMWIEk4aWFGToLY=;
 b=ZkVkE1Ct0I96MfwOJ0i8HM58ts4CXZYL8gdqLFjVYEy7g1HZS1pQ2nOKmktl05b0R4
 eeG9o2uWzKS8bHcR23t5g0VGoWDlov2eegAqabAFIk0gftQPcdSLVMDHbwfheqpMtcaS
 ccgF8GA1EyqyeIrcfFVQ1Jx1wP51S8WMbZ1zI=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=Onr1MyN7x6Lwx8jzFGhgjCtI73LIXMWIEk4aWFGToLY=;
 b=NDL6ULybKWKgXYngTcgoNqh2/9RFW1uB0KIWzyw01NeM6o3rerCoCJ/RbU8BPw2j/1
 ZqO7dC6i/u1CpzLUy/n3UxBW+yskS2YeOF9DWj0NMTbh1OUeN6GRzrK/+bzFkHiW5BWU
 MOj7WoSwsTS4L0RPYrPMpvy2VkMvDAohrHzvnHFtgxROwugVJ3RAARGtbD/GWFjfxax3
 ZaZ83MHJFxZIQTwqxGVDfOyYX4eM+uLuCUarbgq3TCkiL2+Oc4umWeSY6WwVEk3UyvTF
 N0vZx9+XPYgRGMAcYeEwQHA9IfjwUYjDrK5BiNOwmUrkJN8ZTD+k+xbgM5axsbgJ+p7Y
 pADQ==
X-Gm-Message-State: AOPr4FWELZJA912wpahgefXyRkm4AbE//2bpBHJu4KMIj0Q7AyfZTX9Ki1FxjtwEWtjr420jaikILVbcc/pzrMhN
MIME-Version: 1.0
X-Received: by 10.55.186.132 with SMTP id k126mr4213429qkf.176.1463760030523; 
 Fri, 20 May 2016 09:00:30 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Fri, 20 May 2016 09:00:30 -0700 (PDT)
In-Reply-To: <8737pclof1.fsf@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
 <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
 <8737pclof1.fsf@HIDDEN>
Date: Fri, 20 May 2016 17:00:30 +0100
Message-ID: <CAOnWdogS8TgSruY5GF6cvBe3NdWo-9DBpOuZ5n8dLOEddTUf3w@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: multipart/alternative; boundary=94eb2c043626a80ef905334830d2
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--94eb2c043626a80ef905334830d2
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 20 May 2016 at 16:49, Mathieu Lirzin <mthl@HIDDEN> wrote:

> Reuben Thomas <rrt@HIDDEN> writes:
>
> > On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
> >
> >     > It is often easier to write expected-to-fail tests this way (so
> >     that
> >     > they can all look the same), rather than have to have, for
> >     example, an
> >     > extra driver that converts expected errors into success codes
> >     for the
> >     > automake test harness.
> >
> >     What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
> >
> >
> > =E2=80=8BA custom test driver.=E2=80=8B
>
> OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
> checking failures.  IMO the solution Peter proposed is nice and simple.
>

=E2=80=8BWhat Peter proposed is essentially a custom test driver: I would n=
ot
expect to duplicate the logic to check the return code &c. in each test
expected to fail; rather, I would put it in a custom test driver that would
handle expected fails and mark them as passes. (My expected fails are all
of the same type, i.e. a non-zero exit code. It might additionally be
useful, as Peter suggests, to check that an expected error message is
produced.)

--=20
http://rrt.sc3d.org

--94eb2c043626a80ef905334830d2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
20 May 2016 at 16:49, Mathieu Lirzin <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:mthl@HIDDEN" target=3D"_blank">mthl@HIDDEN</a>&gt;</span> wrote:<br></d=
iv><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><span class=3D"">Reuben Thomas &lt;<a href=3D"mailto:rrt@sc3d=
.org">rrt@HIDDEN</a>&gt; writes:<br>
<br>
&gt; On 19 May 2016 at 00:04, Mathieu Lirzin &lt;<a href=3D"mailto:mthl@gnu=
.org">mthl@HIDDEN</a>&gt; wrote:<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; It is often easier to write expected-to-fail t=
ests this way (so<br>
&gt;=C2=A0 =C2=A0 =C2=A0that<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; they can all look the same), rather than have =
to have, for<br>
&gt;=C2=A0 =C2=A0 =C2=A0example, an<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; extra driver that converts expected errors int=
o success codes<br>
&gt;=C2=A0 =C2=A0 =C2=A0for the<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; automake test harness.<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 =C2=A0What do you mean precisely by =E2=80=9Can extra dri=
ver=E2=80=9D?<br>
&gt;<br>
&gt;<br>
&gt; =E2=80=8BA custom test driver.=E2=80=8B<br>
<br>
</span>OK, I wasn&#39;t sure.=C2=A0 Indeed a custom test driver seems a bit=
 heavy just<br>
checking failures.=C2=A0 IMO the solution Peter proposed is nice and simple=
.<br></blockquote><div><br></div><div><div class=3D"gmail_default" style=3D=
"font-size:small;display:inline">=E2=80=8BWhat Peter proposed is essentiall=
y a custom test driver: I would not expect to duplicate the logic to check =
the return code &amp;c. in each test expected to fail; rather, I would put =
it in a custom test driver that would handle expected fails and mark them a=
s passes. (My expected fails are all of the same type, i.e. a non-zero exit=
 code. It might additionally be useful, as Peter suggests, to check that an=
 expected error message is produced.)</div></div></div><div><br></div>-- <b=
r><div class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" target=3D"=
_blank">http://rrt.sc3d.org</a></div>
</div></div>

--94eb2c043626a80ef905334830d2--

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 15 Jul 2017 16:12:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jul 15 12:12:08 2017
Received: from localhost ([127.0.0.1]:40533 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1dWPfr-000549-33
	for submit <at> debbugs.gnu.org; Sat, 15 Jul 2017 12:12:07 -0400
Received: from eggs.gnu.org ([208.118.235.92]:50047)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mthl@HIDDEN>) id 1dWPfp-00053c-6i
 for control <at> debbugs.gnu.org; Sat, 15 Jul 2017 12:12:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mthl@HIDDEN>) id 1dWPff-0007WP-Ou
 for control <at> debbugs.gnu.org; Sat, 15 Jul 2017 12:11:59 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:60996)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <mthl@HIDDEN>)
 id 1dWPff-0007WE-Lk
 for control <at> debbugs.gnu.org; Sat, 15 Jul 2017 12:11:55 -0400
Received: from arennes-653-1-267-199.w109-218.abo.wanadoo.fr
 ([109.218.42.199]:49386 helo=pc10.home)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <mthl@HIDDEN>) id 1dWPff-0000P4-3a
 for control <at> debbugs.gnu.org; Sat, 15 Jul 2017 12:11:55 -0400
Date: Sat, 15 Jul 2017 18:11:51 +0200
Message-Id: <87lgnphe7s.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: Mathieu Lirzin <mthl@HIDDEN>
Subject: control message for bug #23521
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)

severity 23521 wishlist

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 2 Jan 2022 09:43:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jan 02 04:43:08 2022
Received: from localhost ([127.0.0.1]:59913 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1n3xOO-0006hD-9t
	for submit <at> debbugs.gnu.org; Sun, 02 Jan 2022 04:43:08 -0500
Received: from woodpecker.gentoo.org ([140.211.166.183]:56790
 helo=smtp.gentoo.org) by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <vapier@HIDDEN>) id 1n3xOL-0006ge-M9
 for control <at> debbugs.gnu.org; Sun, 02 Jan 2022 04:43:06 -0500
Received: by smtp.gentoo.org (Postfix, from userid 559)
 id 895FD3430C2; Sun,  2 Jan 2022 09:42:59 +0000 (UTC)
From: Mike Frysinger <vapier@HIDDEN>
To: control <at> debbugs.gnu.org
Subject: Control message
Message-Id: <20220102094259.895FD3430C2@HIDDEN>
Date: Sun,  2 Jan 2022 09:42:59 +0000 (UTC)
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

retitle 23521 control test "PASS" exit status on a per-test basis
thankyou

Last modified: Sun, 2 Jan 2022 09:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.