GNU bug report logs - #23521
XFAIL

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: automake; Severity: wishlist; Reported by: Reuben Thomas <rrt@HIDDEN>; dated Thu, 12 May 2016 10:02:02 UTC; Maintainer for automake is bug-automake@HIDDEN.
Severity set to 'wishlist' from 'normal' Request was from Mathieu Lirzin <mthl@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 20 May 2016 16:00:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 12:00:39 2016
Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3mqs-0002NJ-OQ
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 12:00:38 -0400
Received: from mail-qk0-f180.google.com ([209.85.220.180]:35209)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3mqq-0002N3-5s
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 12:00:37 -0400
Received: by mail-qk0-f180.google.com with SMTP id n62so61136393qkc.2
 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 09:00:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=Onr1MyN7x6Lwx8jzFGhgjCtI73LIXMWIEk4aWFGToLY=;
 b=ZkVkE1Ct0I96MfwOJ0i8HM58ts4CXZYL8gdqLFjVYEy7g1HZS1pQ2nOKmktl05b0R4
 eeG9o2uWzKS8bHcR23t5g0VGoWDlov2eegAqabAFIk0gftQPcdSLVMDHbwfheqpMtcaS
 ccgF8GA1EyqyeIrcfFVQ1Jx1wP51S8WMbZ1zI=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=Onr1MyN7x6Lwx8jzFGhgjCtI73LIXMWIEk4aWFGToLY=;
 b=NDL6ULybKWKgXYngTcgoNqh2/9RFW1uB0KIWzyw01NeM6o3rerCoCJ/RbU8BPw2j/1
 ZqO7dC6i/u1CpzLUy/n3UxBW+yskS2YeOF9DWj0NMTbh1OUeN6GRzrK/+bzFkHiW5BWU
 MOj7WoSwsTS4L0RPYrPMpvy2VkMvDAohrHzvnHFtgxROwugVJ3RAARGtbD/GWFjfxax3
 ZaZ83MHJFxZIQTwqxGVDfOyYX4eM+uLuCUarbgq3TCkiL2+Oc4umWeSY6WwVEk3UyvTF
 N0vZx9+XPYgRGMAcYeEwQHA9IfjwUYjDrK5BiNOwmUrkJN8ZTD+k+xbgM5axsbgJ+p7Y
 pADQ==
X-Gm-Message-State: AOPr4FWELZJA912wpahgefXyRkm4AbE//2bpBHJu4KMIj0Q7AyfZTX9Ki1FxjtwEWtjr420jaikILVbcc/pzrMhN
MIME-Version: 1.0
X-Received: by 10.55.186.132 with SMTP id k126mr4213429qkf.176.1463760030523; 
 Fri, 20 May 2016 09:00:30 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Fri, 20 May 2016 09:00:30 -0700 (PDT)
In-Reply-To: <8737pclof1.fsf@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
 <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
 <8737pclof1.fsf@HIDDEN>
Date: Fri, 20 May 2016 17:00:30 +0100
Message-ID: <CAOnWdogS8TgSruY5GF6cvBe3NdWo-9DBpOuZ5n8dLOEddTUf3w@HIDDEN>
Subject: Re: bug#23521: XFAIL
From: Reuben Thomas <rrt@HIDDEN>
To: Mathieu Lirzin <mthl@HIDDEN>
Content-Type: multipart/alternative; boundary=94eb2c043626a80ef905334830d2
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--94eb2c043626a80ef905334830d2
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 20 May 2016 at 16:49, Mathieu Lirzin <mthl@HIDDEN> wrote:

> Reuben Thomas <rrt@HIDDEN> writes:
>
> > On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
> >
> >     > It is often easier to write expected-to-fail tests this way (so
> >     that
> >     > they can all look the same), rather than have to have, for
> >     example, an
> >     > extra driver that converts expected errors into success codes
> >     for the
> >     > automake test harness.
> >
> >     What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
> >
> >
> > =E2=80=8BA custom test driver.=E2=80=8B
>
> OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
> checking failures.  IMO the solution Peter proposed is nice and simple.
>

=E2=80=8BWhat Peter proposed is essentially a custom test driver: I would n=
ot
expect to duplicate the logic to check the return code &c. in each test
expected to fail; rather, I would put it in a custom test driver that would
handle expected fails and mark them as passes. (My expected fails are all
of the same type, i.e. a non-zero exit code. It might additionally be
useful, as Peter suggests, to check that an expected error message is
produced.)

--=20
http://rrt.sc3d.org

--94eb2c043626a80ef905334830d2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
20 May 2016 at 16:49, Mathieu Lirzin <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:mthl@HIDDEN" target=3D"_blank">mthl@HIDDEN</a>&gt;</span> wrote:<br></d=
iv><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><span class=3D"">Reuben Thomas &lt;<a href=3D"mailto:rrt@sc3d=
.org">rrt@HIDDEN</a>&gt; writes:<br>
<br>
&gt; On 19 May 2016 at 00:04, Mathieu Lirzin &lt;<a href=3D"mailto:mthl@gnu=
.org">mthl@HIDDEN</a>&gt; wrote:<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; It is often easier to write expected-to-fail t=
ests this way (so<br>
&gt;=C2=A0 =C2=A0 =C2=A0that<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; they can all look the same), rather than have =
to have, for<br>
&gt;=C2=A0 =C2=A0 =C2=A0example, an<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; extra driver that converts expected errors int=
o success codes<br>
&gt;=C2=A0 =C2=A0 =C2=A0for the<br>
&gt;=C2=A0 =C2=A0 =C2=A0&gt; automake test harness.<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 =C2=A0What do you mean precisely by =E2=80=9Can extra dri=
ver=E2=80=9D?<br>
&gt;<br>
&gt;<br>
&gt; =E2=80=8BA custom test driver.=E2=80=8B<br>
<br>
</span>OK, I wasn&#39;t sure.=C2=A0 Indeed a custom test driver seems a bit=
 heavy just<br>
checking failures.=C2=A0 IMO the solution Peter proposed is nice and simple=
.<br></blockquote><div><br></div><div><div class=3D"gmail_default" style=3D=
"font-size:small;display:inline">=E2=80=8BWhat Peter proposed is essentiall=
y a custom test driver: I would not expect to duplicate the logic to check =
the return code &amp;c. in each test expected to fail; rather, I would put =
it in a custom test driver that would handle expected fails and mark them a=
s passes. (My expected fails are all of the same type, i.e. a non-zero exit=
 code. It might additionally be useful, as Peter suggests, to check that an=
 expected error message is produced.)</div></div></div><div><br></div>-- <b=
r><div class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" target=3D"=
_blank">http://rrt.sc3d.org</a></div>
</div></div>

--94eb2c043626a80ef905334830d2--




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 20 May 2016 15:50:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 11:50:09 2016
Received: from localhost ([127.0.0.1]:59232 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3mgj-000284-Ei
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:09 -0400
Received: from eggs.gnu.org ([208.118.235.92]:39715)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mthl@HIDDEN>) id 1b3mgi-00027s-Gd
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:08 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <mthl@HIDDEN>) id 1b3mgZ-0006Uj-De
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:03 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_40,RP_MATCHES_RCVD
 autolearn=disabled version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50949)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <mthl@HIDDEN>)
 id 1b3mgZ-0006Ue-A1; Fri, 20 May 2016 11:49:59 -0400
Received: from mek33-4-82-236-46-88.fbx.proxad.net ([82.236.46.88]:36512
 helo=godel)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128)
 (Exim 4.82) (envelope-from <mthl@HIDDEN>)
 id 1b3mgX-0007B1-IK; Fri, 20 May 2016 11:49:57 -0400
From: Mathieu Lirzin <mthl@HIDDEN>
To: Reuben Thomas <rrt@HIDDEN>
Subject: Re: bug#23521: XFAIL
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
 <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
Date: Fri, 20 May 2016 17:49:54 +0200
In-Reply-To: <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
 (Reuben Thomas's message of "Thu, 19 May 2016 10:15:33 +0100")
Message-ID: <8737pclof1.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -6.4 (------)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.4 (------)

Reuben Thomas <rrt@HIDDEN> writes:

> On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
>
>     > It is often easier to write expected-to-fail tests this way (so
>     that
>     > they can all look the same), rather than have to have, for
>     example, an
>     > extra driver that converts expected errors into success codes
>     for the
>     > automake test harness.
>=20=20=20=20=20
>     What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
>=20=20=20=20=20
>
> =E2=80=8BA custom test driver.=E2=80=8B

OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
checking failures.  IMO the solution Peter proposed is nice and simple.

--=20
Mathieu Lirzin




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 20 May 2016 15:04:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 11:04:06 2016
Received: from localhost ([127.0.0.1]:59200 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3lyA-0007m4-JN
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 11:04:06 -0400
Received: from mail-qg0-f48.google.com ([209.85.192.48]:35135)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3ly8-0007lW-VJ
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:04:05 -0400
Received: by mail-qg0-f48.google.com with SMTP id j18so13490944qge.2
 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 08:04:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=o3CNYIWm9AKJW/hosEoYm0gjbXfwG22+x1Ji7S2gU6M=;
 b=VkcSWvRCQ1xcefygHGodgm+m8AwpDki7lpL66ZHr8UV8J9CgzmViK9myczfK9xfJew
 kNc5K6X4+nsDxQ7R8Esn8FFjU8J1rEFntxD8q8F3wO2jgTDJDoMs+yWoGso7dGhrQKyB
 oxSKzdjnpU9dK5kf1QcKnXxXp2NnSNCEJ8e0E=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=o3CNYIWm9AKJW/hosEoYm0gjbXfwG22+x1Ji7S2gU6M=;
 b=XC+qsQX82iJm1A92RkuPp7IEWyL/TH/Hm+kUtwK/a5w3MU+LPzIRIUhoTvEkgZVYD5
 jLyuMmG6/Snlwz8Cz7JmILLS9VxoY3JUfIr8T5IZIDBdGb0NvRC7PmG5KIlYRidkIHum
 5K1ZRXi1upTsnuWkFOA/7K4KyBiE9ZDiL9db4mdS7Uf2MgzGJcmvNo8VX3IFocZG5Bdu
 866EBFtqMfSrIY5Oxg7jPXVABM6ocmzaEa83FyJr94nILtixir+jGJuUlRSow+dT367a
 hR4uV9zwSd5hdTY0gyVBpZaZkvu2wJ2P8Z13B0BmGb+Ie9kq3rsSKmEC4A5aB8DukUtr
 xpPg==
X-Gm-Message-State: AOPr4FWZ2S/6IpUEhqhVTHnVRE/1Q6cYRFnRQycX8L5uNBs44Hm0Wl887NZq++YyLlAIJ2DpFWvWDReqDr4kSV4h
MIME-Version: 1.0
X-Received: by 10.140.21.164 with SMTP id 33mr3786018qgl.34.1463756639432;
 Fri, 20 May 2016 08:03:59 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Fri, 20 May 2016 08:03:59 -0700 (PDT)
In-Reply-To: <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
 <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN>
Date: Fri, 20 May 2016 16:03:59 +0100
Message-ID: <CAOnWdoh3FQNeOu3Cvahn6Ggo_esxW91mxzUK4ADV9VP2J32Bfg@HIDDEN>
Subject: Re: bug#23521: XFAIL
From: Reuben Thomas <rrt@HIDDEN>
To: Gavin Smith <gavinsmith0123@HIDDEN>
Content-Type: multipart/alternative; boundary=001a11c13aca8829fa053347669f
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--001a11c13aca8829fa053347669f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 20 May 2016 at 15:58, Gavin Smith <gavinsmith0123@HIDDEN> wrote:

> On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
> >> It is often easier to write expected-to-fail tests this way (so that
> >> they can all look the same), rather than have to have, for example, an
> >> extra driver that converts expected errors into success codes for the
> >> automake test harness.
> >
> > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
>
> This would be a reference to a "custom test driver".
>
> https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom=
-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support
>

=E2=80=8BThanks, that's what I meant.

--=20
http://rrt.sc3d.org

--001a11c13aca8829fa053347669f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
20 May 2016 at 15:58, Gavin Smith <span dir=3D"ltr">&lt;<a href=3D"mailto:g=
avinsmith0123@HIDDEN" target=3D"_blank">gavinsmith0123@HIDDEN</a>&gt;=
</span> wrote:<br></div><div class=3D"gmail_extra"><div class=3D"gmail_quot=
e"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex"><span class=3D"">On 19 May 2016 at 00:04,=
 Mathieu Lirzin &lt;<a href=3D"mailto:mthl@HIDDEN">mthl@HIDDEN</a>&gt; wr=
ote:<br>
</span><span class=3D"">&gt;&gt; It is often easier to write expected-to-fa=
il tests this way (so that<br>
&gt;&gt; they can all look the same), rather than have to have, for example=
, an<br>
&gt;&gt; extra driver that converts expected errors into success codes for =
the<br>
&gt;&gt; automake test harness.<br>
&gt;<br>
&gt; What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?<br>
<br>
</span>This would be a reference to a &quot;custom test driver&quot;.<br>
<a href=3D"https://www.gnu.org/software/automake/manual/html_node/Overview-=
of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support=
" rel=3D"noreferrer" target=3D"_blank">https://www.gnu.org/software/automak=
e/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of=
-Custom-Test-Drivers-Support</a><br>
</blockquote></div><br><div class=3D"gmail_default" style=3D"font-size:smal=
l">=E2=80=8BThanks, that&#39;s what I meant.</div><div><br></div>-- <br><di=
v class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" target=3D"_blan=
k">http://rrt.sc3d.org</a></div>
</div></div>

--001a11c13aca8829fa053347669f--




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 20 May 2016 14:58:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 10:58:09 2016
Received: from localhost ([127.0.0.1]:59195 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3lsO-0007YA-Te
	for submit <at> debbugs.gnu.org; Fri, 20 May 2016 10:58:09 -0400
Received: from mail-yw0-f174.google.com ([209.85.161.174]:33491)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gavinsmith0123@HIDDEN>) id 1b3lsO-0007We-2v
 for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 10:58:08 -0400
Received: by mail-yw0-f174.google.com with SMTP id x194so111436017ywd.0
 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 07:58:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc:content-transfer-encoding;
 bh=eYDZM4mJzWTHm+BsbEhjb3IIzcvoafekfERuphxc8pM=;
 b=WF/GRNCPsBvjDIkoKWNUhmofQQ23n4Ky/KTI21kyeyNNDeUaS8P7Yt/LKrAnzkzCUW
 zPyXTKqBhhdzT3OsWqK8zQnoEyZHCRiNMdwOOAda+O1BTrYtBdWy9iPouJ/gFit2Gu5x
 +clC2ktiNfQJAuIckjseekrBMP55cWIes9NdZ/hcSHD7373xMDBGCiv7wAHYG+93VVQy
 XBOxM9meprz/s/f3Tb6yivmII5nFAdXz5jpUvzRNKs1vJG9PViyfnrVMwAgvfsvdh0Ba
 xiX342rZVyPbOO3MWiwT5ZqKr1X8yX+A+QWpnYeefg4BCyJENYiwstDPxydBk1ldjwKu
 HUTg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc:content-transfer-encoding;
 bh=eYDZM4mJzWTHm+BsbEhjb3IIzcvoafekfERuphxc8pM=;
 b=l1gZ1KGXPfZcto80NHoFCRiv1sZUGuuU21lLaruBJ0iP0+ulrZ1tsWqcAsiun6mWjr
 qOd/ksqILTEOQmmR+HXT9dgEEGVOkJcGovdk6b55mMun9IfYlDUKCiViBQVXI+QvM905
 Oem3lGJkJBmH/jHgdRH4rgisfINans/cOpkrJKeILKX866b3m6wus+Qp7LVpWUJ5Rv6Q
 G/KAKaoahP73w5SEdwgYx8Mn2pCp3E4NiMcNE2EA/z84e33KxALKFE/Uk6zYk451Jxuk
 EKAe0aJ3iIC6MC5h7JLuuEm+mHFHw46LBLQiT9cDtnYiZQxtff8bnKhj3h17Aq4UQeeb
 2UeQ==
X-Gm-Message-State: AOPr4FVLa+fvq1m86tKIsvfcvEIEm+KWiMuA7dkKqAxVxX3w1BMK2Bvh7WfHJNdELaklyxAtCNCkp+BDAunZeA==
MIME-Version: 1.0
X-Received: by 10.129.106.9 with SMTP id f9mr2229650ywc.90.1463756282398; Fri,
 20 May 2016 07:58:02 -0700 (PDT)
Received: by 10.37.215.200 with HTTP; Fri, 20 May 2016 07:58:02 -0700 (PDT)
In-Reply-To: <87futfq877.fsf@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
Date: Fri, 20 May 2016 15:58:02 +0100
Message-ID: <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN>
Subject: Re: bug#23521: XFAIL
From: Gavin Smith <gavinsmith0123@HIDDEN>
To: Mathieu Lirzin <mthl@HIDDEN>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.2 (/)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org, Reuben Thomas <rrt@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.2 (/)

On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:
>> It is often easier to write expected-to-fail tests this way (so that
>> they can all look the same), rather than have to have, for example, an
>> extra driver that converts expected errors into success codes for the
>> automake test harness.
>
> What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?

This would be a reference to a "custom test driver".
https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-T=
est-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 19 May 2016 09:16:36 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 19 05:16:36 2016
Received: from localhost ([127.0.0.1]:57618 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3K4K-00081p-0L
	for submit <at> debbugs.gnu.org; Thu, 19 May 2016 05:16:36 -0400
Received: from mail-qk0-f170.google.com ([209.85.220.170]:34808)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3K4H-00081b-U0
 for 23521 <at> debbugs.gnu.org; Thu, 19 May 2016 05:16:34 -0400
Received: by mail-qk0-f170.google.com with SMTP id y126so9000590qke.1
 for <23521 <at> debbugs.gnu.org>; Thu, 19 May 2016 02:16:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=9QXDUEyA69fDc+y4Y4AAdyEw/TDMoqQ9rIDYYLoEF6s=;
 b=DyRWAHsNaxZAzhGe89ux4j8ZCCPL2cgFShz082YeIk25LJL6GZFJT8C47Led+w843T
 //ub2ERyf6NoTn1PmnvIbLx54fERN3shu7FMEwlEPFc4HEDIyv24HEYMaxh0jvao12G0
 IGOJj6aDg4bfR10w2EKFHBuIBJtW6g5uZDdKs=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=9QXDUEyA69fDc+y4Y4AAdyEw/TDMoqQ9rIDYYLoEF6s=;
 b=W4Lns2lWuUedfx3YiObbEFSiiMWICBmM/ZFEFg1SL49O1GopAVc9XBWyXzVAbgQSRQ
 rkHnniPBXzbAFVEPfFaqnBlWxbwAajY2H2n5Bdf//MzxEtUWCg3PBrTINTHSIu3tTqut
 7kFUZTksXko0QhACG20u8gpzUzroQcZfqyS/QIJuZYl7xyWXBkDZU7YDTBVOsdk1O0HT
 4lCHgI+FuPfFk2wffcf/S0k90v2Dtn3doob9KA4YjVnVz4ZzTEVURiNPac6EYKvTYdyN
 b9+ZCuPEZb0l4CElNpefgJqpfTFG3DRu64ReklV/Rx3d4+tl79vqzi9n2hQwyd3Vs1nO
 v7jA==
X-Gm-Message-State: AOPr4FXg+819s0MD2VdN3i4HupmlXyqUspk4/ZlwOqP3HE5Z9GoxOgZlO/Mz/BSBETThG/ohUWAnE3CPs7PnIrwq
MIME-Version: 1.0
X-Received: by 10.55.186.132 with SMTP id k126mr12880567qkf.176.1463649388628; 
 Thu, 19 May 2016 02:16:28 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Thu, 19 May 2016 02:16:28 -0700 (PDT)
In-Reply-To: <573D0107.1@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN> <573D0107.1@HIDDEN>
Date: Thu, 19 May 2016 10:16:28 +0100
Message-ID: <CAOnWdogbgeiS2YiLYXupMsibk6c5bueHPirm-gfKT0znufEAUw@HIDDEN>
Subject: Re: bug#23521: XFAIL
From: Reuben Thomas <rrt@HIDDEN>
To: Peter Johansson <trojkan@HIDDEN>
Content-Type: multipart/alternative; boundary=94eb2c043626e2b45c05332e6d82
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--94eb2c043626e2b45c05332e6d82
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 19 May 2016 at 00:55, Peter Johansson <trojkan@HIDDEN> wrote:

>
>
> On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>
>> Another common use for "expected failure" is to write tests to check
>>> >that error conditions arise as expected, for example, by checking that
>>> >a program raises an error when given invalid input.
>>>
>> I agree that XFAIL can be ambiguous, however I think this usage is not
>> desirable.  It gives an additional opposite meaning to XFAIL symbol
>> which makes it even more confusing.
>>
>> I agree. When I wanna tests that a program fails with incorrect input, I
> prefer writing a tests that calls the program, check that it fails (exit =
1
> or whatever is expected), and perhaps even parse the error message, and i=
f
> it looks like I expect exit 0 aka PASS.
>

=E2=80=8BThanks. I shall continue with my "deviant" usage for now, but if t=
hat is
not considered normal, I understand that you won't want to document it.

--=20
http://rrt.sc3d.org

--94eb2c043626e2b45c05332e6d82
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
19 May 2016 at 00:55, Peter Johansson <span dir=3D"ltr">&lt;<a href=3D"mail=
to:trojkan@HIDDEN" target=3D"_blank">trojkan@HIDDEN</a>&gt;</span> wr=
ote:<br></div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockq=
uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><span class=3D""><br>
<br>
On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Another common use for &quot;expected failure&quot; is to write tests to ch=
eck<br>
&gt;that error conditions arise as expected, for example, by checking that<=
br>
&gt;a program raises an error when given invalid input.<br>
</blockquote>
I agree that XFAIL can be ambiguous, however I think this usage is not<br>
desirable.=C2=A0 It gives an additional opposite meaning to XFAIL symbol<br=
>
which makes it even more confusing.<br>
<br>
</blockquote></span>
I agree. When I wanna tests that a program fails with incorrect input, I pr=
efer writing a tests that calls the program, check that it fails (exit 1 or=
 whatever is expected), and perhaps even parse the error message, and if it=
 looks like I expect exit 0 aka PASS.<br></blockquote><div><br></div><div><=
div class=3D"gmail_default" style=3D"font-size:small;display:inline">=E2=80=
=8BThanks. I shall continue with my &quot;deviant&quot; usage for now, but =
if that is not considered normal, I understand that you won&#39;t want to d=
ocument it.</div></div></div><div><br></div>-- <br><div class=3D"gmail_sign=
ature"><a href=3D"http://rrt.sc3d.org" target=3D"_blank">http://rrt.sc3d.or=
g</a></div>
</div></div>

--94eb2c043626e2b45c05332e6d82--




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 19 May 2016 09:15:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 19 05:15:40 2016
Received: from localhost ([127.0.0.1]:57614 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3K3Q-000800-NE
	for submit <at> debbugs.gnu.org; Thu, 19 May 2016 05:15:40 -0400
Received: from mail-qg0-f43.google.com ([209.85.192.43]:35912)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b3K3O-0007zn-M0
 for 23521 <at> debbugs.gnu.org; Thu, 19 May 2016 05:15:39 -0400
Received: by mail-qg0-f43.google.com with SMTP id w36so39853253qge.3
 for <23521 <at> debbugs.gnu.org>; Thu, 19 May 2016 02:15:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :cc; bh=H9b5kaCa5pxk3kedrvv5TqWWKXI2dITHPikbDJqB7Ao=;
 b=wI75O6ZJuuk8v0IFz3LlUpxJwKXPObirRLxvWGQm4yvRM+pqW3lit7ZFpst3AiLxZY
 ED1pteQWQPgeEAVEZzddfSG0Ihi4cqkMqCWHtF8UjOdScGrQLdT6yoTuGjGNI+6Qa/ZA
 ICWS3oaj+cjefxnri6DIaZKmLYYkkAGgtKEwY=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:date
 :message-id:subject:from:to:cc;
 bh=H9b5kaCa5pxk3kedrvv5TqWWKXI2dITHPikbDJqB7Ao=;
 b=gbp/d8JY1oE8PaxIK/BAkpkN0aIlLiaPGtVgaX2HAtNvDGM82dQ4onpQDq94cQFc2a
 /yvNMLK/2yUychyR7UPXEoFWNlXIRNqnUqUn3yei/17jmaCucz1QWmuqN/mfgzX6dumU
 5vvTTg9cPRXLMNRXGk0wH3qb//6Z9sAxvavgNuQ4NbVJ/XrwEOVNQD4NPiK6sAFGe/2C
 TTouYompzXY1fDMNsuwvMVNZEe0pR7OBagCjW5Z2i7yfSv2dKx1MhbZfySmjDlxGJwXi
 ZdmSdHHAAOE7maWHr9n89W+ILDYHMpJelEQpudiVBIOHm650CqUt6KMC3XJKx+y1vTPe
 KbVQ==
X-Gm-Message-State: AOPr4FWm439gkdevKAD5okdYG1LUPkQeJu+heM+ngyzMIbg/0PEmDTkJQf7TA9C1BwjIGWHeNwYosTUqn4k3T81l
MIME-Version: 1.0
X-Received: by 10.140.102.117 with SMTP id v108mr12318743qge.103.1463649333173; 
 Thu, 19 May 2016 02:15:33 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Thu, 19 May 2016 02:15:33 -0700 (PDT)
In-Reply-To: <87futfq877.fsf@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
Date: Thu, 19 May 2016 10:15:33 +0100
Message-ID: <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN>
Subject: Re: bug#23521: XFAIL
From: Reuben Thomas <rrt@HIDDEN>
To: Mathieu Lirzin <mthl@HIDDEN>
Content-Type: multipart/alternative; boundary=001a11c16ec494811405332e6a61
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

--001a11c16ec494811405332e6a61
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote:

> > It is often easier to write expected-to-fail tests this way (so that
> > they can all look the same), rather than have to have, for example, an
> > extra driver that converts expected errors into success codes for the
> > automake test harness.
>
> What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?
>

=E2=80=8BA custom test driver.=E2=80=8B

--=20
http://rrt.sc3d.org

--001a11c16ec494811405332e6a61
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On =
19 May 2016 at 00:04, Mathieu Lirzin <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:mthl@HIDDEN" target=3D"_blank">mthl@HIDDEN</a>&gt;</span> wrote:<br></d=
iv><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">&gt; It is often easier to write expected-to-fail tests this =
way (so that<br>
&gt; they can all look the same), rather than have to have, for example, an=
<br>
&gt; extra driver that converts expected errors into success codes for the<=
br>
&gt; automake test harness.<br>
<br>
What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?<br></block=
quote><div><br></div><div><div class=3D"gmail_default" style=3D"font-size:s=
mall">=E2=80=8BA custom test driver.=E2=80=8B</div></div></div><div><br></d=
iv>-- <br><div class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" ta=
rget=3D"_blank">http://rrt.sc3d.org</a></div>
</div></div>

--001a11c16ec494811405332e6a61--




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 18 May 2016 23:56:02 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed May 18 19:56:02 2016
Received: from localhost ([127.0.0.1]:57428 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3BJq-000396-52
	for submit <at> debbugs.gnu.org; Wed, 18 May 2016 19:56:02 -0400
Received: from mail-pf0-f196.google.com ([209.85.192.196]:33706)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <trojkan@HIDDEN>) id 1b3BJo-00038l-Vy
 for 23521 <at> debbugs.gnu.org; Wed, 18 May 2016 19:56:01 -0400
Received: by mail-pf0-f196.google.com with SMTP id y7so6324193pfb.0
 for <23521 <at> debbugs.gnu.org>; Wed, 18 May 2016 16:56:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=subject:to:references:cc:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-transfer-encoding;
 bh=FpB6RoxmRq4DEIPrjfOlMeDgxj35kqjHioS3eXMde5Q=;
 b=rStw0738S2PKh272PeWK6txVbfrZiTy3f2FNLbDRTNtzz0wKnswnWqzTsXPafoHqEU
 2x2kiiym+4ja5Db5HgVuqeBxMaMvFxZii2JOjrEbrCCw+Z4cnA1J1BBo5df8jsZfXAWW
 g5ZWe/QBsN1F3ik4npnd521fa3LlucrkB+sz1NRh7p4zkngRQcSekGQNuCFBy9CN+eDy
 pcu4reWnXU/vVjqhMhiS9fl6Ko9NLwOTKr4D6CN0Rw+X6uaGK6rcGoJXTom/QXOwj2ru
 g+9figeQoWlJTm/a2anFYihrfV4bIhe+Tj1hpOW9S6D+jJ3SbAwieoReHu/kWIRvnvDe
 jIKQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:subject:to:references:cc:from:message-id:date
 :user-agent:mime-version:in-reply-to:content-transfer-encoding;
 bh=FpB6RoxmRq4DEIPrjfOlMeDgxj35kqjHioS3eXMde5Q=;
 b=lTof6hOBQUhvlYuURfD/Zh/vAnuerFuSb6AfAgFe5M4uYhdvTWu8aLvXwq2c2jJzuL
 25t/859jz9HecgEyb9EVTR/cD1CRJrF/GMMvHcjy7IYbrxunko+rynj8QgEtPMivgIL0
 IHUfDVhYw7C41FzLBzlpVu+7WQ2kkQMiZ5+F+zB17M/Ahi+YzANj/TqXHbZjiYmVhNyp
 t0KrTBRepR982LtOOwtcIRvuYSRVaS1AavZR+mV728TiFU3YQmFIVPEkuSbsIzseXZVx
 NzMT+pOdlyBlEk6aS26dz8JoDCVMbILGbGeciq2aE+etm1t2ZW8ZwVtkpfjtd/vrHz5d
 TEDw==
X-Gm-Message-State: AOPr4FW+/lTo3yMBOMAn7Yth2XDQmsDGwRbl7/VdjuxF/Of+ahO9muFqiSCqEE83p3Lm9w==
X-Received: by 10.98.69.213 with SMTP id n82mr15033934pfi.48.1463615755202;
 Wed, 18 May 2016 16:55:55 -0700 (PDT)
Received: from [10.3.81.221] (gateway.qimr.edu.au. [152.98.8.1])
 by smtp.gmail.com with ESMTPSA id g5sm14841513pac.1.2016.05.18.16.55.53
 (version=TLSv1/SSLv3 cipher=OTHER);
 Wed, 18 May 2016 16:55:54 -0700 (PDT)
Subject: Re: bug#23521: XFAIL
To: Mathieu Lirzin <mthl@HIDDEN>, Reuben Thomas <rrt@HIDDEN>
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 <87futfq877.fsf@HIDDEN>
From: Peter Johansson <trojkan@HIDDEN>
Message-ID: <573D0107.1@HIDDEN>
Date: Thu, 19 May 2016 09:55:51 +1000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <87futfq877.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)



On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>> Another common use for "expected failure" is to write tests to check
>> >that error conditions arise as expected, for example, by checking that
>> >a program raises an error when given invalid input.
> I agree that XFAIL can be ambiguous, however I think this usage is not
> desirable.  It gives an additional opposite meaning to XFAIL symbol
> which makes it even more confusing.
>
I agree. When I wanna tests that a program fails with incorrect input, I 
prefer writing a tests that calls the program, check that it fails (exit 
1 or whatever is expected), and perhaps even parse the error message, 
and if it looks like I expect exit 0 aka PASS.

Cheers,
Peter




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at 23521 <at> debbugs.gnu.org:


Received: (at 23521) by debbugs.gnu.org; 18 May 2016 23:04:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed May 18 19:04:39 2016
Received: from localhost ([127.0.0.1]:57408 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b3AW6-0001zh-Pj
	for submit <at> debbugs.gnu.org; Wed, 18 May 2016 19:04:38 -0400
Received: from v-zimmta02.u-bordeaux.fr ([147.210.215.20]:54076)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mthl@HIDDEN>) id 1b3AW5-0001zS-8n
 for 23521 <at> debbugs.gnu.org; Wed, 18 May 2016 19:04:37 -0400
Received: from v-zimmta02.u-bordeaux.fr (localhost [127.0.0.1])
 by v-zimmta02.u-bordeaux.fr (Postfix) with ESMTP id BB3AD8007A2;
 Thu, 19 May 2016 01:04:29 +0200 (CEST)
Received: from godel (mek33-4-82-236-46-88.fbx.proxad.net [82.236.46.88])
 by v-zimmta02.u-bordeaux.fr (Postfix) with ESMTPSA id 5AC2F8000AD;
 Thu, 19 May 2016 01:04:29 +0200 (CEST)
From: Mathieu Lirzin <mthl@HIDDEN>
To: Reuben Thomas <rrt@HIDDEN>
Subject: Re: bug#23521: XFAIL
References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
Date: Thu, 19 May 2016 01:04:28 +0200
In-Reply-To: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
 (Reuben Thomas's message of "Thu, 12 May 2016 11:01:11 +0100")
Message-ID: <87futfq877.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-AV-Checked: ClamAV using ClamSMTP
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: 23521
Cc: 23521 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.4 (-)

Hi,

Reuben Thomas <rrt@HIDDEN> writes:

> The documentation says: "It's not uncommon, especially during early
> development stages, that some tests fail for known reasons, and that
> the developer doesn't want
> to tackle these failures immediately (this is especially true when the
> failing tests deal with corner cases)."
>
> Another common use for "expected failure" is to write tests to check
> that error conditions arise as expected, for example, by checking that
> a program raises an error when given invalid input.

I agree that XFAIL can be ambiguous, however I think this usage is not
desirable.  It gives an additional opposite meaning to XFAIL symbol
which makes it even more confusing.

> If that's a reasonable use of automake's test harness, perhaps the
> documentation could reflect that, e.g. by adding:
>
> "Another use for XFAIL is to mark tests that are supposed to fail, for
> example, to check that a program raises an error when given invalid
> input."
>
> It is often easier to write expected-to-fail tests this way (so that
> they can all look the same), rather than have to have, for example, an
> extra driver that converts expected errors into success codes for the
> automake test harness.

What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?

Thanks

--=20
Mathieu Lirzin




Information forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 12 May 2016 10:01:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 12 06:01:33 2016
Received: from localhost ([127.0.0.1]:48072 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1b0nQz-0003X7-Ex
	for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:33 -0400
Received: from eggs.gnu.org ([208.118.235.92]:33667)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rrt@HIDDEN>) id 1b0nQw-0003Wq-Bv
 for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:31 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <rrt@HIDDEN>) id 1b0nQl-0004RZ-Fz
 for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:25 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,HTML_MESSAGE,
 T_DKIM_INVALID autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:54770)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <rrt@HIDDEN>)
 id 1b0nQl-0004RL-DD
 for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:19 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:52851)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <rrt@HIDDEN>) id 1b0nQi-00073L-IZ
 for bug-automake@HIDDEN; Thu, 12 May 2016 06:01:18 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <rrt@HIDDEN>) id 1b0nQe-0004Ps-20
 for bug-automake@HIDDEN; Thu, 12 May 2016 06:01:16 -0400
Received: from mail-qk0-x22c.google.com ([2607:f8b0:400d:c09::22c]:34176)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <rrt@HIDDEN>)
 id 1b0nQd-0004Pk-SV
 for bug-automake@HIDDEN; Thu, 12 May 2016 06:01:11 -0400
Received: by mail-qk0-x22c.google.com with SMTP id r184so40103544qkc.1
 for <bug-automake@HIDDEN>; Thu, 12 May 2016 03:01:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google;
 h=mime-version:date:message-id:subject:from:to;
 bh=90Bvch9qyJpFx1/F1bH6u18lrgeQl9rS0cF5XMxXNdQ=;
 b=4L4ZlCiSIdUTyheATYwZU7IzveM+fiOYPdGJXliVUHjv6k8XqTbgTbu9wEbVcSrWai
 55T8Q3LCuWieietFHqySa8GZ0/fTPKz1ZQNer3GfLz0QOVu6183OPbpxDaHB6Kn3CyRT
 hP/n64RreDM/oN0H8+GrYTJg3egeeushsGyMQ=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:date:message-id:subject:from:to;
 bh=90Bvch9qyJpFx1/F1bH6u18lrgeQl9rS0cF5XMxXNdQ=;
 b=hYaz+UicAvrcgkG52WACs3K3YGdQcykWb2vvM72u04PpuqtYKEy0w3cSUQ3nkoCIJI
 TlXgaJjjH+20oc9nNGNJ7rwxFZM9cEZ7B+53s+ZAd4A7oamvJLHV9T1gLrXl2KgQWtQJ
 6LBVDVSxZ1yGQP9Yd9Lyt5YblRK16PaW7O3Z0x/yZ07QrcR2vWBVZgGxBVbtODy4cyS8
 UeuLbstlnhh5brqTXupbvNqoJ1CXG1exDGwJVUtVCi1Dd/ZbQOTqxeirPWQvSSWrh9rg
 Q4BsVGNzbsJZOOeb3jZghKAeCR0N6Evio5NnyHA+W0u/ijkBCgMw4Plml+fss1HrouT0
 uhZg==
X-Gm-Message-State: AOPr4FXT6Z1Vj8uQ263cSoOQYsaodZkH75XxFPL5sr0BKU59lbyEjy4Rnd+Yst/gWy/pXbXENEC2COzvnikdBZZ5
MIME-Version: 1.0
X-Received: by 10.55.180.133 with SMTP id d127mr8849603qkf.131.1463047271038; 
 Thu, 12 May 2016 03:01:11 -0700 (PDT)
Received: by 10.55.86.70 with HTTP; Thu, 12 May 2016 03:01:11 -0700 (PDT)
Date: Thu, 12 May 2016 11:01:11 +0100
Message-ID: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN>
Subject: XFAIL
From: Reuben Thomas <rrt@HIDDEN>
To: bug-automake <bug-automake@HIDDEN>
Content-Type: multipart/alternative; boundary=94eb2c06fad6e1575c0532a23cc6
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.1 (----)

--94eb2c06fad6e1575c0532a23cc6
Content-Type: text/plain; charset=UTF-8

The documentation says: "It's not uncommon, especially during early
development stages, that some tests fail for known reasons, and that the
developer doesn't want
to tackle these failures immediately (this is especially true when the
failing tests deal with corner cases)."

Another common use for "expected failure" is to write tests to check that
error conditions arise as expected, for example, by checking that a program
raises an error when given invalid input.

If that's a reasonable use of automake's test harness, perhaps the
documentation could reflect that, e.g. by adding:

"Another use for XFAIL is to mark tests that are supposed to fail, for
example, to check that a program raises an error when given invalid input."

It is often easier to write expected-to-fail tests this way (so that they
can all look the same), rather than have to have, for example, an extra
driver that converts expected errors into success codes for the automake
test harness.

-- 
http://rrt.sc3d.org

--94eb2c06fad6e1575c0532a23cc6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">The=
 documentation says: &quot;It&#39;s not uncommon, especially during early d=
evelopment stages, that some tests fail for known reasons, and that the dev=
eloper doesn&#39;t want</div><div class=3D"gmail_default">to tackle these f=
ailures immediately (this is especially true when the</div><div class=3D"gm=
ail_default">failing tests deal with corner cases).&quot;</div><div class=
=3D"gmail_default"><br></div><div class=3D"gmail_default">Another common us=
e for &quot;expected failure&quot; is to write tests to check that error co=
nditions arise as expected, for example, by checking that a program raises =
an error when given invalid input.</div><div class=3D"gmail_default"><br></=
div><div class=3D"gmail_default">If that&#39;s a reasonable use of automake=
&#39;s test harness, perhaps the documentation could reflect that, e.g. by =
adding:</div><div class=3D"gmail_default"><br></div><div class=3D"gmail_def=
ault">&quot;Another use for XFAIL is to mark tests that are supposed to fai=
l, for example, to check that a program raises an error when given invalid =
input.&quot;</div><div class=3D"gmail_default"><br></div><div class=3D"gmai=
l_default">It is often easier to write expected-to-fail tests this way (so =
that they can all look the same), rather than have to have, for example, an=
 extra driver that converts expected errors into success codes for the auto=
make test harness.</div><div><br></div>-- <br><div class=3D"gmail_signature=
"><a href=3D"http://rrt.sc3d.org" target=3D"_blank">http://rrt.sc3d.org</a>=
</div>
</div>

--94eb2c06fad6e1575c0532a23cc6--




Acknowledgement sent to Reuben Thomas <rrt@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-automake@HIDDEN. Full text available.
Report forwarded to bug-automake@HIDDEN:
bug#23521; Package automake. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sat, 15 Jul 2017 16:15:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.