Mike Frysinger <vapier@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Mathieu Lirzin <mthl@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Received: (at 23521) by debbugs.gnu.org; 20 May 2016 16:00:39 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 12:00:39 2016 Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3mqs-0002NJ-OQ for submit <at> debbugs.gnu.org; Fri, 20 May 2016 12:00:38 -0400 Received: from mail-qk0-f180.google.com ([209.85.220.180]:35209) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <rrt@HIDDEN>) id 1b3mqq-0002N3-5s for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 12:00:37 -0400 Received: by mail-qk0-f180.google.com with SMTP id n62so61136393qkc.2 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 09:00:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=Onr1MyN7x6Lwx8jzFGhgjCtI73LIXMWIEk4aWFGToLY=; b=ZkVkE1Ct0I96MfwOJ0i8HM58ts4CXZYL8gdqLFjVYEy7g1HZS1pQ2nOKmktl05b0R4 eeG9o2uWzKS8bHcR23t5g0VGoWDlov2eegAqabAFIk0gftQPcdSLVMDHbwfheqpMtcaS ccgF8GA1EyqyeIrcfFVQ1Jx1wP51S8WMbZ1zI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=Onr1MyN7x6Lwx8jzFGhgjCtI73LIXMWIEk4aWFGToLY=; b=NDL6ULybKWKgXYngTcgoNqh2/9RFW1uB0KIWzyw01NeM6o3rerCoCJ/RbU8BPw2j/1 ZqO7dC6i/u1CpzLUy/n3UxBW+yskS2YeOF9DWj0NMTbh1OUeN6GRzrK/+bzFkHiW5BWU MOj7WoSwsTS4L0RPYrPMpvy2VkMvDAohrHzvnHFtgxROwugVJ3RAARGtbD/GWFjfxax3 ZaZ83MHJFxZIQTwqxGVDfOyYX4eM+uLuCUarbgq3TCkiL2+Oc4umWeSY6WwVEk3UyvTF N0vZx9+XPYgRGMAcYeEwQHA9IfjwUYjDrK5BiNOwmUrkJN8ZTD+k+xbgM5axsbgJ+p7Y pADQ== X-Gm-Message-State: AOPr4FWELZJA912wpahgefXyRkm4AbE//2bpBHJu4KMIj0Q7AyfZTX9Ki1FxjtwEWtjr420jaikILVbcc/pzrMhN MIME-Version: 1.0 X-Received: by 10.55.186.132 with SMTP id k126mr4213429qkf.176.1463760030523; Fri, 20 May 2016 09:00:30 -0700 (PDT) Received: by 10.55.86.70 with HTTP; Fri, 20 May 2016 09:00:30 -0700 (PDT) In-Reply-To: <8737pclof1.fsf@HIDDEN> References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN> <8737pclof1.fsf@HIDDEN> Date: Fri, 20 May 2016 17:00:30 +0100 Message-ID: <CAOnWdogS8TgSruY5GF6cvBe3NdWo-9DBpOuZ5n8dLOEddTUf3w@HIDDEN> Subject: Re: bug#23521: XFAIL From: Reuben Thomas <rrt@HIDDEN> To: Mathieu Lirzin <mthl@HIDDEN> Content-Type: multipart/alternative; boundary=94eb2c043626a80ef905334830d2 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) --94eb2c043626a80ef905334830d2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 20 May 2016 at 16:49, Mathieu Lirzin <mthl@HIDDEN> wrote: > Reuben Thomas <rrt@HIDDEN> writes: > > > On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote: > > > > > It is often easier to write expected-to-fail tests this way (so > > that > > > they can all look the same), rather than have to have, for > > example, an > > > extra driver that converts expected errors into success codes > > for the > > > automake test harness. > > > > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D? > > > > > > =E2=80=8BA custom test driver.=E2=80=8B > > OK, I wasn't sure. Indeed a custom test driver seems a bit heavy just > checking failures. IMO the solution Peter proposed is nice and simple. > =E2=80=8BWhat Peter proposed is essentially a custom test driver: I would n= ot expect to duplicate the logic to check the return code &c. in each test expected to fail; rather, I would put it in a custom test driver that would handle expected fails and mark them as passes. (My expected fails are all of the same type, i.e. a non-zero exit code. It might additionally be useful, as Peter suggests, to check that an expected error message is produced.) --=20 http://rrt.sc3d.org --94eb2c043626a80ef905334830d2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On = 20 May 2016 at 16:49, Mathieu Lirzin <span dir=3D"ltr"><<a href=3D"mailt= o:mthl@HIDDEN" target=3D"_blank">mthl@HIDDEN</a>></span> wrote:<br></d= iv><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class= =3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd= ing-left:1ex"><span class=3D"">Reuben Thomas <<a href=3D"mailto:rrt@sc3d= .org">rrt@HIDDEN</a>> writes:<br> <br> > On 19 May 2016 at 00:04, Mathieu Lirzin <<a href=3D"mailto:mthl@gnu= .org">mthl@HIDDEN</a>> wrote:<br> ><br> >=C2=A0 =C2=A0 =C2=A0> It is often easier to write expected-to-fail t= ests this way (so<br> >=C2=A0 =C2=A0 =C2=A0that<br> >=C2=A0 =C2=A0 =C2=A0> they can all look the same), rather than have = to have, for<br> >=C2=A0 =C2=A0 =C2=A0example, an<br> >=C2=A0 =C2=A0 =C2=A0> extra driver that converts expected errors int= o success codes<br> >=C2=A0 =C2=A0 =C2=A0for the<br> >=C2=A0 =C2=A0 =C2=A0> automake test harness.<br> ><br> >=C2=A0 =C2=A0 =C2=A0What do you mean precisely by =E2=80=9Can extra dri= ver=E2=80=9D?<br> ><br> ><br> > =E2=80=8BA custom test driver.=E2=80=8B<br> <br> </span>OK, I wasn't sure.=C2=A0 Indeed a custom test driver seems a bit= heavy just<br> checking failures.=C2=A0 IMO the solution Peter proposed is nice and simple= .<br></blockquote><div><br></div><div><div class=3D"gmail_default" style=3D= "font-size:small;display:inline">=E2=80=8BWhat Peter proposed is essentiall= y a custom test driver: I would not expect to duplicate the logic to check = the return code &c. in each test expected to fail; rather, I would put = it in a custom test driver that would handle expected fails and mark them a= s passes. (My expected fails are all of the same type, i.e. a non-zero exit= code. It might additionally be useful, as Peter suggests, to check that an= expected error message is produced.)</div></div></div><div><br></div>-- <b= r><div class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" target=3D"= _blank">http://rrt.sc3d.org</a></div> </div></div> --94eb2c043626a80ef905334830d2--
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 20 May 2016 15:50:09 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 11:50:09 2016 Received: from localhost ([127.0.0.1]:59232 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3mgj-000284-Ei for submit <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:09 -0400 Received: from eggs.gnu.org ([208.118.235.92]:39715) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <mthl@HIDDEN>) id 1b3mgi-00027s-Gd for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <mthl@HIDDEN>) id 1b3mgZ-0006Uj-De for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:50:03 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_40,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50949) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <mthl@HIDDEN>) id 1b3mgZ-0006Ue-A1; Fri, 20 May 2016 11:49:59 -0400 Received: from mek33-4-82-236-46-88.fbx.proxad.net ([82.236.46.88]:36512 helo=godel) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from <mthl@HIDDEN>) id 1b3mgX-0007B1-IK; Fri, 20 May 2016 11:49:57 -0400 From: Mathieu Lirzin <mthl@HIDDEN> To: Reuben Thomas <rrt@HIDDEN> Subject: Re: bug#23521: XFAIL References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN> Date: Fri, 20 May 2016 17:49:54 +0200 In-Reply-To: <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN> (Reuben Thomas's message of "Thu, 19 May 2016 10:15:33 +0100") Message-ID: <8737pclof1.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.4 (------) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.4 (------) Reuben Thomas <rrt@HIDDEN> writes: > On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote: > > > It is often easier to write expected-to-fail tests this way (so > that > > they can all look the same), rather than have to have, for > example, an > > extra driver that converts expected errors into success codes > for the > > automake test harness. >=20=20=20=20=20 > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D? >=20=20=20=20=20 > > =E2=80=8BA custom test driver.=E2=80=8B OK, I wasn't sure. Indeed a custom test driver seems a bit heavy just checking failures. IMO the solution Peter proposed is nice and simple. --=20 Mathieu Lirzin
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 20 May 2016 15:04:06 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 11:04:06 2016 Received: from localhost ([127.0.0.1]:59200 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3lyA-0007m4-JN for submit <at> debbugs.gnu.org; Fri, 20 May 2016 11:04:06 -0400 Received: from mail-qg0-f48.google.com ([209.85.192.48]:35135) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <rrt@HIDDEN>) id 1b3ly8-0007lW-VJ for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 11:04:05 -0400 Received: by mail-qg0-f48.google.com with SMTP id j18so13490944qge.2 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 08:04:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=o3CNYIWm9AKJW/hosEoYm0gjbXfwG22+x1Ji7S2gU6M=; b=VkcSWvRCQ1xcefygHGodgm+m8AwpDki7lpL66ZHr8UV8J9CgzmViK9myczfK9xfJew kNc5K6X4+nsDxQ7R8Esn8FFjU8J1rEFntxD8q8F3wO2jgTDJDoMs+yWoGso7dGhrQKyB oxSKzdjnpU9dK5kf1QcKnXxXp2NnSNCEJ8e0E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=o3CNYIWm9AKJW/hosEoYm0gjbXfwG22+x1Ji7S2gU6M=; b=XC+qsQX82iJm1A92RkuPp7IEWyL/TH/Hm+kUtwK/a5w3MU+LPzIRIUhoTvEkgZVYD5 jLyuMmG6/Snlwz8Cz7JmILLS9VxoY3JUfIr8T5IZIDBdGb0NvRC7PmG5KIlYRidkIHum 5K1ZRXi1upTsnuWkFOA/7K4KyBiE9ZDiL9db4mdS7Uf2MgzGJcmvNo8VX3IFocZG5Bdu 866EBFtqMfSrIY5Oxg7jPXVABM6ocmzaEa83FyJr94nILtixir+jGJuUlRSow+dT367a hR4uV9zwSd5hdTY0gyVBpZaZkvu2wJ2P8Z13B0BmGb+Ie9kq3rsSKmEC4A5aB8DukUtr xpPg== X-Gm-Message-State: AOPr4FWZ2S/6IpUEhqhVTHnVRE/1Q6cYRFnRQycX8L5uNBs44Hm0Wl887NZq++YyLlAIJ2DpFWvWDReqDr4kSV4h MIME-Version: 1.0 X-Received: by 10.140.21.164 with SMTP id 33mr3786018qgl.34.1463756639432; Fri, 20 May 2016 08:03:59 -0700 (PDT) Received: by 10.55.86.70 with HTTP; Fri, 20 May 2016 08:03:59 -0700 (PDT) In-Reply-To: <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN> References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN> Date: Fri, 20 May 2016 16:03:59 +0100 Message-ID: <CAOnWdoh3FQNeOu3Cvahn6Ggo_esxW91mxzUK4ADV9VP2J32Bfg@HIDDEN> Subject: Re: bug#23521: XFAIL From: Reuben Thomas <rrt@HIDDEN> To: Gavin Smith <gavinsmith0123@HIDDEN> Content-Type: multipart/alternative; boundary=001a11c13aca8829fa053347669f X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) --001a11c13aca8829fa053347669f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 20 May 2016 at 15:58, Gavin Smith <gavinsmith0123@HIDDEN> wrote: > On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote: > >> It is often easier to write expected-to-fail tests this way (so that > >> they can all look the same), rather than have to have, for example, an > >> extra driver that converts expected errors into success codes for the > >> automake test harness. > > > > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D? > > This would be a reference to a "custom test driver". > > https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom= -Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support > =E2=80=8BThanks, that's what I meant. --=20 http://rrt.sc3d.org --001a11c13aca8829fa053347669f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On = 20 May 2016 at 15:58, Gavin Smith <span dir=3D"ltr"><<a href=3D"mailto:g= avinsmith0123@HIDDEN" target=3D"_blank">gavinsmith0123@HIDDEN</a>>= </span> wrote:<br></div><div class=3D"gmail_extra"><div class=3D"gmail_quot= e"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left= :1px #ccc solid;padding-left:1ex"><span class=3D"">On 19 May 2016 at 00:04,= Mathieu Lirzin <<a href=3D"mailto:mthl@HIDDEN">mthl@HIDDEN</a>> wr= ote:<br> </span><span class=3D"">>> It is often easier to write expected-to-fa= il tests this way (so that<br> >> they can all look the same), rather than have to have, for example= , an<br> >> extra driver that converts expected errors into success codes for = the<br> >> automake test harness.<br> ><br> > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?<br> <br> </span>This would be a reference to a "custom test driver".<br> <a href=3D"https://www.gnu.org/software/automake/manual/html_node/Overview-= of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support= " rel=3D"noreferrer" target=3D"_blank">https://www.gnu.org/software/automak= e/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of= -Custom-Test-Drivers-Support</a><br> </blockquote></div><br><div class=3D"gmail_default" style=3D"font-size:smal= l">=E2=80=8BThanks, that's what I meant.</div><div><br></div>-- <br><di= v class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" target=3D"_blan= k">http://rrt.sc3d.org</a></div> </div></div> --001a11c13aca8829fa053347669f--
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 20 May 2016 14:58:09 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri May 20 10:58:09 2016 Received: from localhost ([127.0.0.1]:59195 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3lsO-0007YA-Te for submit <at> debbugs.gnu.org; Fri, 20 May 2016 10:58:09 -0400 Received: from mail-yw0-f174.google.com ([209.85.161.174]:33491) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <gavinsmith0123@HIDDEN>) id 1b3lsO-0007We-2v for 23521 <at> debbugs.gnu.org; Fri, 20 May 2016 10:58:08 -0400 Received: by mail-yw0-f174.google.com with SMTP id x194so111436017ywd.0 for <23521 <at> debbugs.gnu.org>; Fri, 20 May 2016 07:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-transfer-encoding; bh=eYDZM4mJzWTHm+BsbEhjb3IIzcvoafekfERuphxc8pM=; b=WF/GRNCPsBvjDIkoKWNUhmofQQ23n4Ky/KTI21kyeyNNDeUaS8P7Yt/LKrAnzkzCUW zPyXTKqBhhdzT3OsWqK8zQnoEyZHCRiNMdwOOAda+O1BTrYtBdWy9iPouJ/gFit2Gu5x +clC2ktiNfQJAuIckjseekrBMP55cWIes9NdZ/hcSHD7373xMDBGCiv7wAHYG+93VVQy XBOxM9meprz/s/f3Tb6yivmII5nFAdXz5jpUvzRNKs1vJG9PViyfnrVMwAgvfsvdh0Ba xiX342rZVyPbOO3MWiwT5ZqKr1X8yX+A+QWpnYeefg4BCyJENYiwstDPxydBk1ldjwKu HUTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-transfer-encoding; bh=eYDZM4mJzWTHm+BsbEhjb3IIzcvoafekfERuphxc8pM=; b=l1gZ1KGXPfZcto80NHoFCRiv1sZUGuuU21lLaruBJ0iP0+ulrZ1tsWqcAsiun6mWjr qOd/ksqILTEOQmmR+HXT9dgEEGVOkJcGovdk6b55mMun9IfYlDUKCiViBQVXI+QvM905 Oem3lGJkJBmH/jHgdRH4rgisfINans/cOpkrJKeILKX866b3m6wus+Qp7LVpWUJ5Rv6Q G/KAKaoahP73w5SEdwgYx8Mn2pCp3E4NiMcNE2EA/z84e33KxALKFE/Uk6zYk451Jxuk EKAe0aJ3iIC6MC5h7JLuuEm+mHFHw46LBLQiT9cDtnYiZQxtff8bnKhj3h17Aq4UQeeb 2UeQ== X-Gm-Message-State: AOPr4FVLa+fvq1m86tKIsvfcvEIEm+KWiMuA7dkKqAxVxX3w1BMK2Bvh7WfHJNdELaklyxAtCNCkp+BDAunZeA== MIME-Version: 1.0 X-Received: by 10.129.106.9 with SMTP id f9mr2229650ywc.90.1463756282398; Fri, 20 May 2016 07:58:02 -0700 (PDT) Received: by 10.37.215.200 with HTTP; Fri, 20 May 2016 07:58:02 -0700 (PDT) In-Reply-To: <87futfq877.fsf@HIDDEN> References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> Date: Fri, 20 May 2016 15:58:02 +0100 Message-ID: <CAKPWYQ30ck1RzA4Susg5Kk07_VuyKAcUU53LG-TV11Q8gXysoQ@HIDDEN> Subject: Re: bug#23521: XFAIL From: Gavin Smith <gavinsmith0123@HIDDEN> To: Mathieu Lirzin <mthl@HIDDEN> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.2 (/) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org, Reuben Thomas <rrt@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 0.2 (/) On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote: >> It is often easier to write expected-to-fail tests this way (so that >> they can all look the same), rather than have to have, for example, an >> extra driver that converts expected errors into success codes for the >> automake test harness. > > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D? This would be a reference to a "custom test driver". https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-T= est-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 19 May 2016 09:16:36 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 19 05:16:36 2016 Received: from localhost ([127.0.0.1]:57618 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3K4K-00081p-0L for submit <at> debbugs.gnu.org; Thu, 19 May 2016 05:16:36 -0400 Received: from mail-qk0-f170.google.com ([209.85.220.170]:34808) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <rrt@HIDDEN>) id 1b3K4H-00081b-U0 for 23521 <at> debbugs.gnu.org; Thu, 19 May 2016 05:16:34 -0400 Received: by mail-qk0-f170.google.com with SMTP id y126so9000590qke.1 for <23521 <at> debbugs.gnu.org>; Thu, 19 May 2016 02:16:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=9QXDUEyA69fDc+y4Y4AAdyEw/TDMoqQ9rIDYYLoEF6s=; b=DyRWAHsNaxZAzhGe89ux4j8ZCCPL2cgFShz082YeIk25LJL6GZFJT8C47Led+w843T //ub2ERyf6NoTn1PmnvIbLx54fERN3shu7FMEwlEPFc4HEDIyv24HEYMaxh0jvao12G0 IGOJj6aDg4bfR10w2EKFHBuIBJtW6g5uZDdKs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=9QXDUEyA69fDc+y4Y4AAdyEw/TDMoqQ9rIDYYLoEF6s=; b=W4Lns2lWuUedfx3YiObbEFSiiMWICBmM/ZFEFg1SL49O1GopAVc9XBWyXzVAbgQSRQ rkHnniPBXzbAFVEPfFaqnBlWxbwAajY2H2n5Bdf//MzxEtUWCg3PBrTINTHSIu3tTqut 7kFUZTksXko0QhACG20u8gpzUzroQcZfqyS/QIJuZYl7xyWXBkDZU7YDTBVOsdk1O0HT 4lCHgI+FuPfFk2wffcf/S0k90v2Dtn3doob9KA4YjVnVz4ZzTEVURiNPac6EYKvTYdyN b9+ZCuPEZb0l4CElNpefgJqpfTFG3DRu64ReklV/Rx3d4+tl79vqzi9n2hQwyd3Vs1nO v7jA== X-Gm-Message-State: AOPr4FXg+819s0MD2VdN3i4HupmlXyqUspk4/ZlwOqP3HE5Z9GoxOgZlO/Mz/BSBETThG/ohUWAnE3CPs7PnIrwq MIME-Version: 1.0 X-Received: by 10.55.186.132 with SMTP id k126mr12880567qkf.176.1463649388628; Thu, 19 May 2016 02:16:28 -0700 (PDT) Received: by 10.55.86.70 with HTTP; Thu, 19 May 2016 02:16:28 -0700 (PDT) In-Reply-To: <573D0107.1@HIDDEN> References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> <573D0107.1@HIDDEN> Date: Thu, 19 May 2016 10:16:28 +0100 Message-ID: <CAOnWdogbgeiS2YiLYXupMsibk6c5bueHPirm-gfKT0znufEAUw@HIDDEN> Subject: Re: bug#23521: XFAIL From: Reuben Thomas <rrt@HIDDEN> To: Peter Johansson <trojkan@HIDDEN> Content-Type: multipart/alternative; boundary=94eb2c043626e2b45c05332e6d82 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) --94eb2c043626e2b45c05332e6d82 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 19 May 2016 at 00:55, Peter Johansson <trojkan@HIDDEN> wrote: > > > On 05/19/2016 09:04 AM, Mathieu Lirzin wrote: > >> Another common use for "expected failure" is to write tests to check >>> >that error conditions arise as expected, for example, by checking that >>> >a program raises an error when given invalid input. >>> >> I agree that XFAIL can be ambiguous, however I think this usage is not >> desirable. It gives an additional opposite meaning to XFAIL symbol >> which makes it even more confusing. >> >> I agree. When I wanna tests that a program fails with incorrect input, I > prefer writing a tests that calls the program, check that it fails (exit = 1 > or whatever is expected), and perhaps even parse the error message, and i= f > it looks like I expect exit 0 aka PASS. > =E2=80=8BThanks. I shall continue with my "deviant" usage for now, but if t= hat is not considered normal, I understand that you won't want to document it. --=20 http://rrt.sc3d.org --94eb2c043626e2b45c05332e6d82 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On = 19 May 2016 at 00:55, Peter Johansson <span dir=3D"ltr"><<a href=3D"mail= to:trojkan@HIDDEN" target=3D"_blank">trojkan@HIDDEN</a>></span> wr= ote:<br></div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockq= uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><span class=3D""><br> <br> On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><blockquote class=3D"gmail_quote" style=3D"m= argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Another common use for "expected failure" is to write tests to ch= eck<br> >that error conditions arise as expected, for example, by checking that<= br> >a program raises an error when given invalid input.<br> </blockquote> I agree that XFAIL can be ambiguous, however I think this usage is not<br> desirable.=C2=A0 It gives an additional opposite meaning to XFAIL symbol<br= > which makes it even more confusing.<br> <br> </blockquote></span> I agree. When I wanna tests that a program fails with incorrect input, I pr= efer writing a tests that calls the program, check that it fails (exit 1 or= whatever is expected), and perhaps even parse the error message, and if it= looks like I expect exit 0 aka PASS.<br></blockquote><div><br></div><div><= div class=3D"gmail_default" style=3D"font-size:small;display:inline">=E2=80= =8BThanks. I shall continue with my "deviant" usage for now, but = if that is not considered normal, I understand that you won't want to d= ocument it.</div></div></div><div><br></div>-- <br><div class=3D"gmail_sign= ature"><a href=3D"http://rrt.sc3d.org" target=3D"_blank">http://rrt.sc3d.or= g</a></div> </div></div> --94eb2c043626e2b45c05332e6d82--
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 19 May 2016 09:15:41 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 19 05:15:40 2016 Received: from localhost ([127.0.0.1]:57614 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3K3Q-000800-NE for submit <at> debbugs.gnu.org; Thu, 19 May 2016 05:15:40 -0400 Received: from mail-qg0-f43.google.com ([209.85.192.43]:35912) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <rrt@HIDDEN>) id 1b3K3O-0007zn-M0 for 23521 <at> debbugs.gnu.org; Thu, 19 May 2016 05:15:39 -0400 Received: by mail-qg0-f43.google.com with SMTP id w36so39853253qge.3 for <23521 <at> debbugs.gnu.org>; Thu, 19 May 2016 02:15:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=H9b5kaCa5pxk3kedrvv5TqWWKXI2dITHPikbDJqB7Ao=; b=wI75O6ZJuuk8v0IFz3LlUpxJwKXPObirRLxvWGQm4yvRM+pqW3lit7ZFpst3AiLxZY ED1pteQWQPgeEAVEZzddfSG0Ihi4cqkMqCWHtF8UjOdScGrQLdT6yoTuGjGNI+6Qa/ZA ICWS3oaj+cjefxnri6DIaZKmLYYkkAGgtKEwY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=H9b5kaCa5pxk3kedrvv5TqWWKXI2dITHPikbDJqB7Ao=; b=gbp/d8JY1oE8PaxIK/BAkpkN0aIlLiaPGtVgaX2HAtNvDGM82dQ4onpQDq94cQFc2a /yvNMLK/2yUychyR7UPXEoFWNlXIRNqnUqUn3yei/17jmaCucz1QWmuqN/mfgzX6dumU 5vvTTg9cPRXLMNRXGk0wH3qb//6Z9sAxvavgNuQ4NbVJ/XrwEOVNQD4NPiK6sAFGe/2C TTouYompzXY1fDMNsuwvMVNZEe0pR7OBagCjW5Z2i7yfSv2dKx1MhbZfySmjDlxGJwXi ZdmSdHHAAOE7maWHr9n89W+ILDYHMpJelEQpudiVBIOHm650CqUt6KMC3XJKx+y1vTPe KbVQ== X-Gm-Message-State: AOPr4FWm439gkdevKAD5okdYG1LUPkQeJu+heM+ngyzMIbg/0PEmDTkJQf7TA9C1BwjIGWHeNwYosTUqn4k3T81l MIME-Version: 1.0 X-Received: by 10.140.102.117 with SMTP id v108mr12318743qge.103.1463649333173; Thu, 19 May 2016 02:15:33 -0700 (PDT) Received: by 10.55.86.70 with HTTP; Thu, 19 May 2016 02:15:33 -0700 (PDT) In-Reply-To: <87futfq877.fsf@HIDDEN> References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> Date: Thu, 19 May 2016 10:15:33 +0100 Message-ID: <CAOnWdoioYFPFGcemdDx7-wyHjnwwHA_wo0zwNvN71jeijM_D=Q@HIDDEN> Subject: Re: bug#23521: XFAIL From: Reuben Thomas <rrt@HIDDEN> To: Mathieu Lirzin <mthl@HIDDEN> Content-Type: multipart/alternative; boundary=001a11c16ec494811405332e6a61 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) --001a11c16ec494811405332e6a61 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 19 May 2016 at 00:04, Mathieu Lirzin <mthl@HIDDEN> wrote: > > It is often easier to write expected-to-fail tests this way (so that > > they can all look the same), rather than have to have, for example, an > > extra driver that converts expected errors into success codes for the > > automake test harness. > > What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D? > =E2=80=8BA custom test driver.=E2=80=8B --=20 http://rrt.sc3d.org --001a11c16ec494811405332e6a61 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">On = 19 May 2016 at 00:04, Mathieu Lirzin <span dir=3D"ltr"><<a href=3D"mailt= o:mthl@HIDDEN" target=3D"_blank">mthl@HIDDEN</a>></span> wrote:<br></d= iv><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class= =3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd= ing-left:1ex">> It is often easier to write expected-to-fail tests this = way (so that<br> > they can all look the same), rather than have to have, for example, an= <br> > extra driver that converts expected errors into success codes for the<= br> > automake test harness.<br> <br> What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D?<br></block= quote><div><br></div><div><div class=3D"gmail_default" style=3D"font-size:s= mall">=E2=80=8BA custom test driver.=E2=80=8B</div></div></div><div><br></d= iv>-- <br><div class=3D"gmail_signature"><a href=3D"http://rrt.sc3d.org" ta= rget=3D"_blank">http://rrt.sc3d.org</a></div> </div></div> --001a11c16ec494811405332e6a61--
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 18 May 2016 23:56:02 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed May 18 19:56:02 2016 Received: from localhost ([127.0.0.1]:57428 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3BJq-000396-52 for submit <at> debbugs.gnu.org; Wed, 18 May 2016 19:56:02 -0400 Received: from mail-pf0-f196.google.com ([209.85.192.196]:33706) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <trojkan@HIDDEN>) id 1b3BJo-00038l-Vy for 23521 <at> debbugs.gnu.org; Wed, 18 May 2016 19:56:01 -0400 Received: by mail-pf0-f196.google.com with SMTP id y7so6324193pfb.0 for <23521 <at> debbugs.gnu.org>; Wed, 18 May 2016 16:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=FpB6RoxmRq4DEIPrjfOlMeDgxj35kqjHioS3eXMde5Q=; b=rStw0738S2PKh272PeWK6txVbfrZiTy3f2FNLbDRTNtzz0wKnswnWqzTsXPafoHqEU 2x2kiiym+4ja5Db5HgVuqeBxMaMvFxZii2JOjrEbrCCw+Z4cnA1J1BBo5df8jsZfXAWW g5ZWe/QBsN1F3ik4npnd521fa3LlucrkB+sz1NRh7p4zkngRQcSekGQNuCFBy9CN+eDy pcu4reWnXU/vVjqhMhiS9fl6Ko9NLwOTKr4D6CN0Rw+X6uaGK6rcGoJXTom/QXOwj2ru g+9figeQoWlJTm/a2anFYihrfV4bIhe+Tj1hpOW9S6D+jJ3SbAwieoReHu/kWIRvnvDe jIKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=FpB6RoxmRq4DEIPrjfOlMeDgxj35kqjHioS3eXMde5Q=; b=lTof6hOBQUhvlYuURfD/Zh/vAnuerFuSb6AfAgFe5M4uYhdvTWu8aLvXwq2c2jJzuL 25t/859jz9HecgEyb9EVTR/cD1CRJrF/GMMvHcjy7IYbrxunko+rynj8QgEtPMivgIL0 IHUfDVhYw7C41FzLBzlpVu+7WQ2kkQMiZ5+F+zB17M/Ahi+YzANj/TqXHbZjiYmVhNyp t0KrTBRepR982LtOOwtcIRvuYSRVaS1AavZR+mV728TiFU3YQmFIVPEkuSbsIzseXZVx NzMT+pOdlyBlEk6aS26dz8JoDCVMbILGbGeciq2aE+etm1t2ZW8ZwVtkpfjtd/vrHz5d TEDw== X-Gm-Message-State: AOPr4FW+/lTo3yMBOMAn7Yth2XDQmsDGwRbl7/VdjuxF/Of+ahO9muFqiSCqEE83p3Lm9w== X-Received: by 10.98.69.213 with SMTP id n82mr15033934pfi.48.1463615755202; Wed, 18 May 2016 16:55:55 -0700 (PDT) Received: from [10.3.81.221] (gateway.qimr.edu.au. [152.98.8.1]) by smtp.gmail.com with ESMTPSA id g5sm14841513pac.1.2016.05.18.16.55.53 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 18 May 2016 16:55:54 -0700 (PDT) Subject: Re: bug#23521: XFAIL To: Mathieu Lirzin <mthl@HIDDEN>, Reuben Thomas <rrt@HIDDEN> References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> <87futfq877.fsf@HIDDEN> From: Peter Johansson <trojkan@HIDDEN> Message-ID: <573D0107.1@HIDDEN> Date: Thu, 19 May 2016 09:55:51 +1000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <87futfq877.fsf@HIDDEN> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) On 05/19/2016 09:04 AM, Mathieu Lirzin wrote: >> Another common use for "expected failure" is to write tests to check >> >that error conditions arise as expected, for example, by checking that >> >a program raises an error when given invalid input. > I agree that XFAIL can be ambiguous, however I think this usage is not > desirable. It gives an additional opposite meaning to XFAIL symbol > which makes it even more confusing. > I agree. When I wanna tests that a program fails with incorrect input, I prefer writing a tests that calls the program, check that it fails (exit 1 or whatever is expected), and perhaps even parse the error message, and if it looks like I expect exit 0 aka PASS. Cheers, Peter
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at 23521) by debbugs.gnu.org; 18 May 2016 23:04:39 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed May 18 19:04:39 2016 Received: from localhost ([127.0.0.1]:57408 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b3AW6-0001zh-Pj for submit <at> debbugs.gnu.org; Wed, 18 May 2016 19:04:38 -0400 Received: from v-zimmta02.u-bordeaux.fr ([147.210.215.20]:54076) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <mthl@HIDDEN>) id 1b3AW5-0001zS-8n for 23521 <at> debbugs.gnu.org; Wed, 18 May 2016 19:04:37 -0400 Received: from v-zimmta02.u-bordeaux.fr (localhost [127.0.0.1]) by v-zimmta02.u-bordeaux.fr (Postfix) with ESMTP id BB3AD8007A2; Thu, 19 May 2016 01:04:29 +0200 (CEST) Received: from godel (mek33-4-82-236-46-88.fbx.proxad.net [82.236.46.88]) by v-zimmta02.u-bordeaux.fr (Postfix) with ESMTPSA id 5AC2F8000AD; Thu, 19 May 2016 01:04:29 +0200 (CEST) From: Mathieu Lirzin <mthl@HIDDEN> To: Reuben Thomas <rrt@HIDDEN> Subject: Re: bug#23521: XFAIL References: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> Date: Thu, 19 May 2016 01:04:28 +0200 In-Reply-To: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> (Reuben Thomas's message of "Thu, 12 May 2016 11:01:11 +0100") Message-ID: <87futfq877.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-AV-Checked: ClamAV using ClamSMTP X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: 23521 Cc: 23521 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.4 (-) Hi, Reuben Thomas <rrt@HIDDEN> writes: > The documentation says: "It's not uncommon, especially during early > development stages, that some tests fail for known reasons, and that > the developer doesn't want > to tackle these failures immediately (this is especially true when the > failing tests deal with corner cases)." > > Another common use for "expected failure" is to write tests to check > that error conditions arise as expected, for example, by checking that > a program raises an error when given invalid input. I agree that XFAIL can be ambiguous, however I think this usage is not desirable. It gives an additional opposite meaning to XFAIL symbol which makes it even more confusing. > If that's a reasonable use of automake's test harness, perhaps the > documentation could reflect that, e.g. by adding: > > "Another use for XFAIL is to mark tests that are supposed to fail, for > example, to check that a program raises an error when given invalid > input." > > It is often easier to write expected-to-fail tests this way (so that > they can all look the same), rather than have to have, for example, an > extra driver that converts expected errors into success codes for the > automake test harness. What do you mean precisely by =E2=80=9Can extra driver=E2=80=9D? Thanks --=20 Mathieu Lirzin
bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.Received: (at submit) by debbugs.gnu.org; 12 May 2016 10:01:33 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu May 12 06:01:33 2016 Received: from localhost ([127.0.0.1]:48072 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1b0nQz-0003X7-Ex for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:33 -0400 Received: from eggs.gnu.org ([208.118.235.92]:33667) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <rrt@HIDDEN>) id 1b0nQw-0003Wq-Bv for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <rrt@HIDDEN>) id 1b0nQl-0004RZ-Fz for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:25 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,HTML_MESSAGE, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:54770) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <rrt@HIDDEN>) id 1b0nQl-0004RL-DD for submit <at> debbugs.gnu.org; Thu, 12 May 2016 06:01:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52851) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <rrt@HIDDEN>) id 1b0nQi-00073L-IZ for bug-automake@HIDDEN; Thu, 12 May 2016 06:01:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <rrt@HIDDEN>) id 1b0nQe-0004Ps-20 for bug-automake@HIDDEN; Thu, 12 May 2016 06:01:16 -0400 Received: from mail-qk0-x22c.google.com ([2607:f8b0:400d:c09::22c]:34176) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <rrt@HIDDEN>) id 1b0nQd-0004Pk-SV for bug-automake@HIDDEN; Thu, 12 May 2016 06:01:11 -0400 Received: by mail-qk0-x22c.google.com with SMTP id r184so40103544qkc.1 for <bug-automake@HIDDEN>; Thu, 12 May 2016 03:01:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sc3d.org; s=google; h=mime-version:date:message-id:subject:from:to; bh=90Bvch9qyJpFx1/F1bH6u18lrgeQl9rS0cF5XMxXNdQ=; b=4L4ZlCiSIdUTyheATYwZU7IzveM+fiOYPdGJXliVUHjv6k8XqTbgTbu9wEbVcSrWai 55T8Q3LCuWieietFHqySa8GZ0/fTPKz1ZQNer3GfLz0QOVu6183OPbpxDaHB6Kn3CyRT hP/n64RreDM/oN0H8+GrYTJg3egeeushsGyMQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=90Bvch9qyJpFx1/F1bH6u18lrgeQl9rS0cF5XMxXNdQ=; b=hYaz+UicAvrcgkG52WACs3K3YGdQcykWb2vvM72u04PpuqtYKEy0w3cSUQ3nkoCIJI TlXgaJjjH+20oc9nNGNJ7rwxFZM9cEZ7B+53s+ZAd4A7oamvJLHV9T1gLrXl2KgQWtQJ 6LBVDVSxZ1yGQP9Yd9Lyt5YblRK16PaW7O3Z0x/yZ07QrcR2vWBVZgGxBVbtODy4cyS8 UeuLbstlnhh5brqTXupbvNqoJ1CXG1exDGwJVUtVCi1Dd/ZbQOTqxeirPWQvSSWrh9rg Q4BsVGNzbsJZOOeb3jZghKAeCR0N6Evio5NnyHA+W0u/ijkBCgMw4Plml+fss1HrouT0 uhZg== X-Gm-Message-State: AOPr4FXT6Z1Vj8uQ263cSoOQYsaodZkH75XxFPL5sr0BKU59lbyEjy4Rnd+Yst/gWy/pXbXENEC2COzvnikdBZZ5 MIME-Version: 1.0 X-Received: by 10.55.180.133 with SMTP id d127mr8849603qkf.131.1463047271038; Thu, 12 May 2016 03:01:11 -0700 (PDT) Received: by 10.55.86.70 with HTTP; Thu, 12 May 2016 03:01:11 -0700 (PDT) Date: Thu, 12 May 2016 11:01:11 +0100 Message-ID: <CAOnWdoimg9vayh4SkGDGR9NY5LNadSGcVysaqk65M2soJ3RD1w@HIDDEN> Subject: XFAIL From: Reuben Thomas <rrt@HIDDEN> To: bug-automake <bug-automake@HIDDEN> Content-Type: multipart/alternative; boundary=94eb2c06fad6e1575c0532a23cc6 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -4.1 (----) --94eb2c06fad6e1575c0532a23cc6 Content-Type: text/plain; charset=UTF-8 The documentation says: "It's not uncommon, especially during early development stages, that some tests fail for known reasons, and that the developer doesn't want to tackle these failures immediately (this is especially true when the failing tests deal with corner cases)." Another common use for "expected failure" is to write tests to check that error conditions arise as expected, for example, by checking that a program raises an error when given invalid input. If that's a reasonable use of automake's test harness, perhaps the documentation could reflect that, e.g. by adding: "Another use for XFAIL is to mark tests that are supposed to fail, for example, to check that a program raises an error when given invalid input." It is often easier to write expected-to-fail tests this way (so that they can all look the same), rather than have to have, for example, an extra driver that converts expected errors into success codes for the automake test harness. -- http://rrt.sc3d.org --94eb2c06fad6e1575c0532a23cc6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">The= documentation says: "It's not uncommon, especially during early d= evelopment stages, that some tests fail for known reasons, and that the dev= eloper doesn't want</div><div class=3D"gmail_default">to tackle these f= ailures immediately (this is especially true when the</div><div class=3D"gm= ail_default">failing tests deal with corner cases)."</div><div class= =3D"gmail_default"><br></div><div class=3D"gmail_default">Another common us= e for "expected failure" is to write tests to check that error co= nditions arise as expected, for example, by checking that a program raises = an error when given invalid input.</div><div class=3D"gmail_default"><br></= div><div class=3D"gmail_default">If that's a reasonable use of automake= 's test harness, perhaps the documentation could reflect that, e.g. by = adding:</div><div class=3D"gmail_default"><br></div><div class=3D"gmail_def= ault">"Another use for XFAIL is to mark tests that are supposed to fai= l, for example, to check that a program raises an error when given invalid = input."</div><div class=3D"gmail_default"><br></div><div class=3D"gmai= l_default">It is often easier to write expected-to-fail tests this way (so = that they can all look the same), rather than have to have, for example, an= extra driver that converts expected errors into success codes for the auto= make test harness.</div><div><br></div>-- <br><div class=3D"gmail_signature= "><a href=3D"http://rrt.sc3d.org" target=3D"_blank">http://rrt.sc3d.org</a>= </div> </div> --94eb2c06fad6e1575c0532a23cc6--
Reuben Thomas <rrt@HIDDEN>:bug-automake@HIDDEN.
Full text available.bug-automake@HIDDEN:bug#23521; Package automake.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.