GNU bug report logs - #58065
UNSUPPORTED test cases still can FAIL

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 58065 in the body.
You can then email your comments to 58065 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arsen Arsenović <arsen <at> aarsen.me>
To: bug-dejagnu <at> gnu.org
Subject: UNSUPPORTED test cases still can FAIL
Date: Sun, 25 Sep 2022 01:39:05 +0200

Hi,

When a testsuite that uses dg.exp returns ::unsupported:: for some test 
from ${tool}-dg-test or ${tool}-dg-prune, dg-{error,warning} markers in 
that test case are still processed. If the ::unsupported:: is returned 
from ${tool}-dg-test, these markers emit a FAIL, since the output does 
not contain their wanted errors. If the ::unsupported:: is returned from 
${tool}-dg-prune, these markers are processed as normal, which can 
sometimes PASS due to compiler resumption, but can also FAIL. I see no 
other way to report ::unsupported:: status from the dg callbacks.
This feels like an oversight, as it makes conditionally skipping tests 
as UNSUPPORTED nigh impossible.

Previous ML discussion:
https://lists.gnu.org/archive/html/dejagnu/2022-09/msg00000.html

Minimized reproduction case:
https://git.sr.ht/~arsen/dejagnu-demo-testcase/
(commit 4e83a1a0820062387985d866286d73299f0691a6)

Suggested solutions:
- Call ${tool}-dg-prune before processing dg-messages.
- Permit ${tool}-dg-test to return a special value to prevent processing 
dg-messages.

Thanks,
-- 
Arsen Arsenović

Message #10 received at 58065 <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Arsen Arsenović <arsen <at> aarsen.me>
Cc: 58065 <at> debbugs.gnu.org
Subject: Re: bug#58065: UNSUPPORTED test cases still can FAIL
Date: Tue, 04 Oct 2022 22:31:17 -0500

A preliminary patch has been pushed on the PR58065 branch at Savannah.  
Please confirm that it resolves this issue.

The preliminary patch works by storing the result of message processing 
before calling ${tool}-dg-prune and discarding the stored results 
instead of reporting them to the framework if ${tool}-dg-prune indicates 
to abort the test.

The descriptions given were sufficient that I was able avoid examining 
the offered test case and thus avoid needing associated copyright paperwork.

Somewhat amusingly, the additional tests included bring the total number 
of tests across all tools in a run of DejaGnu's internal testsuite to 
666.  While I would enjoy tweaking superstitious people by making a 
release like this, unfortunately there are still quite a few other items 
on the checklist for 1.6.4, so there will probably be more tests added 
before the release.


-- Jacob

Message #13 received at 58065 <at> debbugs.gnu.org (full text, mbox):

From: Arsen Arsenović <arsen <at> aarsen.me>
To: jcb62281 <at> gmail.com
Cc: 58065 <at> debbugs.gnu.org
Subject: Re: bug#58065: UNSUPPORTED test cases still can FAIL
Date: Wed, 05 Oct 2022 20:56:21 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Wednesday, 5 October 2022 05:31:17 CEST Jacob Bachmeyer wrote:
> A preliminary patch has been pushed on the PR58065 branch at Savannah.
> Please confirm that it resolves this issue.
It does.

> The preliminary patch works by storing the result of message
> processing before calling ${tool}-dg-prune and discarding the stored
> results instead of reporting them to the framework if
> ${tool}-dg-prune indicates to abort the test.
> 
> The descriptions given were sufficient that I was able avoid examining
> the offered test case and thus avoid needing associated copyright
> paperwork.
Huh, would have looking at the provided test even require such papers? 
That seems a bit excess..

At any rate, this works. Thank you!

Have a great evening,
-- 
Arsen Arsenović

[signature.asc (application/pgp-signature, inline)]

Message #18 received at 58065-done <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Arsen Arsenović <arsen <at> aarsen.me>
Cc: 58065-done <at> debbugs.gnu.org
Subject: Re: bug#58065: UNSUPPORTED test cases still can FAIL
Date: Wed, 05 Oct 2022 20:23:56 -0500

Arsen Arsenović wrote:
> On Wednesday, 5 October 2022 05:31:17 CEST Jacob Bachmeyer wrote:
>   
>> A preliminary patch has been pushed on the PR58065 branch at Savannah.
>> Please confirm that it resolves this issue.
>>     
> It does.
>   

Thank you.  The PR58065 branch has been merged to master and will be 
included in the future 1.6.4 release.

>> [...]
>>
>> The descriptions given were sufficient that I was able avoid examining
>> the offered test case and thus avoid needing associated copyright
>> paperwork.
>>     
> Huh, would have looking at the provided test even require such papers? 
> That seems a bit excess..
>   

There is a possibility of trouble if code subsequently added to DejaGnu, 
such as a regression test, were to be "significant" and sufficiently 
resemble the offered case, and papers were not in order.  Not looking at 
it short-circuits the whole problem.

In general, the GNU project seems to take a "better safe than sorry" 
angle on the issue, and the prospect of being "low-hanging fruit" for 
some future SCO-alike does not appeal to me.


-- Jacob

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.