GNU logs - #68860, boring messages

Message sent to bug-automake@HIDDEN:

Subject: bug#68860: race condition with make recheck
Resent-From: Peter Johansson <trojkan@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 01 Feb 2024 01:13:01 +0000
Resent-Message-ID: <handler.68860.B.170674992419327 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 68860 <at> debbugs.gnu.org
Content-Type: multipart/mixed; boundary="------------5fGjAxjJN9VvfUqcnJVupBXE"
Message-ID: <f40fd580-f3ed-4bca-86ec-f3ea5b125d1f@HIDDEN>
Date: Thu, 1 Feb 2024 11:11:37 +1000
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
From: Peter Johansson <trojkan@HIDDEN>
Received-SPF: pass client-ip=2607:f8b0:4864:20::333;
 envelope-from=trojkan@HIDDEN; helo=mail-ot1-x333.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
 T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

This is a multi-part message in MIME format.
--------------5fGjAxjJN9VvfUqcnJVupBXE
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi automakers,

I think I've found a race condition with 'make recheck' that results in 
a source file being compiled twice in parallel and resulting in a 
failure such as

mv: cannot stat '.deps/foo.Tpo': No such file or directory

In my trimmed down example my Makefile.am looks like:

lib_LIBRARIES = libfoo.a
libfoo_a_SOURCES = foo.cc
check_LIBRARIES = libtest.a
libtest_a_SOURCES = test.cc
TESTS = one.test two.test
TEST_EXTENSIONS = .test
AM_DEFAULT_SOURCE_EXT = .cc
EXTRA_PROGRAMS = $(TESTS)
libtest_a_LIBADD = libfoo.a
LDADD = libtest.a libfoo.a

The problem seems to be that both $(TESTS) and check_LIBRARIES depend on 
libfoo.a and trigger compilation of foo.cc. I haven't managed to get the 
same problem with 'make check', so I thought comparing the generated 
rules for check: and recheck: would be useful.

recheck: all $(check_LIBRARIES)

<long rule running failed TESTS>

all: config.h
     $(MAKE) $(AM_MAKEFLAGS) all-am
...

check-am: all-am
     $(MAKE) $(AM_MAKEFLAGS) $(check_LIBRARIES)
     $(MAKE) $(AM_MAKEFLAGS) check-TESTS
check: check-am

I can see how the "check-am: all-am" works as firewall against the race 
condition. OTOH, in the rule for recheck, 'all' triggers a sub-process 
that will build libfoo.a and in the meantime the main process will build 
$(check_LIBRARIES) which trigger the building of libfoo.a. My 
understanding of parallel make is a bit hazy, but I guess the main 
process and sub-process are only talking wrt how many workers they 
employ and are not talking about which rules to work on.

I suppose this is not by design or that I'm doing something illegal by 
having check_LIBRARIES depend stuff that is built within 'make all'. I'm 
not sure what the best way to fix this would be. One idea would to 
change the rule for recheck to

recheck: all

      $(MAKE) $(AM_MAKEFLAGS) $(check_LIBRARIES)

      <long rule running failed TESTS>


but personally I don't fancy these sub-processes because it feels like 
they are the core of the problem for these sort of race conditions.

I have tested with automake 1.16.5 (ubuntu) and 1.16i.

Please find attached a trimmed down example of the problem.


Best Regards,

Peter

--------------5fGjAxjJN9VvfUqcnJVupBXE
Content-Type: application/x-shellscript; name="automake.sh"
Content-Disposition: attachment; filename="automake.sh"
Content-Transfer-Encoding: base64

IyEvYmluL3NoCgpzZXQgLWUKc2V0IC14Cgpta2RpciAtcCBhdXRvbWFrZQpjZCBhdXRvbWFr
ZQoKY2F0ID4gY29uZmlndXJlLmFjPDxFT0YKQUNfSU5JVChbZm9vXSwgWzJdKQpBQ19DT05G
SUdfU1JDRElSKFtjb25maWcuaC5pbl0pCkFDX0NPTkZJR19IRUFERVJTKFtjb25maWcuaF0p
CkFNX0lOSVRfQVVUT01BS0UoW2ZvcmVpZ24gc2lsZW50LXJ1bGVzIHBhcmFsbGVsLXRlc3Rz
XSkKQUNfTEFORyhbQysrXSkKQUNfUFJPR19DWFgKQUNfUFJPR19DWFhDUFAKQUNfUFJPR19S
QU5MSUIKQUNfQ09ORklHX0ZJTEVTKFtNYWtlZmlsZV0pCkFDX09VVFBVVApFT0YKCmNhdCA+
IE1ha2VmaWxlLmFtIDw8RU9GCmxpYl9MSUJSQVJJRVMgPSBsaWJmb28uYQpsaWJmb29fYV9T
T1VSQ0VTID0gZm9vLmNjCgpjaGVja19MSUJSQVJJRVMgPSBsaWJ0ZXN0LmEKbGlidGVzdF9h
X1NPVVJDRVMgPSB0ZXN0LmNjCgpURVNUUyA9IG9uZS50ZXN0IHR3by50ZXN0ClRFU1RfRVhU
RU5TSU9OUyA9IC50ZXN0CkFNX0RFRkFVTFRfU09VUkNFX0VYVCA9IC5jYwpFWFRSQV9QUk9H
UkFNUyA9IFwkKFRFU1RTKQoKbGlidGVzdF9hX0xJQkFERCA9IGxpYmZvby5hCkxEQUREID0g
bGlidGVzdC5hIGxpYmZvby5hCkVPRgoKY2F0ID4gZm9vLmggPDxFT0YKI2lmbmRlZiBteV9m
b29faAojZGVmaW5lIG15X2Zvb19oCiNpbmNsdWRlIDxzdHJpbmc+CnN0ZDo6c3RyaW5nIGZv
byh2b2lkKTsKI2VuZGlmCkVPRgoKY2F0ID4gZm9vLmNjIDw8RU9GCiNpbmNsdWRlIDxjb25m
aWcuaD4KI2luY2x1ZGUgImZvby5oIgpzdGQ6OnN0cmluZyBmb28odm9pZCkgeyByZXR1cm4g
ImZ1IjsgfQoKRU9GCgpjYXQgPiBvbmUuY2MgPDxFT0YKI2luY2x1ZGUgPGNvbmZpZy5oPgoj
aW5jbHVkZSAiZm9vLmgiCiNpbmNsdWRlICJ0ZXN0LmgiCiNpbmNsdWRlIDxpb3N0cmVhbT4K
I2luY2x1ZGUgPHN0cmluZz4KaW50IG1haW4oaW50IGFyZ2MsIGNoYXIqIGFyZ3ZbXSkKewoJ
aW5pdChhcmdjLCBhcmd2KTsKCXN0ZDo6Y291dCA8PCAiSGVsbG8gV29ybGQhXG4iIDw8IGZv
bygpIDw8ICJcbiI7CglyZXR1cm4gMDsKfQpFT0YKCmNhdCA+IHR3by5jYyA8PEVPRgojaW5j
bHVkZSA8Y29uZmlnLmg+CiNpbmNsdWRlICJmb28uaCIKI2luY2x1ZGUgInRlc3QuaCIKI2lu
Y2x1ZGUgPGlvc3RyZWFtPgojaW5jbHVkZSA8c3RyaW5nPgppbnQgbWFpbihpbnQgYXJnYywg
Y2hhciogYXJndltdKQp7Cglpbml0KGFyZ2MsIGFyZ3YpOwoJc3RkOjpzdHJpbmcgc3RyID0g
Zm9vKCk7CglpZiAoc3RyICE9ICJmb28iKSB7CgkgICBzdGQ6OmNlcnIgPDwgImVycm9yOiBm
b28oKTogIiA8PCBzdHIgPDwgIlxuIjsKCSAgIHJldHVybiAxOwoJfQoJcmV0dXJuIDA7Cn0K
RU9GCgpjYXQgPiB0ZXN0LmggPDxFT0YKI2lmbmRlZiBteV90ZXN0X2gKI2RlZmluZSBteV90
ZXN0X2gKdm9pZCBpbml0KGludCBhcmdjLCBjaGFyKiBhcmd2W10pOwojZW5kaWYKRU9GCgpj
YXQgPiB0ZXN0LmNjIDw8RU9GCiNpbmNsdWRlIDxjb25maWcuaD4KI2luY2x1ZGUgInRlc3Qu
aCIKI2luY2x1ZGUgImZvby5oIgojaW5jbHVkZSA8aW9zdHJlYW0+CnZvaWQgaW5pdChpbnQg
YXJnYywgY2hhciogYXJndltdKQp7CglzdGQ6OmNvdXQgPDwgImluaXQgdGVzdCBmb3IgIiA8
PCBmb28oKSA8PCAiXG4iOwp9CkVPRgoKCmF1dG9yZWNvbmYgLWl2ZgouL2NvbmZpZ3VyZSAt
LWVuYWJsZS1zaWxlbnQtcnVsZXMKbWFrZSBhbGwgLWo4Cm1ha2UgY2hlY2sgLWo4IHx8IGVj
aG8gZmFpbGVkIGFzIGV4cGVjdGVkCgpzZWQgJ3MsImZ1IiwiZm9vIiwnIDwgZm9vLmNjID4g
Zm9vLmNjLXQgJiYgbXYgZm9vLmNjLXQgZm9vLmNjCgptYWtlIHJlY2hlY2sgLWoxNgo=

--------------5fGjAxjJN9VvfUqcnJVupBXE--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Peter Johansson <trojkan@HIDDEN>
Subject: bug#68860: Acknowledgement (race condition with make recheck)
Message-ID: <handler.68860.B.170674992419327.ack <at> debbugs.gnu.org>
References: <f40fd580-f3ed-4bca-86ec-f3ea5b125d1f@HIDDEN>
X-Gnu-PR-Message: ack 68860
X-Gnu-PR-Package: automake
Reply-To: 68860 <at> debbugs.gnu.org
Date: Thu, 01 Feb 2024 01:13:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-automake@HIDDEN

If you wish to submit further information on this problem, please
send it to 68860 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
68860: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D68860
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-automake@HIDDEN:

Subject: bug#68860: race condition with make recheck
References: <f40fd580-f3ed-4bca-86ec-f3ea5b125d1f@HIDDEN>
Resent-From: Karl Berry <karl@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 01 Feb 2024 22:26:01 +0000
Resent-Message-ID: <handler.68860.B68860.170682633521049 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: trojkan@HIDDEN
Cc: 68860 <at> debbugs.gnu.org
Date: Thu, 1 Feb 2024 15:25:20 -0700
Message-Id: <202402012225.411MPKig015007@HIDDEN>
From: Karl Berry <karl@HIDDEN>
In-Reply-To: <f40fd580-f3ed-4bca-86ec-f3ea5b125d1f@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Peter,

    The problem seems to be that both $(TESTS) and check_LIBRARIES depend on 
    libfoo.a and trigger compilation of foo.cc. 

Thanks much for the report and analysis. What you wrote looks sensible
to me.

    My understanding of parallel make is a bit hazy,

Me too :(. If anyone else here has a chance to look into this, that
would be great.

    One idea would to change the rule for recheck to

It looks plausible. Another possibility that comes to mind is to make
the recheck target more parallel to all, i.e., with a recheck-am
target. I', not sure.

    Please find attached a trimmed down example of the problem.

Thanks again. Will ponder. --best, karl.

Last modified: Thu, 1 Feb 2024 22:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.