GNU bug report logs - #68808
subsecond mtime discovery code insufficient

Previous Next

To reply to this bug, email your comments to 68808 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: bug-automake <at> gnu.org, JohnsonE <at> usc.edu
Subject: subsecond mtime discovery code insufficient
Date: Mon, 29 Jan 2024 16:00:36 -0700

Hi Erik - following up on your final comment in #68119:

    P.P.S. It still needs
            export am_cv_sleep_fractional_seconds=false
    to correctly run the tests on macOS.  Without that, 75-90 tests fail.

Maybe we can try to figure out why the code that tries to automatically
discern whether subsecond mtimes are supported is insufficient? It seems
there is something different about the mac filesystem (or something). So
I suspect there will be problems on other systems, too.

If there are any test(s) which more or less consistently fail without
the override the sleep_fractional_seconds, maybe we can discern why
those tests are failing. I am dearly hoping not to have inspect every
single test and scatter $sleep commands everywhere.

Meanwhile, the code that tries to figure out if subsecond mtimes are
supported is in m4/sanity.m4, function
_AM_FILESYSTEM_TIMESTAMP_RESOLUTION.  It essentially writes to three
files and sleeps for the trial resolution between them, starting at a
hundredth of a second. Then, if ls -t sorts in the right order, it
assumes that resolution is ok.

For one thing, just to get a grip on reality, maybe you could try
running those commands by hand and see what the result is.  I'm guessing
the Mac (and its sleep command?) do support subsecond mtimes in reality.

  am_try_res=.001
  echo alpha > conftest.ts1
  sleep $am_try_res
  echo beta > conftest.ts2
  sleep $am_try_res
  echo gamma > conftest.ts3
  ls -t conftest.ts?

The result should be conftest.ts3 first, then ts2, then ts1.

But, looking at that code, I belatedly see that shell arrays are being
used. At least I think so. It wouldn't surprise me if the mac /bin/sh
doesn't support arrays. But that should result in assuming subsecond
mtimes are not supported when they are, instead of the reverse. Anyway,
I don't think we can assume arrays, so I'll have to think about that.

But the important thing is why so many tests fail on your system when
subsecond mtimes are enabled (and succeed when not). --thanks, karl.

Message #8 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Karl Berry" <karl <at> freefriends.org>, 68808 <at> debbugs.gnu.org,
 JohnsonE <at> usc.edu
Subject: Re: bug#68808: subsecond mtime discovery code insufficient
Date: Mon, 29 Jan 2024 19:24:11 -0500

On Mon, Jan 29, 2024, at 6:00 PM, Karl Berry wrote:
> But, looking at that code, I belatedly see that shell arrays are being
> used. At least I think so.

It is absolutely not *supposed* to be using shell arrays. If it is, that was a mistake on my part.

zw

Message #11 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: Erik A Johnson <JohnsonE <at> usc.edu>
To: Karl Berry <karl <at> freefriends.org>
Cc: "68808 <at> debbugs.gnu.org" <68808 <at> debbugs.gnu.org>
Subject: Re: subsecond mtime discovery code insufficient
Date: Tue, 30 Jan 2024 12:08:16 +0000

	I'm guessing the Mac (and its sleep command?) do support subsecond mtimes in reality.

echo a > 1 && echo b > 2 && echo c > 3 && ls -t [123]
consistently gives me the correct order (3 2 1).
The macOS version of stat has a mtime_nsec field that is fully populated and the file modification times differ by about a half of a millisecond.

	But, looking at that code, I belatedly see that shell arrays are being
	used. At least I think so. It wouldn't surprise me if the mac /bin/sh
	doesn't support arrays. But that should result in assuming subsecond
	mtimes are not supported when they are, instead of the reverse. Anyway,
	I don't think we can assume arrays, so I'll have to think about that.

On macOS, /bin/sh is actually /bin/bash executing in POSIX mode.  I _think_ it has arrays but I could check if you have some code you want me to run.

Message #14 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: Erik A Johnson <JohnsonE <at> usc.edu>
To: Karl Berry <karl <at> freefriends.org>
Cc: "68808 <at> debbugs.gnu.org" <68808 <at> debbugs.gnu.org>
Subject: Re: subsecond mtime discovery code insufficient
Date: Tue, 30 Jan 2024 12:46:22 +0000

I found the problem!  The default GNU make on macOS (even Sonoma) is 3.81 from 2006!  The configure script for GNU make versions <4.0 did not have the right test for subsecond resolution in the macOS stat() function, so 3.81 was built with resolution only to the second, not subsecond, thus needing at least one second sleep.  (Why, then, does Apple continue to include 3.81 in the software 18 years later? Beyond me.)

Installing the most recent release of GNU make (4.4.1) fixes the problem.  Tests all run successfully.

Erik


----------------------------------------

Details ... With make 3.81, if I create a simple Makefile with

	foo: bar
		touch foo

then

	touch foo
	touch bar
	make

should make (touch) foo because bar is newer.  However, if these are done immediately after each other

	touch foo; touch bar; make

then, most of the time, make just responds with "make: `foo' is up to date."  The following

	n=0; while make | grep -q 'up to date'; do touch foo; touch bar; printf .; n=$(($n+1)); done; echo $n

varies from a handful of trials to several thousand trials before foo's mtime's integer is one less than bar's mtime's integer.

Message #17 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Erik A Johnson" <JohnsonE <at> usc.edu>, "Karl Berry" <karl <at> freefriends.org>
Cc: "68808 <at> debbugs.gnu.org" <68808 <at> debbugs.gnu.org>
Subject: Re: bug#68808: subsecond mtime discovery code insufficient
Date: Tue, 30 Jan 2024 09:06:12 -0500

On Tue, Jan 30, 2024, at 7:46 AM, Erik A Johnson wrote:
> I found the problem!  The default GNU make on macOS (even Sonoma) is
> 3.81 from 2006!  The configure script for GNU make versions <4.0 did
> not have the right test for subsecond resolution in the macOS stat()
> function, so 3.81 was built with resolution only to the second, not
> subsecond, thus needing at least one second sleep.

Okay, so that means the filesystem timestamp resolution test needs to
probe the behavior of `make` as well as all the other things it tests. I
will work on a patch for that as soon as I have a usable workstation
again, which will probably not be until next week at the earliest; if
anyone wants to beat me to it, feel free.

> (Why, then, does Apple continue to include 3.81 in the software 18
> years later? Beyond me.)

3.81 was the last version distributed under GPLv2; Apple won't touch
anything that's distributed under GPLv3.  Their copy of GNU M4 is also
vintage 2006, for the same reason.  I don't claim to understand why
their legal department thinks GPLv2 is fine but v3 has to be avoided,
but (like many other proprietary Unix vendors) they probably keep
shipping the old GNU tools instead of, like, switching to current BSD
tools, for the sake of compatibility with old scripts.

zw

Message #20 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: Peter Johansson <trojkan <at> gmail.com>
To: Erik A Johnson <JohnsonE <at> usc.edu>, Karl Berry <karl <at> freefriends.org>
Cc: "68808 <at> debbugs.gnu.org" <68808 <at> debbugs.gnu.org>
Subject: Re: bug#68808: subsecond mtime discovery code insufficient
Date: Wed, 31 Jan 2024 00:56:38 +1000

[Message part 1 (text/plain, inline)]

On 30/1/24 22:46, Erik A Johnson wrote:
> (Why, then, does Apple continue to include 3.81 in the software 18 years later? Beyond me.)

Probably because 3.81 was the last version released under GPLv2 or later 
and IIRC Apple avoids shipping things that are licensed with GPLv3.

Cheers,

Peter

[Message part 2 (text/html, inline)]

Message #23 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: zack <at> owlfolio.org
Cc: 68808 <at> debbugs.gnu.org
Subject: Re: bug#68808: subsecond mtime discovery code insufficient
Date: Tue, 30 Jan 2024 16:32:04 -0700

(Off-topic for the original bug, but for the sake of public discussion ...)

Hi Zack,

    It is absolutely not *supposed* to be using shell arrays. 

I guess it's not an array. It's the square bracket syntax that confuses
me. A couple of examples from the fn in sanity.m4:

test "$[*]" != "X conftest.ts1 conftest.ts2" ||
..
test "$[]2" = conftest.ts3 &&
..
etc.

I've never seen anything like that before. Searching online, supposedly
$[...] means arithmetic expansion, but clearly that's not what is going
on here. I don't know what it means. Seems like just "$2" (for example)
is intended? I feel like I'm missing/forgetting something basic.

Sorry I didn't read the code when you submitted it originally :(.

Thanks,
Karl

Message #26 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Karl Berry" <karl <at> freefriends.org>
Cc: 68808 <at> debbugs.gnu.org
Subject: Re: bug#68808: subsecond mtime discovery code insufficient
Date: Tue, 30 Jan 2024 18:34:16 -0500

On Tue, Jan 30, 2024, at 6:32 PM, Karl Berry wrote:
> test "$[*]" != "X conftest.ts1 conftest.ts2" ||
> ..
> test "$[]2" = conftest.ts3 &&
> ..
> etc.
>
> I've never seen anything like that before. 

Oh! Those square brackets are M4 quotes to prevent M4 from expanding $* and $2 itself.

zw

Message #29 received at 68808 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: zack <at> owlfolio.org
Cc: 68808 <at> debbugs.gnu.org
Subject: Re: bug#68808: subsecond mtime discovery code insufficient
Date: Wed, 31 Jan 2024 15:10:33 -0700

    Those square brackets are M4 quotes to prevent M4 from expanding
    $* and $2 itself.

Ah, right. Of course. Good!

The $[*] especially confused me. Just looks so perfectly like an array
reference of some strange kind :). I think it would be a tiny bit
clearer to use $[]*? Or am I missing something again?

More importantly, as far as the make probing goes, I have some other
projects to attend to first. So most likely if you can get to it next
week (fingers crossed for your new machine), that will be sooner than
me. Thanks. -k

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.