GNU bug report logs - #26101
Counterproductive calculation order in date

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26101 in the body.
You can then email your comments to 26101 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ulf Zibis <Ulf.Zibis <at> gmx.de>
To: bug-coreutils <at> gnu.org
Subject: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 01:17:07 +0100

[Message part 1 (text/plain, inline)]

Hi,

with
    $ date -r test
I get:
    Di 10. Jan 22:39:14 CET 2017
but with
    $ date -d "$(($(date -r test +%s)-$(date +%s))) seconds -2 months"
I get:
   Sa 12. Nov 22:39:14 CET 2016

I think the better result would be according to a calculation order from 
left to right:
   Sa 10. Nov 22:39:14 CET 2016

Otherwise it seems impossible to correctly change the time stamp of a 
file relative to its current value.
Example:
    $ touch -t $(date -d "$(($(date -r test +%s)-$(date +%s))) seconds 
-2 months" +%Y%m%d%H%M.%S) test
Additionally the command is not correctly reversible with:
    $ touch -t $(date -d "$(($(date -r test +%s)-$(date +%s))) seconds 
+2 months" +%Y%m%d%H%M.%S) test

Regards
Ulf

[smime.p7s (application/pkcs7-signature, attachment)]

Message #8 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Ulf Zibis <Ulf.Zibis <at> gmx.de>, 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Tue, 14 Mar 2017 20:21:59 -0500

[Message part 1 (text/plain, inline)]

On 03/14/2017 07:17 PM, Ulf Zibis wrote:
> Hi,
> 
> with
>     $ date -r test
> I get:
>     Di 10. Jan 22:39:14 CET 2017
> but with
>     $ date -d "$(($(date -r test +%s)-$(date +%s))) seconds -2 months"
> I get:
>    Sa 12. Nov 22:39:14 CET 2016

Let's try this with the new --debug option of 8.26 (although I'm in a
different timezone, and not using localized strings, hopefully the
concept gets through):

$ date -r ChangeLog
Wed Jan 11 17:11:02 CST 2017
$ date --debug -d "$(($(date -r ChangeLog +%s)-$(date +%s))) seconds -2
months"
date: parsed relative part: -5363514 seconds
date: parsed relative part: -2 month(s) -5363514 seconds
date: input timezone: -06:00 (set from system default)
date: using current time as starting value: '20:02:56'
date: using current date as starting value: '(Y-M-D) 2017-03-14'
date: starting date/time: '(Y-M-D) 2017-03-14 20:02:56 TZ=-06:00'
date: warning: when adding relative months/years, it is recommended to
specify the 15th of the months
date: after date adjustment (+0 years, -2 months, +0 days),
date:     new date/time = '(Y-M-D) 2017-01-14 19:02:56 TZ=-06:00'
date: warning: daylight saving time changed after date adjustment
date: '(Y-M-D) 2017-01-14 19:02:56 TZ=-06:00' = 1484442176 epoch-seconds
date: after time adjustment (+0 hours, +0 minutes, -5363514 seconds, +0 ns),
date:     new time = 1479078662 epoch-seconds
date: output timezone: -06:00 (set from system default)
date: final: 1479078662.653491329 (epoch-seconds)
date: final: (Y-M-D) 2016-11-13 23:11:02 (UTC0)
date: final: (Y-M-D) 2016-11-13 17:11:02 (output timezone TZ=-06:00)
Sun Nov 13 17:11:02 CST 2016

> 
> I think the better result would be according to a calculation order from
> left to right:
>    Sa 10. Nov 22:39:14 CET 2016

What calculation are you trying to achieve? The inner $(()) computes the
difference between two dates: the reference file timestamp, and the
current time, resulting in a negative offset in seconds.  The outer date
computation (where I ran --debug) then sees only two relative values - a
negative seconds offset, and a negative months offset, both applied from
the current time as the starting point.

> 
> Otherwise it seems impossible to correctly change the time stamp of a
> file relative to its current value.

Wait - so you're trying to shift a file's time stamp by a fixed amount,
such as by 2 months?  Doesn't that imply that the outer date should see
the file's timestamp as one of its starting points, rather than being
left to start from the current time as its starting point?

> Example:
>     $ touch -t $(date -d "$(($(date -r test +%s)-$(date +%s))) seconds
> -2 months" +%Y%m%d%H%M.%S) test

So what's wrong with:

$ date --debug -d "Jan 1 1970 $(date -r ChangeLog +%s) seconds -2 months"
date: parsed date part: (Y-M-D) 2017-01-01
date: parsed number part: year: 1970
date: parsed relative part: +1484176262 seconds
date: parsed relative part: -2 month(s) +1484176262 seconds
date: input timezone: -06:00 (set from system default)
date: warning: using midnight as starting time: 00:00:00
date: starting date/time: '(Y-M-D) 1970-01-01 00:00:00 TZ=-06:00'
date: warning: when adding relative months/years, it is recommended to
specify the 15th of the months
date: after date adjustment (+0 years, -2 months, +0 days),
date:     new date/time = '(Y-M-D) 1969-11-01 00:00:00 TZ=-06:00'
date: '(Y-M-D) 1969-11-01 00:00:00 TZ=-06:00' = -5248800 epoch-seconds
date: after time adjustment (+0 hours, +0 minutes, +1484176262 seconds,
+0 ns),
date:     new time = 1478927462 epoch-seconds
date: output timezone: -06:00 (set from system default)
date: final: 1478927462.000000000 (epoch-seconds)
date: final: (Y-M-D) 2016-11-12 05:11:02 (UTC0)
date: final: (Y-M-D) 2016-11-11 23:11:02 (output timezone TZ=-06:00)
Fri Nov 11 23:11:02 CST 2016

which is two months prior to the timestamp?

But you may have found a real bug (or at least something we can
enhance). We document that +%s can be reversed:

$ date -r ChangeLog +%s
1484176262
$ date -r ChangeLog
Wed Jan 11 17:11:02 CST 2017
$ date -d @1484176262
Wed Jan 11 17:11:02 CST 2017

But awkwardly, we refuse to allow @seconds to be used as a starting
point for relative operations:

$ date --debug -d "@$(date -r ChangeLog +%s) -2 months"
date: parsed number of seconds part: number of seconds: 1484176262
date: error: parsing failed, stopped at ' months'
src/date: invalid date ‘@1484176262 -2 months’

Also, since you're using GNU date, you're probably using GNU touch,
which means you can use touch -d instead of touch -t to set the time of
the file, with a LOT less notational overhead, and in a reversible
manner relative to a timestamp:

$ : > foo
$ touch -d '-2 months' -r foo foo
$ ls -l foo
-rw-rw-r--. 1 eblake eblake 0 Jan 14 19:19 foo
$ touch -d '+2 months' -r foo foo
$ ls -l foo
-rw-rw-r--. 1 eblake eblake 0 Mar 14 20:19 foo

I'm leaving this open, in case someone wants to tackle the notion of
making @seconds + relative offset a valid parse. But I think your
initial use case is already feasible without any changes to the current
version of date and touch that you are using.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ulf Zibis <Ulf.Zibis <at> gmx.de>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 13:23:48 +0100

[Message part 1 (text/plain, inline)]

Eric,

much thanks for your detailed examination.


Am 15.03.2017 um 02:21 schrieb Eric Blake:
> Let's try this with the new --debug option of 8.26
Great, but current version of my Ubuntu is 8.25

> $ date --debug -d "$(($(date -r ChangeLog +%s)-$(date +%s))) seconds -2
> months"
> date: parsed relative part: -5363514 seconds
> date: parsed relative part: -2 month(s) -5363514 seconds
Why not:

date: parsed relative part: -5363514 seconds -2 month(s)


A more simple example without touch:
$ date +%F
2017-03-15
$ date -d "-20 day" +%F
2017-02-23
$ date -d "-20 day -2 month" +%F
2016-12-26
$ date -d "-2 month -20 day" +%F
2016-12-26

In the 2nd example I would expect:
2016-12-23

>
> What calculation are you trying to achieve? The inner $(()) computes the
> difference between two dates: the reference file timestamp, and the
> current time, resulting in a negative offset in seconds.  The outer date
> computation (where I ran --debug) then sees only two relative values - a
> negative seconds offset, and a negative months offset, both applied from
> the current time as the starting point.
>
>> Otherwise it seems impossible to correctly change the time stamp of a
>> file relative to its current value.
> Wait - so you're trying to shift a file's time stamp by a fixed amount,
> such as by 2 months?  Doesn't that imply that the outer date should see
> the file's timestamp as one of its starting points, rather than being
> left to start from the current time as its starting point?
Yes, but I failed with (a) combining -r and -d and (b) setting an 
absolute starting point e.g.:
$ date -d "$(date -r test) -2 month"
date: ungültiges Datum »Di 10. Jan 22:39:00 CET 2017 -2 month“

So I did this "trick", because I didn't see a chance to directly allow a 
reference file timestamp as starting point.

> So what's wrong with:
> $ date --debug -d "Jan 1 1970 $(date -r ChangeLog +%s) seconds -2 months"
> date: parsed date part: (Y-M-D) 2017-01-01
> date: parsed number part: year: 1970
> date: parsed relative part: +1484176262 seconds
> date: parsed relative part: -2 month(s) +1484176262 seconds
> date: input timezone: -06:00 (set from system default)
> date: warning: using midnight as starting time: 00:00:00
> date: starting date/time: '(Y-M-D) 1970-01-01 00:00:00 TZ=-06:00'
> date: warning: when adding relative months/years, it is recommended to
> specify the 15th of the months
> date: after date adjustment (+0 years, -2 months, +0 days),
> date:     new date/time = '(Y-M-D) 1969-11-01 00:00:00 TZ=-06:00'
> date: '(Y-M-D) 1969-11-01 00:00:00 TZ=-06:00' = -5248800 epoch-seconds
> date: after time adjustment (+0 hours, +0 minutes, +1484176262 seconds,
> +0 ns),
> date:     new time = 1478927462 epoch-seconds
> date: output timezone: -06:00 (set from system default)
> date: final: 1478927462.000000000 (epoch-seconds)
> date: final: (Y-M-D) 2016-11-12 05:11:02 (UTC0)
> date: final: (Y-M-D) 2016-11-11 23:11:02 (output timezone TZ=-06:00)
> Fri Nov 11 23:11:02 CST 2016
>
> which is two months prior to the timestamp?

This example may work, because the shortness of february is not involved.

> Also, since you're using GNU date, you're probably using GNU touch,
> which means you can use touch -d instead of touch -t to set the time of
> the file, with a LOT less notational overhead, and in a reversible
> manner relative to a timestamp:
>
> $ : > foo
> $ touch -d '-2 months' -r foo foo
> $ ls -l foo
> -rw-rw-r--. 1 eblake eblake 0 Jan 14 19:19 foo
> $ touch -d '+2 months' -r foo foo
> $ ls -l foo
> -rw-rw-r--. 1 eblake eblake 0 Mar 14 20:19 foo
Wow, much thanks, I didn't know that touch (a) would allow relative 
dates and (b) would allow to combine -r and -d and even in the weird 
order of -d -r.

> I'm leaving this open,
Thanks.

So I see several points for enhancement:
- Allow @seconds in -d option, even for touch.
- Document the '@' syntax not only as example.
- For date document if combination of -r -d (and others) is allowed.
- For touch document if combination of -r -d (and others) is allowed.
- For date allow combination of -r -d (maybe even more).
- Calculate terms of -d option from left to right.

[smime.p7s (application/pkcs7-signature, attachment)]

Message #14 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Ulf Zibis <Ulf.Zibis <at> gmx.de>, 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 07:44:43 -0500

[Message part 1 (text/plain, inline)]

On 03/15/2017 07:23 AM, Ulf Zibis wrote:

> 
> A more simple example without touch:
> $ date +%F
> 2017-03-15
> $ date -d "-20 day" +%F
> 2017-02-23
> $ date -d "-20 day -2 month" +%F
> 2016-12-26
> $ date -d "-2 month -20 day" +%F
> 2016-12-26
> 
> In the 2nd example I would expect:
> 2016-12-23

Why?  Both operations are computing: Subtract 20 days, and subtract 2
months (equivalent to 60 days).  Subtraction is commutative, it doesn't
matter which one you subtract first, if you are doing two subtractions.

Maybe you are confused on how date implements "subtract a month".  It
does NOT do "subtract 28, 29, 30, or 31 days as appropriate", but rather
does "subtract 30 days, for lack of anything better to do".  Maybe it
could, but someone would have to write the patch.  Relative dates are
HARD to compute, which is why --debug included this warning:

> date: starting date/time: '(Y-M-D) 1970-01-01 00:00:00 TZ=-06:00'
> date: warning: when adding relative months/years, it is recommended to
> specify the 15th of the months 

It sounds like you are trying to do a piecemeal computation, where you
compute "what month was it two months ago (using the trick of starting
from the 15th of this month, and only outputting the month), and then
subtract 2 days from the given day within the month".  To do piecemeal
computations like that requires piecemeal date invocations, unless you
can propose a patch to the date parser engine to make
additions/subtractions of months smarter based on the order that
relative operations appear in the date string, and prove that it won't
break existing users that have come to rely on the currently-documented
behavior of "month == 30 days".

> 
> So I see several points for enhancement:
> - Allow @seconds in -d option, even for touch.
> - Document the '@' syntax not only as example.
> - For date document if combination of -r -d (and others) is allowed.

Not currently allowed, but maybe it should be.

> - For touch document if combination of -r -d (and others) is allowed.

Yes, it's allowed, but a documentation improvement would not hurt.

> - For date allow combination of -r -d (maybe even more).
> - Calculate terms of -d option from left to right.

Already done (but doesn't matter, if it is commutative) - so apparently
what you are REALLY asking for is a smarter "-1 month" that subtracts a
variable number of days (28-31) rather than a fixed number of days (30).


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ulf Zibis <Ulf.Zibis <at> gmx.de>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 14:43:47 +0100

[Message part 1 (text/plain, inline)]

Am 15.03.2017 um 13:44 schrieb Eric Blake:
> Maybe you are confused on how date implements "subtract a month".  It
> does NOT do "subtract 28, 29, 30, or 31 days as appropriate", but rather
> does "subtract 30 days, for lack of anything better to do".

Are you really sure ???
Here on my 8.25 version I get:
$ date -d "-12 month" +%F
2016-03-15
$ date -d "-360 day" +%F
2016-03-20

So I think my list for enhancements is still fully applicable.

-Ulf

PS: You may drop the copy to my personal email, so I don't get it twice, 
as I'm subscribed to the list.

[smime.p7s (application/pkcs7-signature, attachment)]

Message #20 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 10:40:33 -0400

> On Mar 15, 2017, at 09:43, Ulf Zibis <Ulf.Zibis <at> gmx.de> wrote:
> 
> Am 15.03.2017 um 13:44 schrieb Eric Blake:
>> Maybe you are confused on how date implements "subtract a month".  It
>> does NOT do "subtract 28, 29, 30, or 31 days as appropriate", but rather
>> does "subtract 30 days, for lack of anything better to do".
> 
> Are you really sure ???
> Here on my 8.25 version I get:
> $ date -d "-12 month" +%F
> 2016-03-15
> $ date -d "-360 day" +%F
> 2016-03-20

To give more details about the inter working of date adjustments:

Assuming the current date is 2017-03-15:

"2017-03-15 - 12 months":
   Year  = 2017
   Month = -9 (yes, it is stored temporarily as -9)
   Day   = 15
 Then it is normalized to:
   Year  = 2016
   Month = 3
   Day   = 15

While "2017-03-15 - 360 days" =
   Year  = 2017
   Month = 3
   Day   = -345 (yes, stored as -345)
 then normalized to:
   Year  = 2016
   Month = 3
   Day   = 20


More about the normalization:

When adjusting months (e..g "-12 months),
The value of 'day' isn't touched.
It will be (unexpectedly) modified only in cases where the
month does not have that day, e.g.:
 "2017-03-30 - 1 month" becomes:
   Year = 2017
   Month = 2
   Day = 30
 then normalized to:
   Year = 2017
   Month = 3
   Day = 2
(because "2017-02-30" means "2 days after 2017-02-28" which is "2017-03-02").


When adjusting days, it is equivalent to subtracting
the number of seconds from the current unix epoch (i.e. seconds since 1970-01-01).
Example:

Current unix time:
   $ date -d '20170315' +%s
   1489536000

Number of days you want to subtract, in seconds:
   $ bc -l<<<'360*86400'
   31104000

Resulting unix time:
   $ bc -l<<<'1489536000-360*86400'
   1458432000

Which is equivalent to:
   $ date -d @1458432000 +%F
   2016-03-20



Technical note:
date adjustment (years/months/days) is done directly on the member variables
of a 'struct tm' , which is why it ignores the number of days in months.
time adjustment (hours/minutes/seconds) is done on the unix time, AFTER
the normalized date has been converted to unix time.


> So I think my list for enhancements is still fully applicable.

I'm late-comer to the thread, so I'll verify:
You'd like to be able to do date calculations in some predictable way,
in your case to be able to change a file's timestamp.

However,
consider that adjusting by any scale (years/months/days/hours/minutes) will
have side-effects.

Adjusting by months can shift days (e.g. "2016-03-30 - 1 month" is still march).
Adjusting by days can shift hours (e.g. on Day light saving time),
Adjusting by hours can stay in the same day (e.g. "+24 hours" on a when day light saving results in 25 "hours").
etc.

It is the edge-cases that make day adjustment tricky (e.g. "+1 month" is not "+30 days", "+1 day" is not always "+24 hours", etc.).

If you can come up with reliable and predictable rules for date adjustment that will not have these unexpected results - there's definitely room for improvements.
However, there's also the existing behavior and backwards compatibility that will need to be taken into account.

regards,
 -assaf

Message #23 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Ulf Zibis <Ulf.Zibis <at> gmx.de>, 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 09:46:46 -0500

[Message part 1 (text/plain, inline)]

On 03/15/2017 08:43 AM, Ulf Zibis wrote:
> 
> Am 15.03.2017 um 13:44 schrieb Eric Blake:
>> Maybe you are confused on how date implements "subtract a month".  It
>> does NOT do "subtract 28, 29, 30, or 31 days as appropriate", but rather
>> does "subtract 30 days, for lack of anything better to do".
> 
> Are you really sure ???
> Here on my 8.25 version I get:
> $ date -d "-12 month" +%F
> 2016-03-15
> $ date -d "-360 day" +%F
> 2016-03-20

Interesting (it's been a while since I've tried that; so maybe we really
have made improvements in the meantime).  --debug output is pertinent:

$ src/date -d '-12 month' --debug
date: parsed relative part: -12 month(s)
date: input timezone: -06:00 (set from system default)
date: using current time as starting value: '09:41:25'
date: using current date as starting value: '(Y-M-D) 2017-03-15'
date: starting date/time: '(Y-M-D) 2017-03-15 09:41:25 TZ=-06:00'
date: after date adjustment (+0 years, -12 months, +0 days),
date:     new date/time = '(Y-M-D) 2016-03-15 09:41:25 TZ=-06:00'
...

vs.
$ src/date -d '-360 day' --debug
date: parsed relative part: -360 day(s)
date: input timezone: -06:00 (set from system default)
date: using current time as starting value: '09:41:32'
date: using current date as starting value: '(Y-M-D) 2017-03-15'
date: starting date/time: '(Y-M-D) 2017-03-15 09:41:32 TZ=-06:00'
date: warning: when adding relative days, it is recommended to specify
12:00pm
date: after date adjustment (+0 years, +0 months, -360 days),
date:     new date/time = '(Y-M-D) 2016-03-20 09:41:32 TZ=-06:00'

> 
> So I think my list for enhancements is still fully applicable.
> 
> -Ulf
> 
> PS: You may drop the copy to my personal email, so I don't get it twice,
> as I'm subscribed to the list.

List policy is to reply-to-all, so that we don't have to think about who
is subscribed, while making sure that even unsubscribed readers stay in
the loop on the message they are interested in.  The list server has
settings where you can request that you don't receive duplicate messages
(that is, the list won't send you a second copy of the mail if your
address was listed in to or cc); and you can also set mail-followup-to
when posting to help direct the behavior of reply-to-all when someone
replies to you.  It doesn't scale to make me and every other subscriber
special-case "which people that I'm replying to don't want a duplicate",
compared to you to just tweak settings on your end to avoid the
duplicates and/or set things up so that reply-to-all excludes you
because you prefer to get it through the list.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #26 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>, Ulf Zibis <Ulf.Zibis <at> gmx.de>,
 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 08:53:31 -0700

On 15/03/17 05:44, Eric Blake wrote:
> On 03/15/2017 07:23 AM, Ulf Zibis wrote:
> 
>>
>> A more simple example without touch:
>> $ date +%F
>> 2017-03-15
>> $ date -d "-20 day" +%F
>> 2017-02-23
>> $ date -d "-20 day -2 month" +%F
>> 2016-12-26
>> $ date -d "-2 month -20 day" +%F
>> 2016-12-26
>>
>> In the 2nd example I would expect:
>> 2016-12-23
> 
> Why?  Both operations are computing: Subtract 20 days, and subtract 2
> months (equivalent to 60 days).  Subtraction is commutative, it doesn't
> matter which one you subtract first, if you are doing two subtractions.
> 
> Maybe you are confused on how date implements "subtract a month".  It
> does NOT do "subtract 28, 29, 30, or 31 days as appropriate", but rather
> does "subtract 30 days, for lack of anything better to do".  Maybe it
> could, but someone would have to write the patch.  Relative dates are
> HARD to compute, which is why --debug included this warning:

Yes that's an awkward gotcha (I'm not sure is related to the OP's issue).
I outlined a solution at http://bugs.gnu.org/18159#8
I didn't document in "coreutils gotchas" as it's covered in "coreutils FAQ",
though it might deserve a discussion there.

cheers,
Pádraig

Message #29 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Ulf Zibis <Ulf.Zibis <at> gmx.de>, Eric Blake <eblake <at> redhat.com>,
 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 14:47:56 -0400

Hello, 

> On Mar 15, 2017, at 11:53, Pádraig Brady <P <at> draigBrady.com> wrote:
> 
> Yes that's an awkward gotcha (I'm not sure is related to the OP's issue).
> I outlined a solution at http://bugs.gnu.org/18159#8
> I didn't document in "coreutils gotchas" as it's covered in "coreutils FAQ",
> though it might deserve a discussion there.

If mentioning date adjustment in the 'gotcha' page,
perhaps the following examples will be useful (short but sufficient to illustrate the problems). All examples work in EDT timezone (= "America/New_York" or -05:00).
"--debug" should be useful to understand these cases.


A year is not always 365 days (2016 was a leap year with 366 days):

  $ date -d '2016-03-01 - 1 year' +%F
  2015-03-01
  $ date -d '2016-03-01 - 365 days' +%F
  2015-03-02

Month calculations are tricky at the end of the month:

  $ date -d '2017-09-30 - 1 month' +%F
  2017-08-30
  $ date -d '2017-10-31 - 1 month' +%F
  2017-10-01

Explanation:
what happens is that after the month is adjusted from 2017-10-31 to 2017-09-31,
the invalid date 2017-09-31 is adjusted by the number of invalid days (1 day after
the last valid day 2017-09-30 becomes 2017-10-01).
 

A month is not always 30 days (which is obvious, but more tricky
with February):

   $ date -d '2017-03-30 - 1 month' +%F
   2017-03-02
   $ date -d '2017-03-30 - 30 days' +%F
   2017-02-28

A day is not always 24 hours (Nov 6th 2016 was end of DST in US east-coast timezone):

   $ date -d '2016-11-06 + 1 day' +%F
   2016-11-07
   $ date -d '2016-11-06 + 24 hours' +%F
   2016-11-06

Daylight saving adjustment matter *not only* on the actual
day of the switch, but also if the adjusted period crosses
the date of the switch:

   $ date -d '2016-11-01 + 10 days' +%F
   2016-11-11
   $ date -d '2016-11-01 + 240 hours' +%F
   2016-11-10

   $ date -d '2016-06-01 EDT + 6 months' +%F
   2016-11-30




regards,
 - assaf

Message #32 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 15:39:14 -0400

Correcting my own mistake:

> On Mar 15, 2017, at 10:40, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> 
> When adjusting days, it is equivalent to subtracting
> the number of seconds from the current unix epoch (i.e. seconds since 1970-01-01).
> [...]

Adjusting days is equivalent to adjusting hours/minutes/seconds
ONLY when there are no edge cases like daylight saving time.

"1 day" is not always "24 hours" nor "86400 seconds".

regards,
 - assaf

Message #35 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Ulf Zibis <Ulf.Zibis <at> gmx.de>
To: Assaf Gordon <assafgordon <at> gmail.com>, 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 15 Mar 2017 23:37:59 +0100

[Message part 1 (text/plain, inline)]

Am 15.03.2017 um 15:40 schrieb Assaf Gordon:
> To give more details about the inter working of date adjustments:
Much thanks for outlining these details in great clarity.

> You'd like to be able to do date calculations in some predictable way,
Yes!
Currently the result is not predictable because the order of processing 
the terms is not defined transparently.

> in your case to be able to change a file's timestamp.

Not any more as thanks to Eric I now know how to do that by only help of 
touch.

> However,
> consider that adjusting by any scale (years/months/days/hours/minutes) will
> have side-effects.
>
> Adjusting by months can shift days (e.g. "2016-03-30 - 1 month" is still march).
> Adjusting by days can shift hours (e.g. on Day light saving time),
> Adjusting by hours can stay in the same day (e.g. "+24 hours" on a when day light saving results in 25 "hours").
> etc.
>
> It is the edge-cases that make day adjustment tricky (e.g. "+1 month" is not "+30 days", "+1 day" is not always "+24 hours", etc.).
>
> If you can come up with reliable and predictable rules for date adjustment that will not have these unexpected results - there's definitely room for improvements.

I do not see any problems with adjustment when determining the 
processing order from left to right, but I'm not a real specialist to 
that subject.

> However, there's also the existing behavior and backwards compatibility that will need to be taken into account.

This may be an important issue.
On the other hand, such side effects could also result from arbitrary 
code changes, as the processing order to follow is nowhere determined.
Maybe an intelligent search over all shell scripts of a typical Linux 
installation could discover the usage of such cases.

-Ulf

[smime.p7s (application/pkcs7-signature, attachment)]

Message #38 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Ulf Zibis <Ulf.Zibis <at> gmx.de>
To: Eric Blake <eblake <at> redhat.com>, 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Thu, 16 Mar 2017 00:20:16 +0100

[Message part 1 (text/plain, inline)]

Am 15.03.2017 um 15:46 schrieb Eric Blake:
> List policy is to reply-to-all, so that we don't have to think about who
> is subscribed, while making sure that even unsubscribed readers stay in
> the loop on the message they are interested in.  The list server has
> settings where you can request that you don't receive duplicate messages
> (that is, the list won't send you a second copy of the mail if your
> address was listed in to or cc);

Yes, but I prefer to filter my emails on List-Id, as it is unambiguous 
and simple. With that setting I would loose this possibility.

> and you can also set mail-followup-to
> when posting to help direct the behavior of reply-to-all when someone
> replies to you.

I must amid, I do not really understand this.

> It doesn't scale to make me and every other subscriber
> special-case "which people that I'm replying to don't want a duplicate",
> compared to you to just tweak settings on your end to avoid the
> duplicates and/or set things up so that reply-to-all excludes you
> because you prefer to get it through the list.

Agreed!

-Ulf

[smime.p7s (application/pkcs7-signature, attachment)]

Message #41 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Ulf Zibis <Ulf.Zibis <at> gmx.de>
To: Pádraig Brady <P <at> draigBrady.com>,
 Eric Blake <eblake <at> redhat.com>, 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Thu, 16 Mar 2017 00:27:50 +0100

[Message part 1 (text/plain, inline)]

Am 15.03.2017 um 16:53 schrieb Pádraig Brady:
>  "coreutils FAQ"

By the way, I think there is a typo:

$ date --date="$(date +%Y-%m-15) -1 month" +'Last month was %B.'
Last month was June.

$ date --date="$(date +%Y-%m-15) +1 month" +'Next month will be %B.'
Next month will be May.

Compared with:

$ thismonth=$(date --date="@$now" "+%Y-%m-15 12:00")
$ echo "LastMonth=$(date -d "$thismonth 1 month ago" +%B), NextMonth=$(date -d "$thismonth 1 month" +%B)."
LastMonth=March, NextMonth=May.

I assume, in the first example, it should be:

Last month was *March*.


-Ulf

[Message part 2 (text/html, inline)]

[smime.p7s (application/pkcs7-signature, attachment)]

Message #44 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Vincent Lefevre <vincent <at> vinc17.net>
To: Ulf Zibis <Ulf.Zibis <at> gmx.de>
Cc: 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 7 Feb 2018 01:37:34 +0100

On 2017-03-15 13:23:48 +0100, Ulf Zibis wrote:
> A more simple example without touch:
> $ date +%F
> 2017-03-15
> $ date -d "-20 day" +%F
> 2017-02-23
> $ date -d "-20 day -2 month" +%F
> 2016-12-26
> $ date -d "-2 month -20 day" +%F
> 2016-12-26
> 
> In the 2nd example I would expect:
> 2016-12-23

Similarly:

zira% date +%Y-%m-%d -d '2003-02-01 - 1 month'
2003-01-01
zira% date +%Y-%m-%d -d '2003-02-01 - 31 days'
2003-01-01

but if I add "+ 1 month", I get different results:

zira% date +%Y-%m-%d -d '2003-02-01 - 31 days + 1 month'
2003-01-29
zira% date +%Y-%m-%d -d '2003-02-01 - 1 month + 1 month'
2003-02-01

Unfortunately this behavior, which is due to the fact that operations
are reordered to take into account years, then months, then days, is
not documented in the Coreutils manual.

-- 
Vincent Lefèvre <vincent <at> vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Message #47 received at 26101 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Vincent Lefevre <vincent <at> vinc17.net>, Ulf Zibis <Ulf.Zibis <at> gmx.de>
Cc: 26101 <at> debbugs.gnu.org
Subject: Re: bug#26101: Counterproductive calculation order in date
Date: Wed, 7 Feb 2018 03:45:17 -0700

[Message part 1 (text/plain, inline)]

Hello Vincent and all,

On 2018-02-06 05:37 PM, Vincent Lefevre wrote:
> Similarly:
> 
> zira% date +%Y-%m-%d -d '2003-02-01 - 1 month'
> 2003-01-01
> zira% date +%Y-%m-%d -d '2003-02-01 - 31 days'
> 2003-01-01
> 
> but if I add "+ 1 month", I get different results:
> 
> zira% date +%Y-%m-%d -d '2003-02-01 - 31 days + 1 month'
> 2003-01-29
> zira% date +%Y-%m-%d -d '2003-02-01 - 1 month + 1 month'
> 2003-02-01
> 
> Unfortunately this behavior, which is due to the fact that operations
> are reordered to take into account years, then months, then days, is
> not documented in the Coreutils manual.

That is not accurate - there is no "operation order"
(except in the sense that hours/minutes/seconds are handled separately
after years/months/days).

What happens technically is that when adding years/months/days
the 'date' program operates directly on the "struct tm" member variables 
[1], and then uses the mktime function [2] to normalize invalid dates.

[1] http://pubs.opengroup.org/onlinepubs/7908799/xsh/time.h.html
[2] http://pubs.opengroup.org/onlinepubs/7908799/xsh/mktime.html

In your example:

The date "2003-02-01" becomes the following values in "struct tm":
  tm.tm_year = 103 (years since 1900)
  tm.tm_mon  = 1   (zero-based, 0=Jan, 1=Feb, 2=Mar)
  tm.tm_mday = 1

The date "2003-02-01 - 31 days + 1 month" becomes the following values:

  tm.tm_year = 103
  tm.tm_mon  = 2    (1=Feb + 1 month)
  tm.tm_mday = -30  (1 - 31)

Then calling mktime(3) normalizes it to "2003-01-29".

Please see the attached C program which reproduces the steps above.
The coreutils/gnulib equivalent arithmetics code is in
'parse-datetime.y' line 2161 [3] (or search for the comment "Add 
relative date.").

  [3] 
https://opengrok.housegordon.com/source/xref/gnulib/lib/parse-datetime.y#2161

----

As for "how it should behave",
there are so many edge-cases with date calculations that no single
implementation would satisfy everyone.

To avoid such issues,
it is recommended in the FAQ (and in the --debug output)
to always use the 15th day of the month when adding months
(and similarly, use 12pm when adding days and hours,
and use the middle of year when adding years).



Few examples:

1.
Mar 27th in EEST time zone has only 23 hours due to DST,
which leads to the following difference:

  $ TZ=Europe/Helsinki date -d '2011-03-28 - 24 hours' +%F
  2011-03-26
  $ TZ=Europe/Helsinki date -d '2011-03-28 - yesterday' +%F
  2011-03-27

If you add "12pm" (the middle of the day), then the calculations
are more intuitive:

  $ TZ=Europe/Helsinki date -d '2011-03-28 12pm - yesterday' +%F
  2011-03-27
  $ TZ=Europe/Helsinki date -d '2011-03-28 12pm - 24 hours' +%F
  2011-03-27


2.

February 2003 had 28 days, leading to the following result:

  $ date +%F -d '2003-03-31 - 31 days'
  2003-02-28
  $ date +%F -d '2003-03-31 - 1 month'
  2003-03-03

It becomes clearer with "--debug":

  $ date +%F --debug -d '2003-03-31 - 1 month'
  date: parsed date part: (Y-M-D) 2003-03-31
  [...]
  date: warning: when adding relative months/years, it is recommended to
                 specify the 15th of the months
  [...]
  date: warning: month/year adjustment resulted in shifted dates:
  date:      adjusted Y M D: 2003 02 31
  date:    normalized Y M D: 2003 03 03

But if you had used the 15th of the month, it would "just work":

  $ date +%F -d '2003-03-15 - 1 month'
  2003-02-15

Similarly:

  $ ./date +%F -d '2003-10-30 - 8 months'
  2003-03-02
  $ ./date +%F -d '2003-10-15 - 8 months'
  2003-02-15


3.
months with 30 vs 31 days cause similar issues:

  $ ./date +%F -d '2003-05-30 - 1 month'
  2003-04-30
  $ ./date +%F -d '2003-05-31 - 1 month'
  2003-05-01

  $ ./date +%F --debug -d '2003-05-31 - 1 month'
  date: parsed date part: (Y-M-D) 2003-05-31
  date: parsed relative part: -1 month(s)
  [...]
  date: warning: when adding relative months/years, it is recommended to
                 specify the 15th of the months
  [...]
  date: warning: month/year adjustment resulted in shifted dates:
  date:      adjusted Y M D: 2003 04 31
  date:    normalized Y M D: 2003 05 01
  [...]



4.
And of course, issues with leap years (2016 was a leap year, 2017 was not):

  $ date +%F -d '2016-01-01 + 1 year'
  2017-01-01
  $ date +%F -d '2016-01-01 + 365 days'
  2016-12-31
  $ date +%F -d '2016-02-29 + 1 year'
  2017-03-01

  $ ./date +%F --debug -d '2016-02-29 + 1 year'
  date: parsed date part: (Y-M-D) 2016-02-29
  date: parsed relative part: +1 year(s)
  [...]
  date: warning: month/year adjustment resulted in shifted dates:
  date:      adjusted Y M D: 2017 02 29
  date:    normalized Y M D: 2017 03 01
  [...]


By using the 15th of the month, adding years becomes more intuitive:

  $ date +%F -d '2016-02-15 + 1 year'
  2017-02-15




regards,
 - assaf

[date-arith.c (text/x-csrc, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.