GNU logs - #47883, boring messages

Message sent to bug-coreutils@HIDDEN:

Subject: bug#47883: sort -o loses data when it crashes
Resent-From: "Peter van Dijk" <peter@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Sun, 18 Apr 2021 22:44:01 +0000
Resent-Message-ID: <handler.47883.B.161878579323246 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 47883 <at> debbugs.gnu.org
User-Agent: Cyrus-JMAP/3.5.0-alpha0-273-g8500d2492d-fm-20210323.002-g8500d249
Mime-Version: 1.0
Message-Id: <0910fdcb-ec97-45e2-9128-4bbb369d74d9@HIDDEN>
Date: Sun, 18 Apr 2021 19:46:01 +0200
From: "Peter van Dijk" <peter@HIDDEN>
Content-Type: text/plain
Received-SPF: none client-ip=64.147.123.19; envelope-from=peter@HIDDEN;
 helo=wout3-smtp.messagingengine.com
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001,
 SPF_NONE=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-Mailman-Approved-At: Sun, 18 Apr 2021 18:43:11 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sort.html: -o  output
    Specify the name of an output file to be used instead of the standard output. This file can be the same as one of the input files.

https://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html: "data may be lost if the system crashes or sort encounters an I/O or other serious error while a file is being sorted in place" and "sort with --merge (-m) can open the output file before reading all input"

While the manual (but not the manpage) mentions the data loss, I think it would be great if sort did not have this problem at all, and I think the OpenGroup text also says it should not have this problem. I looked around, and a lot of software does get this right (by opening a randomly-named temp file to write to, and only moving it into place when it is written successfuly) - GNU sed -i, OpenBSD sort, and surely there are more. As a bonus, doing this would also make the `-o someinputfile -m` case safe.

Reproduction of the data loss is easy:

$ seq 10000 > 10000 ; prlimit --fsize=10 sort -R -o 10000 10000 ; wc -l 10000
File size limit exceeded (core dumped)
2 10000


(coreutils shuf has the same problem even though not all code appears to be shared - for example, sorts open the file for writing even before it opens it for reading, while shuf reverses the order of those two operations. That difference makes no difference in the effect, though.)

-- 
  Peter van Dijk
  peter@HIDDEN

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: "Peter van Dijk" <peter@HIDDEN>
Subject: bug#47883: Acknowledgement (sort -o loses data when it crashes)
Message-ID: <handler.47883.B.161878579323246.ack <at> debbugs.gnu.org>
References: <0910fdcb-ec97-45e2-9128-4bbb369d74d9@HIDDEN>
X-Gnu-PR-Message: ack 47883
X-Gnu-PR-Package: coreutils
Reply-To: 47883 <at> debbugs.gnu.org
Date: Sun, 18 Apr 2021 22:44:01 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-coreutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 47883 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
47883: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47883
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-coreutils@HIDDEN:

Subject: bug#47883: sort -o loses data when it crashes
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Thu, 22 Apr 2021 02:12:01 +0000
Resent-Message-ID: <handler.47883.B47883.161905748630762 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Peter van Dijk <peter@HIDDEN>
Cc: 47883 <at> debbugs.gnu.org
References: <0910fdcb-ec97-45e2-9128-4bbb369d74d9@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Message-ID: <31aa9fcc-4be5-7a13-4682-3c320b46091d@HIDDEN>
Date: Wed, 21 Apr 2021 19:11:14 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <0910fdcb-ec97-45e2-9128-4bbb369d74d9@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On 4/18/21 10:46 AM, Peter van Dijk wrote:
> While the manual (but not the manpage) mentions the data loss, I think =
it would be great if sort did not have this problem at all, and I think t=
he OpenGroup text also says it should not have this problem.

I don't know of any 'sort' implementation that does not have the problem=20
at all. For example, FreeBSD 'sort -o file file' can lose 'file' in some=20
(rare) cases. The only portable way to avoid this problem in a shell=20
script is to output to some other file first and make sure that worked,=20
before attempting to replace the input file.

Also, I don't see where the Open Group spec says what you're saying. On=20
the contrary, the spec merely says that '-o output' should cause output=20
to be sent to the output file. If there are multiple hard links to the=20
output file, this suggests 'sort' should update the output file's=20
contents without breaking any hard links. Admittedly the Open Group spec=20
is a bit vague in this area, but I certainly don't see anything implying=20
that GNU 'sort' does not conform to POSIX in this area.

FreeBSD 'sort' has a problem, in that 'sort -o A B' preserves all hard=20
links to A's file, but 'sort -o A A' does not because it breaks the link=20
from A. That's confusing.

Traditional Unix 'sort -o A' behaves the way GNU 'sort' does; it=20
preserves all hard links to A's file. So there is a compatibility=20
argument for doing things the way GNU 'sort' does them, even if that=20
might lead to more data loss in rare cases.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 24 Apr 2021 21:45:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Apr 24 17:45:07 2021
Received: from localhost ([127.0.0.1]:41626 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1laQ5K-0000RB-NR
	for submit <at> debbugs.gnu.org; Sat, 24 Apr 2021 17:45:06 -0400
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:53402)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1laQ5I-0000QQ-Uz
 for control <at> debbugs.gnu.org; Sat, 24 Apr 2021 17:45:05 -0400
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id F0C41160099
 for <control <at> debbugs.gnu.org>; Sat, 24 Apr 2021 14:44:58 -0700 (PDT)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id 5Eg4gGy-DvXV for <control <at> debbugs.gnu.org>;
 Sat, 24 Apr 2021 14:44:58 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 15DDC1600A7
 for <control <at> debbugs.gnu.org>; Sat, 24 Apr 2021 14:44:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id cUQ_RjWKtr3c for <control <at> debbugs.gnu.org>;
 Sat, 24 Apr 2021 14:44:57 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id C1BF5160099
 for <control <at> debbugs.gnu.org>; Sat, 24 Apr 2021 14:44:57 -0700 (PDT)
To: control <at> debbugs.gnu.org
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Subject: merge 47059 with duplicates
Message-ID: <4a4cbb15-da52-124f-669d-4935481cecb8@HIDDEN>
Date: Sat, 24 Apr 2021 14:44:56 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

merge 47059 47883 48002
thanks

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#47883: sort -o loses data when it crashes
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Sat, 24 Apr 2021 22:01:02 +0000
Resent-Message-ID: <handler.47883.B47883.16193016304466 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47883
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: 
To: L A Walsh <coreutils@HIDDEN>
Cc: 47883 <at> debbugs.gnu.org, peter@HIDDEN
Received: via spool by 47883-submit <at> debbugs.gnu.org id=B47883.16193016304466
          (code B ref 47883); Sat, 24 Apr 2021 22:01:02 +0000
Received: (at 47883) by debbugs.gnu.org; 24 Apr 2021 22:00:30 +0000
Received: from localhost ([127.0.0.1]:41640 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1laQKE-00019Z-2Z
	for submit <at> debbugs.gnu.org; Sat, 24 Apr 2021 18:00:30 -0400
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:54650)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1laQKC-00015d-Av
 for 47883 <at> debbugs.gnu.org; Sat, 24 Apr 2021 18:00:28 -0400
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 3554E160099;
 Sat, 24 Apr 2021 15:00:22 -0700 (PDT)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id ECiNihcWpdFf; Sat, 24 Apr 2021 15:00:21 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 7C1E81600A7;
 Sat, 24 Apr 2021 15:00:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id GkCZxJSlWXnR; Sat, 24 Apr 2021 15:00:21 -0700 (PDT)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 52DF8160099;
 Sat, 24 Apr 2021 15:00:21 -0700 (PDT)
References: <0910fdcb-ec97-45e2-9128-4bbb369d74d9@HIDDEN>
 <31aa9fcc-4be5-7a13-4682-3c320b46091d@HIDDEN>
 <608479A6.2000701@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Message-ID: <c0b6a19e-3537-4a2f-9f3a-3c15b1a67cb8@HIDDEN>
Date: Sat, 24 Apr 2021 15:00:21 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <608479A6.2000701@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

As I wrote you privately last month, the coreutils maintainers (who are=20
not me) are pretty busy. The proposed change in bug#47883 would be=20
incompatible with longstanding tradition and would almost certainly=20
break some existing scripts running on GNU/Linux. This is not something=20
to do lightly.

It might be possible to come up with a different change that would=20
address the issue raised without being so disruptive. Whatever change=20
(if any) is chosen, someone needs to think it through, code it up,=20
document it, and test it. Although nobody's found the time to do that,=20
perhaps you could volunteer or find someone who could volunteer; that=20
would surely accelerate the process.

You mentioned that we have multiple bug reports (now 47059, 47883,=20
48002) on basically the same topic, so I have taken the liberty of=20
merging them.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 21 Feb 2022 09:05:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 21 04:05:32 2022
Received: from localhost ([127.0.0.1]:35002 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nM4dP-0000to-SQ
	for submit <at> debbugs.gnu.org; Mon, 21 Feb 2022 04:05:32 -0500
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:46606)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1nM4dO-0000tb-BH
 for control <at> debbugs.gnu.org; Mon, 21 Feb 2022 04:05:30 -0500
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 171C4160103
 for <control <at> debbugs.gnu.org>; Mon, 21 Feb 2022 01:05:25 -0800 (PST)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id vnTSdSJi1XQc for <control <at> debbugs.gnu.org>;
 Mon, 21 Feb 2022 01:05:24 -0800 (PST)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 79449160106
 for <control <at> debbugs.gnu.org>; Mon, 21 Feb 2022 01:05:24 -0800 (PST)
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id m9CYyoW0WpMh for <control <at> debbugs.gnu.org>;
 Mon, 21 Feb 2022 01:05:24 -0800 (PST)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 586BF160103
 for <control <at> debbugs.gnu.org>; Mon, 21 Feb 2022 01:05:24 -0800 (PST)
Message-ID: <80fc1cea-fd38-bd78-af7e-29bc021c0574@HIDDEN>
Date: Mon, 21 Feb 2022 01:05:24 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Content-Language: en-US
To: control <at> debbugs.gnu.org
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Subject: close 48070
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

close 48070

tags 48034 moreinfo
close 48034

unmerge 47883
unmerge 48002

Last modified: Mon, 21 Feb 2022 09:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.