GNU bug report logs - #50988
26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: sb@HIDDEN; dated Sun, 3 Oct 2021 08:35:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 50988 <at> debbugs.gnu.org:


Received: (at 50988) by debbugs.gnu.org; 3 Oct 2021 09:15:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 03 05:15:46 2021
Received: from localhost ([127.0.0.1]:32792 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mWxb0-0000wc-Ex
	for submit <at> debbugs.gnu.org; Sun, 03 Oct 2021 05:15:46 -0400
Received: from eggs.gnu.org ([209.51.188.92]:42214)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mWxaz-0000rl-RE
 for 50988 <at> debbugs.gnu.org; Sun, 03 Oct 2021 05:15:46 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43842)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mWxat-0006Wr-DR; Sun, 03 Oct 2021 05:15:39 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3262
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mWxad-0007e8-5a; Sun, 03 Oct 2021 05:15:38 -0400
Date: Sun, 03 Oct 2021 12:15:08 +0300
Message-Id: <83czoma2qr.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: sb@HIDDEN
In-Reply-To: <86o886jym9.fsf@HIDDEN> (sb@HIDDEN)
Subject: Re: bug#50988: 26.3;
 gnus-cloud gets wrong chunk byte count writing sync from windows
References: <86o886jym9.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 50988
Cc: 50988 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: sb@HIDDEN
> Date: Sun, 03 Oct 2021 10:34:06 +0200
> 
> When gnus-cloud-upload-all-data is run in gnus on emacs on windows, the
> byte count in the sync data chunks is wrong, so that the data is broken
> when attemting to sync on a debian machine.
> 
> The first chunk has the following header:
> (:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
> 
> The byte count here is 16241, but it should have been 15754.
> 
> This results in the contents of the score files that follows .gnus.el to
> be copied into .gnus.el and the .gnus.el file becomes unparsable.
> 
> The difference in length corresponds to the number of lines in .gnus.el,
> so I'm guessing it's because the CR in CRLF line endings is stripped
> away.
> 
> The place where the length is set, is here
>   https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n98
> 
> I've tried taking the length of the string instead of the size of the
> buffer, but the count for .gnus.el still came out as 16241 (so the CR
> stripping doesn't take place there).
> 
> I've also, as an experiment, replaced insert-file-contents-literally,
> with insert-file-contents, and then windows gnus wrote data that
> gnus-cloud-download-all-data on debian gnus could read.

Does the file read by that code have DOS-style CRLF end-of-line
format?  And what does that code have to do with your .gnus.el?  (I
don't use Gnus, so apologies for asking questions whose answers are
trivial.)

In general, if that function is supposed to read user-written files
(as opposed to files Gnus itself writes), then it should not assume
the EOL format is Unix.  But I'm not sure insert-file-contents is the
right solution here, because I don't know what will Gnus do with the
file.  Note that the function reads the file into a unibyte buffer,
while insert-file-contents performs decoding, and these two don't mix
well, usually.

Do you understand why the byte count "should have been 15754"?  If
gnus-cloud synchronizes your files with a remote repository, then the
files on the remote should have the same EOL format as on your
original disk, so where does Emacs strip or ignore the CR characters
to come up with a smaller number of bytes in the actual transfer?

HTH




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#50988; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 3 Oct 2021 08:34:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 03 04:34:44 2021
Received: from localhost ([127.0.0.1]:60963 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mWwx4-0007Qg-Ew
	for submit <at> debbugs.gnu.org; Sun, 03 Oct 2021 04:34:44 -0400
Received: from lists.gnu.org ([209.51.188.17]:52938)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sb@HIDDEN>) id 1mWwx2-0007QW-55
 for submit <at> debbugs.gnu.org; Sun, 03 Oct 2021 04:34:28 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:43394)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <sb@HIDDEN>) id 1mWwx1-000354-TZ
 for bug-gnu-emacs@HIDDEN; Sun, 03 Oct 2021 04:34:27 -0400
Received: from cadalora.default.sbang.bv.iomart.io
 ([2001:41c9:1:424::90]:42936)
 by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <sb@HIDDEN>)
 id 1mWwwz-0003O0-C0
 for bug-gnu-emacs@HIDDEN; Sun, 03 Oct 2021 04:34:27 -0400
Received: from mccoy (unknown [84.210.87.211])
 by cadalora.default.sbang.bv.iomart.io (Postfix) with ESMTPSA id A8BFC1000E3
 for <bug-gnu-emacs@HIDDEN>; Sun,  3 Oct 2021 09:34:07 +0100 (BST)
From: sb@HIDDEN
To: bug-gnu-emacs@HIDDEN
Subject: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows
Date: Sun, 03 Oct 2021 10:34:06 +0200
Message-ID: <86o886jym9.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: none client-ip=2001:41c9:1:424::90; envelope-from=sb@HIDDEN;
 helo=cadalora.default.sbang.bv.iomart.io
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001,
 SPF_NONE=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

This happens for me on emacs 26.3, but has happened ever since I first
tried to use gnus-cloud in 2016, so I've never been able to use
gnus-cloud (I've continued to use its predecessor gnus-sync, but with
emacs 27, gnus-sync no longer worked).

When gnus-cloud-upload-all-data is run in gnus on emacs on windows, the
byte count in the sync data chunks is wrong, so that the data is broken
when attemting to sync on a debian machine.

The first chunk has the following header:
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)

The byte count here is 16241, but it should have been 15754.

This results in the contents of the score files that follows .gnus.el to
be copied into .gnus.el and the .gnus.el file becomes unparsable.

The difference in length corresponds to the number of lines in .gnus.el,
so I'm guessing it's because the CR in CRLF line endings is stripped
away.

The place where the length is set, is here
  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n98

I've tried taking the length of the string instead of the size of the
buffer, but the count for .gnus.el still came out as 16241 (so the CR
stripping doesn't take place there).

I've also, as an experiment, replaced insert-file-contents-literally,
with insert-file-contents, and then windows gnus wrote data that
gnus-cloud-download-all-data on debian gnus could read.

But that's probably not a fix, since insert-file-contents-literally is
used intentionally?

This is with
(gnus-cloud-storage-method nil)
which is what I've been using for debugging because attempts to using
base64 with gzip caused error messages of the type "not base64"
(presumably also because the byte count was wrong?).




Acknowledgement sent to sb@HIDDEN:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#50988; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sun, 3 Oct 2021 09:15:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.