GNU bug report logs - #50988
26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 50988 in the body.
You can then email your comments to 50988 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: sb <at> dod.no
To: bug-gnu-emacs <at> gnu.org
Subject: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows
Date: Sun, 03 Oct 2021 10:34:06 +0200

This happens for me on emacs 26.3, but has happened ever since I first
tried to use gnus-cloud in 2016, so I've never been able to use
gnus-cloud (I've continued to use its predecessor gnus-sync, but with
emacs 27, gnus-sync no longer worked).

When gnus-cloud-upload-all-data is run in gnus on emacs on windows, the
byte count in the sync data chunks is wrong, so that the data is broken
when attemting to sync on a debian machine.

The first chunk has the following header:
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)

The byte count here is 16241, but it should have been 15754.

This results in the contents of the score files that follows .gnus.el to
be copied into .gnus.el and the .gnus.el file becomes unparsable.

The difference in length corresponds to the number of lines in .gnus.el,
so I'm guessing it's because the CR in CRLF line endings is stripped
away.

The place where the length is set, is here
  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n98

I've tried taking the length of the string instead of the size of the
buffer, but the count for .gnus.el still came out as 16241 (so the CR
stripping doesn't take place there).

I've also, as an experiment, replaced insert-file-contents-literally,
with insert-file-contents, and then windows gnus wrote data that
gnus-cloud-download-all-data on debian gnus could read.

But that's probably not a fix, since insert-file-contents-literally is
used intentionally?

This is with
(gnus-cloud-storage-method nil)
which is what I've been using for debugging because attempts to using
base64 with gzip caused error messages of the type "not base64"
(presumably also because the byte count was wrong?).

Message #8 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: sb <at> dod.no
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3;
 gnus-cloud gets wrong chunk byte count writing sync from windows
Date: Sun, 03 Oct 2021 12:15:08 +0300

> From: sb <at> dod.no
> Date: Sun, 03 Oct 2021 10:34:06 +0200
> 
> When gnus-cloud-upload-all-data is run in gnus on emacs on windows, the
> byte count in the sync data chunks is wrong, so that the data is broken
> when attemting to sync on a debian machine.
> 
> The first chunk has the following header:
> (:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
> 
> The byte count here is 16241, but it should have been 15754.
> 
> This results in the contents of the score files that follows .gnus.el to
> be copied into .gnus.el and the .gnus.el file becomes unparsable.
> 
> The difference in length corresponds to the number of lines in .gnus.el,
> so I'm guessing it's because the CR in CRLF line endings is stripped
> away.
> 
> The place where the length is set, is here
>   https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n98
> 
> I've tried taking the length of the string instead of the size of the
> buffer, but the count for .gnus.el still came out as 16241 (so the CR
> stripping doesn't take place there).
> 
> I've also, as an experiment, replaced insert-file-contents-literally,
> with insert-file-contents, and then windows gnus wrote data that
> gnus-cloud-download-all-data on debian gnus could read.

Does the file read by that code have DOS-style CRLF end-of-line
format?  And what does that code have to do with your .gnus.el?  (I
don't use Gnus, so apologies for asking questions whose answers are
trivial.)

In general, if that function is supposed to read user-written files
(as opposed to files Gnus itself writes), then it should not assume
the EOL format is Unix.  But I'm not sure insert-file-contents is the
right solution here, because I don't know what will Gnus do with the
file.  Note that the function reads the file into a unibyte buffer,
while insert-file-contents performs decoding, and these two don't mix
well, usually.

Do you understand why the byte count "should have been 15754"?  If
gnus-cloud synchronizes your files with a remote repository, then the
files on the remote should have the same EOL format as on your
original disk, so where does Emacs strip or ignore the CR characters
to come up with a smaller number of bytes in the actual transfer?

HTH

Message #11 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Steinar Bang <sb <at> dod.no>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3;
 gnus-cloud gets wrong chunk byte count writing sync from windows
Date: Sun, 03 Oct 2021 12:32:51 +0200

>>>>> Eli Zaretskii <eliz <at> gnu.org>:

> Does the file read by that code have DOS-style CRLF end-of-line
> format?

Yes.

> And what does that code have to do with your .gnus.el?  (I don't use
> Gnus, so apologies for asking questions whose answers are trivial.)

The purpose of gnus-cloud is to create IMAP like functionality for NNTP
groups.  Ie. what's read and followed up on on one computer, is marked
as read and followed up on a different computer (I use gnus on 3
different computers and sync state between them).

gnus-cloud also syncs setup in the .gnus.el and the current state of the
score files. The .gnus.el contents is the first chunk in the sync file.

The mechanism of the sync is an IMAP folder with new syncs written as
messages, with the body of the message consisting chunks that have a
lisp-ish header, like this:
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
and then the contents of the chunk, until the next header

An example:
 https://gist.github.com/steinarb/155c43fd102c1a929ea441b0703492a3

When the byte count in "length" is wrong, the contents of bytes past the
end, are added to ~/.gnus.el in the computer synced to.

> In general, if that function is supposed to read user-written files
> (as opposed to files Gnus itself writes), then it should not assume
> the EOL format is Unix.  But I'm not sure insert-file-contents is the
> right solution here, because I don't know what will Gnus do with the
> file.

Yes, I'm pretty sure insert-file-contents isn't the right fix either,
but now it sort of works, as opposed to not working at all.  So I'll
limp along on that as a workaround until a real fix is.

> Note that the function reads the file into a unibyte buffer, while
> insert-file-contents performs decoding, and these two don't mix well,
> usually.

> Do you understand why the byte count "should have been 15754"?

Yes.

The .gnus.el file became unparsable on the synched to machine after
sync, because of stuff that was added at the end (which sort of looked
like gnus score files).

So I looked into the contents of the file and used head to return the
first 16241 bytes of the file following the header
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
and saw that this included the stuff that broke the parsing of
.gnus.el:
 https://gist.github.com/steinarb/4ed3a4b716aab22f855df9af387cce78
(search for :type and you'll find the point where .gnus.el parsing
breaks after sync).

At this point I had concluded that the byte count was wrong, and the
only thing I could think about as the cause, was CRLF vs LF.

My .gnus.el has 487 lines and (- 16241 487) is 15754, so I tried using
head to extract the first 15754 bytes of the chunk and ended up with the
correct results:
 https://gist.github.com/steinarb/650839cec8b9961bb614b95be9c026b2

So the evidence is circumstantial but compelling (compelling to me, at
least). 

> If gnus-cloud synchronizes your files with a remote repository, then
> the files on the remote should have the same EOL format as on your
> original disk, so where does Emacs strip or ignore the CR characters
> to come up with a smaller number of bytes in the actual transfer?

I'm not sure where the conversion happens and have been unable to figure
it out.

But I'm guessing the buffer inserted into here, has eol lf-only and that
this is where/when the conversion happens:
 https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n104

FWIW I didn't really want gnus-cloud to sync .gnus.el, since that file
is handled as a config file checked into git.  But excluding it is not a
part of gnus-cloud config (at least not documented).

Message #14 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Steinar Bang <sb <at> dod.no>
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3;
 gnus-cloud gets wrong chunk byte count writing sync from windows
Date: Sun, 03 Oct 2021 14:04:44 +0300

> From: Steinar Bang <sb <at> dod.no>
> Cc: 50988 <at> debbugs.gnu.org
> Date: Sun, 03 Oct 2021 12:32:51 +0200
> 
> > If gnus-cloud synchronizes your files with a remote repository, then
> > the files on the remote should have the same EOL format as on your
> > original disk, so where does Emacs strip or ignore the CR characters
> > to come up with a smaller number of bytes in the actual transfer?
> 
> I'm not sure where the conversion happens and have been unable to figure
> it out.
> 
> But I'm guessing the buffer inserted into here, has eol lf-only and that
> this is where/when the conversion happens:
>  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n104

Sounds unlikely: inserting a string into a buffer doesn't change the
EOL format, or at least isn't supposed to.

Message #17 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: sb <at> dod.no
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing
 sync from windows
Date: Mon, 12 Sep 2022 12:24:55 +0200

sb <at> dod.no writes:

> The first chunk has the following header:
> (:type :file :file-name "~/.gnus.el" :timestamp
> "2021-09-19T15:04:33+0200" :length 16241)
>
> The byte count here is 16241, but it should have been 15754.

I'm not able to reproduce this exactly, but I think I've identified a
possible problem in this area now.  Do you still see these issues on the
current "master" branch?

Message #22 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Steinar Bang <sb <at> dod.no>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing
 sync from windows
Date: Tue, 13 Sep 2022 19:28:58 +0200

>>>>> Lars Ingebrigtsen <larsi <at> gnus.org>:
> sb <at> dod.no writes:

>> The first chunk has the following header:
>> (:type :file :file-name "~/.gnus.el" :timestamp
>> "2021-09-19T15:04:33+0200" :length 16241)
>> 
>> The byte count here is 16241, but it should have been 15754.

> I'm not able to reproduce this exactly, but I think I've identified a
> possible problem in this area now.  Do you still see these issues on
> the current "master" branch?

No, sorry!

I removed gnus-cloud sync of .gnus.el and .authinfo.gpg on "Wed Oct 6 20:50:09 2021 +0200"

modified   .emacs
@@ -783,6 +783,7 @@ to be something different.")
  '(gnus-cloud-interactive nil)
  '(gnus-cloud-method "nnimap:privat")
  '(gnus-cloud-storage-method nil)
+ '(gnus-cloud-synced-files '((:directory "~/News" :match ".*.SCORE\\'")))
  '(gnus-picon-style 'right)
  '(gnus-treat-fill-long-lines nil)
  '(gnus-treat-newsgroups-picon nil)

(also, I'm now using a debian GNU/linux laptop, so I don't see any
CRLF/LF related issues (which I still think this was, even if Eli
doesn't). I.e. not easy to reproduce)

Message #25 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Steinar Bang <sb <at> dod.no>
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing
 sync from windows
Date: Wed, 14 Sep 2022 14:34:22 +0200

Steinar Bang <sb <at> dod.no> writes:

> (also, I'm now using a debian GNU/linux laptop, so I don't see any
> CRLF/LF related issues (which I still think this was, even if Eli
> doesn't). I.e. not easy to reproduce)

OK, then I guess there won't be any further progress in this bug report,
and I'm closing it.  If you do see this problem again, please respond to
the debbugs address and we'll reopen.

Message #30 received at 50988 <at> debbugs.gnu.org (full text, mbox):

From: Steinar Bang <sb <at> dod.no>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 50988 <at> debbugs.gnu.org
Subject: Re: bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing
 sync from windows
Date: Wed, 14 Sep 2022 17:04:19 +0200

>>>>> Lars Ingebrigtsen <larsi <at> gnus.org>:

> Steinar Bang <sb <at> dod.no> writes:
>> (also, I'm now using a debian GNU/linux laptop, so I don't see any
>> CRLF/LF related issues (which I still think this was, even if Eli
>> doesn't).

(Because if I remember correctly, the diff in byte count matched the
line number...)

>> I.e. not easy to reproduce)

> OK, then I guess there won't be any further progress in this bug report,
> and I'm closing it.  If you do see this problem again, please respond to
> the debbugs address and we'll reopen.

+1

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.