GNU bug report logs - #27141
26.0.50; mml-generate-mime-1 broken

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27141 in the body.
You can then email your comments to 27141 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: bug-gnu-emacs <at> gnu.org
Cc: larsi <at> gnus.org
Subject: 26.0.50; mml-generate-mime-1 broken
Date: Tue, 30 May 2017 15:09:28 +0900

[Message part 1 (text/plain, inline)]

Hi,

I noticed this when I forwarded a mail using Gnus.  The problem
is that non-ASCII letters contained in the header of the original
mail will not be encoded properly.  The rough outline of what
`mml-generate-mime-1' does then are:

・Insert the original mail (decoded) into a unibyte buffer
  (that with-temp-buffer of mml.el-[line:619] generates).
・Detect the charset of the contents.
・Encode the contents by that detected charset.
・Insert the encoded contsnts into the other unibyte buffer
  (that with-temp-buffer of mml.el-[line:612] generates).
・Run `mml-to-mime' that encodes the header of the original mail
  using `mail-encode-encoded-word-buffer' (an alias to
  `rfc2047-encode-message-header').

rfc2047's encoder expects human readable text, however, there
are encoded ones as described above.  So, as for at least mail
forwarding, `mml-generate-mime-1' is seriously broken.

The current code looks like WIP to me.  How about reverting it
to the emacs-25 version?  Slightly modernized[1] one is attached.

[1] Don't use old compat functions and string-as-(multi|uni)byte.

In GNU Emacs 26.0.50 (build 1, i686-pc-cygwin, GTK+ Version 3.18.9)
 of 2017-05-30 built on localhost
Windowing system distributor 'The Cygwin/X Project', version 11.0.11900000

[Message part 2 (application/emacs-lisp, attachment)]

Message #10 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: 27141 <at> debbugs.gnu.org
Cc: larsi <at> gnus.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Thu, 01 Jun 2017 08:21:57 +0900

On Tue, 30 May 2017 15:09:28 +0900, Katsumi Yamaoka wrote:
> So, as for at least mail forwarding, `mml-generate-mime-1' is
> seriously broken.

I've reverted it to the emacs-25 version (with some modifications)
for a emergency fix.

Message #13 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Katsumi Yamaoka <yamaoka <at> jpl.org>
Cc: 27141 <at> debbugs.gnu.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Thu, 01 Jun 2017 07:20:13 +0200

Katsumi Yamaoka <yamaoka <at> jpl.org> writes:

> On Tue, 30 May 2017 15:09:28 +0900, Katsumi Yamaoka wrote:
>> So, as for at least mail forwarding, `mml-generate-mime-1' is
>> seriously broken.
>
> I've reverted it to the emacs-25 version (with some modifications)
> for a emergency fix.

Well, the function was reworked because it destroys files when
forwarding them, I seem to recall?  So this seems like it's just
changing one bug for another...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #16 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 27141 <at> debbugs.gnu.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Thu, 01 Jun 2017 15:15:57 +0900

On Thu, 01 Jun 2017 07:20:13 +0200, Lars Ingebrigtsen wrote:
> Katsumi Yamaoka <yamaoka <at> jpl.org> writes:

>> On Tue, 30 May 2017 15:09:28 +0900, Katsumi Yamaoka wrote:
>>> So, as for at least mail forwarding, `mml-generate-mime-1' is
>>> seriously broken.

>> I've reverted it to the emacs-25 version (with some modifications)
>> for a emergency fix.

> Well, the function was reworked because it destroys files when
> forwarding them, I seem to recall?  So this seems like it's just
> changing one bug for another...

OK, could you please re-revert and fix it?  I did have no idea
to fix the bug in this construction...

(with-temp-buffer
  (set-buffer-multibyte nil)
  (insert
   (with-temp-buffer ;; <- unibyte
     (insert something) ;; <- multibyte
     (encode-coding ...)
     (buffer-string))))

Thanks.

Message #19 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Katsumi Yamaoka <yamaoka <at> jpl.org>
Cc: 27141 <at> debbugs.gnu.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Thu, 01 Jun 2017 09:17:06 +0200

Katsumi Yamaoka <yamaoka <at> jpl.org> writes:

> OK, could you please re-revert and fix it?  I did have no idea
> to fix the bug in this construction...
>
> (with-temp-buffer
>   (set-buffer-multibyte nil)
>   (insert
>    (with-temp-buffer ;; <- unibyte

Hm...  why is this buffer unibyte here?  And if it is, won't
(set-buffer-multibyte t) fix that?

>      (insert something) ;; <- multibyte
>      (encode-coding ...)
>      (buffer-string))))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #22 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 27141 <at> debbugs.gnu.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Thu, 01 Jun 2017 17:32:53 +0900

On Thu, 01 Jun 2017 09:17:06 +0200, Lars Ingebrigtsen wrote:
>> (with-temp-buffer
>>   (set-buffer-multibyte nil)
>>   (insert
>>    (with-temp-buffer ;; <- unibyte

> Hm...  why is this buffer unibyte here?

Oops, for years I misunderstood that the multibyteness of a newly
created buffer defaults to that of the current buffer.  But it's
wrong!

(with-temp-buffer
  (set-buffer-multibyte nil)
  (with-temp-buffer
    enable-multibyte-characters))
 => t

> And if it is, won't (set-buffer-multibyte t) fix that?

No, it doesn't help.  As I wrote the first post,

> ・Insert the original mail (decoded) into a unibyte buffer
                                              ^^^^^^^ multibyte
>   (that with-temp-buffer of mml.el-[line:619] generates).
> ・Detect the charset of the contents.
> ・Encode the contents by that detected charset.
> ・Insert the encoded contsnts into the other unibyte buffer
>   (that with-temp-buffer of mml.el-[line:612] generates).
> ・Run `mml-to-mime' that encodes the header of the original mail
>   using `mail-encode-encoded-word-buffer' (an alias to
>   `rfc2047-encode-message-header').

> rfc2047's encoder expects human readable text, however, there
> are encoded ones as described above.

Message #25 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Katsumi Yamaoka <yamaoka <at> jpl.org>
Cc: 27141 <at> debbugs.gnu.org, larsi <at> gnus.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Sat, 02 Sep 2017 17:05:05 +0300

> Date: Thu, 01 Jun 2017 17:32:53 +0900
> From: Katsumi Yamaoka <yamaoka <at> jpl.org>
> Cc: 27141 <at> debbugs.gnu.org
> 
> On Thu, 01 Jun 2017 09:17:06 +0200, Lars Ingebrigtsen wrote:
> >> (with-temp-buffer
> >>   (set-buffer-multibyte nil)
> >>   (insert
> >>    (with-temp-buffer ;; <- unibyte
> 
> > Hm...  why is this buffer unibyte here?
> 
> Oops, for years I misunderstood that the multibyteness of a newly
> created buffer defaults to that of the current buffer.  But it's
> wrong!
> 
> (with-temp-buffer
>   (set-buffer-multibyte nil)
>   (with-temp-buffer
>     enable-multibyte-characters))
>  => t
> 
> > And if it is, won't (set-buffer-multibyte t) fix that?
> 
> No, it doesn't help.  As I wrote the first post,
> 
> > ・Insert the original mail (decoded) into a unibyte buffer
>                                               ^^^^^^^ multibyte
> >   (that with-temp-buffer of mml.el-[line:619] generates).
> > ・Detect the charset of the contents.
> > ・Encode the contents by that detected charset.
> > ・Insert the encoded contsnts into the other unibyte buffer
> >   (that with-temp-buffer of mml.el-[line:612] generates).
> > ・Run `mml-to-mime' that encodes the header of the original mail
> >   using `mail-encode-encoded-word-buffer' (an alias to
> >   `rfc2047-encode-message-header').
> 
> > rfc2047's encoder expects human readable text, however, there
> > are encoded ones as described above.

Ping!  Any news on this issue?  It currently indicated as blocking the
release of Emacs 26.1, so can we please expedite its resolution?

Thanks.

Message #28 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 27141 <at> debbugs.gnu.org, larsi <at> gnus.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Mon, 04 Sep 2017 08:37:26 +0900

On Sat, 02 Sep 2017 17:05:05 +0300, Eli Zaretskii wrote:
> Ping!  Any news on this issue?  It currently indicated as blocking the
> release of Emacs 26.1, so can we please expedite its resolution?

The version Lars modified was obviously broken at least for the
problem I brought up:

> I noticed this when I forwarded a mail using Gnus.  The problem
> is that non-ASCII letters contained in the header of the original
> mail will not be encoded properly.

Oh, I forgot to write the bottom line, sorry.  It happend when I
forward a message in which the header contains non-ASCII letters.
What I did then (revno 8130d91) was reverting Lars' change.  I'm
using the current version with no problem so far.  Though I don't
know what Lars tried to fix, I think it is not a stopper for the
next release as there seems to be no bug report brought up about
it.  Lars?

Message #31 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Katsumi Yamaoka <yamaoka <at> jpl.org>
Cc: 27141 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Wed, 13 Sep 2017 19:36:47 +0200

Katsumi Yamaoka <yamaoka <at> jpl.org> writes:

> Oh, I forgot to write the bottom line, sorry.  It happend when I
> forward a message in which the header contains non-ASCII letters.
> What I did then (revno 8130d91) was reverting Lars' change.  I'm
> using the current version with no problem so far.  Though I don't
> know what Lars tried to fix, I think it is not a stopper for the
> next release as there seems to be no bug report brought up about
> it.  Lars?

I seem to have forgotten to tag my patch with the corresponding bug
report?  Annoying.  Anyway, the problem my patch fixed was dealing with
text files included in Message with mml tags -- they would (under some
circumstances; there was a test case in the original bug report) be
corrupted.

But my fix was obviously a mistake, since it resulted in pretty much all
multibyte buffers being mishandled when forwarding, so the original
problem will have to be fixed in a different way.

But I don't think this should be a release blocker.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #34 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 27141 <at> debbugs.gnu.org, yamaoka <at> jpl.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Wed, 13 Sep 2017 21:41:30 +0300

unblock 24655 by 27141
thanks

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  27141 <at> debbugs.gnu.org
> Date: Wed, 13 Sep 2017 19:36:47 +0200
> 
> I seem to have forgotten to tag my patch with the corresponding bug
> report?  Annoying.  Anyway, the problem my patch fixed was dealing with
> text files included in Message with mml tags -- they would (under some
> circumstances; there was a test case in the original bug report) be
> corrupted.
> 
> But my fix was obviously a mistake, since it resulted in pretty much all
> multibyte buffers being mishandled when forwarding, so the original
> problem will have to be fixed in a different way.
> 
> But I don't think this should be a release blocker.

Thanks, I'm unblocking it.  Though I hope this will be fixed before
26.1 is out.

Message #39 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 27141 <at> debbugs.gnu.org, yamaoka <at> jpl.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Wed, 13 Sep 2017 20:45:11 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Thanks, I'm unblocking it.  Though I hope this will be fixed before
> 26.1 is out.

Yup.  Does anybody happen to know what the original bug report number
is?  I seem to remember that it had a nice test case that demonstrated
the problem...  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #42 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 27141 <at> debbugs.gnu.org, yamaoka <at> jpl.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Wed, 13 Sep 2017 22:14:45 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: yamaoka <at> jpl.org,  27141 <at> debbugs.gnu.org
> Date: Wed, 13 Sep 2017 20:45:11 +0200
> 
> Does anybody happen to know what the original bug report number is?
> I seem to remember that it had a nice test case that demonstrated
> the problem...  :-)

Not sure what you mean.  Did you mean this:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27141#5

I see no other bug number cited in the discussion of this one.

Message #45 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 27141 <at> debbugs.gnu.org, yamaoka <at> jpl.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Wed, 13 Sep 2017 21:18:54 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Not sure what you mean.  Did you mean this:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27141#5
>
> I see no other bug number cited in the discussion of this one.

There is no other bug number mentioned, which is why I asked whether
someone (by chance) remembered the other bug number.  It was a long
shot.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #48 received at 27141 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: 27141 <at> debbugs.gnu.org,  yamaoka <at> jpl.org
Subject: Re: bug#27141: 26.0.50; mml-generate-mime-1 broken
Date: Mon, 23 Sep 2019 20:15:53 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Eli Zaretskii <eliz <at> gnu.org> writes:
>
>> Not sure what you mean.  Did you mean this:
>>
>>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27141#5
>>
>> I see no other bug number cited in the discussion of this one.
>
> There is no other bug number mentioned, which is why I asked whether
> someone (by chance) remembered the other bug number.  It was a long
> shot.  :-)

I've looked around, and I can't find it.  Without a test case, I don't
think we're going to get further on that problem, and this bug report is
about the bug my fix introduced, so I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.