GNU bug report logs - #45657
27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 45657 in the body.
You can then email your comments to 45657 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Justus-dev <at> Piater.name
To: bug-gnu-emacs <at> gnu.org
Subject: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding
Date: Mon, 04 Jan 2021 18:17:51 +0100

Since Emacs 27.1 and still in the current development version, non-ASCII
characters display incorrectly if the encoding is 7bit or 8bit, at least
in my Gnus setup with nnmaildir backend.  This holds for both text/plain
and text/html content types.

To test this, send yourself a MIME message such as the following:

<#multipart type=mixed>
<#part type=text/plain charset="utf-8" encoding="7bit" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#part type=text/plain charset="utf-8" encoding="8bit" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#part type=text/plain charset="utf-8" encoding="quoted-printable" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#part type=text/plain charset="utf-8" encoding="base64" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#/multipart>

The first two parts will show non-ASCII characters in \ddd form (in HTML
parts, this would be printable but wrong characters instead), while the
third and fourth parts will display correctly.

I have not bothered reporting this till now because there is an easy
workaround (hit e (to edit) followed immediately by C-c C-c (to commit
the void edit), which will reencode the message in quoted-printable),
and because I figured I cannot be the only one who observes this
problem...???


This bug persists in my current Emacs build:

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.17.4)
 of 2021-01-03 built on piater
Repository revision: dc5dab469242c40803891518efe9c9f6bd650cea
Repository branch: makepkg
Windowing system distributor 'System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-sound=alsa --with-modules --without-gconf --without-gsettings
 --with-nativecomp --with-pgtk --with-x-toolkit=gtk3 --without-xaw3d
 --without-m17n-flt --with-cairo --with-xwidgets
 --without-compress-install 'CFLAGS=-march=x86-64 -mtune=generic -O2
 -pipe -fno-plt -g -fuse-ld=gold -g -fuse-ld=gold'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL
GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3
PGTK XIM MODULES NATIVE_COMP THREADS XWIDGETS LIBSYSTEMD JSON PDUMPER
LCMS2

Important settings:
  value of $LC_COLLATE: de_AT.UTF-8
  value of $LC_MONETARY: de_AT.UTF-8
  value of $LC_TIME: de_AT.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  semantic-minor-modes-format: ((:eval (if (or semantic-highlight-edits-mode semantic-show-unmatched-syntax-mode)  S)))
  gnus-agent-group-mode: t
  gnus-topic-mode: t
  shell-dirtrack-mode: t
  gnus-undo-mode: t
  show-paren-mode: t
  override-global-mode: t
  pdfgrep-mode: t
  pdf-occur-global-minor-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/haskell-mode/w3m-haddock hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/w3m-haddock
/usr/share/emacs/site-lisp/haskell-mode/inf-haskell hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/inf-haskell
/usr/share/emacs/site-lisp/haskell-mode/highlight-uses-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/highlight-uses-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell
/usr/share/emacs/site-lisp/haskell-mode/haskell-utils hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-utils
/usr/share/emacs/site-lisp/haskell-mode/haskell-unicode-input-method hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-unicode-input-method
/usr/share/emacs/site-lisp/haskell-mode/haskell-string hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-string
/usr/share/emacs/site-lisp/haskell-mode/haskell-sort-imports hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-sort-imports
/usr/share/emacs/site-lisp/haskell-mode/haskell-session hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-session
/usr/share/emacs/site-lisp/haskell-mode/haskell-sandbox hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-sandbox
/usr/share/emacs/site-lisp/haskell-mode/haskell-repl hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-repl
/usr/share/emacs/site-lisp/haskell-mode/haskell-process hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-process
/usr/share/emacs/site-lisp/haskell-mode/haskell-presentation-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-presentation-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell-navigate-imports hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-navigate-imports
/usr/share/emacs/site-lisp/haskell-mode/haskell-move-nested hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-move-nested
/usr/share/emacs/site-lisp/haskell-mode/haskell-modules hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-modules
/usr/share/emacs/site-lisp/haskell-mode/haskell-menu hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-menu
/usr/share/emacs/site-lisp/haskell-mode/haskell-load hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-load
/usr/share/emacs/site-lisp/haskell-mode/haskell-lexeme hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-lexeme
/usr/share/emacs/site-lisp/haskell-mode/haskell-interactive-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-interactive-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell-indentation hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-indentation
/usr/share/emacs/site-lisp/haskell-mode/haskell-indent hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-indent
/usr/share/emacs/site-lisp/haskell-mode/haskell-hoogle hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-hoogle
/usr/share/emacs/site-lisp/haskell-mode/haskell-font-lock hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-font-lock
/usr/share/emacs/site-lisp/haskell-mode/haskell-doc hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-doc
/usr/share/emacs/site-lisp/haskell-mode/haskell-decl-scan hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-decl-scan
/usr/share/emacs/site-lisp/haskell-mode/haskell-debug hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-debug
/usr/share/emacs/site-lisp/haskell-mode/haskell-customize hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-customize
/usr/share/emacs/site-lisp/haskell-mode/haskell-completions hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-completions
/usr/share/emacs/site-lisp/haskell-mode/haskell-complete-module hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-complete-module
/usr/share/emacs/site-lisp/haskell-mode/haskell-compile hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-compile
/usr/share/emacs/site-lisp/haskell-mode/haskell-compat hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-compat
/usr/share/emacs/site-lisp/haskell-mode/haskell-commands hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-commands
/usr/share/emacs/site-lisp/haskell-mode/haskell-collapse hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-collapse
/usr/share/emacs/site-lisp/haskell-mode/haskell-checkers hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-checkers
/usr/share/emacs/site-lisp/haskell-mode/haskell-cabal hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-cabal
/usr/share/emacs/site-lisp/haskell-mode/haskell-c2hs hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-c2hs
/usr/share/emacs/site-lisp/haskell-mode/haskell-align-imports hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-align-imports
/usr/share/emacs/site-lisp/haskell-mode/ghci-script-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/ghci-script-mode
/usr/share/emacs/site-lisp/haskell-mode/ghc-core hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/ghc-core

Features:
(shadow emacsbug timezone vc-hg vc-git vc-bzr pdf-sync pdf-annot
pdf-outline pdf-links pdf-history smerge-mode diff diff-mode dabbrev rx
term ehelp shr-color flow-fill edebug debug backtrace pop3 gnus-gravatar
gravatar dns gnus-cus gnus-html gnus-kill gnus-logic gnus-mh mh-comp
mh-scan mh-gnus mh-e mh-compat mh-buffers mh-loaddefs gnus-registry
registry gnus-salt gnus-vm cl-print shortdoc locate mule-util ffap
spam-stat nnmairix nnml gnus-uu yenc gnus-sieve gnus-notifications
gnus-fun notifications gnus-eform gnus-dup gnus-diary nndiary gnus-demon
gnus-delay gnus-bookmark deuglify tramp-cmds misearch multi-isearch view
smiley utf-7 imap rfc2104 epa-file mailalias smtpmail sendmail
gnus-bcklg gnus-draft gnus-cite mail-extr gnus-async qp sort gnus-ml
disp-table gnus-topic mm-archive gnutls network-stream url-http url-gw
nsm url-cache url-auth nnrss nndraft nnmh nnfolder nnmaildir gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp gnus-cache
tramp-cache tramp-sh tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat native-complete shell ls-lisp cus-theme xwidget
wid-browse tree-widget dired-aux org-clock diary-lib diary-loaddefs
cal-iso org-indent ol-eww eww xdg url-queue thingatpt mm-url ol-rmail
ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search eieio-opt speedbar
ezimage dframe gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-sum shr kinsoku svg dom gnus-group gnus-undo gnus-start gnus-dbus
dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec
gnus-int gnus-range message rmc puny rfc822 mml mml-sec epa epg
epg-config mailabbrev gmm-utils mailheader gnus-win ol-docview ol-bibtex
bibtex ol-bbdb ol-w3m poly-org polymode poly-lock polymode-base
polymode-weave polymode-export polymode-compat polymode-methods
polymode-core polymode-classes eieio-custom eieio-base color matlab
derived matlab-compat pulse face-remap org-agenda paren gnus nnheader
gnus-util rmail rmail-loaddefs mail-utils gnus-dired parse-time iso8601
use-package-bind-key bind-key quelpa-use-package cl-extra
use-package-core quelpa mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr lisp-mnt help-fns
radix-tree help-mode ox-odt rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar ox-html table ox-ascii ox-publish ox org-element avl-tree
generator org-id org-refile haskell-mode-autoloads yaml-mode shell-here
org-recoll edmacro kmacro doc-view org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
pcomplete org-list org-faces org-entities time-date noutline outline
easy-mmode org-version ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs cal-menu calendar cal-loaddefs
pdfgrep grep pdf-occur ibuf-ext ibuffer ibuffer-loaddefs tablist
tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local find-func cedet
pdf-isearch let-alist pdf-misc imenu pdf-tools compile comint ansi-color
ring cus-edit cus-start cus-load wid-edit pdf-view bookmark
text-property-search pp jka-compr pdf-cache pdf-info tq pdf-util advice
format-spec image-mode dired dired-loaddefs exif server info package
easymenu browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/pgtk-win
pgtk-win term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face pcase macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify dynamic-setting font-render-setting cairo move-toolbar gtk
x-toolkit pgtk lcms2 multi-tty make-network-process nativecomp emacs)

Memory information:
((conses 16 2133617 264148)
 (symbols 48 55146 3)
 (strings 32 1087649 82508)
 (string-bytes 1 63791349)
 (vectors 16 393221)
 (vector-slots 8 6065362 537361)
 (floats 8 598 661)
 (intervals 56 87491 968)
 (buffers 984 90))

Message #8 received at 45657 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Justus-dev <at> Piater.name
Cc: 45657 <at> debbugs.gnu.org
Subject: Re: bug#45657: 27.1;
 Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding
Date: Mon, 04 Jan 2021 19:50:59 +0200

> From: Justus-dev <at> Piater.name
> Date: Mon, 04 Jan 2021 18:17:51 +0100
> 
> Since Emacs 27.1 and still in the current development version, non-ASCII
> characters display incorrectly if the encoding is 7bit or 8bit, at least
> in my Gnus setup with nnmaildir backend.  This holds for both text/plain
> and text/html content types.
> 
> To test this, send yourself a MIME message such as the following:
> 
> <#multipart type=mixed>
> <#part type=text/plain charset="utf-8" encoding="7bit" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#part type=text/plain charset="utf-8" encoding="8bit" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#part type=text/plain charset="utf-8" encoding="quoted-printable" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#part type=text/plain charset="utf-8" encoding="base64" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#/multipart>
> 
> The first two parts will show non-ASCII characters in \ddd form (in HTML
> parts, this would be printable but wrong characters instead), while the
> third and fourth parts will display correctly.
> 
> I have not bothered reporting this till now because there is an easy
> workaround (hit e (to edit) followed immediately by C-c C-c (to commit
> the void edit), which will reencode the message in quoted-printable),
> and because I figured I cannot be the only one who observes this
> problem...???

Isn't this a duplicate of bug#44307?

Message #11 received at 45657 <at> debbugs.gnu.org (full text, mbox):

From: Justus-dev <at> Piater.name
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Justus-dev <at> Piater.name, 45657 <at> debbugs.gnu.org
Subject: Re: bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit
 and 8bit encoding
Date: Mon, 04 Jan 2021 19:57:10 +0100

Eli Zaretskii <eliz <at> gnu.org> wrote on Mon, 04 Jan 2021 19:50:59 +0200:

> Isn't this a duplicate of bug#44307?

Yes, it is.  Sorry, my bad.  I checked only "emacs" bugs, not "gnus"
bugs.

Justus

Message #14 received at 45657 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Justus-dev <at> Piater.name
Cc: 45657 <at> debbugs.gnu.org
Subject: Re: bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit
 and 8bit encoding
Date: Mon, 04 Jan 2021 21:22:20 +0200

merge 45657 44307
thanks

> From: Justus-dev <at> Piater.name
> Cc: Justus-dev <at> Piater.name, 45657 <at> debbugs.gnu.org
> Date: Mon, 04 Jan 2021 19:57:10 +0100
> 
> Eli Zaretskii <eliz <at> gnu.org> wrote on Mon, 04 Jan 2021 19:50:59 +0200:
> 
> > Isn't this a duplicate of bug#44307?
> 
> Yes, it is.  Sorry, my bad.  I checked only "emacs" bugs, not "gnus"
> bugs.

No sweat, I merged them.

Regardless, I hope someone of the Gnus bunch looks into this real
soon, as I think we should fix this on the release branch for the
upcoming Emacs 27.2.

Message #19 received at 45657 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Justus-dev <at> Piater.name, 45657 <at> debbugs.gnu.org
Subject: Re: bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit
 and 8bit encoding
Date: Tue, 05 Jan 2021 10:31:55 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> Regardless, I hope someone of the Gnus bunch looks into this real
> soon, as I think we should fix this on the release branch for the
> upcoming Emacs 27.2.

I'd love to, as soon as somebody can give me a case to reproduce the
error, but so far all the attempts have been garbled files...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.