GNU bug report logs - #54154
29.0.50; [PATCH] `sieve-manage-getscript' fails if script contains multibyte characters

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>; Keywords: patch; dated Fri, 25 Feb 2022 09:20:01 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 28 Feb 2022 13:08:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 28 08:08:31 2022
Received: from localhost ([127.0.0.1]:60486 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nOflP-0005G1-Ky
	for submit <at> debbugs.gnu.org; Mon, 28 Feb 2022 08:08:31 -0500
Received: from quimby.gnus.org ([95.216.78.240]:36842)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1nOflA-0005FS-Il
 for 54154 <at> debbugs.gnu.org; Mon, 28 Feb 2022 08:08:30 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:Date:References:
 In-Reply-To:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=qT7RZeukXNAue7KzIfwIV1x8GtEiSFpipo6FBUtVqhg=; b=JdRJ7XxcnELrj4maFcNmFQ9bft
 Nf8ABBQi/0pF1NHDi2SSLlT7B1dXdkelWJlmflkFsZyvNTbL8MsPNsiwvODGKjJOENxavnyb8FsGD
 AxdbeUe94eo171imDTcaQx1Eiq9tsXG0GsZNDM1gcoe0DARzhwUeCofBEGSY4kzedyBc=;
Received: from [84.212.220.105] (helo=giant)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1nOfl2-00085Y-Fx; Mon, 28 Feb 2022 14:08:10 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: emacs@HIDDEN
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
In-Reply-To: <87o82rw4ml.fsf@HIDDEN> (Kai Tetzlaff's message of "Mon, 28
 Feb 2022 13:35:30 +0100")
References: <87wnhj5nbk.fsf@HIDDEN> <878rtzxhnc.fsf@HIDDEN>
 <875yp3w0pr.fsf@HIDDEN> <87ilt33pi3.fsf@HIDDEN>
 <877d9hvf8d.fsf@HIDDEN> <87o82rw4ml.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAACBj
 SFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAD1BMVEWdypd5tIRkoHEY
 JSX////pchdQAAAAAWJLR0QEj2jZUQAAAAd0SU1FB+YCHAwdMyVgHRYAAAGxSURBVDjLdZNhouMg
 CITFEwC9gOAFXsP977YDatLX3bXNj/g5DCBpJNJIWiNuTZjaXtSUiJS6sxiz5Q4vrIJtISEWlgHd
 ESVQuV+f5aLqBZjXjsE1FQZkfMxXqHy0pXn7e8EDv/8BhKUhYiOL4lWUZ6gsEek2rjq2wlyhUN0K
 6rzMvTyQsaBAIyb0SAqYQrEAKmE5AJtproIw4r4KKoB3PM4wnxFhv8wdP6gil/xSkELxKjBOE/eC
 Yha4Tq+QxlbEDUpheRrpGiyuJPd9VFb5jyEOwF8KeLPofADalHUU4A/FThf90OAUPEALoH1h5T4O
 oG0uIfEDyc/dkjJHgeExAN6rwBxQ5ey1vIa0BVaovj2aDu5xgzVwWYejpGzwew+cYBjW7PZhrycr
 WOMsrTtXnx+gmo6s3KZks95y98prhFzmyMu69vi40BaZX3W7V0nCEfdco8e+LHwY955i23xuFA/I
 488+gGT0HCC83bEKoA7MYsfh+QmQJnWrO415fYSy9blVzX7FA6jVJ1tBtuA6CnSz+3yibI+GMZTu
 8bWQFSGrf4Icqn4CPZVjPBBK57fiDzF0aGQ0w9opAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIyLTAy
 LTI4VDEyOjI5OjUwKzAwOjAw4J1uWQAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMi0wMi0yOFQxMjoy
 OTo1MCswMDowMJHA1uUAAAAASUVORK5CYII=
X-Now-Playing: Blaine L. Reininger's _Songs From The Rain Palace_: "Where
 Are You"
Date: Mon, 28 Feb 2022 14:08:06 +0100
Message-ID: <87v8wzm955.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  (Re-sending because the previous mail went to to-online.de, 
 which rejected it.) "Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes: > Ok,
 I'm attaching two patches which fix all issues I noticed. 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 54154
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

(Re-sending because the previous mail went to to-online.de, which
rejected it.)

"Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes:

> Ok, I'm attaching two patches which fix all issues I noticed.

The patches look good to me, but I haven't tried them myself, because I
don't use sieve-manage.  If somebody who does could try the patches,
that would be helpful.  Anybody?

> What I ended up with is quite a bit more than the initial attempt. Since
> these changes are non-trivial, I will need to do the copyright
> assignment. About a week ago I actually sent an email to assign@HIDDEN
> to get the process started. But I haven't received a reply. Could you
> please send me the necessary papers? I'm in Germany, so my understanding
> is that it should be possible to do this electronically?

It sometimes takes a while to get the process going -- if you don't get
a response from the copyright clerk within a couple more days, send me
an email and I'll get in touch with them to see what's up.





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 28 Feb 2022 13:07:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 28 08:07:07 2022
Received: from localhost ([127.0.0.1]:60477 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nOfk3-0005Cz-2f
	for submit <at> debbugs.gnu.org; Mon, 28 Feb 2022 08:07:07 -0500
Received: from quimby.gnus.org ([95.216.78.240]:36796)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1nOfk1-0005CQ-Op
 for 54154 <at> debbugs.gnu.org; Mon, 28 Feb 2022 08:07:06 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=sUYL8Ltm6kcIncs9acQLiIs5Xs41wlxFJhjF1giMczo=; b=NdBj0fyqkq52LGpAwpBOUFeLSK
 ztL+lDlBGrmcvBQr8DM1nrAYZ2VwVNVCnD+PfJkc7SY6duhH485EdmVJN5TIRJ7CWmw4TRyKjWj23
 dIRa9O6K4Dd0z+XfKQ1iqoBdu0G+JcOuuUMwQtOROtUAkzpnQMzjz2G/PsaGl2S1Z3sQ=;
Received: from [84.212.220.105] (helo=giant)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1nOfjt-00084n-CW; Mon, 28 Feb 2022 14:06:59 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
References: <87wnhj5nbk.fsf@HIDDEN> <878rtzxhnc.fsf@HIDDEN>
 <875yp3w0pr.fsf@HIDDEN> <87ilt33pi3.fsf@HIDDEN>
 <877d9hvf8d.fsf@HIDDEN> <87o82rw4ml.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAACBj
 SFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAD1BMVEWdypd5tIRkoHEY
 JSX////pchdQAAAAAWJLR0QEj2jZUQAAAAd0SU1FB+YCHAwdMyVgHRYAAAGxSURBVDjLdZNhouMg
 CITFEwC9gOAFXsP977YDatLX3bXNj/g5DCBpJNJIWiNuTZjaXtSUiJS6sxiz5Q4vrIJtISEWlgHd
 ESVQuV+f5aLqBZjXjsE1FQZkfMxXqHy0pXn7e8EDv/8BhKUhYiOL4lWUZ6gsEek2rjq2wlyhUN0K
 6rzMvTyQsaBAIyb0SAqYQrEAKmE5AJtproIw4r4KKoB3PM4wnxFhv8wdP6gil/xSkELxKjBOE/eC
 Yha4Tq+QxlbEDUpheRrpGiyuJPd9VFb5jyEOwF8KeLPofADalHUU4A/FThf90OAUPEALoH1h5T4O
 oG0uIfEDyc/dkjJHgeExAN6rwBxQ5ey1vIa0BVaovj2aDu5xgzVwWYejpGzwew+cYBjW7PZhrycr
 WOMsrTtXnx+gmo6s3KZks95y98prhFzmyMu69vi40BaZX3W7V0nCEfdco8e+LHwY955i23xuFA/I
 488+gGT0HCC83bEKoA7MYsfh+QmQJnWrO415fYSy9blVzX7FA6jVJ1tBtuA6CnSz+3yibI+GMZTu
 8bWQFSGrf4Icqn4CPZVjPBBK57fiDzF0aGQ0w9opAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIyLTAy
 LTI4VDEyOjI5OjUwKzAwOjAw4J1uWQAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMi0wMi0yOFQxMjoy
 OTo1MCswMDowMJHA1uUAAAAASUVORK5CYII=
X-Now-Playing: Blaine L. Reininger's _Songs From The Rain Palace_: "Where
 Are You"
Date: Mon, 28 Feb 2022 14:06:56 +0100
In-Reply-To: <87o82rw4ml.fsf@HIDDEN> (Kai Tetzlaff's message of "Mon, 28
 Feb 2022 13:35:30 +0100")
Message-ID: <87y21vm973.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  "Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes: > Ok, I'm
 attaching two patches which fix all issues I noticed. The patches look good
 to me, but I haven't tried them myself, because I don't use sieve-manage.
 If somebody who does could try the patches, that would be helpful. Anybody?
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 54154
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

"Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes:

> Ok, I'm attaching two patches which fix all issues I noticed.

The patches look good to me, but I haven't tried them myself, because I
don't use sieve-manage.  If somebody who does could try the patches,
that would be helpful.  Anybody?

> What I ended up with is quite a bit more than the initial attempt. Since
> these changes are non-trivial, I will need to do the copyright
> assignment. About a week ago I actually sent an email to assign@HIDDEN
> to get the process started. But I haven't received a reply. Could you
> please send me the necessary papers? I'm in Germany, so my understanding
> is that it should be possible to do this electronically?

It sometimes takes a while to get the process going -- if you don't get
a response from the copyright clerk within a couple more days, send me
an email and I'll get in touch with them to see what's up.





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 28 Feb 2022 12:35:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 28 07:35:44 2022
Received: from localhost ([127.0.0.1]:60398 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nOfFf-0004Hh-Mf
	for submit <at> debbugs.gnu.org; Mon, 28 Feb 2022 07:35:44 -0500
Received: from mailout11.t-online.de ([194.25.134.85]:34128)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kai.tetzlaff@HIDDEN>) id 1nOfFd-0004HW-IG
 for 54154 <at> debbugs.gnu.org; Mon, 28 Feb 2022 07:35:42 -0500
Received: from fwd86.dcpf.telekom.de (fwd86.aul.t-online.de [10.223.144.112])
 by mailout11.t-online.de (Postfix) with SMTP id 15CE418472;
 Mon, 28 Feb 2022 13:35:40 +0100 (CET)
Received: from mail.tetzco.de ([188.192.172.49]) by fwd86.t-online.de
 with (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384 encrypted)
 esmtp id 1nOfFa-4XC9mD0; Mon, 28 Feb 2022 13:35:39 +0100
Received: from moka (moka.tetzco.de [172.30.42.200])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest
 SHA256) (No client certificate requested)
 (Authenticated sender: kai@HIDDEN)
 by mail.tetzco.de (Postfix) with ESMTPSA id DDD1B6C00B7;
 Mon, 28 Feb 2022 13:35:31 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tetzco.de; s=20210624; 
 t=1646051732;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
 in-reply-to:in-reply-to:references:references;
 bh=QNXwyAVrDrx67iwZo0vfxJsfMtRmNd7AVvVnttje4iE=;
 b=tlo/IEk/TZw1GTGhi3yLO5paWt1cvQY59N19sj4jVP6zd2eekApzlzN45AgdcAVqrVC8Dr
 NJBvewMWBiIQn8mlHp3VaAoUYJwN4YPIR7k9M+xs7KjgXPojzPhqRIl31565/6qHitzrJK
 kYq/Y4GNtkl+xFbnrxv9mboQW3c84gzj5IvF3LguURjyyDv9KfJA9+xAXKcZyyr3zu43yc
 POo3vYPaUYryL+wh1brBFlk+X7ZgIF9NJvwzceH/pMP8i4nmbDMdNxAw28OqMtBkTvVhhy
 PWHoyCOanEI/RQ2hlVJwwOlaPNvcmjJVkHEP/cJWnvzKsRQXjC5MI+9M6lzII2A+2tfAef
 HbIn6hbfvZvT+t3cXj/MNoISO/BfRR8BU5gRp/YJy4XY8zffAObxo3HG2mTUGrSZwpXaP4
 dL62px9XPBy0XCK/xPNxNlcLOt8096871hJaUT5QzYkeoeak5ZRFCmb7zBLv8A2eb4HcSZ
 IDxruMpizsB5X5qBbrGCqmgWnZf+y2SYZfVf2zFtLr9CbDwAkuhjkW6yYOIRrBfXoQNaG8
 g4hIPA/IYpMxoA6aDsU0vI2Gu0sEQPLAGBxLqCJL9FmWC/j3p+vFzhleASZbVvw/s8bPgE
 xOMejvhobb/7MDSU5XixmDQxSIkXEREZYITM49t5ZXr5rW1oq13+4=
From: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
In-Reply-To: <877d9hvf8d.fsf@HIDDEN> (Lars Ingebrigtsen's message of "Sat,
 26 Feb 2022 16:07:14 +0100")
References: <87wnhj5nbk.fsf@HIDDEN> <878rtzxhnc.fsf@HIDDEN>
 <875yp3w0pr.fsf@HIDDEN> <87ilt33pi3.fsf@HIDDEN>
 <877d9hvf8d.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
Date: Mon, 28 Feb 2022 13:35:30 +0100
Message-ID: <87o82rw4ml.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Rspamd-Queue-Id: DDD1B6C00B7
X-Spamd-Result: default: False [-6.79 / 30.00]; BAYES_HAM(-3.00)[100.00%];
 NEURAL_HAM(-3.00)[-1.000];
 GENERIC_REPUTATION(-0.69)[-0.68704150986515];
 MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-diff];
 RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[];
 MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+]; RCPT_COUNT_TWO(0.00)[2];
 DKIM_SIGNED(0.00)[tetzco.de:s=20210624];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[];
 FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]
X-Rspamd-Server: rakaposhi
X-TOI-EXPURGATEID: 150726::1646051739-000116DC-48D18D04/0/0 CLEAN NORMAL
X-TOI-MSGID: c3085302-4321-4847-b496-06633083348e
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 54154
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

--=-=-=
Content-Type: text/plain

Oh - the previous version of the first patch in my last email still
contained a bug (I forgot to re-run `git format-patch` before sending
the mail).

Lars Ingebrigtsen <larsi@HIDDEN> writes:

> Kai Tetzlaff <emacs@HIDDEN> writes:
>
>> So just reverting it won't work. I will try to undo the parts relevant
>> to this issue.
>
> Sounds good.

Ok, I'm attaching two patches which fix all issues I noticed.

What I ended up with is quite a bit more than the initial attempt. Since
these changes are non-trivial, I will need to do the copyright
assignment. About a week ago I actually sent an email to assign@HIDDEN
to get the process started. But I haven't received a reply. Could you
please send me the necessary papers? I'm in Germany, so my understanding
is that it should be possible to do this electronically?


The first (and major) set of fixes are in sieve-manage.el for the issues
with multibyte characters in sieve scripts
(sieve-manage-getscript/putscript). This also adds supports for
multibyte characters in script names
(sieve-manage-listscripts/getscript/putscript/havespace/deletescript/setactive).

There is now also some handling of getscript errors reported by the
server and improved logging.


--=-=-=
Content-Type: text/x-diff
Content-Disposition: inline;
 filename=0001-Fix-mostly-multibyte-issues-in-sieve-manage.el-Bug-5.patch
Content-Description: fix multibyte issues

From 3a4ecad9f680d130fba9e792b87824e1f5e6a6eb Mon Sep 17 00:00:00 2001
From: Kai Tetzlaff <emacs@HIDDEN>
Date: Mon, 28 Feb 2022 11:08:07 +0100
Subject: [PATCH 1/2] Fix (mostly multibyte) issues in sieve-manage.el 
 (Bug#54154)

The managesieve protocol (s. RFC5804) requires support for (a sightly
restricted variant of) UTF-8 in script content and script names. This
commit fixes/improves the handling of multibyte characters.

In addition, `sieve-manage-getscript' now properly handles NO
responses from the server instead of inflooping.

There are also some logging improvements.

* lisp/net/sieve-manage.el
(sieve-manage--append-to-log):
(sieve-manage--message):
(sieve-manage--error):
(sieve-manage-encode):
(sieve-manage-decode):
(sieve-manage-no-p): New functions.
(sieve-manage-make-process-buffer): Switch process buffer to unibyte.
(sieve-manage-open-server): Add `:coding 'raw-text-unix` to
`open-network-stream' call. Use unix EOLs in order to keep matching
CRLF (aka "\r\n") intact.
(sieve-manage-send): Make sure that UTF-8 multibyte characters are
properly encoded before sending data to the server.
(sieve-manage-getscript):
(sieve-manage-putscript): Use the changes above to fix down/uploading
scripts containing UTF-8 multibyte characters.
(sieve-manage-listscripts):
(sieve-manage-havespace)
(sieve-manage-getscript)
(sieve-manage-putscript):
(sieve-manage-deletescript):
(sieve-manage-setactive): Use the changes above to fix handling of
script names which contain UTF-8 multibyte characters.
(sieve-manage-parse-string):
(sieve-manage-getscript): Add handling of server responses with type
NO. Abort `sieve-manage-getscript' and show error message in message
area.
(sieve-manage-erase):
(sieve-manage-drop-next-answer):
(sieve-manage-parse-crlf): Return erased/dropped data (instead of nil).
(sieve-sasl-auth):
(sieve-manage-getscript):
(sieve-manage-erase):
(sieve-manage-open-server):
(sieve-manage-open):
(sieve-manage-send): Improve logging.
---
 lisp/net/sieve-manage.el | 126 +++++++++++++++++++++++++++------------
 1 file changed, 87 insertions(+), 39 deletions(-)

diff --git a/lisp/net/sieve-manage.el b/lisp/net/sieve-manage.el
index 50342b9105..4a36f94431 100644
--- a/lisp/net/sieve-manage.el
+++ b/lisp/net/sieve-manage.el
@@ -167,7 +167,53 @@ sieve-manage-process
 (defvar sieve-manage-capability nil)
 
 ;; Internal utility functions
-(autoload 'mm-enable-multibyte "mm-util")
+(defun sieve-manage--append-to-log (&rest args)
+  "Append ARGS to sieve-manage log buffer.
+
+ARGS can be a string or a list of strings.
+The buffer to use for logging is specifified via
+`sieve-manage-log'. If it is nil, logging is disabled."
+  (when sieve-manage-log
+    (with-current-buffer (or (get-buffer sieve-manage-log)
+                             (with-current-buffer
+                                 (get-buffer-create sieve-manage-log)
+                               (set-buffer-multibyte nil)
+                               (buffer-disable-undo)
+                               (current-buffer)))
+      (goto-char (point-max))
+      (apply #'insert args))))
+
+(defun sieve-manage--message (format-string &rest args)
+  "Wrapper around `message' which also logs to sieve manage log.
+
+See `sieve-manage--append-to-log'."
+  (let ((ret (apply #'message
+                    (concat "sieve-manage: " format-string)
+                    args)))
+    (sieve-manage--append-to-log ret "\n")
+    ret))
+
+(defun sieve-manage--error (format-string &rest args)
+  "Wrapper around `error' which also logs to sieve manage log.
+
+See `sieve-manage--append-to-log'."
+  (let ((msg (apply #'format
+                    (concat "sieve-manage/ERROR: " format-string)
+                    args)))
+    (sieve-manage--append-to-log msg "\n")
+    (error msg)))
+
+(defun sieve-manage-encode (utf8-string)
+  "Convert UTF8-STRING to managesieve protocol octets."
+  (encode-coding-string utf8-string 'raw-text t))
+
+(defun sieve-manage-decode (octets &optional buffer)
+  "Convert managesieve protocol OCTETS to utf-8 string.
+
+If optional BUFFER is non-nil, insert decoded string into BUFFER."
+  (when octets
+    ;; eol type unix is required to preserve "\r\n"
+    (decode-coding-string octets 'utf-8-unix t buffer)))
 
 (defun sieve-manage-make-process-buffer ()
   (with-current-buffer
@@ -175,22 +221,19 @@ sieve-manage-make-process-buffer
                                    sieve-manage-server
                                    sieve-manage-port))
     (mapc #'make-local-variable sieve-manage-local-variables)
-    (mm-enable-multibyte)
+    (set-buffer-multibyte nil)
+    (setq-local after-change-functions nil)
     (buffer-disable-undo)
     (current-buffer)))
 
 (defun sieve-manage-erase (&optional p buffer)
-  (let ((buffer (or buffer (current-buffer))))
-    (and sieve-manage-log
-	 (with-current-buffer (get-buffer-create sieve-manage-log)
-	   (mm-enable-multibyte)
-	   (buffer-disable-undo)
-	   (goto-char (point-max))
-	   (insert-buffer-substring buffer (with-current-buffer buffer
-					     (point-min))
-				    (or p (with-current-buffer buffer
-					    (point-max)))))))
-  (delete-region (point-min) (or p (point-max))))
+  (with-current-buffer (or buffer (current-buffer))
+    (let* ((start (point-min))
+           (end (or p (point-max)))
+           (logdata (buffer-substring-no-properties start end)))
+      (sieve-manage--append-to-log logdata)
+      (delete-region start end)
+      logdata)))
 
 (defun sieve-manage-open-server (server port &optional stream buffer)
   "Open network connection to SERVER on PORT.
@@ -202,6 +245,8 @@ sieve-manage-open-server
                  (open-network-stream
                   "SIEVE" buffer server port
                   :type stream
+                  ;; eol type unix is required to preserve "\r\n"
+                  :coding 'raw-text-unix
                   :capability-command "CAPABILITY\r\n"
                   :end-of-command "^\\(OK\\|NO\\).*\n"
                   :success "^OK.*\n"
@@ -224,7 +269,7 @@ sieve-manage-open-server
 ;; Authenticators
 (defun sieve-sasl-auth (buffer mech)
   "Login to server using the SASL MECH method."
-  (message "sieve: Authenticating using %s..." mech)
+  (sieve-manage--message "Authenticating using %s..." mech)
   (with-current-buffer buffer
     (let* ((auth-info (auth-source-search :host sieve-manage-server
                                           :port "sieve"
@@ -275,11 +320,15 @@ sieve-sasl-auth
             (if (and (setq step (sasl-next-step client step))
                      (setq data (sasl-step-data step)))
                 ;; We got data for server but it's finished
-                (error "Server not ready for SASL data: %s" data)
+                (sieve-manage--error
+                 "Server not ready for SASL data: %s" data)
               ;; The authentication process is finished.
+              (sieve-manage--message "Logged in as %s using %s"
+                                     user-name mech)
               (throw 'done t)))
           (unless (stringp rsp)
-            (error "Server aborted SASL authentication: %s" (caddr rsp)))
+            (sieve-manage--error
+             "Server aborted SASL authentication: %s" (caddr rsp)))
           (sasl-step-set-data step (base64-decode-string rsp))
           (setq step (sasl-next-step client step))
           (sieve-manage-send
@@ -288,8 +337,7 @@ sieve-sasl-auth
                        (base64-encode-string (sasl-step-data step)
                                              'no-line-break)
                        "\"")
-             ""))))
-      (message "sieve: Login using %s...done" mech))))
+             "")))))))
 
 (defun sieve-manage-cram-md5-p (buffer)
   (sieve-manage-capability "SASL" "CRAM-MD5" buffer))
@@ -353,7 +401,7 @@ sieve-manage-open
                                   sieve-manage-default-stream)
           sieve-manage-auth   (or auth
                                   sieve-manage-auth))
-    (message "sieve: Connecting to %s..." sieve-manage-server)
+    (sieve-manage--message "Connecting to %s..." sieve-manage-server)
     (sieve-manage-open-server sieve-manage-server
                               sieve-manage-port
                               sieve-manage-stream
@@ -368,7 +416,8 @@ sieve-manage-open
             (setq sieve-manage-auth auth)
             (cl-return)))
         (unless sieve-manage-auth
-          (error "Couldn't figure out authenticator for server")))
+          (sieve-manage--error
+           "Couldn't figure out authenticator for server")))
       (sieve-manage-erase)
       (current-buffer))))
 
@@ -433,11 +482,7 @@ sieve-manage-havespace
 (defun sieve-manage-putscript (name content &optional buffer)
   (with-current-buffer (or buffer (current-buffer))
     (sieve-manage-send (format "PUTSCRIPT \"%s\" {%d+}%s%s" name
-                               ;; Here we assume that the coding-system will
-                               ;; replace each char with a single byte.
-                               ;; This is always the case if `content' is
-                               ;; a unibyte string.
-			       (length content)
+			       (length (sieve-manage-encode content))
 			       sieve-manage-client-eol content))
     (sieve-manage-parse-okno)))
 
@@ -449,11 +494,10 @@ sieve-manage-deletescript
 (defun sieve-manage-getscript (name output-buffer &optional buffer)
   (with-current-buffer (or buffer (current-buffer))
     (sieve-manage-send (format "GETSCRIPT \"%s\"" name))
-    (let ((script (sieve-manage-parse-string)))
-      (sieve-manage-parse-crlf)
-      (with-current-buffer output-buffer
-	(insert script))
-      (sieve-manage-parse-okno))))
+    (sieve-manage-decode (sieve-manage-parse-string)
+                         output-buffer)
+    (sieve-manage-parse-crlf)
+    (sieve-manage-parse-okno)))
 
 (defun sieve-manage-setactive (name &optional buffer)
   (with-current-buffer (or buffer (current-buffer))
@@ -478,6 +522,9 @@ sieve-manage-drop-next-answer
 (defun sieve-manage-ok-p (rsp)
   (string= (downcase (or (car-safe rsp) "")) "ok"))
 
+(defun sieve-manage-no-p (rsp)
+  (string= (downcase (or (car-safe rsp) "")) "no"))
+
 (defun sieve-manage-is-okno ()
   (when (looking-at (concat
 		     "^\\(OK\\|NO\\)\\( (\\([^)]+\\))\\)?\\( \\(.*\\)\\)?"
@@ -528,7 +575,11 @@ sieve-manage-parse-string
     (while (null rsp)
       (accept-process-output (get-buffer-process (current-buffer)) 1)
       (goto-char (point-min))
-      (setq rsp (sieve-manage-is-string)))
+      (unless (setq rsp (sieve-manage-is-string))
+        (when (sieve-manage-no-p (sieve-manage-is-okno))
+          ;; simple `error' is enough since `sieve-manage-erase'
+          ;; already adds the server response to the log
+          (error (sieve-manage-erase)))))
     (sieve-manage-erase (point))
     rsp))
 
@@ -540,7 +591,8 @@ sieve-manage-parse-listscripts
   (let (tmp rsp data)
     (while (null rsp)
       (while (null (or (setq rsp (sieve-manage-is-okno))
-		       (setq tmp (sieve-manage-is-string))))
+                       (setq tmp (sieve-manage-decode
+                                  (sieve-manage-is-string)))))
 	(accept-process-output (get-buffer-process (current-buffer)) 1)
 	(goto-char (point-min)))
       (when tmp
@@ -559,13 +611,9 @@ sieve-manage-parse-listscripts
       rsp)))
 
 (defun sieve-manage-send (cmdstr)
-  (setq cmdstr (concat cmdstr sieve-manage-client-eol))
-  (and sieve-manage-log
-       (with-current-buffer (get-buffer-create sieve-manage-log)
-	 (mm-enable-multibyte)
-	 (buffer-disable-undo)
-	 (goto-char (point-max))
-	 (insert cmdstr)))
+  (setq cmdstr (sieve-manage-encode
+                (concat cmdstr sieve-manage-client-eol)))
+  (sieve-manage--append-to-log cmdstr)
   (process-send-string sieve-manage-process cmdstr))
 
 (provide 'sieve-manage)
-- 
2.34.1


--=-=-=
Content-Type: text/plain


Both, the (internal) process/protocol buffer and the log buffer are now
unibyte. The conversion to multibyte UTF-8 is only done for user visible
(UI) buffers.

To properly handle the protocol line termination (CRLF), I added
`:coding 'raw-text-unix` (with explicit unix EOL convention) to the
`open-network-stream' call (also in the new `manage-sieve-encode'
function. This was needed to allow keep the various (looking-at
"...\r\n" ...) calls working. This is something which still feels a bit
weird, but I haven't found another way to do it. I did some tests with
(setq-default buffer-file-coding-system 'utf-8-unix/'utf-8-dos) which
did not show any issues.

I would also add some ERT tests, probably in a separate commit?


In addition, I found that `sieve-manage-quit' in sieve.el had the
tendency to kill unrelated buffers in case of errors during earlier
steps. For this, I created a sepate patch:


--=-=-=
Content-Type: text/x-diff
Content-Disposition: inline;
 filename=0002-Improve-robustnes-of-sieve-manage-quit-in-case-of-er.patch
Content-Description: avoid killing unrelated buffers

From 83ab45907c7b528ae4db98f33415e05e679c312e Mon Sep 17 00:00:00 2001
From: Kai Tetzlaff <emacs@HIDDEN>
Date: Mon, 28 Feb 2022 11:33:56 +0100
Subject: [PATCH 2/2] Improve robustnes of `sieve-manage-quit' in case of
 errors

* lisp/net/sieve.el (sieve-manage-quit): Avoid killing buffers it's
not supposed to touch.
---
 lisp/net/sieve.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/net/sieve.el b/lisp/net/sieve.el
index 630ea04070..5680526389 100644
--- a/lisp/net/sieve.el
+++ b/lisp/net/sieve.el
@@ -154,7 +154,8 @@ sieve-manage-quit
   (interactive)
   (sieve-manage-close sieve-manage-buffer)
   (kill-buffer sieve-manage-buffer)
-  (kill-buffer (current-buffer)))
+  (when-let ((buffer (get-buffer sieve-buffer)))
+    (kill-buffer buffer)))
 
 (defun sieve-bury-buffer ()
   "Bury the Manage Sieve buffer without closing the connection."
-- 
2.34.1


--=-=-=--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 26 Feb 2022 15:07:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Feb 26 10:07:30 2022
Received: from localhost ([127.0.0.1]:56086 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nNyfS-000383-0U
	for submit <at> debbugs.gnu.org; Sat, 26 Feb 2022 10:07:30 -0500
Received: from quimby.gnus.org ([95.216.78.240]:42110)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1nNyfP-00037r-Tt
 for 54154 <at> debbugs.gnu.org; Sat, 26 Feb 2022 10:07:28 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=9BTDYko715Xwl68dwKfdSQ3HpveO7vvWdgCShAGr7ko=; b=I+/9NrNxdOkXjtiU0nfy9tXtQH
 bBnAewQ466P3xJl+hyqyH8KCJbjSkxJebzv0XYDwS1n/VWhKGFDrlBFnTacXyYJLsMCr6lshOXln2
 hvqO7KlRQp+CvewdqRo8l9HRlJElOUxFigV19z96ZPC4YmCaoZAaXSxbiheqHcJ+oBtc=;
Received: from [84.212.220.105] (helo=giant)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1nNyfH-0006LE-9Z; Sat, 26 Feb 2022 16:07:22 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Kai Tetzlaff <kai@HIDDEN>
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
References: <87wnhj5nbk.fsf@HIDDEN> <878rtzxhnc.fsf@HIDDEN>
 <875yp3w0pr.fsf@HIDDEN> <87ilt33pi3.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAACBj
 SFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAElBMVEX8+vnY08+tpaGE
 e3ZANzH////1ngwYAAAAAWJLR0QF+G/pxwAAAAd0SU1FB+YCGg8DJFXfRosAAAGwSURBVDjLtZPb
 gYQgDEVBGwCcAgSmADUUMEPSf02bhIfu/i8fo+SY5N7AGPMvy3v3Z7/rI3qbHsSHDB95aiz4CWJI
 EhtV7MzZ/+xnytHArLD8zng0Ha+1VwoJrugeKSSbkAPp2qNZegp9jdmIjgZOQjdANW+O1LcCxELH
 yDheEuKfApL+jrGDYvVbAsLCGpel5dB3+/DXeCVZzkS3sgYBhCCSQNYVLTip7AQwKk0T4Wk2bYlO
 AQ1Qk7NmJU1pX3I9aXERRGNTFnUSvmcLqZy+DVv9TrD5grEb1BFNgECHvcF3Dp1tHsE1gM+UcK4u
 u5lB9XE7DAwg/o77BK0A9SG1cNYyC/oOnv3t3oCfgNy4PWtzMidV9bYaAecvABA1zib35ghKH2Qf
 zFYDsLAJgLqujKqFIwpyDf2OvgkST3+c3QtTB4VO729QsHtceBPtBCwNukVWHsNutG3zIaqsddmn
 GJoPkcHnc/8XfBildqvzCdF5D5Bihg6QL/9KkOUVxGwazcvHvLDU19By3WNfWKXxKfU53KBcF3e3
 fmv7+zwwI3e+nGgseGeom1ILD8OHUugHUE2KyKSnXokAAAAldEVYdGRhdGU6Y3JlYXRlADIwMjIt
 MDItMjZUMTU6MDM6MzYrMDA6MDBXnYsfAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDIyLTAyLTI2VDE1
 OjAzOjM2KzAwOjAwJsAzowAAAABJRU5ErkJggg==
X-Now-Playing: Laraaji's _Moon Piano_: "Through This Moment"
Date: Sat, 26 Feb 2022 16:07:14 +0100
In-Reply-To: <87ilt33pi3.fsf@HIDDEN> (Kai Tetzlaff's message of "Fri, 25
 Feb 2022 17:00:36 +0100")
Message-ID: <877d9hvf8d.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  Kai Tetzlaff <kai@HIDDEN> writes: > So just reverting
 it won't work. I will try to undo the parts relevant > to this issue. Sounds
 good. Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 54154
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Kai Tetzlaff <kai@HIDDEN> writes:

> So just reverting it won't work. I will try to undo the parts relevant
> to this issue.

Sounds good.

> For clarification: The original code before Alberts change was using
> this macro (which seemingly contains an error in the doc string):
>
> (defmacro sieve-manage-disable-multibyte ()
>   "Enable multibyte in the current buffer."
>   (unless (featurep 'xemacs)
>     '(set-buffer-multibyte nil)))
>
> to *disable* multibyte handling in the response/protocol buffer. If
> using `set-buffer-multibyte' is not the right thing, what should be used
> instead?

Using (set-buffer-multibyte nil) is the right thing to do to make a
buffer unibyte -- but usually only when the buffer is empty.  There's
been some discussion about making `set-buffer-multibyte' signal an error
if used in a non-empty buffer, because in 99.7% of the cases where
people do that, it's the wrong thing to do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 25 Feb 2022 17:26:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 25 12:26:55 2022
Received: from localhost ([127.0.0.1]:53630 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nNeMn-0007U7-HO
	for submit <at> debbugs.gnu.org; Fri, 25 Feb 2022 12:26:54 -0500
Received: from mout-p-102.mailbox.org ([80.241.56.152]:45712)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kai@HIDDEN>) id 1nNd1g-0002k1-46
 for 54154 <at> debbugs.gnu.org; Fri, 25 Feb 2022 11:01:01 -0500
Received: from smtp102.mailbox.org (smtp102.mailbox.org [80.241.60.233])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest
 SHA256) (No client certificate requested)
 by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4K4vdK5N1Mz9sV4;
 Fri, 25 Feb 2022 17:00:53 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tetzlaff.eu; s=20210624;
 t=1645804839;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
 in-reply-to:in-reply-to:references:references;
 bh=MFS0aNH9Y2gV4HS794o0V2/dUOOHjpqxcDPZmgIli+c=;
 b=UyzA2i4c3QWsqHKtduWsiarraWWw2b/lz34MmSCzkVAEC0Mj/myG3syER59QKd+9n+1W1z
 zaDmAXBhWKZcUQOxb3ClUlxOWz5DnETJOskDCf+aFe+BpvO0XC5KvbcIjMfvXwi0DohOdA
 apMLYTSLdDQEr/KGjO4Ie26ZkoGHSjxBahr8AF59v2KHZ16rxPve233tMg/W5xI/fhSrvG
 W7DwFJDSieBNMu7IOI0sGah898vdyHDTB7yujjYedMFffIidA9LTt3z8v4cLPK9m25M/eb
 O7jtek7/+UnEmYGW8RrSQdpxh/sBHTPVxUSrTsX1OFo1MHzf0WwDnNjFkpQjVQGIuSyPOw
 bRWkBnbLn9hqlUW6pP7JIrWvzUHJPNyYx1yTUiveo8+Lur/33AOEVELXpfwVawooUop6N8
 yQ1wkBvN9unxqKvb/mw5wQRuzSbbMiOcGkwEsFBen67wM5nzwCxqy9nHrwGf7Nu2JahOPd
 IJTgYqVngXXJPWq+gf6Byiwd+XPt76H+Jj62o+UDeQIhh32m9DcWZ+zJ9FMxiYj8U5Y5aL
 oTRKGPnXgyGWNaYl9DQNfp9yRedhUNbhB0jKtjq2rH6MKwi0DNbljq/LrzMKdCWySFBkqd
 yASDDNSmRaZ+TbemkwXBj3UO1LNKa5zd73G7QuHgZ1OuV8pgZbeB0=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tetzlaff.eu; s=MBO0001;
 t=1645804852;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
 in-reply-to:in-reply-to:references:references;
 bh=MFS0aNH9Y2gV4HS794o0V2/dUOOHjpqxcDPZmgIli+c=;
 b=il7c/x5sqvcl0hxJsj90tFWhv2knOcAx5399E7Nn4ze/n+4ikwJ/eOp8sXsQgRQX5K+Ee3
 fVl/QV5G3eLn79CkwQVwr2QkOy7pOcQFJn5Z1bfBKnwMQn9xrSpYCuSf2r8ScBCfIVSL2p
 PLJ7x/Hre9MS2aRMO1edjh/NgN0T+n0aCQFOwV0x36UAGkEKruQ+FHNdHK2QiHm/bkR1bj
 OUm6yExUnBwPTcpk8euW+reoLBuVIfCivlyV5Is+bUaLkC3k2+slAX8A1JjmXnsI2NWFXj
 kosDAz7oV6nqgLG2HdCIhQCLsUqg0JVf04YAPvg8hDV5lkGHHLGiHS0m9ONZCQ==
From: Kai Tetzlaff <kai@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
References: <87wnhj5nbk.fsf@HIDDEN> <878rtzxhnc.fsf@HIDDEN>
 <875yp3w0pr.fsf@HIDDEN>
Date: Fri, 25 Feb 2022 17:00:36 +0100
In-Reply-To: <875yp3w0pr.fsf@HIDDEN> (Lars Ingebrigtsen's message of "Fri,
 25 Feb 2022 14:10:56 +0100")
Message-ID: <87ilt33pi3.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Rspamd-Queue-Id: 1D20B6C00B7
X-Spamd-Result: default: False [-6.79 / 30.00]; BAYES_HAM(-3.00)[99.99%];
 NEURAL_HAM(-3.00)[-1.000];
 GENERIC_REPUTATION(-0.69)[-0.68619817412468];
 MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[];
 ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+];
 DKIM_SIGNED(0.00)[tetzlaff.eu:s=20210624];
 RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[];
 RCPT_COUNT_TWO(0.00)[2]
X-Rspamd-Server: rakaposhi
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 54154
X-Mailman-Approved-At: Fri, 25 Feb 2022 12:26:50 -0500
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Lars Ingebrigtsen <larsi@HIDDEN> writes:

> The mail bounced with:
>
>   kai.tetzlaff@HIDDEN
>     host mx03.t-online.de [194.25.134.73]
>     SMTP error from remote mail server after initial connection:
>     554 IP=95.216.78.240 - A problem occurred. (Ask your postmaster for help or to contact tosa@HIDDEN to clarify.)

Sorry, not sure what is happening there. Using a different From:
address, now (hopefully - using the t-online address was accidental
anyway).

>>>    (with-current-buffer (or buffer (current-buffer))
>>>      (sieve-manage-send (format "GETSCRIPT \"%s\"" name))
>>> +    (set-buffer-multibyte nil)
>>>      (let ((script (sieve-manage-parse-string)))
>>> +      (set-buffer-multibyte t)
>>
>> Changing multibyteness in a buffer like this is (virtually) never the
>> right thing to do -- it usually leads to obscure breakages.
>>
>>> In general, it is also not clear to me why the response (or process)
>>> buffer needs to be multibyte enabled at all as it should only be used
>>> for the line/byte oriented protocol data. But the commit message of
>>> 8e16fb987df9b which introduced the multibyte handling states:
>>>
>>>     commit 8e16fb987df9b80b8328e9dbf80351a5f9d85bbb
>>>     Author: Albert Krewinkel <krewinkel@HIDDEN>
>>>     Date:   2013-06-11 07:32:25 +0000
>>>     ...
>>>         * Enable Multibyte for SieveManage buffers: The parser won't properly
>>>           handle umlauts and line endings unless multibyte is turned on in the
>>>           process buffer.
>>>     ...
>>>
>>> so this was obviously done on purpose. I contacted Albert about this but
>>> he couldn't remember the details (it's been nearly 10 years).
>>
>> I don't see why this buffer should be multibyte, either.  The
>> communication with the server is done using bytes, not characters.  When
>> we need to have characters, we should decode the data and put it in a
>> multibyte buffer.
>>
>> So can you try to back out that commit and see whether it fixes the
>> problem instead?

Most of the referenced commit was about changes related to STARTTLS
handling. Here's the full commit message:

    lisp/gnus/sievel-manage.el: fully support STARTTLS, fix bit rot
    
    * Make sieve-manage-open work with STARTTLS: shorten stream managing
      functions by using open-protocol-stream to do most of the work.
      Has the nice benefit of enabling STARTTLS.

    * Remove unneeded functions and options: the following functions and
      options are neither in the API, nor called by any other function,
      so they are deleted:
      - sieve-manage-network-p
      - sieve-manage-network-open
      - sieve-manage-starttls-p
      - sieve-manage-starttls-open
      - sieve-manage-forward
      - sieve-manage-streams
      - sieve-manage-stream-alist

      The options could not be applied in a meaningful way anymore; they
      didn't happen to have much effect before.

    * Cosmetic changes and code clean-up

    * Enable Multibyte for SieveManage buffers: The parser won't properly
      handle umlauts and line endings unless multibyte is turned on in the
      process buffer.

    * Wait for capabilities after STARTTLS: following RFC5804, the server
      sends new capabilities after successfully establishing a TLS
      connection with the client.  The client should update the cached list
      of capabilities, but we just ignore the answer for now.

So just reverting it won't work. I will try to undo the parts relevant
to this issue.

For clarification: The original code before Alberts change was using
this macro (which seemingly contains an error in the doc string):

(defmacro sieve-manage-disable-multibyte ()
  "Enable multibyte in the current buffer."
  (unless (featurep 'xemacs)
    '(set-buffer-multibyte nil)))

to *disable* multibyte handling in the response/protocol buffer. If
using `set-buffer-multibyte' is not the right thing, what should be used
instead?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 25 Feb 2022 13:11:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 25 08:11:07 2022
Received: from localhost ([127.0.0.1]:51547 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nNaNH-00060P-JA
	for submit <at> debbugs.gnu.org; Fri, 25 Feb 2022 08:11:07 -0500
Received: from smtp.domeneshop.no ([194.63.252.55]:35145)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eyesore1@HIDDEN>) id 1nNaNE-0005zm-0O
 for 54154 <at> debbugs.gnu.org; Fri, 25 Feb 2022 08:11:06 -0500
Received: from [84.212.220.105] (port=34652 helo=giant)
 by smtp.domeneshop.no with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <eyesore1@HIDDEN>)
 id 1nNaN7-0008W1-HY; Fri, 25 Feb 2022 14:10:57 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
In-Reply-To: <878rtzxhnc.fsf@HIDDEN> (Lars Ingebrigtsen's message of "Fri,
 25 Feb 2022 13:19:51 +0100")
References: <87wnhj5nbk.fsf@HIDDEN> <878rtzxhnc.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAACBj
 SFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAGFBMVEX27dH07NTb0rSr
 pI7m2rjGuaOHiHr///+41XY3AAAAAWJLR0QHFmGI6wAAAAd0SU1FB+YCGQs2C0mDvlYAAAG6SURB
 VDjLbZJBlpswDECl1N2bDN0TYbqeGZesjWv2FEIvwJvc/wgjAU4MQS/hgb/1ZcsGyF4C5QHP122A
 /C7zq96Dk7WfaRZG8GZN+e9IVX5g/nkAdKt/XQYUg95m6J/Dx4BRHiFgpn/fncUDFd7GvH6tgVnb
 6B/dY5lnyhvZF4B+q6lGBA0Yfecmz+SrsgVqBFkYnrO8WVWY8eRMA8g4nKkiMt5wcR5myg+2shmX
 vrJYQ8MZMIdgXuai8oEMXSCGhuNAAOdAHZAQHCwA5R8FKhgXwWMqGwKHiiBOdqKR2LkdhJWsGRjl
 oLjAmiK6UJFM4U8XvAC1cXnHCkUUixTJkgRTotrk+RmoF6BmoBQLFDyYrwq1qP53YNK988IZmIFv
 LOx36YNp7dXaXYm5SGknazvZ4BaU9nqfbAe7DqswXO/3qUt2v/IwTFc7OWX2RXxZT8OX4xaB21Xv
 rTcwZ7htEV84M59Jf9uC0POoe70m3BHDR+ALaN+fdDlcIhp7gr9/TmSedTxVIweDgVvpisd14dlM
 qpHAvqclTlTNGRWDOl1wywX6ik0MbNouGut6JBJVfUsyTr14GHj6BnxrY2eh47n+AAAAJXRFWHRk
 YXRlOmNyZWF0ZQAyMDIyLTAyLTI1VDExOjU0OjExKzAwOjAwquXbswAAACV0RVh0ZGF0ZTptb2Rp
 ZnkAMjAyMi0wMi0yNVQxMTo1NDoxMSswMDowMNu4Yw8AAAAASUVORK5CYII=
X-Now-Playing: Patrik Torsson's =?utf-8?Q?=5FKolv=C3=A4teserenader=5F=3A?=
 "Nattglimmer"
Date: Fri, 25 Feb 2022 14:10:56 +0100
Message-ID: <875yp3w0pr.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 1.2 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: The mail bounced with: kai.tetzlaff@HIDDEN host
 mx03.t-online.de
 [194.25.134.73] SMTP error from remote mail server after initial connection:
 554 IP=95.216.78.240 - A problem occurred. (Ask your postmaster for help
 or [...] 
 Content analysis details:   (1.2 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 0.0 RCVD_IN_MSPIKE_H5      RBL: Excellent reputation (+5)
 [194.63.252.55 listed in wl.mailspike.net]
 0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
 mail domains are different
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 1.0 SPF_SOFTFAIL           SPF: sender does not match SPF record (softfail)
 0.0 RCVD_IN_MSPIKE_WL      Mailspike good senders
 -0.0 T_SCC_BODY_TEXT_LINE   No description available.
X-Debbugs-Envelope-To: 54154
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.2 (/)

The mail bounced with:

  kai.tetzlaff@HIDDEN
    host mx03.t-online.de [194.25.134.73]
    SMTP error from remote mail server after initial connection:
    554 IP=95.216.78.240 - A problem occurred. (Ask your postmaster for help or to contact tosa@HIDDEN to clarify.)

Trying to send via a different SMTP server...

Lars Ingebrigtsen <larsi@HIDDEN> writes:

> "Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes:
>
>>    (with-current-buffer (or buffer (current-buffer))
>>      (sieve-manage-send (format "GETSCRIPT \"%s\"" name))
>> +    (set-buffer-multibyte nil)
>>      (let ((script (sieve-manage-parse-string)))
>> +      (set-buffer-multibyte t)
>
> Changing multibyteness in a buffer like this is (virtually) never the
> right thing to do -- it usually leads to obscure breakages.
>
>> In general, it is also not clear to me why the response (or process)
>> buffer needs to be multibyte enabled at all as it should only be used
>> for the line/byte oriented protocol data. But the commit message of
>> 8e16fb987df9b which introduced the multibyte handling states:
>>
>>     commit 8e16fb987df9b80b8328e9dbf80351a5f9d85bbb
>>     Author: Albert Krewinkel <krewinkel@HIDDEN>
>>     Date:   2013-06-11 07:32:25 +0000
>>     ...
>>         * Enable Multibyte for SieveManage buffers: The parser won't properly
>>           handle umlauts and line endings unless multibyte is turned on in the
>>           process buffer.
>>     ...
>>
>> so this was obviously done on purpose. I contacted Albert about this but
>> he couldn't remember the details (it's been nearly 10 years).
>
> I don't see why this buffer should be multibyte, either.  The
> communication with the server is done using bytes, not characters.  When
> we need to have characters, we should decode the data and put it in a
> multibyte buffer.
>
> So can you try to back out that commit and see whether it fixes the
> problem instead?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at 54154 <at> debbugs.gnu.org:


Received: (at 54154) by debbugs.gnu.org; 25 Feb 2022 12:20:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 25 07:20:08 2022
Received: from localhost ([127.0.0.1]:51488 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nNZZw-0002L8-Dy
	for submit <at> debbugs.gnu.org; Fri, 25 Feb 2022 07:20:08 -0500
Received: from quimby.gnus.org ([95.216.78.240]:57796)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1nNZZq-0002KS-F6
 for 54154 <at> debbugs.gnu.org; Fri, 25 Feb 2022 07:20:07 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=sFFuLKAnldBhuLGOgN5GT5SW0WnftzEVR5vYyOAZqWo=; b=nJ0/bdPIvPdjh4MD0S6x5NTJ/5
 LX9uEGzvXz1Kn1E6GJ8M28Jhme9OcIZf9ihQSFI4FHAvJvT0KIviOTaG4RTqq7/tBSbCnSEhg0Aml
 g5/4zPPVOh0zo+wqoBKrHk2E5I+DfmT4VsZl7c1PtKQwxVLBOs4FGKJeCWe2q/ONfarI=;
Received: from [84.212.220.105] (helo=giant)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1nNZZf-0001QT-Rb; Fri, 25 Feb 2022 13:19:56 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>
Subject: Re: bug#54154: 29.0.50; [PATCH] `sieve-manage-getscript' fails if
 script contains multibyte characters
References: <87wnhj5nbk.fsf@HIDDEN>
X-Now-Playing: Patrik Torsson's =?utf-8?Q?=5FKolv=C3=A4teserenader=5F=3A?=
 =?utf-8?Q?_=22Avm=C3=B6nstring=22?=
Date: Fri, 25 Feb 2022 13:19:51 +0100
In-Reply-To: <87wnhj5nbk.fsf@HIDDEN> (Kai Tetzlaff's message of "Fri, 25
 Feb 2022 10:04:47 +0100")
Message-ID: <878rtzxhnc.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes: >
 (with-current-buffer
 (or buffer (current-buffer)) > (sieve-manage-send (format "GETSCRIPT \"%s\""
 name)) > + (set-buffer-multibyte nil) > (let ((script
 (sieve-manage-parse-string))) > + (set-buffer- [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 54154
Cc: 54154 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

"Kai Tetzlaff" <kai.tetzlaff@HIDDEN> writes:

>    (with-current-buffer (or buffer (current-buffer))
>      (sieve-manage-send (format "GETSCRIPT \"%s\"" name))
> +    (set-buffer-multibyte nil)
>      (let ((script (sieve-manage-parse-string)))
> +      (set-buffer-multibyte t)

Changing multibyteness in a buffer like this is (virtually) never the
right thing to do -- it usually leads to obscure breakages.

> In general, it is also not clear to me why the response (or process)
> buffer needs to be multibyte enabled at all as it should only be used
> for the line/byte oriented protocol data. But the commit message of
> 8e16fb987df9b which introduced the multibyte handling states:
>
>     commit 8e16fb987df9b80b8328e9dbf80351a5f9d85bbb
>     Author: Albert Krewinkel <krewinkel@HIDDEN>
>     Date:   2013-06-11 07:32:25 +0000
>     ...
>         * Enable Multibyte for SieveManage buffers: The parser won't properly
>           handle umlauts and line endings unless multibyte is turned on in the
>           process buffer.
>     ...
>
> so this was obviously done on purpose. I contacted Albert about this but
> he couldn't remember the details (it's been nearly 10 years).

I don't see why this buffer should be multibyte, either.  The
communication with the server is done using bytes, not characters.  When
we need to have characters, we should decode the data and put it in a
multibyte buffer.

So can you try to back out that commit and see whether it fixes the
problem instead?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 25 Feb 2022 09:19:38 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 25 04:19:38 2022
Received: from localhost ([127.0.0.1]:51270 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nNWlF-0005PB-LS
	for submit <at> debbugs.gnu.org; Fri, 25 Feb 2022 04:19:38 -0500
Received: from lists.gnu.org ([209.51.188.17]:33064)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kai.tetzlaff@HIDDEN>) id 1nNWXC-00050h-Pn
 for submit <at> debbugs.gnu.org; Fri, 25 Feb 2022 04:05:07 -0500
Received: from eggs.gnu.org ([209.51.188.92]:59570)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <kai.tetzlaff@HIDDEN>)
 id 1nNWXC-00012j-1T
 for bug-gnu-emacs@HIDDEN; Fri, 25 Feb 2022 04:05:06 -0500
Received: from mailout06.t-online.de ([194.25.134.19]:59334)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <kai.tetzlaff@HIDDEN>)
 id 1nNWX9-00043g-5D
 for bug-gnu-emacs@HIDDEN; Fri, 25 Feb 2022 04:05:05 -0500
Received: from fwd89.dcpf.telekom.de (fwd89.aul.t-online.de [10.223.144.115])
 by mailout06.t-online.de (Postfix) with SMTP id 95FB6208E1
 for <bug-gnu-emacs@HIDDEN>; Fri, 25 Feb 2022 10:04:53 +0100 (CET)
Received: from mail.tetzco.de ([188.192.172.49]) by fwd89.t-online.de
 with (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384 encrypted)
 esmtp id 1nNWWz-3QF0Ph0; Fri, 25 Feb 2022 10:04:53 +0100
Received: from moka (moka.tetzco.de [172.30.42.200])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest
 SHA256) (No client certificate requested)
 (Authenticated sender: kai@HIDDEN)
 by mail.tetzco.de (Postfix) with ESMTPSA id D219A6C00B7
 for <bug-gnu-emacs@HIDDEN>; Fri, 25 Feb 2022 10:04:48 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tetzco.de; s=20210624; 
 t=1645779888;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:mime-version:mime-version:content-type:content-type;
 bh=PwDgaHjyL8rA/3fBFfSZbaZYA0yGb7cVYkFEX3BdMa4=;
 b=j+Qm8m1edEiKhn1n/ay7M5dIMFHshuIwGvM/2fxAhyMjQ5CM8xcthdGduNtmQv+TshT0Y3
 RR1xFqRuW4FbiKd7vA1DMScovItq3rPANK2d4ySvSu87zvzXyDxfZMT2vuizE+voekVHFw
 KrkvxbNXIWN/XkpScqKTgTU0E16Kv/kU/LeuK/GJ9bktCcfxb4TBsV52cmNgr3TDzqHWVy
 NjaYVAWw8NxdEdm433kv5fx8lLi7AMGLYGn/uIhyHRzOQe5Odf2bEblBaWADg20DP2OaWb
 37CEs9j845ckkUVJhVmJWgJxqHX1TUQs9d464mUAyOrPbEvMaHY4EYAIY91Hwd+CtHACTf
 HJpQNG7ArecXvJ8fJl/S+hLV3bIGz3Z/nB0eA62B3iLZL8jucTnlUbvq8q/FwK1EsRgqTt
 1v++50Zg7pJKdOTjW91K45U70Kg4jXmaGt1sLnudL56ioHUl30W2qqAk2/htrjKhLXnOoF
 Owcnrj/8SCykY+P3Zxy/nSTl8Dkbv8Hc+DOtJCO8UCpIUcLGFDE0yNsRPWl2yiuI91RsGh
 krbPh8PIwu8ZbhsK0W8Nng/X1bLTJqUzl+Ko+XCcmQGRoLg5H3uzILqHEequ/rIBCkm7eb
 EMjWAKqeR9JXhOI2Y0UXT3OCMkL6HVDn8Ed6/+r5r7OZsudlLnSDI=
From: "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 29.0.50; [PATCH] `sieve-manage-getscript' fails if script contains
 multibyte characters
Date: Fri, 25 Feb 2022 10:04:47 +0100
Message-ID: <87wnhj5nbk.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Rspamd-Queue-Id: D219A6C00B7
X-Spamd-Result: default: False [-6.69 / 30.00]; BAYES_HAM(-3.00)[100.00%];
 NEURAL_HAM(-3.00)[-1.000];
 GENERIC_REPUTATION(-0.69)[-0.68619248898889];
 MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-diff];
 MIME_UNKNOWN(0.10)[application/emacs-lisp];
 MIME_TRACE(0.00)[0:+,1:+,2:~,3:+,4:+,5:+];
 RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[];
 MID_RHS_MATCH_FROM(0.00)[];
 DKIM_SIGNED(0.00)[tetzco.de:s=20210624]; TO_DN_NONE(0.00)[];
 RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]
X-Rspamd-Server: rakaposhi
X-TOI-EXPURGATEID: 150726::1645779893-0000F5B3-056C70B4/0/0 CLEAN NORMAL
X-TOI-MSGID: 8bbb26c9-1b3f-4aef-bf78-94f83fce10fb
Received-SPF: none client-ip=194.25.134.19;
 envelope-from=kai.tetzlaff@HIDDEN; helo=mailout06.t-online.de
X-Spam_score_int: -6
X-Spam_score: -0.7
X-Spam_bar: /
X-Spam_report: (-0.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1,
 DKIM_SIGNED=0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001,
 RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001,
 TONLINE_FAKE_DKIM=1,
 T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Fri, 25 Feb 2022 04:19:36 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

--=-=-=
Content-Type: text/plain

The sieve-manage package uses the managesieve protocol (s.
https://datatracker.ietf.org/doc/html/draft-ietf-sieve-managesieve) to
communicate with a sieve server process.

When the sieve-manage client retrieves a script from the server it uses
the `sieve-manage-getscript' function to send command `GETSCRIPT
"<scriptname>"<CRLF>` to the server and tries to parse the response.

If the downloaded sieve script contains multibyte characters the attempt
to parse the response results in an infloop (in
`sieve-manage-parse-okno').

To reproduce, save the following code


--=-=-=
Content-Type: application/emacs-lisp; charset=utf-8
Content-Disposition: inline;
 filename=sieve-manage-getscript-minimal-example.el
Content-Transfer-Encoding: quoted-printable
Content-Description: minimal example

(require 'sieve-manage)
(require 'cl) ; for flet below
(let* ((script-name "test.sieve")
       ;; variables `sieve-manage-server' and `sieve-manage-port' are
       ;; used in `sieve-manage-make-process-buffer'
       (sieve-manage-server)
       (sieve-manage-port "sieve")
       (sieve-buffer (sieve-manage-make-process-buffer))
       (output-buffer (generate-new-buffer script-name)))
  (with-current-buffer sieve-buffer
    (goto-char (point-min))
    ;; simulate managesieve response-getscript with a single multibyte
    ;; character: `=C3=A4`
    (insert "{32}\r\nif body :matches \"=C3=A4\" { stop; }\n\r\nOK \"Getscr=
ipt completed.\"\r\n"))
  ;; use flet to mock some functions in call chain of sieve-manage-getscript
  (flet ((sieve-manage-send (_) nil)
         (accept-process-output (&optional _ _ _ _) nil)
         (get-buffer-process (_) nil))
    ;; watch `sieve-manage-getscript' infloop
    (sieve-manage-getscript script-name output-buffer sieve-buffer)
    (kill-buffer sieve-buffer)))

;; Local Variables:
;; coding: utf-8-unix
;; End:

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

to a file and run: emacs -Q -l <file>


* Detailed analysis:

The example code sets up a response buffer for a successful managesieve
`response-getscript` defined as:

    response-getscript =3D (sieve-script CRLF response-ok)

Here's the buffer content:
```
1: {32}<CRLF>
2: if body :matches "=C3=A4" { stop; }<LF>
3: <CRLF>
4: OK "Getscript completed."<CRLF>
```

It comprises:

1. lines 1-2 (`sieve-script`): encoded as a managesieve `literal-s2c`
   string which:
   a. starts with a length in the form '{<OCTETS>}<CRLF>' (i.e. 32)
   b. followed by the string data (i.e. the actual script: 'if
      body :matches "=C3=A4" { stop;}<LF>') using UTF-8 encoding

2. line 3 (`CRLF`)

3. line 4 (`response-ok`): 'OK' SP "Getscript completed." (the latter is
   an optional `quoted` string which can be shown to the user)

The sieve-manage code is parsing the length into an integer and uses it
to skip over `sieve-script` to get to the start of line 3 (<CRLF> -
empty) which is then also skipped to get to line 4 in order to parse the
result ('OK').


Now the problem:

Since sieve-manage explicitly enables multibyte support in the response
buffer (by calling '(mm-enable-multibyte)' in
`sieve-manage-make-process-buffer`) and uses `goto-char' for the purpose
of skipping/jumping over `sieve-script`, each multibyte character in
`sieve-script` causes the jump to go 1 (2, 3) character(s) too far. In
the example above there's only a single 2 byte character (`=C3=A4`), so
instead of skipping to the beginning of line 3, we land in the middle of
<CRLF>: <CR(point)LF>.

This causes the following attempt to parse the result code (i.e. the 'OK
"Getscript completed."<CRLF>' line) to infloop in
`sieve-manage-parse-okno'.


* An attempt of a fix:

As far as I can tell, the attached patch fixes the issue for the
GETSCRIPT command.


--=-=-=
Content-Type: text/x-diff
Content-Disposition: inline;
 filename=sieve-manage-getscript-multibyte-fix.patch
Content-Description: Fix for multibyte issue in `sieve-manage-getscript'

diff --git a/lisp/net/sieve-manage.el b/lisp/net/sieve-manage.el
index 50342b9105..8020e6fdca 100644
--- a/lisp/net/sieve-manage.el
+++ b/lisp/net/sieve-manage.el
@@ -449,10 +449,19 @@ sieve-manage-deletescript
 (defun sieve-manage-getscript (name output-buffer &optional buffer)
   (with-current-buffer (or buffer (current-buffer))
     (sieve-manage-send (format "GETSCRIPT \"%s\"" name))
+    (set-buffer-multibyte nil)
     (let ((script (sieve-manage-parse-string)))
+      (set-buffer-multibyte t)
       (sieve-manage-parse-crlf)
       (with-current-buffer output-buffer
-	(insert script))
+        (insert (decode-coding-string
+                 script
+                 ;; not sure if using `buffer-file-coding-system' is
+                 ;; the right approach, it might be better to hardcode
+                 ;; it to utf-8-* (managesieve requires UTF-8
+                 ;; encoding) but in that case, which variant of
+                 ;; utf-8-unix/dos/...  is to be used?
+                 buffer-file-coding-system t)))
       (sieve-manage-parse-okno))))
 
 (defun sieve-manage-setactive (name &optional buffer)

--=-=-=
Content-Type: text/plain



* Additional remarks:

There might be more problems. E.g. `sieve-manage-putscript' contains the
following comment:

    ;; Here we assume that the coding-system will
    ;; replace each char with a single byte.
    ;; This is always the case if `content' is
    ;; a unibyte string.

which seems to indicate that it might also have an issue with multibyte
content (even though I have not experienced any uploading issues). I
will try do some more testing to check that.


In general, it is also not clear to me why the response (or process)
buffer needs to be multibyte enabled at all as it should only be used
for the line/byte oriented protocol data. But the commit message of
8e16fb987df9b which introduced the multibyte handling states:

    commit 8e16fb987df9b80b8328e9dbf80351a5f9d85bbb
    Author: Albert Krewinkel <krewinkel@HIDDEN>
    Date:   2013-06-11 07:32:25 +0000
    ...
        * Enable Multibyte for SieveManage buffers: The parser won't properly
          handle umlauts and line endings unless multibyte is turned on in the
          process buffer.
    ...

so this was obviously done on purpose. I contacted Albert about this but
he couldn't remember the details (it's been nearly 10 years).



In GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.31, cairo version 1.16.0)
 of 2022-02-18 built on moka
Repository revision: 51e51ce2df46fc0c6e17a97e74b00366bb9c09d8
Repository branch: master
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --with-pgtk --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

--=-=-=--




Acknowledgement sent to "Kai Tetzlaff" <kai.tetzlaff@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#54154; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 28 Feb 2022 13:15:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.