GNU bug report logs - #33919
[PATCH 0/5] Add Zstandard compression for installation and utilities

Previous Next

Package: emacs;

Reported by: Alex <agrambot <at> gmail.com>

Date: Sun, 30 Dec 2018 06:47:01 UTC

Severity: wishlist

Tags: patch

Fixed in version 27.1

Done: Lars Ingebrigtsen <larsi <at> gnus.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 33919 in the body.
You can then email your comments to 33919 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Sun, 30 Dec 2018 06:47:01 GMT) Full text and rfc822 format available.

Acknowledgement sent to Alex <agrambot <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Sun, 30 Dec 2018 06:47:01 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alex <agrambot <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: [PATCH 0/5] Add Zstandard compression for installation and utilities
Date: Sun, 30 Dec 2018 00:46:08 -0600
[Message part 1 (text/plain, inline)]
I figured that it would be nice to support other compression programs
for installed files, and Zstandard[1] seems like a good alternative
for gzip in this area due to its fast decompression and wide range of
options.

This patch series depends on the patch I posted for bug#33880[2].

I have a few questions/comments:

1) I tested about everything except the Tramp part. One possible issue
is that zstd prints a progress/summary line to stdout without the -q
flag when compressing a single file; does Tramp care about that here?

2) Is there a convenient method to avoid the duplication of setting up
the COMPR* (and perhaps other) variables in the doc/*/Makefile.in
files?  What about making a common file that other files load at the
beginning?

3) Hopefully using eval is not an issue here.  I believe the security
should be the same, since I double quoted the filename strings (which
I confirmed could be used to execute code without the extra quotes),
and the old GZIP_PROG could also be exploited.

[1] https://facebook.github.io/zstd/
[2]
https://lists.gnu.org/archive/html/bug-gnu-emacs/2018-12/msg00926.html

Alexander Gramiak (5):
  Add Zstandard compression support for etags
  Add Zstandard compression support for dired
  Add Zstandard compression support for Tramp
  * make-dist: Add option for zstd compression
  Add Zstandard compression option for make install

 INSTALL                   | 34 +++++++++++++++++++----
 Makefile.in               | 57 ++++++++++++++++++++++++++-------------
 admin/admin.el            |  2 +-
 configure.ac              | 36 ++++++++++++++++++-------
 doc/emacs/Makefile.in     | 25 ++++++++++++-----
 doc/lispintro/Makefile.in | 25 ++++++++++++-----
 doc/lispref/Makefile.in   | 25 ++++++++++++-----
 doc/man/etags.1           |  2 +-
 doc/misc/Makefile.in      | 25 ++++++++++++-----
 doc/misc/tramp.texi       |  9 ++++---
 etc/NEWS                  | 30 +++++++++++++++++++++
 lib-src/etags.c           |  3 ++-
 lisp/dired-aux.el         | 21 ++++++++++-----
 lisp/jka-cmpr-hook.el     |  4 +--
 lisp/net/tramp-archive.el |  8 +++---
 lisp/net/tramp-sh.el      |  1 +
 make-dist                 |  8 +++++-
 17 files changed, 239 insertions(+), 76 deletions(-)

[0001-Add-Zstandard-compression-support-for-etags.patch (text/x-patch, attachment)]
[0002-Add-Zstandard-compression-support-for-dired.patch (text/x-patch, attachment)]
[0003-Add-Zstandard-compression-support-for-Tramp.patch (text/x-patch, attachment)]
[0004-make-dist-Add-option-for-zstd-compression.patch (text/x-patch, attachment)]
[0005-Add-Zstandard-compression-option-for-make-install.patch (text/x-patch, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Sun, 30 Dec 2018 10:07:01 GMT) Full text and rfc822 format available.

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Alex <agrambot <at> gmail.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: [PATCH 0/5] Add Zstandard compression for installation and
 utilities
Date: Sun, 30 Dec 2018 11:06:10 +0100
Alex <agrambot <at> gmail.com> writes:

Hi Alex,

> I figured that it would be nice to support other compression programs
> for installed files, and Zstandard[1] seems like a good alternative
> for gzip in this area due to its fast decompression and wide range of
> options.

Thanks.

> 1) I tested about everything except the Tramp part. One possible issue
> is that zstd prints a progress/summary line to stdout without the -q
> flag when compressing a single file; does Tramp care about that here?

I cannot test your integration into tramp-archive.el, my Ubuntu 18.10
carries libarchive 3.2.2 only. Next year, @work, I have a Fedora 29
machine, which I can use for testing. Will report then.

Integration into tramp-sh.el works. The "-q" argument is not needed,
because Tramp redirects stderr to /dev/null for the compression
commands. The progress line seems to go there.

> diff --git a/etc/NEWS b/etc/NEWS
> index af99e29e40..e65823413b 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -927,6 +927,10 @@ ad-hoc multi-hop file names must match the previous hop.
>  timeout, after which the underlying session is disabled.  This is for
>  security reasons.
>
> ++++
> +*** Zstandard compression is now supported when using libarchive 3.3
> +or newer.
> +
>  ** Rcirc

This is superfluous. tramp-archive.el is new in Emacs 27, and it is
mentioned in etc/NEWS. Changes on that file do not need to be documented
additionally.

Changes in compression of tramp-sh.el are not user visible, no need to
document them in etc/NEWS. This might change, if we decide to give the
user an option to decide, which compression has to be applied in
Tramp. That's not possible for now.

> --- a/lisp/net/tramp-archive.el
> +++ b/lisp/net/tramp-archive.el
> +;; ".gz", ".lrz", ".lz", ".lz4", ".lzma", ".lzo", ".uu", ".xz",".Z",
> +;; and .zst.  A valid archive file name would be

This shall be 

;; ".gz", ".lrz", ".lz", ".lz4", ".lzma", ".lzo", ".uu", ".xz", ".Z",
;; and ".zst".  A valid archive file name would be

> +    "tar" "tbz" "tgz" "tlz" "txz" ".tzst" ;; (Compressed) tape archives.

This shall be

    "tar" "tbz" "tgz" "tlz" "txz" "tzst" ;; (Compressed) tape archives.

Best regards, Michael.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Sun, 30 Dec 2018 15:22:02 GMT) Full text and rfc822 format available.

Message #11 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alex <agrambot <at> gmail.com>
Cc: 33919 <at> debbugs.gnu.org
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Sun, 30 Dec 2018 17:21:11 +0200
> From: Alex <agrambot <at> gmail.com>
> Date: Sun, 30 Dec 2018 00:46:08 -0600
> 
> I figured that it would be nice to support other compression programs
> for installed files, and Zstandard[1] seems like a good alternative
> for gzip in this area due to its fast decompression and wide range of
> options.

Thanks.

I'm okay with adding support for zstd to Emacs, but I'm not sure we
should complicate the build with offering yet another compression
option.  Up front, I don't think the advantages of zstd are really so
overwhelming that they will justify the complexity.

So maybe split the patch series in two; the part that adds zstd
support to Emacs can be installed already.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Wed, 02 Jan 2019 13:05:01 GMT) Full text and rfc822 format available.

Message #14 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Alex <agrambot <at> gmail.com>
Cc: 33919 <at> debbugs.gnu.org
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Wed, 02 Jan 2019 14:04:43 +0100
Michael Albinus <michael.albinus <at> gmx.de> writes:

Hi Alex,

>> 1) I tested about everything except the Tramp part. One possible issue
>> is that zstd prints a progress/summary line to stdout without the -q
>> flag when compressing a single file; does Tramp care about that here?
>
> I cannot test your integration into tramp-archive.el, my Ubuntu 18.10
> carries libarchive 3.2.2 only. Next year, @work, I have a Fedora 29
> machine, which I can use for testing. Will report then.

Tests performed, everything looks good.

Best regards, Michael.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Sun, 23 Jun 2019 18:33:02 GMT) Full text and rfc822 format available.

Message #17 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 33919 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
 Alex <agrambot <at> gmail.com>
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Sun, 23 Jun 2019 20:32:05 +0200
Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Alex <agrambot <at> gmail.com>
>> Date: Sun, 30 Dec 2018 00:46:08 -0600
>> 
>> I figured that it would be nice to support other compression programs
>> for installed files, and Zstandard[1] seems like a good alternative
>> for gzip in this area due to its fast decompression and wide range of
>> options.
>
> Thanks.
>
> I'm okay with adding support for zstd to Emacs, but I'm not sure we
> should complicate the build with offering yet another compression
> option.  Up front, I don't think the advantages of zstd are really so
> overwhelming that they will justify the complexity.
>
> So maybe split the patch series in two; the part that adds zstd
> support to Emacs can be installed already.

Nothing further was done here (I think), but Michael confirmed that the
Tramp part of the changes work.  And the functionality is split into
patches -- I can apply the first three patches (the ones that add the
support for external zstd binaries) and skip the last two (that add a
compilation option for the zstd library)?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Sun, 23 Jun 2019 18:50:02 GMT) Full text and rfc822 format available.

Message #20 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 33919 <at> debbugs.gnu.org,
 Alex <agrambot <at> gmail.com>
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Sun, 23 Jun 2019 20:49:04 +0200
Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Nothing further was done here (I think), but Michael confirmed that the
> Tramp part of the changes work.

Yes, with the small comments I've added.

> And the functionality is split into
> patches -- I can apply the first three patches (the ones that add the
> support for external zstd binaries) and skip the last two (that add a
> compilation option for the zstd library)?

No objection from my side.

Best regards, Michael.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Mon, 24 Jun 2019 14:09:02 GMT) Full text and rfc822 format available.

Message #23 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 33919 <at> debbugs.gnu.org,
 Alex <agrambot <at> gmail.com>
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Mon, 24 Jun 2019 16:07:56 +0200
Michael Albinus <michael.albinus <at> gmx.de> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> Nothing further was done here (I think), but Michael confirmed that the
>> Tramp part of the changes work.
>
> Yes, with the small comments I've added.
>
>> And the functionality is split into
>> patches -- I can apply the first three patches (the ones that add the
>> support for external zstd binaries) and skip the last two (that add a
>> compilation option for the zstd library)?
>
> No objection from my side.

I've now applied the first three patches and made some additional
commits that I think covers Michael's comments.  (It was supposed to be
a single addition commit but I had vc-dir in the wrong directory.
:-/)

So I'm now closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




bug marked as fixed in version 27.1, send any further explanations to 33919 <at> debbugs.gnu.org and Alex <agrambot <at> gmail.com> Request was from Lars Ingebrigtsen <larsi <at> gnus.org> to control <at> debbugs.gnu.org. (Mon, 24 Jun 2019 14:09:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Mon, 24 Jun 2019 14:45:02 GMT) Full text and rfc822 format available.

Message #28 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 33919 <at> debbugs.gnu.org, michael.albinus <at> gmx.de, agrambot <at> gmail.com
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Mon, 24 Jun 2019 17:44:37 +0300
> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: Alex <agrambot <at> gmail.com>,  33919 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>
> Date: Sun, 23 Jun 2019 20:32:05 +0200
> 
> > I'm okay with adding support for zstd to Emacs, but I'm not sure we
> > should complicate the build with offering yet another compression
> > option.  Up front, I don't think the advantages of zstd are really so
> > overwhelming that they will justify the complexity.
> >
> > So maybe split the patch series in two; the part that adds zstd
> > support to Emacs can be installed already.
> 
> Nothing further was done here (I think), but Michael confirmed that the
> Tramp part of the changes work.  And the functionality is split into
> patches -- I can apply the first three patches (the ones that add the
> support for external zstd binaries) and skip the last two (that add a
> compilation option for the zstd library)?

When you ask a question, I'd appreciate if next time you waited a bit
longer for the answer.  (I've only seen the question this morning,
i.e. 12 hours ago, and didn't yet have time to re-read the discussions
and the patches.)  TIA.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#33919; Package emacs. (Mon, 24 Jun 2019 16:28:02 GMT) Full text and rfc822 format available.

Message #31 received at 33919 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 33919 <at> debbugs.gnu.org, michael.albinus <at> gmx.de, agrambot <at> gmail.com
Subject: Re: bug#33919: [PATCH 0/5] Add Zstandard compression for installation
 and utilities
Date: Mon, 24 Jun 2019 18:27:51 +0200
Eli Zaretskii <eliz <at> gnu.org> writes:

>> > I'm okay with adding support for zstd to Emacs, but I'm not sure we
>> > should complicate the build with offering yet another compression
>> > option.  Up front, I don't think the advantages of zstd are really so
>> > overwhelming that they will justify the complexity.
>> >
>> > So maybe split the patch series in two; the part that adds zstd
>> > support to Emacs can be installed already.

[...]

> When you ask a question, I'd appreciate if next time you waited a bit
> longer for the answer.  (I've only seen the question this morning,
> i.e. 12 hours ago, and didn't yet have time to re-read the discussions
> and the patches.)  TIA.

Sorry; I assumed that you were on board with this already based on your
previous comment, so I was really soliciting answers from others
(Michael, in particular).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Tue, 23 Jul 2019 11:24:09 GMT) Full text and rfc822 format available.

This bug report was last modified 4 years and 250 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.