GNU bug report logs - #26225
[PATCH] guix: Compress and decompress xz archives in parallel.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26225 in the body.
You can then email your comments to 26225 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
Subject: [PATCH] guix: Compress and decompress xz archives in parallel.
Date: Thu, 23 Mar 2017 11:47:32 +0200

* guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
* guix/utils.scm (decompressed-port, compressed-port,
compressed-output-port): Same.
---
 guix/scripts/pack.scm | 3 ++-
 guix/utils.scm        | 7 ++++---
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 626c592e1..9e91bc22a 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2017 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2017 Efraim Flashner <efraim <at> flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -59,7 +60,7 @@
         (compressor "lzip"  "lz"
                     #~(#+(file-append lzip "/bin/lzip") "-9"))
         (compressor "xz"    "xz"
-                    #~(#+(file-append xz "/bin/xz") "-e"))
+                    #~(#+(file-append xz "/bin/xz") "-e -T0"))
         (compressor "bzip2" "bz2"
                     #~(#+(file-append bzip2 "/bin/bzip2") "-9"))))
 
diff --git a/guix/utils.scm b/guix/utils.scm
index bc90686de..fb962df8b 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2014 Ian Denhardt <ian <at> zenhack.net>
 ;;; Copyright © 2016 Mathieu Lirzin <mthl <at> gnu.org>
 ;;; Copyright © 2015 David Thompson <davet <at> gnu.org>
+;;; Copyright © 2017 Efraim Flashner <efraim <at> flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -154,7 +155,7 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
-    ('xz           (filtered-port `(,%xz "-dc") input))
+    ('xz           (filtered-port `(,%xz "-dc -T0") input))
     ('gzip         (filtered-port `(,%gzip "-dc") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -164,7 +165,7 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-c") input))
-    ('xz           (filtered-port `(,%xz "-c") input))
+    ('xz           (filtered-port `(,%xz "-c -T0") input))
     ('gzip         (filtered-port `(,%gzip "-c") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."
   (match compression
     ((or #f 'none) (values output '()))
     ('bzip2        (filtered-output-port `(,%bzip2 "-c" ,@options) output))
-    ('xz           (filtered-output-port `(,%xz "-c" ,@options) output))
+    ('xz           (filtered-output-port `(,%xz "-c -T0" ,@options) output))
     ('gzip         (filtered-output-port `(,%gzip "-c" ,@options) output))
     (else          (error "unsupported compression scheme" compression))))
 
-- 
2.12.1

Message #10 received at 26225 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: 26225 <at> debbugs.gnu.org
Cc: Efraim Flashner <efraim <at> flashner.co.il>
Subject: Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in
 parallel.
Date: Fri, 24 Mar 2017 15:36:15 -0400

Reassigned to guix-patches.
(You must include a Package: line if reporting by bcc(?) ).

Efraim Flashner wrote:

> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> * guix/utils.scm (decompressed-port, compressed-port,
> compressed-output-port): Same.
> ---
>  guix/scripts/pack.scm | 3 ++-
>  guix/utils.scm        | 7 ++++---
>  2 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
> index 626c592e1..9e91bc22a 100644
> --- a/guix/scripts/pack.scm
> +++ b/guix/scripts/pack.scm
> @@ -1,5 +1,6 @@
>  ;;; GNU Guix --- Functional package management for GNU
>  ;;; Copyright © 2015, 2017 Ludovic Courtès <ludo <at> gnu.org>
> +;;; Copyright © 2017 Efraim Flashner <efraim <at> flashner.co.il>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -59,7 +60,7 @@
>          (compressor "lzip"  "lz"
>                      #~(#+(file-append lzip "/bin/lzip") "-9"))
>          (compressor "xz"    "xz"
> -                    #~(#+(file-append xz "/bin/xz") "-e"))
> +                    #~(#+(file-append xz "/bin/xz") "-e -T0"))
>          (compressor "bzip2" "bz2"
>                      #~(#+(file-append bzip2 "/bin/bzip2") "-9"))))
>  
> diff --git a/guix/utils.scm b/guix/utils.scm
> index bc90686de..fb962df8b 100644
> --- a/guix/utils.scm
> +++ b/guix/utils.scm
> @@ -5,6 +5,7 @@
>  ;;; Copyright © 2014 Ian Denhardt <ian <at> zenhack.net>
>  ;;; Copyright © 2016 Mathieu Lirzin <mthl <at> gnu.org>
>  ;;; Copyright © 2015 David Thompson <davet <at> gnu.org>
> +;;; Copyright © 2017 Efraim Flashner <efraim <at> flashner.co.il>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>    (match compression
>      ((or #f 'none) (values input '()))
>      ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
> -    ('xz           (filtered-port `(,%xz "-dc") input))
> +    ('xz           (filtered-port `(,%xz "-dc -T0") input))
>      ('gzip         (filtered-port `(,%gzip "-dc") input))
>      (else          (error "unsupported compression scheme" compression))))
>  
> @@ -164,7 +165,7 @@ a symbol such as 'xz."
>    (match compression
>      ((or #f 'none) (values input '()))
>      ('bzip2        (filtered-port `(,%bzip2 "-c") input))
> -    ('xz           (filtered-port `(,%xz "-c") input))
> +    ('xz           (filtered-port `(,%xz "-c -T0") input))
>      ('gzip         (filtered-port `(,%gzip "-c") input))
>      (else          (error "unsupported compression scheme" compression))))
>  
> @@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."
>    (match compression
>      ((or #f 'none) (values output '()))
>      ('bzip2        (filtered-output-port `(,%bzip2 "-c" ,@options) output))
> -    ('xz           (filtered-output-port `(,%xz "-c" ,@options) output))
> +    ('xz           (filtered-output-port `(,%xz "-c -T0" ,@options) output))
>      ('gzip         (filtered-output-port `(,%gzip "-c" ,@options) output))
>      (else          (error "unsupported compression scheme" compression))))

Message #13 received at 26225 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 26225 <at> debbugs.gnu.org, efraim <at> flashner.co.il
Subject: Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in
 parallel.
Date: Thu, 30 Mar 2017 18:22:35 -0400

[Message part 1 (text/plain, inline)]

Efraim Flashner wrote:
> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> * guix/utils.scm (decompressed-port, compressed-port,
> compressed-output-port): Same.

> --- a/guix/utils.scm
> +++ b/guix/utils.scm
> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>    (match compression
>      ((or #f 'none) (values input '()))
>      ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
> -    ('xz           (filtered-port `(,%xz "-dc") input))
> +    ('xz           (filtered-port `(,%xz "-dc -T0") input))
>      ('gzip         (filtered-port `(,%gzip "-dc") input))
>      (else          (error "unsupported compression scheme" compression))))

Parallel decompression isn't implemented yet, but the extra '-T0' option
doesn't seem to break anything.

The changes LGTM.

I sent a companion patch to <https://bugs.gnu.org/26316> that enables
threaded compression when building source tarballs.

[signature.asc (application/pgp-signature, inline)]

Message #16 received at 26225 <at> debbugs.gnu.org (full text, mbox):

From: Kei Kebreau <kei <at> openmailbox.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: efraim <at> flashner.co.il, 26225 <at> debbugs.gnu.org
Subject: Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in
 parallel.
Date: Fri, 31 Mar 2017 13:56:37 -0400

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> Efraim Flashner wrote:
>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
>> * guix/utils.scm (decompressed-port, compressed-port,
>> compressed-output-port): Same.
>
>> --- a/guix/utils.scm
>> +++ b/guix/utils.scm
>> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>>    (match compression
>>      ((or #f 'none) (values input '()))
>>      ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
>> -    ('xz           (filtered-port `(,%xz "-dc") input))
>> +    ('xz           (filtered-port `(,%xz "-dc -T0") input))
>>      ('gzip         (filtered-port `(,%gzip "-dc") input))
>>      (else          (error "unsupported compression scheme" compression))))
>
> Parallel decompression isn't implemented yet, but the extra '-T0' option
> doesn't seem to break anything.
>
> The changes LGTM.
>
> I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> threaded compression when building source tarballs.

LGTM as well.

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 26225 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Kei Kebreau <kei <at> openmailbox.org>
Cc: efraim <at> flashner.co.il, 26225 <at> debbugs.gnu.org
Subject: Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in
 parallel.
Date: Fri, 31 Mar 2017 19:01:58 -0400

[Message part 1 (text/plain, inline)]

On Fri, Mar 31, 2017 at 01:56:37PM -0400, Kei Kebreau wrote:
> Leo Famulari <leo <at> famulari.name> writes:
> 
> > Efraim Flashner wrote:
> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> >> * guix/utils.scm (decompressed-port, compressed-port,
> >> compressed-output-port): Same.
> >
> >> --- a/guix/utils.scm
> >> +++ b/guix/utils.scm
> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> >>    (match compression
> >>      ((or #f 'none) (values input '()))
> >>      ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
> >> -    ('xz           (filtered-port `(,%xz "-dc") input))
> >> +    ('xz           (filtered-port `(,%xz "-dc -T0") input))
> >>      ('gzip         (filtered-port `(,%gzip "-dc") input))
> >>      (else          (error "unsupported compression scheme" compression))))
> >
> > Parallel decompression isn't implemented yet, but the extra '-T0' option
> > doesn't seem to break anything.
> >
> > The changes LGTM.
> >
> > I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> > threaded compression when building source tarballs.
> 
> LGTM as well.

AFAICT, this doesn't cause mass rebuilds.

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 26225 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Leo Famulari <leo <at> famulari.name>, 26225 <at> debbugs.gnu.org,
 efraim <at> flashner.co.il
Subject: Re: bug#26225: [PATCH] guix: Compress and decompress xz archives
 in	parallel.
Date: Sat, 01 Apr 2017 08:52:28 +0200

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> Efraim Flashner wrote:
>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
>> * guix/utils.scm (decompressed-port, compressed-port,
>> compressed-output-port): Same.
>
>> --- a/guix/utils.scm
>> +++ b/guix/utils.scm
>> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>>    (match compression
>>      ((or #f 'none) (values input '()))
>>      ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
>> -    ('xz           (filtered-port `(,%xz "-dc") input))
>> +    ('xz           (filtered-port `(,%xz "-dc -T0") input))
>>      ('gzip         (filtered-port `(,%gzip "-dc") input))
>>      (else          (error "unsupported compression scheme" compression))))
>
> Parallel decompression isn't implemented yet, but the extra '-T0' option
> doesn't seem to break anything.
>
> The changes LGTM.
>
> I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> threaded compression when building source tarballs.

Is (parallel-job-count) accessible here? It would be nice to respect it.

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 26225 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: efraim <at> flashner.co.il, 26225 <at> debbugs.gnu.org
Subject: Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in
 parallel.
Date: Sun, 2 Apr 2017 17:21:18 -0400

[Message part 1 (text/plain, inline)]

On Sat, Apr 01, 2017 at 08:52:28AM +0200, Marius Bakke wrote:
> > Efraim Flashner wrote:
> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> >> * guix/utils.scm (decompressed-port, compressed-port,
> >> compressed-output-port): Same.
> >
> >> --- a/guix/utils.scm
> >> +++ b/guix/utils.scm
> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> >>    (match compression
> >>      ((or #f 'none) (values input '()))
> >>      ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
> >> -    ('xz           (filtered-port `(,%xz "-dc") input))
> >> +    ('xz           (filtered-port `(,%xz "-dc -T0") input))
> >>      ('gzip         (filtered-port `(,%gzip "-dc") input))
> >>      (else          (error "unsupported compression scheme" compression))))
> 
> Is (parallel-job-count) accessible here? It would be nice to respect it.

I agree, it would be good if we respected it here, if possible.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.