GNU bug report logs - #45773
[PATCH core-updates 0/1] Allow patch-and-repack to work with plain files.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 45773 in the body.
You can then email your comments to 45773 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH core-updates 0/1] Allow patch-and-repack to work with plain
 files.
Date: Sun, 10 Jan 2021 15:02:31 -0500

While attempting to reproduce the now closed issue
<http://issues.guix.gnu.org/30116>, I stumbled upon another annoyance, which
is that the patch-and-repack procedure used as part of an origin derivation
didn't support single files, so the following failed, for example:

<#part type="text/plain" filename="/home/maxim/src/guix-core-updates/repro.scm" disposition=inline description="Script exhibiting problem">
<#/part>

The following patch fixes that, allowing to use plain files with any of the
already supported compressors, or non-tarball archived that would include a
single directory (the same convention as used for our tarballs).

Maxim Cournoyer (1):
  guix: packages: Allow patch-and-repack to work with plain files.

 guix/packages.scm  | 96 +++++++++++++++++++++++++++++++---------------
 tests/packages.scm | 87 +++++++++++++++++++++++++++++++++++++++--
 2 files changed, 149 insertions(+), 34 deletions(-)

-- 
2.29.2

Message #10 received at 45773 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 45773 <at> debbugs.gnu.org
Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to
 work with plain files.
Date: Sun, 10 Jan 2021 17:07:43 -0500

[Message part 1 (text/plain, inline)]

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> While attempting to reproduce the now closed issue
> <http://issues.guix.gnu.org/30116>, I stumbled upon another annoyance, which
> is that the patch-and-repack procedure used as part of an origin derivation
> didn't support single files, so the following failed, for example:
>
> <#part type="text/plain" filename="/home/maxim/src/guix-core-updates/repro.scm" disposition=inline description="Script exhibiting problem">
> <#/part>

That didn't work out well :-).

Here's the attachment that I meant to send along the cover letter.

[repro.scm (text/plain, inline)]

;;; Run script with: ./pre-inst-env guile -e main -s repro.scm

(use-modules (gnu packages base)
             (gnu packages bootstrap)
             (guix build utils)
             (guix derivations)
             (guix download)
             (guix packages)
             (guix store))

(define %test-file-uri
  "https://raw.githubusercontent.com/realgud/realgud/master/realgud/common/bp-image-data.el")

(define (main _)
  (let ((source (origin
                  (method url-fetch)
                  (uri %test-file-uri)
                  (modules '((guix build utils)))
                  (patch-inputs `(("tar"   ,%bootstrap-coreutils&co)
                                  ("xz"    ,%bootstrap-coreutils&co)
                                  ("locales" ,glibc-utf8-locales)))
                  (snippet
                   '(begin
                      (with-fluids ((%default-port-encoding "ISO-8859-1")
                                    (%default-port-conversion-strategy 'error))
                        (substitute* "bp-image-data.el"
                          (("something")
                           "something else")))))
                  (sha256
                   (base32
                    "1qpn2zhh2qw579bhgjyxvf670r4kibaxls589hkm2yhwfvsjvs68")))))
    (with-store store
      (build-derivations store
                         (list (run-with-store store
                                 (origin->derivation source)))))))

[Message part 3 (text/plain, inline)]

Thanks,

Maxim

Message #13 received at 45773 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 45774 <at> debbugs.gnu.org, 45773 <at> debbugs.gnu.org
Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to
 work with plain files.
Date: Thu, 14 Jan 2021 18:48:04 +0100

Hi!

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> Before this change, only plain directories, tar or zip archives were supported
> as the source of a package for the GNU build system; anything else would cause
> the unpack phase to fail.  Origins relying on snippets would suffer from the
> same problem.
>
> This change adds the support to use files of the following extensions: .gz,
> .Z, .bz2, .lz, and .xz, even when they are not tarballs.  Files of unknown
> extensions are treated as uncompressed files and supported as well.
>
> * guix/packages.scm (patch-and-repack): Only add the compressor utility to the
> PATH when the file is compressed.  Bind more inputs in the mlet, and use them
> for decompressing single files.  Adjust decompression and compression routines.
> [decompression-type]: Return #f when no known compression extension is used.
> [tarball?]: New nested procedure.
> * guix/build/utils.scm (compressor, tarball?): New procedures.  Move
> %xz-parallel-args to the new 'compression helpers' section.
> * tests/packages.scm: Add tests.  Add missing copyright year for Jan.
> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
> sub-directory was found.
> (unpack): Support more file types, including uncompressed plain files.
> ---
>  guix/build/gnu-build-system.scm |  24 ++++++--
>  guix/build/utils.scm            |  47 ++++++++++-----
>  guix/packages.scm               | 100 +++++++++++++++++---------------
>  guix/tests.scm                  |  40 ++++++++++++-
>  tests/builders.scm              |  40 ++++++++++++-
>  tests/packages.scm              |  69 +++++++++++++++++++++-
>  6 files changed, 247 insertions(+), 73 deletions(-)

How frequent is it for an origin to be a regular file other than an
archive?  The underlying question for me is: will this generalization
and increased complexity pay off?  WDYT?

There are aspects of the patch that I find welcome regardless, such as
the improved handling of compression helpers.

Thanks,
Ludo’.

Message #16 received at 45773 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45774 <at> debbugs.gnu.org, 45773 <at> debbugs.gnu.org
Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to
 work with plain files.
Date: Thu, 14 Jan 2021 13:43:43 -0500

Hi Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi!
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> Before this change, only plain directories, tar or zip archives were supported
>> as the source of a package for the GNU build system; anything else would cause
>> the unpack phase to fail.  Origins relying on snippets would suffer from the
>> same problem.
>>
>> This change adds the support to use files of the following extensions: .gz,
>> .Z, .bz2, .lz, and .xz, even when they are not tarballs.  Files of unknown
>> extensions are treated as uncompressed files and supported as well.
>>
>> * guix/packages.scm (patch-and-repack): Only add the compressor utility to the
>> PATH when the file is compressed.  Bind more inputs in the mlet, and use them
>> for decompressing single files.  Adjust decompression and compression routines.
>> [decompression-type]: Return #f when no known compression extension is used.
>> [tarball?]: New nested procedure.
>> * guix/build/utils.scm (compressor, tarball?): New procedures.  Move
>> %xz-parallel-args to the new 'compression helpers' section.
>> * tests/packages.scm: Add tests.  Add missing copyright year for Jan.
>> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
>> sub-directory was found.
>> (unpack): Support more file types, including uncompressed plain files.
>> ---
>>  guix/build/gnu-build-system.scm |  24 ++++++--
>>  guix/build/utils.scm            |  47 ++++++++++-----
>>  guix/packages.scm               | 100 +++++++++++++++++---------------
>>  guix/tests.scm                  |  40 ++++++++++++-
>>  tests/builders.scm              |  40 ++++++++++++-
>>  tests/packages.scm              |  69 +++++++++++++++++++++-
>>  6 files changed, 247 insertions(+), 73 deletions(-)
>
> How frequent is it for an origin to be a regular file other than an
> archive?  The underlying question for me is: will this generalization
> and increased complexity pay off?  WDYT?

I think consistency is the main driver here.  The url-fetch method
supports single file sources; it makes sense that the other components
handling sources support it as well.  It's hard to judge of the
popularity of such a feature when it's never been available; but some
use cases come to mind such as single Emacs package file.  I've made use
of such feature for the new texlive-updmap.cfg definition in
<http://issues.guix.gnu.org/45870>.

> There are aspects of the patch that I find welcome regardless, such as
> the improved handling of compression helpers.

Great!  Thanks for looking at it.

Maxim

Message #23 received at 45773-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45774-done <at> debbugs.gnu.org, 45773-done <at> debbugs.gnu.org
Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to
 work with plain files.
Date: Tue, 26 Jan 2021 22:57:22 -0500

Hi Ludo,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi Ludovic,
>
> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> Hi!
>>
>> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>>
>>> Before this change, only plain directories, tar or zip archives were supported
>>> as the source of a package for the GNU build system; anything else would cause
>>> the unpack phase to fail.  Origins relying on snippets would suffer from the
>>> same problem.
>>>
>>> This change adds the support to use files of the following extensions: .gz,
>>> .Z, .bz2, .lz, and .xz, even when they are not tarballs.  Files of unknown
>>> extensions are treated as uncompressed files and supported as well.
>>>
>>> * guix/packages.scm (patch-and-repack): Only add the compressor utility to the
>>> PATH when the file is compressed.  Bind more inputs in the mlet, and use them
>>> for decompressing single files.  Adjust decompression and compression routines.
>>> [decompression-type]: Return #f when no known compression extension is used.
>>> [tarball?]: New nested procedure.
>>> * guix/build/utils.scm (compressor, tarball?): New procedures.  Move
>>> %xz-parallel-args to the new 'compression helpers' section.
>>> * tests/packages.scm: Add tests.  Add missing copyright year for Jan.
>>> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
>>> sub-directory was found.
>>> (unpack): Support more file types, including uncompressed plain files.
>>> ---
>>>  guix/build/gnu-build-system.scm |  24 ++++++--
>>>  guix/build/utils.scm            |  47 ++++++++++-----
>>>  guix/packages.scm               | 100 +++++++++++++++++---------------
>>>  guix/tests.scm                  |  40 ++++++++++++-
>>>  tests/builders.scm              |  40 ++++++++++++-
>>>  tests/packages.scm              |  69 +++++++++++++++++++++-
>>>  6 files changed, 247 insertions(+), 73 deletions(-)
>>
>> How frequent is it for an origin to be a regular file other than an
>> archive?  The underlying question for me is: will this generalization
>> and increased complexity pay off?  WDYT?
>
> I think consistency is the main driver here.  The url-fetch method
> supports single file sources; it makes sense that the other components
> handling sources support it as well.  It's hard to judge of the
> popularity of such a feature when it's never been available; but some
> use cases come to mind such as single Emacs package file.  I've made use
> of such feature for the new texlive-updmap.cfg definition in
> <http://issues.guix.gnu.org/45870>.

I've been building a sizable part of core-updates on top of this change
now for nearly two weeks, and in this time it's already come handy
twice.

I've made sure the tests ran fine and pushed to core-updates as commit
cfcead2e515c0dae02127e5a76496463898be6b6.

Let me know if anything breaks :-).

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.