GNU bug report logs - #27438
[PATCH] Specify native search path for all ruby packages

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27438 in the body.
You can then email your comments to 27438 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: guix-patches <at> gnu.org
Subject: [PATCH] Specify native search path for all ruby packages
Date: Wed, 21 Jun 2017 07:36:41 +0100

[Message part 1 (text/plain, inline)]

Without specifying this explicitly in each definition, the GEM_PATH is
inherited and the version is that of the inherited package.

[signature.asc (application/pgp-signature, attachment)]

Message #8 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 27438 <at> debbugs.gnu.org
Subject: [PATCH 1/3] gnu: ruby-1.8: Fix search path specification.
Date: Wed, 21 Jun 2017 07:38:34 +0100

* gnu/packages/ruby.scm (ruby-1.8)[native-search-paths]: Specify the native
  search path, such that the version included in the path is 1.8, rather than
  the inherited version.
---
 gnu/packages/ruby.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 7680f4eae..d9eb93643 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -197,7 +197,13 @@ a focus on simplicity and productivity.")
                           "lib/mkmf.rb"
                           "process.c")
              (("/bin/sh") (which "sh"))))
-         %standard-phases)))))
+         %standard-phases)))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "GEM_PATH")
+            (files (list (string-append "lib/ruby/gems/"
+                                        (version-major+minor version)
+                                        ".0"))))))))
 
 (define-public ruby-hoe
   (package
-- 
2.13.1

Message #11 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 27438 <at> debbugs.gnu.org
Subject: [PATCH 2/3] gnu: ruby-2.2: Fix search path specification.
Date: Wed, 21 Jun 2017 07:38:35 +0100

* gnu/packages/ruby.scm (ruby-2.2)[native-search-paths]: Specify the native
  search path, such that the version included in the path is 2.2, rather than
  the inherited version.
---
 gnu/packages/ruby.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index d9eb93643..098024fc0 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -134,7 +134,13 @@ a focus on simplicity and productivity.")
                            "/ruby-" version ".tar.xz"))
        (sha256
         (base32
-         "0lyb7gnbbhs3a3v9grsjgbaixm20wxz6x3h0czyrxnj3cpp8lk13"))))))
+         "0lyb7gnbbhs3a3v9grsjgbaixm20wxz6x3h0czyrxnj3cpp8lk13"))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "GEM_PATH")
+            (files (list (string-append "lib/ruby/gems/"
+                                        (version-major+minor version)
+                                        ".0"))))))))
 
 (define-public ruby-2.1
   (package (inherit ruby)
-- 
2.13.1

Message #14 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 27438 <at> debbugs.gnu.org
Subject: [PATCH 3/3] gnu: ruby-2.3: Fix search path specification.
Date: Wed, 21 Jun 2017 07:38:36 +0100

* gnu/packages/ruby.scm (ruby-2.3)[native-search-paths]: Specify the native
  search path, such that the version included in the path is 2.3, rather than
  the inherited version.
---
 gnu/packages/ruby.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 098024fc0..69835a771 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -120,7 +120,13 @@ a focus on simplicity and productivity.")
        (snippet `(begin
                    ;; Remove bundled libffi
                    (delete-file-recursively "ext/fiddle/libffi-3.2.1")
-                   #t))))))
+                   #t))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "GEM_PATH")
+            (files (list (string-append "lib/ruby/gems/"
+                                        (version-major+minor version)
+                                        ".0"))))))))
 
 (define-public ruby-2.2
   (package (inherit ruby)
-- 
2.13.1

Message #17 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
To: Christopher Baines <mail <at> cbaines.net>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Wed, 21 Jun 2017 21:47:54 +1000

Hi Chris,


On 21/06/17 16:36, Christopher Baines wrote:
> Without specifying this explicitly in each definition, the GEM_PATH is
> inherited and the version is that of the inherited package.
>
I'm not sure if this is by design, but the version of the gems folder is 
embedded in the build of each rubygem e.g. 'ruby-hoe' includes
/gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0

So, in this environment, "require 'hoe'" fails after applying your patch
./pre-inst-env guix environment -C --ad-hoc ruby <at> 2.2 ruby-hoe

Having said that, the gems bundled with each ruby are in a directory 
with the actual version e.g. ruby <at> 2.2 has this path
/gnu/store/f43dirk254lfhmgjs3sf515cpfl8d46q-ruby-2.2.7/lib/ruby/gems/2.2.0

This isn't ideal, any ideas on what we should do? Specify both paths maybe?
Thanks,
ben

Message #20 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Christopher Baines <mail <at> cbaines.net>
Cc: 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Wed, 21 Jun 2017 15:10:17 +0200

Christopher Baines <mail <at> cbaines.net> skribis:

> Without specifying this explicitly in each definition, the GEM_PATH is
> inherited and the version is that of the inherited package.

All 3 patches LGTM, thank you!

Ludo’.

Message #23 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
Cc: Christopher Baines <mail <at> cbaines.net>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Wed, 21 Jun 2017 15:12:18 +0200

Ben Woodcroft <b.woodcroft <at> uq.edu.au> skribis:

> On 21/06/17 16:36, Christopher Baines wrote:
>> Without specifying this explicitly in each definition, the GEM_PATH is
>> inherited and the version is that of the inherited package.
>>
> I'm not sure if this is by design, but the version of the gems folder
> is embedded in the build of each rubygem e.g. 'ruby-hoe' includes
> /gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0

Or should the search path spec include both lib/ruby/gems/2.2.0 and
lib/ruby/gems/2.4.0, in this order?

Ludo’, who has no experience with Ruby.

Message #26 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Christopher Baines <mail <at> cbaines.net>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Thu, 22 Jun 2017 15:27:22 +1000

On 21/06/17 23:12, Ludovic Courtès wrote:
> Ben Woodcroft <b.woodcroft <at> uq.edu.au> skribis:
>
>> On 21/06/17 16:36, Christopher Baines wrote:
>>> Without specifying this explicitly in each definition, the GEM_PATH is
>>> inherited and the version is that of the inherited package.
>>>
>> I'm not sure if this is by design, but the version of the gems folder
>> is embedded in the build of each rubygem e.g. 'ruby-hoe' includes
>> /gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0
> Or should the search path spec include both lib/ruby/gems/2.2.0 and
> lib/ruby/gems/2.4.0, in this order?
Exactly.

Chris, what is your experience? Did you propose the patch because you 
ran into a particular issue?

Ta, ben.

Message #29 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ben Woodcroft <b.woodcroft <at> uq.edu.au>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Thu, 22 Jun 2017 06:40:17 +0100

[Message part 1 (text/plain, inline)]

On 22/06/17 06:27, Ben Woodcroft wrote:
> On 21/06/17 23:12, Ludovic Courtès wrote:
>> Ben Woodcroft <b.woodcroft <at> uq.edu.au> skribis:
>>
>>> On 21/06/17 16:36, Christopher Baines wrote:
>>>> Without specifying this explicitly in each definition, the GEM_PATH is
>>>> inherited and the version is that of the inherited package.
>>>>
>>> I'm not sure if this is by design, but the version of the gems folder
>>> is embedded in the build of each rubygem e.g. 'ruby-hoe' includes
>>> /gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0
>>>
>> Or should the search path spec include both lib/ruby/gems/2.2.0 and
>> lib/ruby/gems/2.4.0, in this order?
> Exactly.
> 
> Chris, what is your experience? Did you propose the patch because you
> ran into a particular issue?

Yep, I ran in to problems trying to use the guix ruby-2.3 package with
the guix bundler package, when I build bundler with ruby-2.3.

Ben's email got me thinking about how this works in Debian, and it looks
like Debian uses a different location /usr/lib/ruby/vendor_ruby/ .

I think there might be benefits from doing similarly, but this needs a
bit of thought and testing, as I'm unsure how this might work,
especially in cases where libraries include native code that links
against ruby.

I've got a patch for the ruby-build-system to make a change roughly like
this, and I'll send that up soon. Relating this back to the issue at
hand, moving to a version independent directory would mean that the
GEM_PATH wouldn't be version specific.

Thanks,

Chris

[signature.asc (application/pgp-signature, attachment)]

Message #32 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ben Woodcroft <b.woodcroft <at> uq.edu.au>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Sun, 16 Jul 2017 18:37:05 +0100

[Message part 1 (text/plain, inline)]

On Thu, 22 Jun 2017 06:40:17 +0100
Christopher Baines <mail <at> cbaines.net> wrote:

> On 22/06/17 06:27, Ben Woodcroft wrote:
> > On 21/06/17 23:12, Ludovic Courtès wrote:  
> >> Ben Woodcroft <b.woodcroft <at> uq.edu.au> skribis:
> >>  
> >>> On 21/06/17 16:36, Christopher Baines wrote:  
> >>>> Without specifying this explicitly in each definition, the
> >>>> GEM_PATH is inherited and the version is that of the inherited
> >>>> package. 
> >>> I'm not sure if this is by design, but the version of the gems
> >>> folder is embedded in the build of each rubygem e.g. 'ruby-hoe'
> >>> includes /gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0
> >>>  
> >> Or should the search path spec include both lib/ruby/gems/2.2.0 and
> >> lib/ruby/gems/2.4.0, in this order?  
> > Exactly.
> > 
> > Chris, what is your experience? Did you propose the patch because
> > you ran into a particular issue?  
> 
> Yep, I ran in to problems trying to use the guix ruby-2.3 package with
> the guix bundler package, when I build bundler with ruby-2.3.
> 
> Ben's email got me thinking about how this works in Debian, and it
> looks like Debian uses a different
> location /usr/lib/ruby/vendor_ruby/ .
> 
> I think there might be benefits from doing similarly, but this needs a
> bit of thought and testing, as I'm unsure how this might work,
> especially in cases where libraries include native code that links
> against ruby.
> 
> I've got a patch for the ruby-build-system to make a change roughly
> like this, and I'll send that up soon. Relating this back to the
> issue at hand, moving to a version independent directory would mean
> that the GEM_PATH wouldn't be version specific.

So... I kind of forgot about this, sorry about that. But I'll send up
the patch shortly.

As an example of what this change means, here is a example of loading
ruby gems with different versions of ruby, with the current and
alternative situations.

I think arel doesn't use native extensions, whereas nokogiri does.
Also, ruby <at> 2.1 is used, as the search path for the guix package
reflects the version of ruby (whereas many others don't).

* Current

** With the version of ruby used to build the gem

→ guix environment --pure --ad-hoc ruby-arel ruby -- ruby -e "puts require 'arel'"
true

(I'm pretty sure the "true" means that it worked)

→ guix environment --pure --ad-hoc ruby-nokogiri ruby -- ruby -e "puts require 'nokogiri'"
true

** With a different version of ruby

→ guix environment --pure --ad-hoc ruby-arel ruby <at> 2.1 -- ruby -e "puts require 'arel'"
/gnu/store/gdg5m75axmrciwh3zhfnsc038q349ymg-ruby-2.1.10/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- arel (LoadError)
	from /gnu/store/gdg5m75axmrciwh3zhfnsc038q349ymg-ruby-2.1.10/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from -e:1:in `<main>'

→ guix environment --pure --ad-hoc ruby-arel ruby <at> 2.1 -- ruby -e "puts require 'nokogiri'"
/gnu/store/gdg5m75axmrciwh3zhfnsc038q349ymg-ruby-2.1.10/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- nokogiri (LoadError)
	from /gnu/store/gdg5m75axmrciwh3zhfnsc038q349ymg-ruby-2.1.10/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from -e:1:in `<main>'


* Alternative

** With the version of ruby used to build the gem

→ guix environment --pure --ad-hoc ruby-arel ruby -- ruby -e "puts require 'arel'"
true

→ guix environment --pure --ad-hoc ruby-nokogiri ruby -- ruby -e "puts require 'nokogiri'"
true

** With a different version of ruby

→ guix environment --pure --ad-hoc ruby-arel ruby <at> 2.1 -- ruby -e "puts require 'arel'"
true

→ guix environment --pure --ad-hoc ruby-nokogiri ruby <at> 2.1 -- ruby -e "puts require 'nokogiri'"
/gnu/store/gdg5m75axmrciwh3zhfnsc038q349ymg-ruby-2.1.10/lib/ruby/2.1.0/fileutils.rb:250:in `mkdir': Read-only file system @ dir_s_mkdir - /gnu/store/jxmi5lr8dbll3pzlzwkafpd9mj1wwxma-profile/lib/ruby/vendor_ruby/extensions/x86_64-linux/2.1.0-static (Errno::EROFS)
	from /gnu/store/gdg5m75axmrciwh3zhfnsc038q349ymg-ruby-2.1.10/lib/ruby/2.1.0/fileutils.rb:250:in `fu_mkdir'
        ...


So, putting the gems in a single location regardless of the version of
ruby they were built with means that a different version of ruby will
at least see them, however, it may still fail to load them.

I think this is an improvement, but I'm very uncertain about ruby. Does
anyone else have opinions on this?

[Message part 2 (application/pgp-signature, inline)]

Message #35 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 27438 <at> debbugs.gnu.org
Subject: [PATCH] guix: build: ruby-build-system: Install to the vendor
 directory
Date: Sun, 16 Jul 2017 18:40:14 +0100

From: Christopher Baines <christopher.baines <at> digital.cabinet-office.gov.uk>

* guix/build/ruby-build-system.scm (install): Install gems to the vendor
  directory, rather than the GEM_HOME. The vendor directory does not include
  the version of ruby used to install the gem in the path, which makes it
  easier to add it to the GEM_PATH for all versions of ruby to use.
* gnu/packages/ruby.scm (ruby,ruby-2.1)[native-search-paths]: Switch to
  lib/ruby/vendor_ruby.
---
 gnu/packages/ruby.scm            | 12 ++----------
 guix/build/ruby-build-system.scm | 38 +++++++++++++++++++++++---------------
 2 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 7eba68444..f634d5e71 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -93,9 +93,7 @@
     (native-search-paths
      (list (search-path-specification
             (variable "GEM_PATH")
-            (files (list (string-append "lib/ruby/gems/"
-                                        (version-major+minor version)
-                                        ".0"))))))
+            (files (list (string-append "lib/ruby/vendor_ruby"))))))
     (synopsis "Programming language interpreter")
     (description "Ruby is a dynamic object-oriented programming language with
 a focus on simplicity and productivity.")
@@ -159,13 +157,7 @@ a focus on simplicity and productivity.")
                           "lib/mkmf.rb"
                           "process.c")
              (("/bin/sh") (which "sh"))))
-         %standard-phases)))
-    (native-search-paths
-     (list (search-path-specification
-            (variable "GEM_PATH")
-            (files (list (string-append "lib/ruby/gems/"
-                                        (version-major+minor version)
-                                        ".0"))))))))
+         %standard-phases)))))
 
 (define-public ruby-1.8
   (package (inherit ruby)
diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-system.scm
index c2d276627..f4ce2e5d8 100644
--- a/guix/build/ruby-build-system.scm
+++ b/guix/build/ruby-build-system.scm
@@ -129,40 +129,48 @@ GEM-FLAGS are passed to the 'gem' invokation, if present."
                                          (assoc-ref inputs "ruby"))
                            1))
          (out (assoc-ref outputs "out"))
-         (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0"))
+         (vendor-dir (string-append out "/lib/ruby/vendor_ruby"))
          (gem-file (first-matching-file "\\.gem$"))
          (gem-file-basename (basename gem-file))
          (gem-name (substring gem-file-basename
                               0
-                              (- (string-length gem-file-basename) 4)))
-         (gem-directory (string-append gem-home "/gems/" gem-name)))
-    (setenv "GEM_HOME" gem-home)
-    (mkdir-p gem-home)
-    (and (apply system* "gem" "install" gem-file
-                "--local" "--ignore-dependencies"
-                ;; Executables should go into /bin, not /lib/ruby/gems.
-                "--bindir" (string-append out "/bin")
-                gem-flags)
+                              (- (string-length gem-file-basename) 4))))
+    (setenv "GEM_VENDOR" vendor-dir)
+    (and (let ((install-succeeded?
+                (zero?
+                 (apply system* "gem" "install" gem-file
+                        "--local" "--ignore-dependencies" "--vendor"
+                        ;; Executables should go into /bin, not
+                        ;; /lib/ruby/gems.
+                        "--bindir" (string-append out "/bin")
+                        gem-flags))))
+           (or install-succeeded?
+               (begin
+                 (simple-format #t "installation failed\n")
+                 (let ((failed-output-dir (string-append (getcwd) "/out")))
+                   (mkdir failed-output-dir)
+                   (copy-recursively out failed-output-dir))
+                 #f)))
          (begin
            ;; Remove the cached gem file as this is unnecessary and contains
            ;; timestamped files rendering builds not reproducible.
-           (let ((cached-gem (string-append gem-home "/cache/" gem-file)))
+           (let ((cached-gem (string-append vendor-dir "/cache/" gem-file)))
              (log-file-deletion cached-gem)
              (delete-file cached-gem))
            ;; For gems with native extensions, several Makefile-related files
            ;; are created that contain timestamps or other elements making
            ;; them not reproducible.  They are unnecessary so we remove them.
-           (if (file-exists? (string-append gem-directory "/ext"))
+           (if (file-exists? (string-append vendor-dir "/ext"))
                (begin
                  (for-each (lambda (file)
                              (log-file-deletion file)
                              (delete-file file))
                            (append
-                            (find-files (string-append gem-home "/doc")
+                            (find-files (string-append vendor-dir "/doc")
                                         "page-Makefile.ri")
-                            (find-files (string-append gem-home "/extensions")
+                            (find-files (string-append vendor-dir "/extensions")
                                         "gem_make.out")
-                            (find-files (string-append gem-directory "/ext")
+                            (find-files (string-append vendor-dir "/ext")
                                         "Makefile")))))
            #t))))
 
-- 
2.13.1

Message #38 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
To: Christopher Baines <mail <at> cbaines.net>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Thu, 20 Jul 2017 09:39:13 +1000

Hi Chris,


On 17/07/17 03:37, Christopher Baines wrote:
> On Thu, 22 Jun 2017 06:40:17 +0100
> Christopher Baines <mail <at> cbaines.net> wrote:
>
>> On 22/06/17 06:27, Ben Woodcroft wrote:
>>> On 21/06/17 23:12, Ludovic Courtès wrote:
>>>> Ben Woodcroft <b.woodcroft <at> uq.edu.au> skribis:
>>>>   
>>>>> On 21/06/17 16:36, Christopher Baines wrote:
>>>>>> Without specifying this explicitly in each definition, the
>>>>>> GEM_PATH is inherited and the version is that of the inherited
>>>>>> package.
>>>>> I'm not sure if this is by design, but the version of the gems
>>>>> folder is embedded in the build of each rubygem e.g. 'ruby-hoe'
>>>>> includes /gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0
>>>>>   
>>>> Or should the search path spec include both lib/ruby/gems/2.2.0 and
>>>> lib/ruby/gems/2.4.0, in this order?
>>> Exactly.
>>>
>>> Chris, what is your experience? Did you propose the patch because
>>> you ran into a particular issue?
>> Yep, I ran in to problems trying to use the guix ruby-2.3 package with
>> the guix bundler package, when I build bundler with ruby-2.3.
>>
>> Ben's email got me thinking about how this works in Debian, and it
>> looks like Debian uses a different
>> location /usr/lib/ruby/vendor_ruby/ .
>>
>> I think there might be benefits from doing similarly, but this needs a
>> bit of thought and testing, as I'm unsure how this might work,
>> especially in cases where libraries include native code that links
>> against ruby.
>>
>> I've got a patch for the ruby-build-system to make a change roughly
>> like this, and I'll send that up soon. Relating this back to the
>> issue at hand, moving to a version independent directory would mean
>> that the GEM_PATH wouldn't be version specific.
> [..]
>
> So, putting the gems in a single location regardless of the version of
> ruby they were built with means that a different version of ruby will
> at least see them, however, it may still fail to load them.
>
> I think this is an improvement, but I'm very uncertain about ruby. Does
> anyone else have opinions on this?
Thanks for working on this. A few thoughts:

What happens to the default gems that come bundled with ruby itself? I'm 
interpreting from your patch that these will not be available?

In general, except for some special circumstances, we don't support old 
versions of software. To fix the issue that you are encountering 
properly with nokogiri probably requires new package definitions using 
"package-with-ruby-2.3" or similar to be made, I suppose. Ludo did some 
nice work making this easier (see 
https://lists.gnu.org/archive/html/guix-patches/2017-04/msg00126.html), 
but I worry in general about the resources required to support older 
Ruby versions. WDYT?

The gem-home procedure in ruby-build-system will now return an incorrect 
path, I think.

Perhaps I'm slow, but what are the advantages of the "vendor_ruby" 
method over exporting multiple GEM_PATHs as Ludo and I suggested? 
Changing the directory seems like a heavier touch and so more likely to 
misbehave. WDYT?

Apologies if this comes across a little negative, I'm just trying to 
consider all the issues. Thanks again for your efforts on this.
ben

Message #41 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Sat, 22 Jul 2017 11:06:30 +0100

[Message part 1 (text/plain, inline)]

On Thu, 20 Jul 2017 09:39:13 +1000
Ben Woodcroft <b.woodcroft <at> uq.edu.au> wrote:

> Hi Chris,
> 
> 
> On 17/07/17 03:37, Christopher Baines wrote:
> > On Thu, 22 Jun 2017 06:40:17 +0100
> > Christopher Baines <mail <at> cbaines.net> wrote:
> >  
> >> On 22/06/17 06:27, Ben Woodcroft wrote:  
> >>> On 21/06/17 23:12, Ludovic Courtès wrote:  
> >>>> Ben Woodcroft <b.woodcroft <at> uq.edu.au> skribis:
> >>>>     
> >>>>> On 21/06/17 16:36, Christopher Baines wrote:  
> >>>>>> Without specifying this explicitly in each definition, the
> >>>>>> GEM_PATH is inherited and the version is that of the inherited
> >>>>>> package.  
> >>>>> I'm not sure if this is by design, but the version of the gems
> >>>>> folder is embedded in the build of each rubygem e.g. 'ruby-hoe'
> >>>>> includes /gnu/store/d867l5i2dqd5qnq4qlsrcwwb0x3443fl-ruby-hoe-3.16.0/lib/ruby/gems/2.4.0
> >>>>>     
> >>>> Or should the search path spec include both lib/ruby/gems/2.2.0
> >>>> and lib/ruby/gems/2.4.0, in this order?  
> >>> Exactly.
> >>>
> >>> Chris, what is your experience? Did you propose the patch because
> >>> you ran into a particular issue?  
> >> Yep, I ran in to problems trying to use the guix ruby-2.3 package
> >> with the guix bundler package, when I build bundler with ruby-2.3.
> >>
> >> Ben's email got me thinking about how this works in Debian, and it
> >> looks like Debian uses a different
> >> location /usr/lib/ruby/vendor_ruby/ .
> >>
> >> I think there might be benefits from doing similarly, but this
> >> needs a bit of thought and testing, as I'm unsure how this might
> >> work, especially in cases where libraries include native code that
> >> links against ruby.
> >>
> >> I've got a patch for the ruby-build-system to make a change roughly
> >> like this, and I'll send that up soon. Relating this back to the
> >> issue at hand, moving to a version independent directory would mean
> >> that the GEM_PATH wouldn't be version specific.  
> > [..]
> >
> > So, putting the gems in a single location regardless of the version
> > of ruby they were built with means that a different version of ruby
> > will at least see them, however, it may still fail to load them.
> >
> > I think this is an improvement, but I'm very uncertain about ruby.
> > Does anyone else have opinions on this?  
> Thanks for working on this. A few thoughts:
> 
> What happens to the default gems that come bundled with ruby itself?
> I'm interpreting from your patch that these will not be available?

They seem to be:

→ guix environment --container --ad-hoc ruby -- irb
...
for 1 packages... done in 0.028 s 
irb(main):001:0> ENV
=> {"PS1"=>"\\u@\\h \\w [env]\\$ ", "TMPDIR"=>"/tmp",
"TEMPDIR"=>"/tmp", "TMP"=>"/tmp", "TEMP"=>"/tmp",
"HOME"=>"/home/chris",
"PATH"=>"/gnu/store/2dh34pi9yjq17qf1bw3dmgcdpri20fjs-profile/bin",
"GEM_PATH"=>"/gnu/store/2dh34pi9yjq17qf1bw3dmgcdpri20fjs-profile/lib/ruby/vendor_ruby",
"GUIX_ENVIRONMENT"=>"/gnu/store/2dh34pi9yjq17qf1bw3dmgcdpri20fjs-profile",
"LINES"=>"76", "COLUMNS"=>"190"}
irb(main):003:0> require 'rake'
=> true
irb(main):004:0> require 'minitest'
=> true


> In general, except for some special circumstances, we don't support
> old versions of software. To fix the issue that you are encountering 
> properly with nokogiri probably requires new package definitions
> using "package-with-ruby-2.3" or similar to be made, I suppose. Ludo
> did some nice work making this easier (see 
> https://lists.gnu.org/archive/html/guix-patches/2017-04/msg00126.html), 
> but I worry in general about the resources required to support older 
> Ruby versions. WDYT?

I'm not aware of any particular problems if you are working with the
package definitions in Guile, as it should be possible to make them use
the single ruby version that you want.

With the guix environment command I posted:

  guix environment --pure --ad-hoc ruby-nokogiri ruby <at> 2.1 -- ruby -e
  "puts require 'nokogiri'"

It would be ideal if there was some way of telling guix environment to
rewrite the package definitions of all packages to use ruby <at> 2.1 in
place of whatever ruby they might be using.

> The gem-home procedure in ruby-build-system will now return an
> incorrect path, I think.

Good spot. I'll have a look at that.

> Perhaps I'm slow, but what are the advantages of the "vendor_ruby" 
> method over exporting multiple GEM_PATHs as Ludo and I suggested? 
> Changing the directory seems like a heavier touch and so more likely
> to misbehave. WDYT?

I agree that it is heavier in some sense, but I like the simplicity of
getting rid of the version from the path.

The best documentation I've found for this is the NEWS of the release
where it was added [1]. While Guix blurs the lines between the "package
system" and the "user", using this vendor directory might come in
useful.

1: http://svn.ruby-lang.org/repos/ruby/tags/v1_8_7/NEWS

> Apologies if this comes across a little negative, I'm just trying to 
> consider all the issues. Thanks again for your efforts on this.

No need to apologise, I too think its important to consider the
details of the support for ruby, which is not something I'm
particularly knowledgeable on.

[Message part 2 (application/pgp-signature, inline)]

Message #44 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
To: Christopher Baines <mail <at> cbaines.net>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Sat, 5 Aug 2017 13:59:56 +1000

Hi Chris, sorry for the delay on this.


On 22/07/17 20:06, Christopher Baines wrote:
> On Thu, 20 Jul 2017 09:39:13 +1000
> Ben Woodcroft <b.woodcroft <at> uq.edu.au> wrote:
>
>> Hi Chris,
>>
>>
>> [..]
>>
>> What happens to the default gems that come bundled with ruby itself?
>> I'm interpreting from your patch that these will not be available?
> They seem to be:
OK, excellent.

>> In general, except for some special circumstances, we don't support
>> old versions of software. To fix the issue that you are encountering
>> properly with nokogiri probably requires new package definitions
>> using "package-with-ruby-2.3" or similar to be made, I suppose. Ludo
>> did some nice work making this easier (see
>> https://lists.gnu.org/archive/html/guix-patches/2017-04/msg00126.html),
>> but I worry in general about the resources required to support older
>> Ruby versions. WDYT?
> I'm not aware of any particular problems if you are working with the
> package definitions in Guile, as it should be possible to make them use
> the single ruby version that you want.
>
> With the guix environment command I posted:
>
>    guix environment --pure --ad-hoc ruby-nokogiri ruby <at> 2.1 -- ruby -e
>    "puts require 'nokogiri'"
>
> It would be ideal if there was some way of telling guix environment to
> rewrite the package definitions of all packages to use ruby <at> 2.1 in
> place of whatever ruby they might be using.
Is "package-mapping" sufficient?

[..]

>> Perhaps I'm slow, but what are the advantages of the "vendor_ruby"
>> method over exporting multiple GEM_PATHs as Ludo and I suggested?
>> Changing the directory seems like a heavier touch and so more likely
>> to misbehave. WDYT?
> I agree that it is heavier in some sense, but I like the simplicity of
> getting rid of the version from the path.
>
> The best documentation I've found for this is the NEWS of the release
> where it was added [1]. While Guix blurs the lines between the "package
> system" and the "user", using this vendor directory might come in
> useful.
>
> 1: http://svn.ruby-lang.org/repos/ruby/tags/v1_8_7/NEWS
Ah, OK. I hadn't realised there was support for this baked into Ruby 
itself. Seems obvious in hindsight.

If all Ruby dependencies build with this change, then the change seems 
reasonable to me, details aside.
ben

Message #47 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Sat, 5 Aug 2017 22:55:52 +0100

[Message part 1 (text/plain, inline)]

On Sat, 5 Aug 2017 13:59:56 +1000
Ben Woodcroft <b.woodcroft <at> uq.edu.au> wrote:

> Hi Chris, sorry for the delay on this.

No problem :)

> On 22/07/17 20:06, Christopher Baines wrote:
> > On Thu, 20 Jul 2017 09:39:13 +1000
> > Ben Woodcroft <b.woodcroft <at> uq.edu.au> wrote:
> >  
> >> Hi Chris,
> >>
> >>
> >> [..]
> >>
> >> What happens to the default gems that come bundled with ruby
> >> itself? I'm interpreting from your patch that these will not be
> >> available?  
> > They seem to be:  
> OK, excellent.
> 
> >> In general, except for some special circumstances, we don't support
> >> old versions of software. To fix the issue that you are
> >> encountering properly with nokogiri probably requires new package
> >> definitions using "package-with-ruby-2.3" or similar to be made, I
> >> suppose. Ludo did some nice work making this easier (see
> >> https://lists.gnu.org/archive/html/guix-patches/2017-04/msg00126.html),
> >> but I worry in general about the resources required to support
> >> older Ruby versions. WDYT?  
> > I'm not aware of any particular problems if you are working with the
> > package definitions in Guile, as it should be possible to make them
> > use the single ruby version that you want.
> >
> > With the guix environment command I posted:
> >
> >    guix environment --pure --ad-hoc ruby-nokogiri ruby <at> 2.1 -- ruby
> > -e "puts require 'nokogiri'"
> >
> > It would be ideal if there was some way of telling guix environment
> > to rewrite the package definitions of all packages to use ruby <at> 2.1
> > in place of whatever ruby they might be using.  
> Is "package-mapping" sufficient?

I don't think so. The ruby used is in the case of the ruby-build-system
is an argument to the build system, so you need to traverse part of the
dependency graph, altering the arguments of packages using the
ruby-build-system.

Or, perhaps do the transformation at a lower level abstraction than the
package record...

> >> Perhaps I'm slow, but what are the advantages of the "vendor_ruby"
> >> method over exporting multiple GEM_PATHs as Ludo and I suggested?
> >> Changing the directory seems like a heavier touch and so more
> >> likely to misbehave. WDYT?  
> > I agree that it is heavier in some sense, but I like the simplicity
> > of getting rid of the version from the path.
> >
> > The best documentation I've found for this is the NEWS of the
> > release where it was added [1]. While Guix blurs the lines between
> > the "package system" and the "user", using this vendor directory
> > might come in useful.
> >
> > 1: http://svn.ruby-lang.org/repos/ruby/tags/v1_8_7/NEWS  
> Ah, OK. I hadn't realised there was support for this baked into Ruby 
> itself. Seems obvious in hindsight.
> 
> If all Ruby dependencies build with this change, then the change
> seems reasonable to me, details aside.

Ok, does anyone know a good process for testing if lots of packages
build? I think I've heard of Hydra building branches?

[Message part 2 (application/pgp-signature, inline)]

Message #50 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
To: Christopher Baines <mail <at> cbaines.net>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 27438 <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] Specify native search path for all ruby
 packages
Date: Sun, 6 Aug 2017 17:17:41 +1000

Hi,

[..]
>>>> In general, except for some special circumstances, we don't support
>>>> old versions of software. To fix the issue that you are
>>>> encountering properly with nokogiri probably requires new package
>>>> definitions using "package-with-ruby-2.3" or similar to be made, I
>>>> suppose. Ludo did some nice work making this easier (see
>>>> https://lists.gnu.org/archive/html/guix-patches/2017-04/msg00126.html),
>>>> but I worry in general about the resources required to support
>>>> older Ruby versions. WDYT?
>>> I'm not aware of any particular problems if you are working with the
>>> package definitions in Guile, as it should be possible to make them
>>> use the single ruby version that you want.
>>>
>>> With the guix environment command I posted:
>>>
>>>     guix environment --pure --ad-hoc ruby-nokogiri ruby <at> 2.1 -- ruby
>>> -e "puts require 'nokogiri'"
>>>
>>> It would be ideal if there was some way of telling guix environment
>>> to rewrite the package definitions of all packages to use ruby <at> 2.1
>>> in place of whatever ruby they might be using.
>> Is "package-mapping" sufficient?
> I don't think so. The ruby used is in the case of the ruby-build-system
> is an argument to the build system, so you need to traverse part of the
> dependency graph, altering the arguments of packages using the
> ruby-build-system.
>
> Or, perhaps do the transformation at a lower level abstraction than the
> package record...
For most compiled gems, I think it should be possible to write some 
procedure so that this would provide you with what you want:

guix environment --ad-hoc ruby <at> 2.1 
--environment='(package-with-explicit-ruby ruby-2.1 ruby-nokogiri)'

For this specific case though, ruby-2.1 might not be the best example, 
since it is explicitly not supported any more.
>> If all Ruby dependencies build with this change, then the change
>> seems reasonable to me, details aside.
> Ok, does anyone know a good process for testing if lots of packages
> build? I think I've heard of Hydra building branches?
I believe we can go by the old standby "guix refresh -l ruby", so 659 
packages are rebuilt. Given this it would be appropriate for the staging 
branch, due to be merged after core-updates.

ben

Message #53 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 27438 <at> debbugs.gnu.org
Cc: guix-devel <at> gnu.org, Ludovic Courtès <ludo <at> gnu.org>,
 Ben Woodcroft <b.woodcroft <at> uq.edu.au>
Subject: [bug#27438] [PATCH] guix: build: ruby-build-system: Install to the
 vendor directory
Date: Sun, 17 Dec 2017 14:19:37 +0000

[Message part 1 (text/plain, inline)]

Hey,

It's been a while since I looked at this, but I think it's still
relevant.

As a reminder, currently the search paths for the Ruby packages in Guix
are a bit off [1], which makes it difficult to use Ruby gems with
different versions of Ruby.

Rubygems by default installs gems to directories specific to the version
of Ruby, and this can be helpful when managing multiple versions of Ruby
and different sets of gem packages in a shared directory, but with Guix,
this isn't necessary, as the store can be used.

Ruby/Rubygems supports installing gems in to a vendor directory, which
is for gems installed by the package system [2]. While Guix blurs the
lines between the "package system" and the "user", using this vendor
directory might come in useful.

I've attached an updated patch, and successfully built every Ruby
package [3], as well as all of the packages that were changed in the
patch. The only failure was ruby-pg, which failed to build before this
patch.

Ideally when using specific versions of ruby, any gems that need to be
build with that version of ruby, would be. While this change doesn't do
that, it does simplify the search paths of ruby packages and the
packages for ruby gems.

Reading the bug report again, I think there was nearly consensus about
merging this, but I wanted to check if anyone has any comments or
experience with Ruby/Rubygems before merging this?

Thanks,

Chris

1:
scheme@(guile-user)> (use-modules (gnu packages) (guix packages) (guix search-paths) (ice-9 match))
scheme@(guile-user)> (map (lambda (ruby) (cons (package-version ruby) (match (package-native-search-paths ruby) (() #f) ((($ <search-path-specification> _ files)) files)))) (find-packages-by-name "ruby"))
$1 = (("2.4.3" "lib/ruby/gems/2.4.0") ("2.4.2" "lib/ruby/gems/2.4.0") ("2.3.6" "lib/ruby/gems/2.4.0") ("2.2.9" "lib/ruby/gems/2.4.0") ("2.1.10" "lib/ruby/gems/2.1.0") ("1.8.7-p374" . #f))

2: http://svn.ruby-lang.org/repos/ruby/tags/v1_8_7/NEWS

3: guix build --keep-going $(guix package -s ruby- | recsel -P name | tr '\n' ' ')

[0001-guix-build-ruby-build-system-Install-to-the-vendor-d.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #56 received at 27438 <at> debbugs.gnu.org (full text, mbox):

From: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
To: Christopher Baines <mail <at> cbaines.net>, 27438 <at> debbugs.gnu.org
Cc: guix-devel <at> gnu.org, Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [bug#27438] [PATCH] guix: build: ruby-build-system: Install to
 the vendor directory
Date: Fri, 12 Jan 2018 17:17:31 +1000

Hi Chris,


On 18/12/17 00:19, Christopher Baines wrote:
> Hey,
>
> It's been a while since I looked at this, but I think it's still
> relevant.
>
> As a reminder, currently the search paths for the Ruby packages in Guix
> are a bit off [1], which makes it difficult to use Ruby gems with
> different versions of Ruby.
>
> Rubygems by default installs gems to directories specific to the version
> of Ruby, and this can be helpful when managing multiple versions of Ruby
> and different sets of gem packages in a shared directory, but with Guix,
> this isn't necessary, as the store can be used.
>
> Ruby/Rubygems supports installing gems in to a vendor directory, which
> is for gems installed by the package system [2]. While Guix blurs the
> lines between the "package system" and the "user", using this vendor
> directory might come in useful.
>
> I've attached an updated patch, and successfully built every Ruby
> package [3], as well as all of the packages that were changed in the
> patch. The only failure was ruby-pg, which failed to build before this
> patch.
>
> Ideally when using specific versions of ruby, any gems that need to be
> build with that version of ruby, would be. While this change doesn't do
> that, it does simplify the search paths of ruby packages and the
> packages for ruby gems.
>
> Reading the bug report again, I think there was nearly consensus about
> merging this, but I wanted to check if anyone has any comments or
> experience with Ruby/Rubygems before merging this?
I think go ahead with this one. It will be useful particularly when we 
update ruby to 2.5.

ben

Message #61 received at 27438-done <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ben Woodcroft <b.woodcroft <at> uq.edu.au>
Cc: guix-devel <at> gnu.org, Ludovic Courtès <ludo <at> gnu.org>,
 27438-done <at> debbugs.gnu.org
Subject: Re: [bug#27438] [PATCH] guix: build: ruby-build-system: Install to
 the vendor directory
Date: Sun, 14 Jan 2018 22:24:17 +0000

[Message part 1 (text/plain, inline)]

Ben Woodcroft <b.woodcroft <at> uq.edu.au> writes:

> Hi Chris,
>
>
> On 18/12/17 00:19, Christopher Baines wrote:
>> Hey,
>>
>> It's been a while since I looked at this, but I think it's still
>> relevant.
>>
>> As a reminder, currently the search paths for the Ruby packages in Guix
>> are a bit off [1], which makes it difficult to use Ruby gems with
>> different versions of Ruby.
>>
>> Rubygems by default installs gems to directories specific to the version
>> of Ruby, and this can be helpful when managing multiple versions of Ruby
>> and different sets of gem packages in a shared directory, but with Guix,
>> this isn't necessary, as the store can be used.
>>
>> Ruby/Rubygems supports installing gems in to a vendor directory, which
>> is for gems installed by the package system [2]. While Guix blurs the
>> lines between the "package system" and the "user", using this vendor
>> directory might come in useful.
>>
>> I've attached an updated patch, and successfully built every Ruby
>> package [3], as well as all of the packages that were changed in the
>> patch. The only failure was ruby-pg, which failed to build before this
>> patch.
>>
>> Ideally when using specific versions of ruby, any gems that need to be
>> build with that version of ruby, would be. While this change doesn't do
>> that, it does simplify the search paths of ruby packages and the
>> packages for ruby gems.
>>
>> Reading the bug report again, I think there was nearly consensus about
>> merging this, but I wanted to check if anyone has any comments or
>> experience with Ruby/Rubygems before merging this?
> I think go ahead with this one. It will be useful particularly when we
> update ruby to 2.5.

Great, I've pushed this along with the other ruby build system change.

Thanks for reviewing,

Chris

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.