GNU bug report logs - #75879
with-parameters does not work generally for packages

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75879 in the body.
You can then email your comments to 75879 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Elsing <david.elsing <at> posteo.net>
To: bug-guix <at> gnu.org
Subject: with-parameters does not work generally for packages
Date: Sun, 26 Jan 2025 21:11:07 +0000

[Message part 1 (text/plain, inline)]

Hello,

I noticed that 'with-parameters' from (guix gexp) does not work with
Guile parameters used in package definitions. They are still set
in 'lower-object', but not anymore when the monadic procedure returned
by 'lower-object' is evaluated.

Attached is an example for a package wrapped by 'with-parameters', which
results in a file with "D" instead of "C" (it is not "A", because the
'arguments' field of <package> is thunked).

Is this intentional? I'm not really sure how (or whether) this should be
changed though.

Best,
David

[test-parameter-package.scm (text/plain, inline)]

(use-modules
 (guix build-system trivial)
 (guix gexp)
 (guix derivations)
 (guix packages)
 (guix store)
 (gnu packages base))

(define %param
  (make-parameter "A"))

(define testp
  (package
    (name "testp")
    (version "0")
    (source #f)
    (build-system trivial-build-system)
    (arguments
     (list
      #:builder
      #~(let ((port (open-file (string-append #$output) "w")))
          (display (string-append #$(%param) "\n") port)
          (close-port port))))
    (home-page #f)
    (synopsis #f)
    (description #f)
    (license #f)))

(%param "B")

(define obj
  (with-parameters
      ((%param "C"))
    testp))

(%param "D")

obj

Message #8 received at 75879 <at> debbugs.gnu.org (full text, mbox):

From: David Elsing <david.elsing <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 75879 <at> debbugs.gnu.org
Subject: Re: [bug#70895] [PATCH] grafts: Only compute necessary graft
 derivations.
Date: Fri, 31 Jan 2025 23:02:47 +0000

Hello,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Uh, looks like this is a real bug.  I’m surprised because we do have
> tests for that in ‘tests/gexp.scm’ (and it’s actually used in a few
> important places), but maybe they’re not exercising the right thing.

Yes indeed, 'with-parameters' is tested for %current-system and
%current-target-system, which are evaluated earlier as a special case in
the gexp-compiler of <parameterized>, and an additional parameter is
only tested by immediately evaluating it.

Best,
David

Message #13 received at 75879 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: David Elsing <david.elsing <at> posteo.net>
Cc: dev <at> jpoiret.xyz, zimon.toutoune <at> gmail.com, othacehe <at> gnu.org, me <at> tobias.gr,
 guix <at> cbaines.net, 75879 <at> debbugs.gnu.org
Subject: Re: bug#75879: with-parameters does not work generally for packages
Date: Sat, 15 Feb 2025 21:59:06 +0100

Hi David,

David Elsing <david.elsing <at> posteo.net> skribis:

> I noticed that 'with-parameters' from (guix gexp) does not work with
> Guile parameters used in package definitions. They are still set
> in 'lower-object', but not anymore when the monadic procedure returned
> by 'lower-object' is evaluated.
>
> Attached is an example for a package wrapped by 'with-parameters', which
> results in a file with "D" instead of "C" (it is not "A", because the
> 'arguments' field of <package> is thunked).
>
> Is this intentional? I'm not really sure how (or whether) this should be
> changed though.

Something just came to mind: the object cache.  The cache is keyed by
object + system + target + grafts?; if there’s anything that influences
what the object lowers to, changes are the object->derivation mapping is
already cached and that other thing will be ignored.

That’s a problem generally speaking with using ‘with-parameters’ with
parameters other than ‘%current-system’, ‘%current-target-system’, and
‘%graft?’.

I wonder if it’s the only thing at play here though.

Ludo’.

Message #16 received at 75879 <at> debbugs.gnu.org (full text, mbox):

From: David Elsing <david.elsing <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: dev <at> jpoiret.xyz, zimon.toutoune <at> gmail.com, othacehe <at> gnu.org, me <at> tobias.gr,
 guix <at> cbaines.net, 75879 <at> debbugs.gnu.org
Subject: Re: bug#75879: with-parameters does not work generally for packages
Date: Mon, 17 Feb 2025 18:59:19 +0000

[Message part 1 (text/plain, inline)]

Hi Ludo',

Ludovic Courtès <ludo <at> gnu.org> writes:

> Something just came to mind: the object cache.  The cache is keyed by
> object + system + target + grafts?; if there’s anything that influences
> what the object lowers to, changes are the object->derivation mapping is
> already cached and that other thing will be ignored.

Oh that's true, this is an additional problem. Would it make sense to
add the parameters passed to `with-parameters' to the keys used in
`mcached'? Then the parameters would have to be passed around quite a
bit however...
Alternatively, would it be possible to query the fluids with
`current-dynamic-state' instead and use that for the keys in `mcached'?
I don't see any way to inspect a dynamic state though.

> That’s a problem generally speaking with using ‘with-parameters’ with
> parameters other than ‘%current-system’, ‘%current-target-system’, and
> ‘%graft?’.

I think `with-parameters' only works for `%current-system' and
`%current-target-system', not for `%graft?': When setting `%graft?' to
#f in the second `with-parameters' wrapping in
`graft-derivation/shallow', it still evaluates to the same derivation as
with `%graft?' set to #t.

> I wonder if it’s the only thing at play here though.

No, attached is a simplified example without a package using a record
(named <test>) with a (thunked) field. Removing the `mcached' calls in
(guix gexp) results in the same derivation. I think this is because
`lower-object' returns a monadic procedure, which does not keep the
fluids set by `with-parameters', so when the (thunked) field is
evaluated, the parameter has changed.

Maybe it would be good to pass the parameters from the gexp compiler of
<parameterized> to `lower-object' as additional argument and use
`with-fluids*' there (just before `lower' is called)? This worked for
the <test> record defined in the attachment, but not for a package, I
guess because of the expander? I'm not sure how to pass the parameters
in that case though.

Best,
David

[test-parameter-thunked.scm (text/plain, inline)]

(use-modules
 (guix gexp)
 (guix monads)
 (guix records)
 (guix store)
 (guix utils))

(define-record-type* <test>
  test make-test
  test?
  this-test
  (field test-field (thunked)))

(define* (test->derivation test)
  (with-monad %store-monad
   (gexp->derivation
    "test"
    #~(let ((port (open-file #$output "w")))
        (display
         (string-append #$(test-field test) "\n") port)
        (close-port port)))))

(define-gexp-compiler (test-compiler (test <test>) system target)
  (test->derivation test))

(define %param
  (make-parameter "A"))

(define testvalue
  (test
   (field (%param))))

(%param "B")

(define testvalue2
  (with-parameters
      ((%param "C"))
    testvalue))

(%param "D")

testvalue2

Message #19 received at 75879 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: David Elsing <david.elsing <at> posteo.net>
Cc: dev <at> jpoiret.xyz, zimon.toutoune <at> gmail.com, othacehe <at> gnu.org, me <at> tobias.gr,
 guix <at> cbaines.net, 75879 <at> debbugs.gnu.org
Subject: Re: bug#75879: with-parameters does not work generally for packages
Date: Sat, 22 Feb 2025 16:11:18 +0100

Hi David,

David Elsing <david.elsing <at> posteo.net> skribis:

> Oh that's true, this is an additional problem. Would it make sense to
> add the parameters passed to `with-parameters' to the keys used in
> `mcached'? Then the parameters would have to be passed around quite a
> bit however...
> Alternatively, would it be possible to query the fluids with
> `current-dynamic-state' instead and use that for the keys in `mcached'?
> I don't see any way to inspect a dynamic state though.

Problem is that the key must remain “small” so that computing its hash
is fast.  It cannot grow further than its current size, I’m afraid.

Ideally the key would contain only parameters known to be influential,
but we cannot tell which are influential and which are not among those
in the current dynamic state.

> I think `with-parameters' only works for `%current-system' and
> `%current-target-system', not for `%graft?': When setting `%graft?' to
> #f in the second `with-parameters' wrapping in
> `graft-derivation/shallow', it still evaluates to the same derivation as
> with `%graft?' set to #t.

The patch I just sent¹ fixes this issue.

The solution was to do the same as ‘mparameterize’ to ensure that the
extent during which parameters are set is the correct one given this is
in a monadic context.

Let me know what you think!

Thanks,
Ludo’.

¹ “gexp: ‘with-parameters’ properly handles ‘%graft?’.”
  I cannot see it yet in Debbugs.

Message #22 received at 75879 <at> debbugs.gnu.org (full text, mbox):

From: David Elsing <david.elsing <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: dev <at> jpoiret.xyz, zimon.toutoune <at> gmail.com, othacehe <at> gnu.org, me <at> tobias.gr,
 guix <at> cbaines.net, 75879 <at> debbugs.gnu.org
Subject: Re: bug#75879: with-parameters does not work generally for packages
Date: Tue, 25 Feb 2025 23:05:20 +0000

Hi Ludo',

Ludovic Courtès <ludo <at> gnu.org> writes:

> Problem is that the key must remain “small” so that computing its hash
> is fast.  It cannot grow further than its current size, I’m afraid.

What if the hash is calculated in `compile-parameterized' instead (as
that is the only supported way to set the parameters) and passed to
`lower-object'?

> Ideally the key would contain only parameters known to be influential,
> but we cannot tell which are influential and which are not among those
> in the current dynamic state.

Yes that's true, I think the parameters affecting the key should only be
the ones passed to `with-parameters', other parameters would then be
assumed to be constant.

> The solution was to do the same as ‘mparameterize’ to ensure that the
> extent during which parameters are set is the correct one given this is
> in a monadic context.

I think that's a good idea, but I'm not so convinced it makes sense for
a general monad [1]. An alternative solution [2] using `with-fluids*'
has a different problem however, as it doesn't seem to be compatible
with prompts.

Thanks,
David

[1] https://issues.guix.gnu.org/76485#2
[2] https://issues.guix.gnu.org/76485#3

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.