GNU bug report logs -
#53668
Updating substitutes on LAN hosts dies unexpectedly
Previous Next
Reported by: Simon Streit <simon <at> netpanic.org>
Date: Mon, 31 Jan 2022 18:38:02 UTC
Severity: normal
Done: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 53668 in the body.
You can then email your comments to 53668 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Mon, 31 Jan 2022 18:38:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Simon Streit <simon <at> netpanic.org>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Mon, 31 Jan 2022 18:38:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hello,
quite often, and quite randomly I run into this situation that whenever
Guix tries to rebuild a profile, and sometimes while downloading from
local Guix hosts sharing their store items, the process will crash with
the following error:
--8<---------------cut here---------------start------------->8---
~ $1 reconfigure
substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
substitute: In ice-9/boot-9.scm:
substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In unknown file:
substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
substitute: In ice-9/boot-9.scm:
substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
substitute: In ice-9/eval.scm:
substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
substitute: In guix/ui.scm:
substitute: 2206:7 13 (run-guix . _)
substitute: 2169:10 12 (run-guix-command _ . _)
substitute: In ice-9/boot-9.scm:
substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In guix/scripts/substitute.scm:
substitute: 757:18 9 (_)
substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
substitute: In guix/substitutes.scm:
substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
substitute: In ice-9/boot-9.scm:
substitute: 1685:16 4 (raise-exception _ #:continuable? _)
substitute: 1685:16 3 (raise-exception _ #:continuable? _)
substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
substitute: 1685:16 1 (raise-exception _ #:continuable? _)
substitute: 1685:16 0 (raise-exception _ #:continuable? _)
substitute:
substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
substitute: Wrong type (expecting exact integer): #f
guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
--8<---------------cut here---------------end--------------->8---
Unfortunately this crash happens at random. Other times it goes
through. Current checkout is ff14bc60e56fb0c6636da1da9a850b7b04abb367,
which isn't the most current, I know. I've been observing this behavior
since some time now, and haven't figured out what the reason is behind
it yet. The error message looks similar to [1].
The way this error appears is, that I usually have one host that I
upgrade first, and then share the checkout and the store between hosts
to speed up the upgrading process locally. Unfortunately the updater
will crash randomly whenever the host starts scanning other hosts that
are found through mDNS. Sometimes this happens while fetching new
packages into a profile.
I've set up publishing:
--8<---------------cut here---------------start------------->8---
(service guix-publish-service-type
(guix-publish-configuration (host "0.0.0.0")
(port 3000)
(ttl #f)
(advertise? #t)))
--8<---------------cut here---------------end--------------->8---
and of course host discovery in guix-service-type too.
Kind regards
Simon
[1] https://issues.guix.gnu.org/52464
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Mon, 31 Jan 2022 20:24:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 53668 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Simon Streit <simon <at> netpanic.org> skribis:
> Hello,
>
> quite often, and quite randomly I run into this situation that whenever
> Guix tries to rebuild a profile, and sometimes while downloading from
> local Guix hosts sharing their store items, the process will crash with
> the following error:
>
> ~ $1 reconfigure
> substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In unknown file:
> substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
> substitute: In ice-9/boot-9.scm:
> substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> substitute: In ice-9/eval.scm:
> substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
> substitute: In guix/ui.scm:
> substitute: 2206:7 13 (run-guix . _)
> substitute: 2169:10 12 (run-guix-command _ . _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In guix/scripts/substitute.scm:
> substitute: 757:18 9 (_)
> substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
> substitute: In guix/substitutes.scm:
> substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
> substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
> substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1685:16 4 (raise-exception _ #:continuable? _)
> substitute: 1685:16 3 (raise-exception _ #:continuable? _)
> substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
> substitute: 1685:16 1 (raise-exception _ #:continuable? _)
> substitute: 1685:16 0 (raise-exception _ #:continuable? _)
> substitute:
> substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> substitute: Wrong type (expecting exact integer): #f
> guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
>
>
> Unfortunately this crash happens at random. Other times it goes
> through. Current checkout is ff14bc60e56fb0c6636da1da9a850b7b04abb367,
> which isn't the most current, I know. I've been observing this behavior
> since some time now, and haven't figured out what the reason is behind
> it yet. The error message looks similar to [1].
>
> The way this error appears is, that I usually have one host that I
> upgrade first, and then share the checkout and the store between hosts
> to speed up the upgrading process locally. Unfortunately the updater
> will crash randomly whenever the host starts scanning other hosts that
> are found through mDNS. Sometimes this happens while fetching new
> packages into a profile.
>
> I've set up publishing:
>
> (service guix-publish-service-type
> (guix-publish-configuration (host "0.0.0.0")
> (port 3000)
> (ttl #f)
> (advertise? #t)))
>
> and of course host discovery in guix-service-type too.
>
>
> Kind regards
> Simon
>
>
> [1] https://issues.guix.gnu.org/52464
Hi,
I have also seen this kind of crashes on my LAN.
I don't know what causes this issue, but I noticed that it has happened
less often since I activated the cache for the guix-publish service.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Mon, 31 Jan 2022 21:39:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Hello Guillaume,
Guillaume Le Vaillant <glv <at> posteo.net> writes:
> I have also seen this kind of crashes on my LAN.
> I don't know what causes this issue, but I noticed that it has happened
> less often since I activated the cache for the guix-publish service.
I just activated the cache. It hasn't crashed so far.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Mon, 31 Jan 2022 21:56:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Simon Streit <simon <at> netpanic.org> writes:
> I just activated the cache. It hasn't crashed so far.
No, it did crash again.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Tue, 01 Feb 2022 02:07:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Hi Simon,
Simon Streit <simon <at> netpanic.org> writes:
> Hello,
>
> quite often, and quite randomly I run into this situation that whenever
> Guix tries to rebuild a profile, and sometimes while downloading from
> local Guix hosts sharing their store items, the process will crash with
> the following error:
>
> ~ $1 reconfigure
> substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In unknown file:
> substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
> substitute: In ice-9/boot-9.scm:
> substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> substitute: In ice-9/eval.scm:
> substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
> substitute: In guix/ui.scm:
> substitute: 2206:7 13 (run-guix . _)
> substitute: 2169:10 12 (run-guix-command _ . _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
> substitute: In guix/scripts/substitute.scm:
> substitute: 757:18 9 (_)
> substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
> substitute: In guix/substitutes.scm:
> substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
> substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
> substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
> substitute: In ice-9/boot-9.scm:
> substitute: 1685:16 4 (raise-exception _ #:continuable? _)
> substitute: 1685:16 3 (raise-exception _ #:continuable? _)
> substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
> substitute: 1685:16 1 (raise-exception _ #:continuable? _)
> substitute: 1685:16 0 (raise-exception _ #:continuable? _)
> substitute:
> substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> substitute: Wrong type (expecting exact integer): #f
> guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
I suffer from this routinely on my slow HDD machine. I attribute the
failure to some timeout that isn't reported correctly by guile-ssh.
Perhaps related to
https://github.com/artyom-poptsov/guile-ssh/issues/29; there's a new
update of guile-ssh to try that Artyom has put up on guix-patches; we
should test it.
Thanks,
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Wed, 11 May 2022 14:59:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Hello!
Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
> Simon Streit <simon <at> netpanic.org> writes:
>
>> Hello,
>>
>> quite often, and quite randomly I run into this situation that whenever
>> Guix tries to rebuild a profile, and sometimes while downloading from
>> local Guix hosts sharing their store items, the process will crash with
>> the following error:
>>
>> ~ $1 reconfigure
>> substitute: updating substitutes from 'http://192.168.0.157:3000'... 56.3%Backtrace:
>> substitute: In ice-9/boot-9.scm:
>> substitute: 1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
>> substitute: In unknown file:
>> substitute: 16 (apply-smob/0 #<thunk 7f309a1caf60>)
>> substitute: In ice-9/boot-9.scm:
>> substitute: 724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
>> substitute: In ice-9/eval.scm:
>> substitute: 619:8 14 (_ #(#(#<directory (guile-user) 7f309a1c4c80>)))
>> substitute: In guix/ui.scm:
>> substitute: 2206:7 13 (run-guix . _)
>> substitute: 2169:10 12 (run-guix-command _ . _)
>> substitute: In ice-9/boot-9.scm:
>> substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
>> substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
>> substitute: In guix/scripts/substitute.scm:
>> substitute: 757:18 9 (_)
>> substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
>> substitute: In guix/substitutes.scm:
>> substitute: 365:27 7 (lookup-narinfos/diverse _ _ #<procedure 7f309690d320 …> …)
>> substitute: 322:31 6 (lookup-narinfos _ _ #:open-connection _ # _)
>> substitute: 245:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
>> substitute: In ice-9/boot-9.scm:
>> substitute: 1685:16 4 (raise-exception _ #:continuable? _)
>> substitute: 1685:16 3 (raise-exception _ #:continuable? _)
>> substitute: 1780:13 2 (_ #<&compound-exception components: (#<&assertion-fail…>)
>> substitute: 1685:16 1 (raise-exception _ #:continuable? _)
>> substitute: 1685:16 0 (raise-exception _ #:continuable? _)
>> substitute:
>> substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>> substitute: Wrong type (expecting exact integer): #f
>> guix system: error: `/gnu/store/kcc8zh1fhp05wgw2m48w3gk228j39f5q-guix-1.3.0-21.e427593/bin/guix substitute' died unexpectedly
>
> I suffer from this routinely on my slow HDD machine. I attribute the
> failure to some timeout that isn't reported correctly by guile-ssh.
Guile-SSH is not involved here though.
However, this may well be this ‘guix publish’ bug that was recently
fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
replies:
https://issues.guix.gnu.org/54723
Could you confirm?
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Wed, 11 May 2022 15:03:03 GMT)
Full text and
rfc822 format available.
Message #23 received at 53668 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
> However, this may well be this ‘guix publish’ bug that was recently
> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
> replies:
>
> https://issues.guix.gnu.org/54723
>
> Could you confirm?
>
> Ludo’.
Even then, the error message says ‘wrong type error ...’, shouldn't it
say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
Greetings,
Maxime.
[signature.asc (application/pgp-signature, inline)]
Reply sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
You have taken responsibility.
(Wed, 11 May 2022 15:07:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Simon Streit <simon <at> netpanic.org>
:
bug acknowledged by developer.
(Wed, 11 May 2022 15:07:02 GMT)
Full text and
rfc822 format available.
Message #28 received at 53668-done <at> debbugs.gnu.org (full text, mbox):
Hi Ludovic,
Ludovic Courtès <ludo <at> gnu.org> writes:
[...]
> However, this may well be this ‘guix publish’ bug that was recently
> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
> replies:
>
> https://issues.guix.gnu.org/54723
>
> Could you confirm?
I've reconfigured my offload machine and haven't seen the issue yet.
Tentatively closing.
Thank you for investigating and fixing the issue!
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Thu, 12 May 2022 01:18:02 GMT)
Full text and
rfc822 format available.
Message #31 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Hi Maxime,
Maxime Devos <maximedevos <at> telenet.be> writes:
> Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
>> However, this may well be this ‘guix publish’ bug that was recently
>> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
>> replies:
>>
>> https://issues.guix.gnu.org/54723
>>
>> Could you confirm?
>>
>> Ludo’.
>
> Even then, the error message says ‘wrong type error ...’, shouldn't it
> say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
I agree that a more accurate error message would have been useful. If
you have the bandwidth for a patch, I'd be more than happy reviewing it
:-).
Thanks,
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Thu, 12 May 2022 08:22:01 GMT)
Full text and
rfc822 format available.
Message #34 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Hi,
Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
> Maxime Devos <maximedevos <at> telenet.be> writes:
>
>> Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
>>> However, this may well be this ‘guix publish’ bug that was recently
>>> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
>>> replies:
>>>
>>> https://issues.guix.gnu.org/54723
>>>
>>> Could you confirm?
>>>
>>> Ludo’.
>>
>> Even then, the error message says ‘wrong type error ...’, shouldn't it
>> say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
>
> I agree that a more accurate error message would have been useful. If
> you have the bandwidth for a patch, I'd be more than happy reviewing it
> :-).
HTTP response parsing happens in (web response) though, not in Guix, so
this is where things should be patch.
Currently that code pretty much expects it’s talking to a well-behaved
HTTP server. Changing it to gracefully deal with any kind of garbage
might turn out to be quite an effort.
Food for thought!
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#53668
; Package
guix
.
(Thu, 12 May 2022 13:48:02 GMT)
Full text and
rfc822 format available.
Message #37 received at 53668 <at> debbugs.gnu.org (full text, mbox):
Hello,
Ludovic Courtès <ludo <at> gnu.org> writes:
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> Maxime Devos <maximedevos <at> telenet.be> writes:
>>
>>> Ludovic Courtès schreef op wo 11-05-2022 om 16:58 [+0200]:
>>>> However, this may well be this ‘guix publish’ bug that was recently
>>>> fixed, whereby ‘guix publish’ without ‘--cache’ would send bogus HTTP
>>>> replies:
>>>>
>>>> https://issues.guix.gnu.org/54723
>>>>
>>>> Could you confirm?
>>>>
>>>> Ludo’.
>>>
>>> Even then, the error message says ‘wrong type error ...’, shouldn't it
>>> say something like ‘bogus HTTP’ or ‘bogus narinfo’ or such instead?
>>
>> I agree that a more accurate error message would have been useful. If
>> you have the bandwidth for a patch, I'd be more than happy reviewing it
>> :-).
>
> HTTP response parsing happens in (web response) though, not in Guix, so
> this is where things should be patch.
>
> Currently that code pretty much expects it’s talking to a well-behaved
> HTTP server. Changing it to gracefully deal with any kind of garbage
> might turn out to be quite an effort.
Hmm. I see. Maxime, perhaps an issue can be logged against Guile to at
least remember it exists, to tackle by anyone with the
interest/bandwidth. I think it'd be valuable in the long run.
Thank you!
Maxim
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Fri, 10 Jun 2022 11:24:07 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 293 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.