GNU bug report logs -
#30290
guix-daemon slows to a crawl when a substitute server is offline
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30290 in the body.
You can then email your comments to 30290 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Tue, 30 Jan 2018 03:09:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Tue, 30 Jan 2018 03:09:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
When a substitute server used by guix-daemon is offline, the daemon will
keep attempting to connect to it, even when it shouldn't need any data
(ran 'sudo guix system reconfigure my-config.scm' multiple times in a
row.
With the disconnected server (bayfront in my case), that command would
take close to 8 minutes, with many system calls like:
--8<---------------cut here---------------start------------->8---
connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
--8<---------------cut here---------------end--------------->8---
which wasted 5 seconds each time.
After removing this server from my substitute servers list, the same
operation (system reconfigure) is 8 times faster (1 minute).
Suggestion: the daemon should stop trying to use the offline substitute
server after trying for X times, and print a warning about it.
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Thu, 03 Dec 2020 00:26:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi Maxim,
On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
> When a substitute server used by guix-daemon is offline, the daemon will
> keep attempting to connect to it, even when it shouldn't need any data
> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> row.
>
> With the disconnected server (bayfront in my case), that command would
> take close to 8 minutes, with many system calls like:
>
> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>
> which wasted 5 seconds each time.
>
> After removing this server from my substitute servers list, the same
> operation (system reconfigure) is 8 times faster (1 minute).
>
> Suggestion: the daemon should stop trying to use the offline substitute
> server after trying for X times, and print a warning about it.
This looks like as a wishlist, right? Do it make sense to include such
feature to the recent discussions about the revamp of offloading,
Cuirass, publish, etc.
All the best,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Sat, 19 Dec 2020 03:05:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi!
zimoun <zimon.toutoune <at> gmail.com> writes:
> Hi Maxim,
>
> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
>> When a substitute server used by guix-daemon is offline, the daemon will
>> keep attempting to connect to it, even when it shouldn't need any data
>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>> row.
>>
>> With the disconnected server (bayfront in my case), that command would
>> take close to 8 minutes, with many system calls like:
>>
>> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>
>> which wasted 5 seconds each time.
>>
>> After removing this server from my substitute servers list, the same
>> operation (system reconfigure) is 8 times faster (1 minute).
>>
>> Suggestion: the daemon should stop trying to use the offline substitute
>> server after trying for X times, and print a warning about it.
>
> This looks like as a wishlist, right? Do it make sense to include such
> feature to the recent discussions about the revamp of offloading,
> Cuirass, publish, etc.
To me it's an issue more than a feature request, especially in a build
farm setting; having a substitute machine down shouldn't cause a slow
down for as long as it's down!
I'm not sure if the recent offloading work that Mathieu did touched that
topic. I'd need to test the scenario. Perhaps a system test would be
useful.
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Tue, 22 Dec 2020 15:17:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi,
Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
> When a substitute server used by guix-daemon is offline, the daemon will
> keep attempting to connect to it, even when it shouldn't need any data
> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> row.
>
> With the disconnected server (bayfront in my case), that command would
> take close to 8 minutes, with many system calls like:
>
> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>
> which wasted 5 seconds each time.
Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
(Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
substitute --query’ process won’t retry connections to an unreachable
host.
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Mon, 28 Dec 2020 12:20:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 30290 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Tue, Dec 22, 2020 at 04:16:08PM +0100, Ludovic Courtès wrote:
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
> > When a substitute server used by guix-daemon is offline, the daemon will
> > keep attempting to connect to it, even when it shouldn't need any data
> > (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> > row.
> >
> > With the disconnected server (bayfront in my case), that command would
> > take close to 8 minutes, with many system calls like:
> >
> > connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
> >
> > which wasted 5 seconds each time.
>
> Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
> (Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
> substitute --query’ process won’t retry connections to an unreachable
> host.
>
> Ludo’.
>
Occasionally my internet drops itself, and I find I'm left forever
waiting for a timeout to see what sources I have cached locally.
--
Efraim Flashner <efraim <at> flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Wed, 24 Mar 2021 21:59:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi,
On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
> When a substitute server used by guix-daemon is offline, the daemon will
> keep attempting to connect to it, even when it shouldn't need any data
> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> row.
>
> With the disconnected server (bayfront in my case), that command would
> take close to 8 minutes, with many system calls like:
>
> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>
> which wasted 5 seconds each time.
>
> After removing this server from my substitute servers list, the same
> operation (system reconfigure) is 8 times faster (1 minute).
>
> Suggestion: the daemon should stop trying to use the offline substitute
> server after trying for X times, and print a warning about it.
What is the status of this bug? Especially with the recent additions in
Cuirass?
Is it still an issue? Is some timeout still happening?
Well, in summary, the 3 relevant messages are:
-------------------- Start of forwarded message --------------------
From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Date: Fri, 18 Dec 2020 22:04:04 -0500
I'm not sure if the recent offloading work that Mathieu did touched that
topic. I'd need to test the scenario. Perhaps a system test would be
useful.
-------------------- End of forwarded message --------------------
-------------------- Start of forwarded message --------------------
From: Ludovic Courtès <ludo <at> gnu.org>
Date: Tue, 22 Dec 2020 16:16:08 +0100
Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
(Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
substitute --query’ process won’t retry connections to an unreachable
host.
-------------------- End of forwarded message --------------------
-------------------- Start of forwarded message --------------------
Date: Mon, 28 Dec 2020 14:19:02 +0200
From: Efraim Flashner <efraim <at> flashner.co.il>
Occasionally my internet drops itself, and I find I'm left forever
waiting for a timeout to see what sources I have cached locally.
-------------------- End of forwarded message --------------------
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Wed, 09 Jun 2021 21:44:03 GMT)
Full text and
rfc822 format available.
Message #23 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi,
On Wed, 24 Mar 2021 at 22:55, zimoun <zimon.toutoune <at> gmail.com> wrote:
> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
>> When a substitute server used by guix-daemon is offline, the daemon will
>> keep attempting to connect to it, even when it shouldn't need any data
>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>> row.
>>
>> With the disconnected server (bayfront in my case), that command would
>> take close to 8 minutes, with many system calls like:
>>
>> connect(14, {sa_family=AF_INET, sin_port=htons(443),
>> sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>
>> which wasted 5 seconds each time.
>>
>> After removing this server from my substitute servers list, the same
>> operation (system reconfigure) is 8 times faster (1 minute).
>>
>> Suggestion: the daemon should stop trying to use the offline substitute
>> server after trying for X times, and print a warning about it.
[...]
> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
> Date: Fri, 18 Dec 2020 22:04:04 -0500 (24 weeks, 4 days, 18 hours ago)
>
> I'm not sure if the recent offloading work that Mathieu did touched that
> topic. I'd need to test the scenario. Perhaps a system test would be
> useful.
> ----------
>
> From: Ludovic Courtès <ludo <at> gnu.org>
> Date: Tue, 22 Dec 2020 16:16:08 +0100
> Date: Tue, 22 Dec 2020 16:16:08 +0100 (24 weeks, 1 day, 6 hours ago)
>
> Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
> (Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
> substitute --query’ process won’t retry connections to an unreachable
> host.
> ----------
>
> From: Efraim Flashner <efraim <at> flashner.co.il>
> Date: Mon, 28 Dec 2020 14:19:02 +0200
> Date: Mon, 28 Dec 2020 14:19:02 +0200 (23 weeks, 2 days, 9 hours ago)
>
> Occasionally my internet drops itself, and I find I'm left forever
> waiting for a timeout to see what sources I have cached locally.
> ----------
What is the current stats of this bug? Is it still happening with the
recent improvements of Cuirass?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Tue, 13 Jul 2021 09:03:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi,
What is the status of this old bug#30290 [1]?
1: <http://issues.guix.gnu.org/issue/30290>
On Wed, 09 Jun 2021 at 23:34, zimoun <zimon.toutoune <at> gmail.com> wrote:
> On Wed, 24 Mar 2021 at 22:55, zimoun <zimon.toutoune <at> gmail.com> wrote:
>> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>> wrote:
>>> When a substitute server used by guix-daemon is offline, the daemon will
>>> keep attempting to connect to it, even when it shouldn't need any data
>>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>>> row.
>>>
>>> With the disconnected server (bayfront in my case), that command would
>>> take close to 8 minutes, with many system calls like:
>>>
>>> connect(14, {sa_family=AF_INET, sin_port=htons(443),
>>> sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>>
>>> which wasted 5 seconds each time.
>>>
>>> After removing this server from my substitute servers list, the same
>>> operation (system reconfigure) is 8 times faster (1 minute).
>>>
>>> Suggestion: the daemon should stop trying to use the offline substitute
>>> server after trying for X times, and print a warning about it.
>
> [...]
>
>> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>> Date: Fri, 18 Dec 2020 22:04:04 -0500 (24 weeks, 4 days, 18 hours ago)
>>
>> I'm not sure if the recent offloading work that Mathieu did touched that
>> topic. I'd need to test the scenario. Perhaps a system test would be
>> useful.
>> ----------
>>
>> From: Ludovic Courtès <ludo <at> gnu.org>
>> Date: Tue, 22 Dec 2020 16:16:08 +0100
>> Date: Tue, 22 Dec 2020 16:16:08 +0100 (24 weeks, 1 day, 6 hours ago)
>>
>> Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
>> (Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
>> substitute --query’ process won’t retry connections to an unreachable
>> host.
>> ----------
>>
>> From: Efraim Flashner <efraim <at> flashner.co.il>
>> Date: Mon, 28 Dec 2020 14:19:02 +0200
>> Date: Mon, 28 Dec 2020 14:19:02 +0200 (23 weeks, 2 days, 9 hours ago)
>>
>> Occasionally my internet drops itself, and I find I'm left forever
>> waiting for a timeout to see what sources I have cached locally.
>> ----------
>
> What is the current stats of this bug? Is it still happening with the
> recent improvements of Cuirass?
After reading all this, I think this bug can be closed. WDYT?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Wed, 18 Aug 2021 11:37:01 GMT)
Full text and
rfc822 format available.
Message #29 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi Maxim,
Reading the discussion…
On Tue, 13 Jul 2021 at 10:49, zimoun <zimon.toutoune <at> gmail.com> wrote:
> What is the status of this old bug#30290 [1]?
>
> 1: <http://issues.guix.gnu.org/issue/30290>
>
> On Wed, 09 Jun 2021 at 23:34, zimoun <zimon.toutoune <at> gmail.com> wrote:
>> On Wed, 24 Mar 2021 at 22:55, zimoun <zimon.toutoune <at> gmail.com> wrote:
>>> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>>> wrote:
>>>> When a substitute server used by guix-daemon is offline, the daemon will
>>>> keep attempting to connect to it, even when it shouldn't need any data
>>>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>>>> row.
>>>>
>>>> With the disconnected server (bayfront in my case), that command would
>>>> take close to 8 minutes, with many system calls like:
>>>>
>>>> connect(14, {sa_family=AF_INET, sin_port=htons(443),
>>>> sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>>>
>>>> which wasted 5 seconds each time.
>>>>
>>>> After removing this server from my substitute servers list, the same
>>>> operation (system reconfigure) is 8 times faster (1 minute).
>>>>
>>>> Suggestion: the daemon should stop trying to use the offline substitute
>>>> server after trying for X times, and print a warning about it.
>>
>> [...]
>>
>>> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>>> Date: Fri, 18 Dec 2020 22:04:04 -0500 (24 weeks, 4 days, 18 hours ago)
>>>
>>> I'm not sure if the recent offloading work that Mathieu did touched that
>>> topic. I'd need to test the scenario. Perhaps a system test would be
>>> useful.
>>> ----------
>>>
>>> From: Ludovic Courtès <ludo <at> gnu.org>
>>> Date: Tue, 22 Dec 2020 16:16:08 +0100
>>> Date: Tue, 22 Dec 2020 16:16:08 +0100 (24 weeks, 1 day, 6 hours ago)
>>>
>>> Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
>>> (Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
>>> substitute --query’ process won’t retry connections to an unreachable
>>> host.
>>> ----------
>>>
>>> From: Efraim Flashner <efraim <at> flashner.co.il>
>>> Date: Mon, 28 Dec 2020 14:19:02 +0200
>>> Date: Mon, 28 Dec 2020 14:19:02 +0200 (23 weeks, 2 days, 9 hours ago)
>>>
>>> Occasionally my internet drops itself, and I find I'm left forever
>>> waiting for a timeout to see what sources I have cached locally.
>>> ----------
>>
>> What is the current stats of this bug? Is it still happening with the
>> recent improvements of Cuirass?
>
> After reading all this, I think this bug can be closed. WDYT?
…I appears to me that this bug could be close. WDYT?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Wed, 18 Aug 2021 13:19:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi,
And sorry for failing to produce a reply earlier :-).
zimoun <zimon.toutoune <at> gmail.com> writes:
[...]
>>> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>>> Date: Fri, 18 Dec 2020 22:04:04 -0500 (24 weeks, 4 days, 18 hours ago)
>>>
>>> I'm not sure if the recent offloading work that Mathieu did touched that
>>> topic. I'd need to test the scenario. Perhaps a system test would be
>>> useful.
>>> ----------
>>>
>>> From: Ludovic Courtès <ludo <at> gnu.org>
>>> Date: Tue, 22 Dec 2020 16:16:08 +0100
>>> Date: Tue, 22 Dec 2020 16:16:08 +0100 (24 weeks, 1 day, 6 hours ago)
>>>
>>> Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
>>> (Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
>>> substitute --query’ process won’t retry connections to an unreachable
>>> host.
>>> ----------
>>>
>>> From: Efraim Flashner <efraim <at> flashner.co.il>
>>> Date: Mon, 28 Dec 2020 14:19:02 +0200
>>> Date: Mon, 28 Dec 2020 14:19:02 +0200 (23 weeks, 2 days, 9 hours ago)
>>>
>>> Occasionally my internet drops itself, and I find I'm left forever
>>> waiting for a timeout to see what sources I have cached locally.
>>> ----------
>>
>> What is the current stats of this bug? Is it still happening with the
>> recent improvements of Cuirass?
>
> After reading all this, I think this bug can be closed. WDYT?
Were you able to replay a scenario in which a substitute server is made
unreachable? That's the information that I'd like to have/see before
closing. I don't come across unreachable substitute servers often, and
can't think of a way to easily test this.
I could make it hang by dropping the input/output connections with
iptables to a remote guix publish server, but then SSH also hangs, so
perhaps that's expected.
I'll try to configure a couple local machines to act as publish servers,
and disconnect them from the network to see what happens.
Thanks,
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Thu, 19 Aug 2021 01:56:01 GMT)
Full text and
rfc822 format available.
Message #35 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hello,
Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
> Hi,
>
> And sorry for failing to produce a reply earlier :-).
>
> zimoun <zimon.toutoune <at> gmail.com> writes:
>
> [...]
>
>>>> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>>>> Date: Fri, 18 Dec 2020 22:04:04 -0500 (24 weeks, 4 days, 18 hours ago)
>>>>
>>>> I'm not sure if the recent offloading work that Mathieu did touched that
>>>> topic. I'd need to test the scenario. Perhaps a system test would be
>>>> useful.
>>>> ----------
>>>>
>>>> From: Ludovic Courtès <ludo <at> gnu.org>
>>>> Date: Tue, 22 Dec 2020 16:16:08 +0100
>>>> Date: Tue, 22 Dec 2020 16:16:08 +0100 (24 weeks, 1 day, 6 hours ago)
>>>>
>>>> Is it still a problem? Commit 4f5234be0378368e6af25925db46612838d25e58
>>>> (Nov. 2019) added a table of unreachable hosts. That way, a ‘guix
>>>> substitute --query’ process won’t retry connections to an unreachable
>>>> host.
>>>> ----------
>>>>
>>>> From: Efraim Flashner <efraim <at> flashner.co.il>
>>>> Date: Mon, 28 Dec 2020 14:19:02 +0200
>>>> Date: Mon, 28 Dec 2020 14:19:02 +0200 (23 weeks, 2 days, 9 hours ago)
>>>>
>>>> Occasionally my internet drops itself, and I find I'm left forever
>>>> waiting for a timeout to see what sources I have cached locally.
>>>> ----------
>>>
>>> What is the current stats of this bug? Is it still happening with the
>>> recent improvements of Cuirass?
>>
>> After reading all this, I think this bug can be closed. WDYT?
>
> Were you able to replay a scenario in which a substitute server is made
> unreachable? That's the information that I'd like to have/see before
> closing. I don't come across unreachable substitute servers often, and
> can't think of a way to easily test this.
>
> I could make it hang by dropping the input/output connections with
> iptables to a remote guix publish server, but then SSH also hangs, so
> perhaps that's expected.
>
> I'll try to configure a couple local machines to act as publish servers,
> and disconnect them from the network to see what happens.
>
> Thanks,
>
> Maxim
I managed to get some problematic behavior from guix substitute:
My test protocole was roughly like this:
1. Setup a 2nd machine (machine B) to act as a substitute server, and
guix pull to the same commit as that of my main machine (machine A).
2. Run guix build -m manifest.scm on machine B (IP: 192.168.10.172).
3. On the machine A, run the command below, explicitly listing machine A
as a substitute URL, along ci.guix.gnu.org. During a download from A,
break the connection (I pulled the wifi USB dongle out):
$ guix build -m ~/stow/guix/manifest.scm --substitute-urls='http://192.168.10.172 https://ci.guix.gnu.org' --no-offload
substitute: updating substitutes from 'http://192.168.10.172:80'... 100.0%
substitute: updating substitutes from 'http://192.168.10.172'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/lxm7brkbrkkv58c4kzlw1lh3wc0bm8wz-gimp-2.10.24.drv
/gnu/store/ddv8jyzwk92nsg1dkv9n3scf6f7w83g5-keepassxc-2.6.6.drv
/gnu/store/xky1y32mccplxsb448ziq68by2mvkdaz-ruby-asciidoctor-2.0.10.drv
/gnu/store/0ph0sjib0d13n2fsl8w9prnky8g5fkvf-ruby-haml-5.0.4.drv
/gnu/store/4dfwfj9qinw4vs6290gdy5qbnqbczm2v-ruby-temple-0.8.2.drv
/gnu/store/b12krypa196yg6gzk2bvrh35i1fg5c7x-ruby-tilt-2.0.10.drv
/gnu/store/131d8193hi1485ylnb9w8jm3jnlv3iyx-ruby-slim-4.1.0.drv
/gnu/store/489nq0jqjby92kv3c6nwrdfqg45l40nw-ruby-sinatra-2.0.8.1.drv
/gnu/store/yay3sa8nnq4j0ixwhp3bxfg5vfisfmf1-sflvault-client-0.9.2-1.8de3902.drv
/gnu/store/2n1xyy0y3nnkrp3mpdifn8r7wf6pzpb0-sflvault-0.9.2-1.8de3902-checkout.drv
/gnu/store/jsyhy4vxzr9yyg66kzk7w28xffyx050c-python-keyring-1.6.1.drv
/gnu/store/kiwn3x2la23f1pa3a5ypsihhc6ja19y5-python-keyring-1.6.1-checkout.drv
The following files will be downloaded:
/gnu/store/2qphwngpvawl6f06d33b2jr18vk1hyc9-module-import-compiled
/gnu/store/r7vsb0vl4y66jbq7b56zmrm60q2507zl-wireshark-3.4.7
/gnu/store/wnzx9anjdkmbnkcg5qdd3j77q1w2j1bd-yelp-3.32.2
/gnu/store/vcxwcwlwhvhxj15ma8ik8lghmz8sb2vq-vinagre-3.22.0
/gnu/store/yg8r6kz95p8v03gz0rglpwzrj21npzzw-spacefm-1.0.6
/gnu/store/bn35x60w72ad59a5pd7gmvxgjwgkqvag-youtube-dl-2021.06.06
/gnu/store/xkn540dzpz75hr9cx19xgd3b1r7vgswi-mpv-0.33.1
/gnu/store/6abwn23grk710qvzvvg1384bs3kc2f8i-linphone-desktop-4.2.5-debug
/gnu/store/4h8ixlh5by2l09vv3rvknmlxv2gm9d6s-linphone-desktop-4.2.5
/gnu/store/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2
/gnu/store/zczjaxs118155n3mx8w91c24izhx0h0f-ruby-asciimath-2.0.1
/gnu/store/zll4p79a29hw95d2gsh4vjdvd856ry4s-ruby-cucumber-html-formatter-7.0.0
[...]
/gnu/store/sa6hvh9bnw73mpplasbjb3idlv71rvcb-gnome-boxes-3.36.6
/gnu/store/6gy957mhm07zaa001avzkv2d8zhjdl5h-poppler-data-0.4.10
/gnu/store/7kwgmhlsy6qal56h3z19anxmw4c7pf35-diffoscope-177
/gnu/store/hxvlcb4wgw0fpyi9ssc4x6f8w3hlng55-gst-plugins-good-1.18.2
/gnu/store/7bqpzvzanmvb4g1g6gqb1jmrw2j8gv3d-gst-plugins-bad-1.18.2
/gnu/store/f8hzmmnp8cm4yqq0y9cf7rgxl05hf423-cheese-3.38.0
substituting /gnu/store/7kwgmhlsy6qal56h3z19anxmw4c7pf35-diffoscope-177...
substituting /gnu/store/ns4n01xgbk6ccvd2z127v71d806rnr6f-inkscape-1.1...
substituting /gnu/store/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2...
substituting /gnu/store/4h8ixlh5by2l09vv3rvknmlxv2gm9d6s-linphone-desktop-4.2.5...
downloading from http://192.168.10.172/nar/zstd/7kwgmhlsy6qal56h3z19anxmw4c7pf35-diffoscope-177 ...
diffoscope-177 10.5MiB/s 00:00 | 128KiB transferreddownloading from http://192.168.10.172/nar/zstd/ns4n01xgbk6ccvd2z127v71d806rnr6f-inkscape-1.1 ...
downloading from http://192.168.10.172/nar/zstd/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2 ...
downloading from http://192.168.10.172/nar/zstd/4h8ixlh5by2l09vv3rvknmlxv2gm9d6s-linphone-desktop-4.2.5 ...
substitution of /gnu/store/4h8ixlh5by2l09vv3rvknmlxv2gm9d6s-linphone-desktop-4.2.5 complete
substituting /gnu/store/7bqpzvzanmvb4g1g6gqb1jmrw2j8gv3d-gst-plugins-bad-1.18.2...
downloading from http://192.168.10.172/nar/zstd/7bqpzvzanmvb4g1g6gqb1jmrw2j8gv3d-gst-plugins-bad-1.18.2 ...
substitution of /gnu/store/7kwgmhlsy6qal56h3z19anxmw4c7pf35-diffoscope-177 complete
substituting /gnu/store/hxvlcb4wgw0fpyi9ssc4x6f8w3hlng55-gst-plugins-good-1.18.2...
downloading from http://192.168.10.172/nar/zstd/hxvlcb4wgw0fpyi9ssc4x6f8w3hlng55-gst-plugins-good-1.18.2 ...
^ It hung up there, waiting indefinitely.
What I would have expected instead, would have been to find out about
the network failure, and retry from the other available substitute URL,
else build locally.
At that time, all the 'substitute' processes are blocked on a read(2)
call, while one of the guix-daemon is also, and 2 others are blocked on
select.
That's not the same as the original report though; let's try to
reproduce that one by running the same command again, while the
substitute server B is still disconnected:
--8<---------------cut here---------------start------------->8---
$ time guix build -m ~/stow/guix/manifest.scm --substitute-urls='http://192.168.10.172 https://ci.guix.gnu.org' --no-offload
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%guix substitute: warning: 192.168.10.172: connection failed: No route to host
substitute:
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/lxm7brkbrkkv58c4kzlw1lh3wc0bm8wz-gimp-2.10.24.drv
/gnu/store/ddv8jyzwk92nsg1dkv9n3scf6f7w83g5-keepassxc-2.6.6.drv
/gnu/store/xky1y32mccplxsb448ziq68by2mvkdaz-ruby-asciidoctor-2.0.10.drv
/gnu/store/0ph0sjib0d13n2fsl8w9prnky8g5fkvf-ruby-haml-5.0.4.drv
/gnu/store/4dfwfj9qinw4vs6290gdy5qbnqbczm2v-ruby-temple-0.8.2.drv
/gnu/store/b12krypa196yg6gzk2bvrh35i1fg5c7x-ruby-tilt-2.0.10.drv
/gnu/store/131d8193hi1485ylnb9w8jm3jnlv3iyx-ruby-slim-4.1.0.drv
/gnu/store/489nq0jqjby92kv3c6nwrdfqg45l40nw-ruby-sinatra-2.0.8.1.drv
/gnu/store/yay3sa8nnq4j0ixwhp3bxfg5vfisfmf1-sflvault-client-0.9.2-1.8de3902.drv
/gnu/store/2n1xyy0y3nnkrp3mpdifn8r7wf6pzpb0-sflvault-0.9.2-1.8de3902-checkout.drv
/gnu/store/jsyhy4vxzr9yyg66kzk7w28xffyx050c-python-keyring-1.6.1.drv
/gnu/store/kiwn3x2la23f1pa3a5ypsihhc6ja19y5-python-keyring-1.6.1-checkout.drv
The following files will be downloaded:
/gnu/store/2qphwngpvawl6f06d33b2jr18vk1hyc9-module-import-compiled
/gnu/store/r7vsb0vl4y66jbq7b56zmrm60q2507zl-wireshark-3.4.7
/gnu/store/wnzx9anjdkmbnkcg5qdd3j77q1w2j1bd-yelp-3.32.2
/gnu/store/vcxwcwlwhvhxj15ma8ik8lghmz8sb2vq-vinagre-3.22.0
/gnu/store/yg8r6kz95p8v03gz0rglpwzrj21npzzw-spacefm-1.0.6
[...]
/gnu/store/zvnnafb7hmiklj8wpvn9qdc85w8rdprl-gnucash-4.2-doc
/gnu/store/rp2ai59zvx5m0k6db0cnkx6nn9n41qjd-gnucash-4.2
/gnu/store/hmy026sjdl489sy3i25r2kz9f70h3awm-gnucash-4.2-python
/gnu/store/1bspzx0103mr17mxhgw0d9zdlgca2psq-spice-gtk-0.37
/gnu/store/bribnmf6djvh1d3rjr2vs5y97141ad97-osinfo-db-20201218
/gnu/store/r1a25sizf07nmh388ri4qybshzlcxbqd-libosinfo-1.7.1
/gnu/store/2z7p7ynamiarxkx4hnk8dk377xqgm3zl-tracker-2.3.5
/gnu/store/458bw9h0f0ybjdqwg4zm5gjjsmfxbalx-webkitgtk-2.32.3
/gnu/store/sa6hvh9bnw73mpplasbjb3idlv71rvcb-gnome-boxes-3.36.6
/gnu/store/6gy957mhm07zaa001avzkv2d8zhjdl5h-poppler-data-0.4.10
/gnu/store/hxvlcb4wgw0fpyi9ssc4x6f8w3hlng55-gst-plugins-good-1.18.2
/gnu/store/7bqpzvzanmvb4g1g6gqb1jmrw2j8gv3d-gst-plugins-bad-1.18.2
/gnu/store/f8hzmmnp8cm4yqq0y9cf7rgxl05hf423-cheese-3.38.0
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substituting /gnu/store/ns4n01xgbk6ccvd2z127v71d806rnr6f-inkscape-1.1...
substituting /gnu/store/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2...
substituting /gnu/store/6abwn23grk710qvzvvg1384bs3kc2f8i-linphone-desktop-4.2.5-debug...
substituting /gnu/store/bribnmf6djvh1d3rjr2vs5y97141ad97-osinfo-db-20201218...
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
guix substitute: error: connect*: No route to host
guix substitute: error: connect*: No route to host
guix substitute: error: connect*: No route to host
guix substitute: warning: 192.168.10.172: connection failed: No route to host
downloading from https://ci.guix.gnu.org/nar/lzip/bribnmf6djvh1d3rjr2vs5y97141ad97-osinfo-db-20201218 ...
osinfo-db-20201218 88KiB 5.9MiB/s 00:00 [############# ] 73.1%substitution of /gnu/store/ns4n01xgbk6ccvd2z127v71d806rnr6f-inkscape-1.1 failed
substitution of /gnu/store/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2 failed
substitution of /gnu/store/6abwn23grk710qvzvvg1384bs3kc2f8i-linphone-desktop-4.2.5-debug failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f16de01c230>
real 1m13.549s
user 0m25.348s
sys 0m0.721s
--8<---------------cut here---------------end--------------->8---
Hmm.
Let's try again,
--8<---------------cut here---------------start------------->8---
$ time guix build -m ~/stow/guix/manifest.scm --substitute-urls='http://192.168.10.172 https://ci.guix.gnu.org' --no-offload
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%guix substitute: warning: 192.168.10.172: connection failed: No route to host
substitute:
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
The following derivations will be built:
/gnu/store/lxm7brkbrkkv58c4kzlw1lh3wc0bm8wz-gimp-2.10.24.drv
/gnu/store/ddv8jyzwk92nsg1dkv9n3scf6f7w83g5-keepassxc-2.6.6.drv
/gnu/store/xky1y32mccplxsb448ziq68by2mvkdaz-ruby-asciidoctor-2.0.10.drv
/gnu/store/0ph0sjib0d13n2fsl8w9prnky8g5fkvf-ruby-haml-5.0.4.drv
/gnu/store/4dfwfj9qinw4vs6290gdy5qbnqbczm2v-ruby-temple-0.8.2.drv
/gnu/store/b12krypa196yg6gzk2bvrh35i1fg5c7x-ruby-tilt-2.0.10.drv
/gnu/store/131d8193hi1485ylnb9w8jm3jnlv3iyx-ruby-slim-4.1.0.drv
/gnu/store/489nq0jqjby92kv3c6nwrdfqg45l40nw-ruby-sinatra-2.0.8.1.drv
/gnu/store/yay3sa8nnq4j0ixwhp3bxfg5vfisfmf1-sflvault-client-0.9.2-1.8de3902.drv
/gnu/store/2n1xyy0y3nnkrp3mpdifn8r7wf6pzpb0-sflvault-0.9.2-1.8de3902-checkout.drv
/gnu/store/jsyhy4vxzr9yyg66kzk7w28xffyx050c-python-keyring-1.6.1.drv
/gnu/store/kiwn3x2la23f1pa3a5ypsihhc6ja19y5-python-keyring-1.6.1-checkout.drv
The following files will be downloaded:
/gnu/store/2qphwngpvawl6f06d33b2jr18vk1hyc9-module-import-compiled
/gnu/store/r7vsb0vl4y66jbq7b56zmrm60q2507zl-wireshark-3.4.7
/gnu/store/wnzx9anjdkmbnkcg5qdd3j77q1w2j1bd-yelp-3.32.2
[...]
/gnu/store/sa6hvh9bnw73mpplasbjb3idlv71rvcb-gnome-boxes-3.36.6
/gnu/store/6gy957mhm07zaa001avzkv2d8zhjdl5h-poppler-data-0.4.10
/gnu/store/hxvlcb4wgw0fpyi9ssc4x6f8w3hlng55-gst-plugins-good-1.18.2
/gnu/store/7bqpzvzanmvb4g1g6gqb1jmrw2j8gv3d-gst-plugins-bad-1.18.2
/gnu/store/f8hzmmnp8cm4yqq0y9cf7rgxl05hf423-cheese-3.38.0
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
substituting /gnu/store/ns4n01xgbk6ccvd2z127v71d806rnr6f-inkscape-1.1...
substituting /gnu/store/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2...
substituting /gnu/store/6abwn23grk710qvzvvg1384bs3kc2f8i-linphone-desktop-4.2.5-debug...
substituting /gnu/store/bribnmf6djvh1d3rjr2vs5y97141ad97-osinfo-db-20201218...
substitute: updating substitutes from 'http://192.168.10.172:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.172'... 0.0%
guix substitute: error: connect*: No route to host
guix substitute: error: connect*: No route to host
guix substitute: error: connect*: No route to host
guix substitute: warning: 192.168.10.172: connection failed: No route to host
downloading from https://ci.guix.gnu.org/nar/lzip/bribnmf6djvh1d3rjr2vs5y97141ad97-osinfo-db-20201218 ...
osinfo-db-20201218 88KiB 6.0MiB/s 00:00 [############# ] 73.1%substitution of /gnu/store/ns4n01xgbk6ccvd2z127v71d806rnr6f-inkscape-1.1 failed
substitution of /gnu/store/f10an83xvya46ndh61y59qaw5vvs5f7n-libreoffice-7.1.4.2 failed
substitution of /gnu/store/6abwn23grk710qvzvvg1384bs3kc2f8i-linphone-desktop-4.2.5-debug failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f1471840230>
real 1m15.216s
user 0m24.963s
sys 0m0.702s
--8<---------------cut here---------------end--------------->8---
Same thing, the daemon is still trying really hard to get something from
that dead substitute server, slowing things down.
That corrupted archive failure is curious, I wonder if it may be
related.
We'll have to keep this bug open I'm afraid :-/.
Thanks,
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Thu, 19 Aug 2021 02:26:02 GMT)
Full text and
rfc822 format available.
Message #38 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Extra note: the problems reported earlier (hang or backtrace instead of
graceful fallback to other substitute servers) also affect the scenario
where substitutes are fetched from uDNS discovered substitute servers (I
just tried).
Thanks,
Maxim
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Wed, 23 Mar 2022 10:52:02 GMT)
Full text and
rfc822 format available.
Message #41 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi Maxim,
On Wed, 18 Aug 2021 at 22:25, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
> Extra note: the problems reported earlier (hang or backtrace instead of
> graceful fallback to other substitute servers) also affect the scenario
> where substitutes are fetched from uDNS discovered substitute servers (I
> just tried).
I guess this old bug [1] about the daemon hanging is still there, right?
How such issue could be tackled?
1: <http://issues.guix.gnu.org/issue/30290>
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org
:
bug#30290
; Package
guix
.
(Sat, 08 Oct 2022 15:18:01 GMT)
Full text and
rfc822 format available.
Message #44 received at 30290 <at> debbugs.gnu.org (full text, mbox):
Hi Maxim,
On Wed, 23 Mar 2022 at 11:50, zimoun <zimon.toutoune <at> gmail.com> wrote:
> On Wed, 18 Aug 2021 at 22:25, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
>> Extra note: the problems reported earlier (hang or backtrace instead of
>> graceful fallback to other substitute servers) also affect the scenario
>> where substitutes are fetched from uDNS discovered substitute servers (I
>> just tried).
>
> I guess this old bug [1] about the daemon hanging is still there, right?
>
> How such issue could be tackled?
>
> 1: <http://issues.guix.gnu.org/issue/30290>
What could be actionable for tackling this old bug?
Cheers,
simon
Reply sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
You have taken responsibility.
(Mon, 10 Oct 2022 03:04:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
:
bug acknowledged by developer.
(Mon, 10 Oct 2022 03:04:02 GMT)
Full text and
rfc822 format available.
Message #49 received at 30290-done <at> debbugs.gnu.org (full text, mbox):
Hi Simon,
zimoun <zimon.toutoune <at> gmail.com> writes:
> Hi Maxim,
>
> On Wed, 23 Mar 2022 at 11:50, zimoun <zimon.toutoune <at> gmail.com> wrote:
>> On Wed, 18 Aug 2021 at 22:25, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
>
>>> Extra note: the problems reported earlier (hang or backtrace instead of
>>> graceful fallback to other substitute servers) also affect the scenario
>>> where substitutes are fetched from uDNS discovered substitute servers (I
>>> just tried).
>>
>> I guess this old bug [1] about the daemon hanging is still there, right?
>>
>> How such issue could be tackled?
>>
>> 1: <http://issues.guix.gnu.org/issue/30290>
>
> What could be actionable for tackling this old bug?
I can't seem to reproduce the original issue, which was that an offline
server would slow down fetching of substitutes by repeatedly trying to
reach it, at least when setting e.g.:
--8<---------------cut here---------------start------------->8---
127.0.0.2 ci.guix.gnu.org
--8<---------------cut here---------------end--------------->8---
In my /etc/hosts file to simulate it's unreachable, with at least
another substitute server available in my --substitute-urls list. It
goes right to that second server without loosing any time.
Closing for now, we can always reopen the issue if it manifests itself
in the future.
--
Thanks,
Maxim
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Mon, 07 Nov 2022 12:24:11 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 170 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.