GNU bug report logs - #41878
'guix substitute' and 'guix pull' fail gracelessly on flaky networks

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 41878 in the body.
You can then email your comments to 41878 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Quinten Gruenthal <quintengruenthal <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: Handshake Error
Date: Mon, 15 Jun 2020 09:30:34 -0700

[Message part 1 (text/plain, inline)]

I received the following error when performing a guix pull:

|substitute: guix substitute: error: TLS error in procedure 'handshake':
Error in the pull function.
killing process 8828
Backtrace:
          11 (primitive-load
"/gnu/store/pl48b057h6yg8w6f7hafiilcc44d0fn6-compute-guix-derivation")
In ice-9/eval.scm:
    155:9 10 (_ _)
    159:9  9 (_ #(#(#(#(#(#(#(#(#(#(#(#(#(#<directory (guile-user)
7fc5ff085f?> ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?))
In ./guix/store.scm:
  2025:24  8 (run-with-store #<store-connection 256.99 7fc5fd407640> _
#:guile-for-build _ #:system _ #:target _)
   1859:8  7 (_ _)
In ./guix/gexp.scm:
   243:18  6 (_ _)
   1061:2  5 (_ _)
    921:2  4 (_ _)
    782:4  3 (_ _)
In ./guix/store.scm:
  1907:12  2 (_ #<store-connection 256.99 7fc5fd407640>)
   1356:5  1 (map/accumulate-builds #<store-connection 256.99 7fc5fd407640>
_ _)
  1367:15  0 (_ #<store-connection 256.99 7fc5fd407640> _ _)

./guix/store.scm:1367:15: ERROR:
  1. &store-protocol-error:
      message: "`/gnu/store/5r3sb6bj6pppn4h35a35956mv5qrd011-guix-command
substitute' died unexpectedly"
      status: 1
guix pull: error: You found a bug: the program
'/gnu/store/pl48b057h6yg8w6f7hafiilcc44d0fn6-compute-guix-derivation'
failed to compute the derivation for Guix (version:
"cf48f0fc4c40a2ec0b38a445e1e13f37722a0ade"; system: "x86_64-linux";
host version: "ecf92194a55188a9c217d76617378749db063453"; pull-version: 1).
Please report it by email to <bug-guix <at> gnu.org>.

I've seen this kind of error before in other software and I believe it
occurs when a handshake can't be performed in time and the connection is
dropped; I was able to successfully pull a few minutes later under better
conditions. That being said I bring this up because a problem in my network
(I think) is being treated like an error in your program.

[Message part 2 (text/html, inline)]

Message #8 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Quinten Gruenthal <quintengruenthal <at> gmail.com>
Cc: 41878 <at> debbugs.gnu.org
Subject: Re: bug#41878: Handshake Error
Date: Mon, 15 Jun 2020 21:42:46 +0200

Hi,

Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:

> |substitute: guix substitute: error: TLS error in procedure 'handshake':
> Error in the pull function.

[...]

> I've seen this kind of error before in other software and I believe it
> occurs when a handshake can't be performed in time and the connection is
> dropped; I was able to successfully pull a few minutes later under better
> conditions. That being said I bring this up because a problem in my network
> (I think) is being treated like an error in your program.

Yes, it looks like a connectivity issue, either on your side or on the
side of https://ci.guix.gnu.org (though I’m unaware of downtime today).

Could you try again?

Thanks,
Ludo’.

Message #11 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Quinten Gruenthal <quintengruenthal <at> gmail.com>
Cc: 41878 <at> debbugs.gnu.org
Subject: Re: bug#41878: Handshake Error
Date: Tue, 16 Jun 2020 10:07:51 +0200

Hi,

Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:

> Sure; though, as I mentioned before I did try again and it did work. Doing
> a pull once more I failed with:
>
> guix substitute: error: TLS error in procedure 'handshake': Error in the
> pull function.
> substitution of
> /gnu/store/k1da7bv579d8gvwdrakd9l4hxswknff2-guix-module-union failed
> guix pull: error: some substitutes for the outputs of derivation
> `/gnu/store/k2gdq7ahjaqmdgjr9xwywqpxvzazn467-guix-e07573432.drv' failed
> (usually happens due to networking issues); try `--fallback' to build
> derivation from source

Could it be that you’re on a flaky network connection?

> As with before, I was also able to succeed on a subsequent pull; but this
> error seemed to be handled better as it suggested both the cause of the
> problem and a possible solution.

If the connection is dropped, there’s little we can do, but maybe you’re
suggesting better error reporting?

Thanks,
Ludo’.

PS: Please keep the bug Cc’d.

Message #14 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Bengt Richter <bokr <at> bokr.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 41878 <at> debbugs.gnu.org, Quinten Gruenthal <quintengruenthal <at> gmail.com>
Subject: Re: bug#41878: Handshake Error
Date: Tue, 16 Jun 2020 11:31:12 +0200

Hi,

On +2020-06-16 10:07:51 +0200, Ludovic Courtès wrote:
> Hi,
> 
> Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:
> 
> > Sure; though, as I mentioned before I did try again and it did work. Doing
> > a pull once more I failed with:
> >
> > guix substitute: error: TLS error in procedure 'handshake': Error in the
> > pull function.
> > substitution of
> > /gnu/store/k1da7bv579d8gvwdrakd9l4hxswknff2-guix-module-union failed
> > guix pull: error: some substitutes for the outputs of derivation
> > `/gnu/store/k2gdq7ahjaqmdgjr9xwywqpxvzazn467-guix-e07573432.drv' failed
> > (usually happens due to networking issues); try `--fallback' to build
> > derivation from source
> 
> Could it be that you’re on a flaky network connection?
> 
> > As with before, I was also able to succeed on a subsequent pull; but this
> > error seemed to be handled better as it suggested both the cause of the
> > problem and a possible solution.
> 
> If the connection is dropped, there’s little we can do, but maybe you’re
> suggesting better error reporting?

Hm, are gexp's checkpointable?

Some ftp can reconnect and continue, IUUC. Idk how that works with https.
Can one adjust timeouts? Continue on an alternate mirror?
E.g. for downloading substitutes?

In general, I hate it when I get 1.0GB through a 1.1GB ISO and lose it all,
esp when I pay by GBs. Can guix help? :)

BTW, I think I have heard of security risks in restarting, but perhaps
with end-to-end integrity checks it is not an issue despite poss MM trying?

> 
> Thanks,
> Ludo’.
> 
> PS: Please keep the bug Cc’d.
> 
> 
> 

-- 
Regards,
Bengt Richter

Message #17 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Quinten Gruenthal <quintengruenthal <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#41878: Handshake Error
Date: Mon, 15 Jun 2020 14:55:20 -0700

[Message part 1 (text/plain, inline)]

Sure; though, as I mentioned before I did try again and it did work. Doing
a pull once more I failed with:

guix substitute: error: TLS error in procedure 'handshake': Error in the
pull function.
substitution of
/gnu/store/k1da7bv579d8gvwdrakd9l4hxswknff2-guix-module-union failed
guix pull: error: some substitutes for the outputs of derivation
`/gnu/store/k2gdq7ahjaqmdgjr9xwywqpxvzazn467-guix-e07573432.drv' failed
(usually happens due to networking issues); try `--fallback' to build
derivation from source

As with before, I was also able to succeed on a subsequent pull; but this
error seemed to be handled better as it suggested both the cause of the
problem and a possible solution.

On Mon, Jun 15, 2020 at 12:42 PM Ludovic Courtès <ludo <at> gnu.org> wrote:

> Hi,
>
> Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:
>
> > |substitute: guix substitute: error: TLS error in procedure 'handshake':
> > Error in the pull function.
>
> [...]
>
> > I've seen this kind of error before in other software and I believe it
> > occurs when a handshake can't be performed in time and the connection is
> > dropped; I was able to successfully pull a few minutes later under better
> > conditions. That being said I bring this up because a problem in my
> network
> > (I think) is being treated like an error in your program.
>
> Yes, it looks like a connectivity issue, either on your side or on the
> side of https://ci.guix.gnu.org (though I’m unaware of downtime today).
>
> Could you try again?
>
> Thanks,
> Ludo’.
>

[Message part 2 (text/html, inline)]

Message #20 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Quinten Gruenthal <quintengruenthal <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 41878 <at> debbugs.gnu.org
Subject: Re: bug#41878: Handshake Error
Date: Tue, 16 Jun 2020 07:57:26 -0700

[Message part 1 (text/plain, inline)]

Yes and yes. I didn't state explicitly that it was a problem with my
network only because I didn't want to appear overconfident in my amateur
diagnosis and I definitely prefer the second error message to the one that
dumps a trace and prompts the filing of a bug. This:

substitute: guix substitute: error: TLS error in procedure 'handshake':
Error in the pull function.
killing process 8828
Backtrace:
          11 (primitive-load
"/gnu/store/pl48b057h6yg8w6f7hafiilcc44d0fn6-compute-guix-derivation")
In ice-9/eval.scm:
    155:9 10 (_ _)
    159:9  9 (_ #(#(#(#(#(#(#(#(#(#(#(#(#(#<directory (guile-user)
7fc5ff085f?> ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?) ?))
In ./guix/store.scm:
  2025:24  8 (run-with-store #<store-connection 256.99 7fc5fd407640> _
#:guile-for-build _ #:system _ #:target _)
   1859:8  7 (_ _)
In ./guix/gexp.scm:
   243:18  6 (_ _)
   1061:2  5 (_ _)
    921:2  4 (_ _)
    782:4  3 (_ _)
In ./guix/store.scm:
  1907:12  2 (_ #<store-connection 256.99 7fc5fd407640>)
   1356:5  1 (map/accumulate-builds #<store-connection 256.99 7fc5fd407640>
_ _)
  1367:15  0 (_ #<store-connection 256.99 7fc5fd407640> _ _)

./guix/store.scm:1367:15: ERROR:
  1. &store-protocol-error:
      message: "`/gnu/store/5r3sb6bj6pppn4h35a35956mv5qrd011-guix-command
substitute' died unexpectedly"
      status: 1
guix pull: error: You found a bug: the program
'/gnu/store/pl48b057h6yg8w6f7hafiilcc44d0fn6-compute-guix-derivation'
failed to compute the derivation for Guix (version:
"cf48f0fc4c40a2ec0b38a445e1e13f37722a0ade"; system: "x86_64-linux";
host version: "ecf92194a55188a9c217d76617378749db063453"; pull-version: 1).
Please report it by email to <bug-guix <at> gnu.org>.

is much different than this:

guix substitute: error: TLS error in procedure 'handshake': Error in the
pull function.
substitution of
/gnu/store/k1da7bv579d8gvwdrakd9l4hxswknff2-guix-module-union failed
guix pull: error: some substitutes for the outputs of derivation
`/gnu/store/k2gdq7ahjaqmdgjr9xwywqpxvzazn467-guix-e07573432.drv' failed
(usually happens due to networking issues); try `--fallback' to build
derivation from source

especially to one as new to scheme as myself. I can't say why the two
errors were different as I'm not familiar mechanically with what phases
pull goes through but I can say the first error occurred during the
beginning of the pull and the second occurred in the wrap up thing it does
towards the end.


On Tue, Jun 16, 2020 at 1:07 AM Ludovic Courtès <ludo <at> gnu.org> wrote:

> Hi,
>
> Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:
>
> > Sure; though, as I mentioned before I did try again and it did work.
> Doing
> > a pull once more I failed with:
> >
> > guix substitute: error: TLS error in procedure 'handshake': Error in the
> > pull function.
> > substitution of
> > /gnu/store/k1da7bv579d8gvwdrakd9l4hxswknff2-guix-module-union failed
> > guix pull: error: some substitutes for the outputs of derivation
> > `/gnu/store/k2gdq7ahjaqmdgjr9xwywqpxvzazn467-guix-e07573432.drv' failed
> > (usually happens due to networking issues); try `--fallback' to build
> > derivation from source
>
> Could it be that you’re on a flaky network connection?
>
> > As with before, I was also able to succeed on a subsequent pull; but this
> > error seemed to be handled better as it suggested both the cause of the
> > problem and a possible solution.
>
> If the connection is dropped, there’s little we can do, but maybe you’re
> suggesting better error reporting?
>
> Thanks,
> Ludo’.
>
> PS: Please keep the bug Cc’d.
>

[Message part 2 (text/html, inline)]

Message #23 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Quinten Gruenthal <quintengruenthal <at> gmail.com>
Cc: 41878 <at> debbugs.gnu.org
Subject: Re: bug#41878: Handshake Error
Date: Wed, 17 Jun 2020 14:03:08 +0200

Hi,

Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:

> Yes and yes. I didn't state explicitly that it was a problem with my
> network only because I didn't want to appear overconfident in my amateur
> diagnosis and I definitely prefer the second error message to the one that
> dumps a trace and prompts the filing of a bug. This:
>
> substitute: guix substitute: error: TLS error in procedure 'handshake':
> Error in the pull function.

I see.  I’ll take a look and see how we can improve on this.

Thanks,
Ludo’.

Message #28 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: Brendan Tildesley <mail <at> brendan.scot>
To: 41878 <at> debbugs.gnu.org
Subject: Re: 'guix substitute' and 'guix pull' fail gracelessly on flaky
 networks
Date: Sun, 30 Aug 2020 20:06:58 +1000

I have not looked closely but from observation I think currently guix 
first decides if it is going to commit to using a substitute, or falling 
back to building locally, by checking if substitutes are available then 
committing to a method. This differs from the concept of a fallback in 
my head, which would involve trying option B only after option A has 
been tried and failed. guix's way means there are a class of failures 
where guix simply gives up and stops instead of falling back.

In my experience, probably 10% of the time I try a guix pull; guix 
package -u ., there is some weird network error that doesn't happen the 
second time I run it. Perhaps it would be sufficient to simply try twice 
for every substitute; accumulate a list of failed substitutes and retry 
them after iterating through the list of substitutes to download, then 
if that fails try building from source. only then are we allowed to give up.

Message #31 received at 41878 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Brendan Tildesley <mail <at> brendan.scot>
Cc: 41878 <at> debbugs.gnu.org
Subject: Re: bug#41878: 'guix substitute' and 'guix pull' fail gracelessly
 on flaky networks
Date: Fri, 11 Sep 2020 16:17:05 +0200

Dear,

On Sun, 30 Aug 2020 at 20:06, Brendan Tildesley <mail <at> brendan.scot> wrote:
> I have not looked closely but from observation I think currently guix first
> decides if it is going to commit to using a substitute, or falling back to
> building locally, by checking if substitutes are available then committing to
> a method. This differs from the concept of a fallback in my head, which would
> involve trying option B only after option A has been tried and failed. guix's
> way means there are a class of failures where guix simply gives up and stops
> instead of falling back.

What do you mean?

> In my experience, probably 10% of the time I try a guix pull; guix package -u
> ., there is some weird network error that doesn't happen the second time I run
> it. Perhaps it would be sufficient to simply try twice for every substitute;
> accumulate a list of failed substitutes and retry them after iterating through
> the list of substitutes to download, then if that fails try building from
> source. only then are we allowed to give up.

It rings a bell to me.  Something about the configuration of Cuirass and
the build farm serving the substitutes; related to caching.  But I am
not able to find the relevant pointer.


All the best,
simon

Message #36 received at 41878-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 41878-done <at> debbugs.gnu.org, Quinten Gruenthal <quintengruenthal <at> gmail.com>
Subject: Re: bug#41878: 'guix substitute' and 'guix pull' fail gracelessly
 on flaky networks
Date: Tue, 18 Mar 2025 10:35:37 +0900

Hi Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi,
>
> Quinten Gruenthal <quintengruenthal <at> gmail.com> skribis:
>
>> Yes and yes. I didn't state explicitly that it was a problem with my
>> network only because I didn't want to appear overconfident in my amateur
>> diagnosis and I definitely prefer the second error message to the one that
>> dumps a trace and prompts the filing of a bug. This:
>>
>> substitute: guix substitute: error: TLS error in procedure 'handshake':
>> Error in the pull function.
>
> I see.  I’ll take a look and see how we can improve on this.

I believe that's been fixed in commit
fa70c141552c76cf4dc9666f577bf6b471fa0d50 ("substitute: Reopen connection
upon “Error in the push function” from GnuTLS.").

Closing.

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.