GNU bug report logs - #31825
guix offload fails with guix-authenticate error

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 31825 in the body.
You can then email your comments to 31825 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug-guix <bug-guix <at> gnu.org>
Subject: guix offload fails with guix-authenticate error
Date: Wed, 13 Jun 2018 23:54:15 -0400

Hello!

I've read the documentation carefully many times, but I still can't make
`guix offload' work. It always fails like so:

--8<---------------cut here---------------start------------->8---
guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
sending 1 store item to '192.168.1.105'...
exporting path `/gnu/store/wrv01knf5xa76j73afscj066pbqq1na3-export-test'
guix offload: error: build failed: program `guix-authenticate' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

In GDB, I've attempted to follow the forks but without success. The
most useful hint I've got was the following message about
guix-authenticate not being found:

--8<---------------cut here---------------start------------->8---
sudo gdb --args guix-daemon --build-users-group=guixbuild
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
[...]
(gdb) r
Starting program: /home/mcournoyer/src/guix/guix-daemon --build-users-group=guixbuild
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/gnu/store/4sqaib7c2dfjv62ivrg9b8wa7bh226la-glibc-2.26.105-g0890d5379c/lib/libthread_db.so.1".
accepted connection from pid 2241, user mcournoyer
accepted connection from pid 29024, user mcournoyer
accepted connection from pid 2241, user mcournoyer
accepted connection from pid 29068, user mcournoyer
error: executing `guix-authenticate': No such file or directory
--8<---------------cut here---------------end--------------->8---

Things I've tried without success:

1. Installing guix into my user's profile and then this to the ~/.bashrc
on the remote machine:

--8<---------------cut here---------------start------------->8---
GUIX_PROFILE=$HOME/.guix-profile
source $GUIX_PROFILE/etc/profile

# Hack to make guix-authenticate available
export PATH=$PATH:$GUIX_PROFILE/libexec
--8<---------------cut here---------------end--------------->8---

2. Noticing that the guix-daemon I was debugging (the one at
~/src/guix/guix-daemon) was using nixLibexecDir = "/usr/local/libexec"
and that the guix-daemon has code doing this in guix-daemon.cc:469:

--8<---------------cut here---------------start------------->8---
      /* Hackily help 'local-store.cc' find our 'guix-authenticate' program, which
	 is known as 'OPENSSL_PATH' here.  */
      std::string search_path;
      search_path = settings.nixLibexecDir;
      if (getenv ("PATH") != NULL)
	{
	  search_path += ":";
	  search_path += getenv ("PATH");
	}

      setenv ("PATH", search_path.c_str (), 1);
--8<---------------cut here---------------end--------------->8---

I tried:

--8<---------------cut here---------------start------------->8---
sudo ln -s /var/guix/profiles/per-user/mcournoyer/guix-profile/libexec /usr/local/libexec
--8<---------------cut here---------------end--------------->8---

I'm motivated to get this working; would be pleased to debug further
given fresh ideas.

Thanks,

Maxim

Message #8 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Fri, 15 Jun 2018 00:08:49 +0200

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> I've read the documentation carefully many times, but I still can't make
> `guix offload' work. It always fails like so:
>
> guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> sending 1 store item to '192.168.1.105'...
> exporting path `/gnu/store/wrv01knf5xa76j73afscj066pbqq1na3-export-test'
> guix offload: error: build failed: program `guix-authenticate' failed with exit code 1

Presumably what this means is that the remote machine rejected the store
item we sent it.

To fix it, you need to authorize the signing key of the first machine on
the second machine, using ‘guix archive --authorize’.

You also need to do the reverse and ‘guix offload test’ will also check
that.

Can you make sure the machines are authorized by each other?  (Check
/etc/guix/acl on each.)

HTH,
Ludo’.

Message #11 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Sun, 17 Jun 2018 22:31:33 -0400

Hi Ludo,

ludo <at> gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> I've read the documentation carefully many times, but I still can't make
>> `guix offload' work. It always fails like so:
>>
>> guix offload test
>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
>> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>> sending 1 store item to '192.168.1.105'...
>> exporting path `/gnu/store/wrv01knf5xa76j73afscj066pbqq1na3-export-test'
>> guix offload: error: build failed: program `guix-authenticate' failed with exit code 1
>
> Presumably what this means is that the remote machine rejected the store
> item we sent it.
>
> To fix it, you need to authorize the signing key of the first machine on
> the second machine, using ‘guix archive --authorize’.
> You also need to do the reverse and ‘guix offload test’ will also check
> that.
>
> Can you make sure the machines are authorized by each other?  (Check
> /etc/guix/acl on each.)

I've verified this a couple times, following the manual
carefully. Here's a sample of what I did:

* On the main machine

$ sudo guix archive --generate-key
guix archive: error: key pair exists under '/etc/guix'; remove it first

$ cat /etc/guix/signing-key.pub 
(public-key 
 (ecc 
  (curve Ed25519)
  (q #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)
  )
 )

$ scp /etc/guix/signing-key.pub x220:/tmp
signing-key.pub    100%  118    46.5KB/s   00:00

* On the offload machine

$ ssh x220
$ sudo -E guix archive --authorize < /tmp/signing-key.pub   # on x220 machine

On my offload machine, the authorized key is added to
/usr/local/etc/guix/acl rather than /etc/guix/acl. I'm not sure why that
is, since this Guix was guix pulled, so it should be standard. But it
shouldn't matter since that running guix-daemon in gdb allowed me to see
that it was using a nixConfDir value set to "/usr/local/etc/guix".

$ sudo cat /etc/guix/acl
--8<---------------cut here---------------start------------->8---
(acl 
 (entry 
  (public-key 
   (rsa 
    (n #00DB1634E3D9DFAC97AE4734DAE968CCB15EE4815C82BDC254883DBB49FE1EF32268E82D4BBE0E35298C481C9DA1551642FAFF05AEC1A60712F1BB4BE7D25D7EFF7A4F89704A5A9AC232870CB9F2476C3B538A0E990A8825DEB73081D317001FB8A188600F2FEF5F5F570E857F3EE4355077A3C3918ED72723A56BA55C466D400658974D7DAD1F6B7B63C192B9C2704D98BBFF1C3BD5B8EF11A8ADC83ACB8FD8E9F1E792FDAD262415D13F2DEE55F330908CFDA9C3C8C32B64F7DD088457D34F445E2E2C83C6D680549DC9B6E6573B89496567204ED285E67A279F2F667080BA941D80D015CE87B0FB6A91A99CECC7D91D2D210B00E4B6E611DA51DB008F1DFE3FCAC6B27393FA781D45F9A15FC7B8785A3E86BA6592B2916CA22CF1E40FC85F85CACA590461154F58F3580B16398908EF32076F411299C28727C94D88B6A618F84DD73AEBED8270BCB6690928CB1BF250C35E1F6BF3B1B30D05BA246ECE8F69D9065DE26F4B3E0D814D70A9C27CB5B7B050C9090590D3A9EF83374F2643E5446FBD39DDB124DBF6DFDAA6D18E2560AD0CBFA11C959C9B7316BF19963A191967054E9FD97DC14D71082B30B1C90A46E8996682474C3BCB51BA0882958897B6DD35E41B5174D0A6BCDE97B89043E95BD1B70DE61DA666893B417196A180005466BC3A742FDF04E89B04460E3E6BC72E7F1B5FEA5B3092FEE551A3C447C12E104E65#)
    (e #010001#)
    )
   )
  (tag 
   (guix import)
   )
  )
 )
--8<---------------cut here---------------end--------------->8---

$ sudo cat /usr/local/etc/guix/acl 
--8<---------------cut here---------------start------------->8---
(acl 
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #5ED0F681F77731AD676285A6DB5986DA5252DE1AA597DFC56835FF948C150834#)
    )
   )
  (tag 
   (guix import)
   )
  )
 )
--8<---------------cut here---------------end--------------->8---

Notice that the same key can be added multiple times by using the
--authorize command, but cleaning up the file doesn't seem to help.

$ sudo -E guix archive --generate-key
guix archive: error: key pair exists under '/usr/local/etc/guix'; remove it first

$ cat /usr/local/etc/guix/signing-key.pub 
(public-key 
 (ecc 
  (curve Ed25519)
  (q #5ED0F681F77731AD676285A6DB5986DA5252DE1AA597DFC56835FF948C150834#)
  )
 )

* Back to my main machine

$ scp x220:/usr/local/etc/guix/signing-key.pub /tmp
signing-key.pub   100%  118    35.3KB/s   00:00

$ sudo -E guix archive --authorize < /tmp/signing-key.pub

$ sudo cat /etc/guix/acl
--8<---------------cut here---------------start------------->8---
(acl 
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #5ED0F681F77731AD676285A6DB5986DA5252DE1AA597DFC56835FF948C150834#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #5ED0F681F77731AD676285A6DB5986DA5252DE1AA597DFC56835FF948C150834#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #5ED0F681F77731AD676285A6DB5986DA5252DE1AA597DFC56835FF948C150834#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #5ED0F681F77731AD676285A6DB5986DA5252DE1AA597DFC56835FF948C150834#)
    )
   )
  (tag 
   (guix import)
   )
  )
 (entry 
  (public-key 
   (ecc 
    (curve Ed25519)
    (q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#)
    )
   )
  (tag 
   (guix import)
   )
  )
 )
--8<---------------cut here---------------end--------------->8---

$ guix offload test
--8<---------------cut here---------------start------------->8---
Connection to 192.168.1.105 closed.
maxim <at> apteryx ~$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
sending 1 store item to '192.168.1.105'...
exporting path `/gnu/store/smgzvgc9krglk0mjpcscg5450l05w4dg-export-test'
guix offload: error: build failed: program `guix-authenticate' failed
with exit code 1
--8<---------------cut here---------------end--------------->8---

Any other ideas?

Thank you!

Maxim

Message #14 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Mon, 18 Jun 2018 10:32:03 +0200

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> maxim <at> apteryx ~$ guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> sending 1 store item to '192.168.1.105'...
> exporting path `/gnu/store/smgzvgc9krglk0mjpcscg5450l05w4dg-export-test'
> guix offload: error: build failed: program `guix-authenticate' failed
> with exit code 1

On closer inspection, it looks like it may be ‘guix-authenticate’ on
“apteryx” that fails.  This could happen if there’s no signing key, for
instance, but you said there’s one, so I don’t know.

Could you attach strace to guix-daemon on “apteryx” and run “guix
offload test” again?

Use something like:

  # strace -p $(pidof guix-daemon) -f -s 345 -o /tmp/log

and then:

  # guix offload test

TIA,
Ludo’.

Message #17 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Mon, 18 Jun 2018 22:35:36 -0400

[Message part 1 (text/plain, inline)]

Hi Ludo,

ludo <at> gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> maxim <at> apteryx ~$ guix offload test
>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
>> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>> sending 1 store item to '192.168.1.105'...
>> exporting path `/gnu/store/smgzvgc9krglk0mjpcscg5450l05w4dg-export-test'
>> guix offload: error: build failed: program `guix-authenticate' failed
>> with exit code 1
>
> On closer inspection, it looks like it may be ‘guix-authenticate’ on
> “apteryx” that fails.  This could happen if there’s no signing key, for
> instance, but you said there’s one, so I don’t know.
>
> Could you attach strace to guix-daemon on “apteryx” and run “guix
> offload test” again?
>
> Use something like:
>
>   # strace -p $(pidof guix-daemon) -f -s 345 -o /tmp/log
>
> and then:
>
>   # guix offload test

I've ran exactly those commands, and this produced the log attached
(half a megabyte of text!). I looked at it but I can't seem to see
what's wrong in there. I hope your trained eyes can see differently!

Thanks again!

Maxim

[log (text/plain, attachment)]

Message #20 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Tue, 19 Jun 2018 11:28:31 +0200

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> Hello,
>>
>> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>>
>>> maxim <at> apteryx ~$ guix offload test
>>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>>> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
>>> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>>> sending 1 store item to '192.168.1.105'...
>>> exporting path `/gnu/store/smgzvgc9krglk0mjpcscg5450l05w4dg-export-test'
>>> guix offload: error: build failed: program `guix-authenticate' failed
>>> with exit code 1
>>
>> On closer inspection, it looks like it may be ‘guix-authenticate’ on
>> “apteryx” that fails.  This could happen if there’s no signing key, for
>> instance, but you said there’s one, so I don’t know.
>>
>> Could you attach strace to guix-daemon on “apteryx” and run “guix
>> offload test” again?
>>
>> Use something like:
>>
>>   # strace -p $(pidof guix-daemon) -f -s 345 -o /tmp/log
>>
>> and then:
>>
>>   # guix offload test
>
> I've ran exactly those commands, and this produced the log attached
> (half a megabyte of text!). I looked at it but I can't seem to see
> what's wrong in there. I hope your trained eyes can see differently!

I don’t see “program `guix-authenticate' failed”, and indeed ‘guix
authenticate’ exits with code 0 (indicating success.)

Are you still getting the error above?

If so it may be that my diagnostic was wrong and that the authentication
failure happens on the other machine.  Could you strace guix-daemon on
that other machine similarly?

However, note that the log ended up containing a copy of your secret
key, /etc/guix/signing-key.sec; sorry for not thinking about this
before.  So once we’re done debugging, you should consider throwing away
that key and re-running “guix archive --generate-key”.  In the next
strace log you send, you might want to remove the private key (look for
an sexp starting with “(private-key …”).

Thanks,
Ludo’.

Message #23 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Tue, 19 Jun 2018 08:43:08 -0400

[Message part 1 (text/plain, inline)]

Hi Ludo!

ludo <at> gnu.org (Ludovic Courtès) writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> ludo <at> gnu.org (Ludovic Courtès) writes:
>>
>>> Hello,
>>>
>>> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>>>
>>>> maxim <at> apteryx ~$ guix offload test
>>>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>>>> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
>>>> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>>>> sending 1 store item to '192.168.1.105'...
>>>> exporting path `/gnu/store/smgzvgc9krglk0mjpcscg5450l05w4dg-export-test'
>>>> guix offload: error: build failed: program `guix-authenticate' failed
>>>> with exit code 1
>>>
>>> On closer inspection, it looks like it may be ‘guix-authenticate’ on
>>> “apteryx” that fails.  This could happen if there’s no signing key, for
>>> instance, but you said there’s one, so I don’t know.
>>>
>>> Could you attach strace to guix-daemon on “apteryx” and run “guix
>>> offload test” again?
>>>
>>> Use something like:
>>>
>>>   # strace -p $(pidof guix-daemon) -f -s 345 -o /tmp/log
>>>
>>> and then:
>>>
>>>   # guix offload test
>>
>> I've ran exactly those commands, and this produced the log attached
>> (half a megabyte of text!). I looked at it but I can't seem to see
>> what's wrong in there. I hope your trained eyes can see differently!
>
> I don’t see “program `guix-authenticate' failed”, and indeed ‘guix
> authenticate’ exits with code 0 (indicating success.)
>
> Are you still getting the error above?

Yes.

> If so it may be that my diagnostic was wrong and that the authentication
> failure happens on the other machine.  Could you strace guix-daemon on
> that other machine similarly?
>
> However, note that the log ended up containing a copy of your secret
> key, /etc/guix/signing-key.sec; sorry for not thinking about this
> before.  So once we’re done debugging, you should consider throwing away
> that key and re-running “guix archive --generate-key”.  In the next
> strace log you send, you might want to remove the private key (look for
> an sexp starting with “(private-key …”).

Oh, thanks for letting me know. I will recreate both keys when we're
done.

Attached is the log for the offloading machine.

From what I can see, the guix-daemon is trying to find the authorized
key in /etc/guix/acl, but the key is added by Guix to
/usr/local/etc/guix/acl...

So, the original two issues:

1. `guix offload' doesn't seem to honor nixConfDir (which the daemon on the
offload machine knows as "/usr/local/etc/guix").

2. The error message should capture the complete error output to ease
debugging: we can see the useful message "25056 write(2, "guix
authenticate: error: error: unauthorized public key: (public-key \n (ecc
\n  (curve Ed25519)\n  (q
#EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)\n
)\n )\n", 176) = 176" in strace. Had I seen this from the start, we
would have saved some debugging time :).

What do you think?

I could work around the issue by copying manually the authorized key
sexp to /etc/guix/acl; I now see:

--8<---------------cut here---------------start------------->8---
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
sending 1 store item to '192.168.1.105'...
exporting path `/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
guix offload: '192.168.1.105' successfully imported '/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
retrieving 1 store item from '192.168.1.105'...
guix offload: error: build failed: implementation cannot deal with > 32-bit integers
--8<---------------cut here---------------end--------------->8---

This appear to be caused by the 'apteryx' GuixSD machine where I run
'guix offload test'. I've attached another strace output capturing this.

Thanks!

Maxim

[log-on-offload-machine.txt (text/plain, attachment)]

[log-32bit-error.txt (text/plain, attachment)]

Message #26 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Tue, 19 Jun 2018 16:49:31 +0200

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> Attached is the log for the offloading machine.
>
> From what I can see, the guix-daemon is trying to find the authorized
> key in /etc/guix/acl, but the key is added by Guix to
> /usr/local/etc/guix/acl...

Hmm you may be using two different ‘guix’ commands no?

> 2. The error message should capture the complete error output to ease
> debugging: we can see the useful message "25056 write(2, "guix
> authenticate: error: error: unauthorized public key: (public-key \n (ecc
> \n  (curve Ed25519)\n  (q
> #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)\n
> )\n )\n", 176) = 176" in strace. Had I seen this from the start, we
> would have saved some debugging time :).

I agree.

> I could work around the issue by copying manually the authorized key
> sexp to /etc/guix/acl; I now see:
>
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> sending 1 store item to '192.168.1.105'...
> exporting path `/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
> guix offload: '192.168.1.105' successfully imported '/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
> retrieving 1 store item from '192.168.1.105'...
> guix offload: error: build failed: implementation cannot deal with > 32-bit integers

The log has this:

--8<---------------cut here---------------start------------->8---
10529 write(4, "atad\0\0\0\0\0\200\0\0\0\0\0\0", 16) = 16
10529 read(4, "W\1\0\0\0\0\0\0\1\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0nix-archive-1\0\0\0\1\0\0\0\0\0\0\0(\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0type\0\0\0\0\7\0\0\0\0\0\0\0regular\0\10\0\0\0\0\0\0\0contents\23\0\0\0\0\0\0\000192.168.1.105-83353\0\0\0\0\0\1\0\0\0\0\0\0\0)\0\0\0\0\0\0\0NIXE\0\0\0\0007\0\0\0\0\0\0\0/gnu/store/wf774mzvfjpw306y5x06wid80d9k90qq-import-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0(protocol-error 1 \"getting status of `/etc/guix/signing-key.sec': Aucun fichier ou dossier de ce "..., 32768) = 352
--8<---------------cut here---------------end--------------->8---

Again the error should be reported…

HTH,
Ludo’.

Message #29 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Tue, 19 Jun 2018 23:01:02 -0400

Hi!

ludo <at> gnu.org (Ludovic Courtès) writes:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> Attached is the log for the offloading machine.
>>
>> From what I can see, the guix-daemon is trying to find the authorized
>> key in /etc/guix/acl, but the key is added by Guix to
>> /usr/local/etc/guix/acl...
>
> Hmm you may be using two different ‘guix’ commands no?
>
>> 2. The error message should capture the complete error output to ease
>> debugging: we can see the useful message "25056 write(2, "guix
>> authenticate: error: error: unauthorized public key: (public-key \n (ecc
>> \n  (curve Ed25519)\n  (q
>> #EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)\n
>> )\n )\n", 176) = 176" in strace. Had I seen this from the start, we
>> would have saved some debugging time :).
>
> I agree.
>
>> I could work around the issue by copying manually the authorized key
>> sexp to /etc/guix/acl; I now see:
>>
>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
>> guix offload: Guix is usable on '192.168.1.105' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>> sending 1 store item to '192.168.1.105'...
>> exporting path `/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
>> guix offload: '192.168.1.105' successfully imported '/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
>> retrieving 1 store item from '192.168.1.105'...
>> guix offload: error: build failed: implementation cannot deal with > 32-bit integers
>
> The log has this:
>
> 10529 write(4, "atad\0\0\0\0\0\200\0\0\0\0\0\0", 16) = 16
> 10529 read(4, "W\1\0\0\0\0\0\0\1\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0nix-archive-1\0\0\0\1\0\0\0\0\0\0\0(\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0type\0\0\0\0\7\0\0\0\0\0\0\0regular\0\10\0\0\0\0\0\0\0contents\23\0\0\0\0\0\0\000192.168.1.105-83353\0\0\0\0\0\1\0\0\0\0\0\0\0)\0\0\0\0\0\0\0NIXE\0\0\0\0007\0\0\0\0\0\0\0/gnu/store/wf774mzvfjpw306y5x06wid80d9k90qq-import-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0(protocol-error 1 \"getting status of `/etc/guix/signing-key.sec': Aucun fichier ou dossier de ce "..., 32768) = 352
>
> Again the error should be reported…

Yes, this error was totally wrong, thanks for pointing it out. The
actual error was the 192.168.1.105 offload machine not finding the key
at /etc/guix/singning-key.sec (since it using the prefix
/usr/local/etc/guix for some reason).

I just did:

--8<---------------cut here---------------start------------->8---
sudo cp /usr/local/etc/guix/signing* /etc/guix/
--8<---------------cut here---------------end--------------->8---

And it is now working. Ouf!

Summarizing this adventure:

0) Make sure your .bashrc doesn't exit early when it is executed in
non-interactive mode (as is the case in Ubuntu).

1) Make sure the guix-authenticate program is available on the host as
well as the offload machines, by installing guix (guix package -i guix)
in the corresponding user profiles and sourcing
$HOME/guix.profile/etc/profile in the ~/.bashrc.

2) Make sure all your guix-daemons are configured to use /etc/guix as
their sysconfdir, as Guix offload currently seems hardcoded to only look
things under /etc/guix.

3) Don't trust any errors output by guix offload ;)

It'd be nice if this was as simple as setting up a Jenkins node... You
tell Guix which machine you want to use and give it SSH access, and it
does the required setup without having the user messing around with keys
and what not.

But I'm seeing far ahead. For now, we could start by adding some points
to the `guix offload` info manual. Then we can try to modify the code to
better capture the error messages. 

I'll start with the documentation.

Thank you,

Maxim

Message #32 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: swedebugia <swedebugia <at> riseup.net>
To: bug-guix <at> gnu.org,Maxim Cournoyer <maxim.cournoyer <at> gmail.com>,ludo <at> gnu.org
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Wed, 20 Jun 2018 05:54:44 +0200

[Message part 1 (text/plain, inline)]

Hi

On June 20, 2018 5:01:02 AM GMT+02:00, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> wrote:
>Hi!
>
>ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>>
>>> Attached is the log for the offloading machine.
>>>
>>> From what I can see, the guix-daemon is trying to find the
>authorized
>>> key in /etc/guix/acl, but the key is added by Guix to
>>> /usr/local/etc/guix/acl...
>>
>> Hmm you may be using two different ‘guix’ commands no?
>>
>>> 2. The error message should capture the complete error output to
>ease
>>> debugging: we can see the useful message "25056 write(2, "guix
>>> authenticate: error: error: unauthorized public key: (public-key \n
>(ecc
>>> \n  (curve Ed25519)\n  (q
>>>
>#EEA139318243D36EB4C728DB96856AB15C47AB64C765FA134CCFB12444B82A7C#)\n
>>> )\n )\n", 176) = 176" in strace. Had I seen this from the start, we
>>> would have saved some debugging time :).
>>
>> I agree.
>>
>>> I could work around the issue by copying manually the authorized key
>>> sexp to /etc/guix/acl; I now see:
>>>
>>> guix offload: testing 1 build machines defined in
>'/etc/guix/machines.scm'...
>>> guix offload: '192.168.1.105' is running guile (GNU Guile) 2.2.3
>>> guix offload: Guix is usable on '192.168.1.105' (test returned
>"/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>>> sending 1 store item to '192.168.1.105'...
>>> exporting path
>`/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
>>> guix offload: '192.168.1.105' successfully imported
>'/gnu/store/np9jwqvxjvasz41nrrh6g3gyn4rpkscw-export-test'
>>> retrieving 1 store item from '192.168.1.105'...
>>> guix offload: error: build failed: implementation cannot deal with >
>32-bit integers
>>
>> The log has this:
>>
>> 10529 write(4, "atad\0\0\0\0\0\200\0\0\0\0\0\0", 16) = 16
>> 10529 read(4,
>"W\1\0\0\0\0\0\0\1\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0nix-archive-1\0\0\0\1\0\0\0\0\0\0\0(\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0type\0\0\0\0\7\0\0\0\0\0\0\0regular\0\10\0\0\0\0\0\0\0contents\23\0\0\0\0\0\0\000192.168.1.105-83353\0\0\0\0\0\1\0\0\0\0\0\0\0)\0\0\0\0\0\0\0NIXE\0\0\0\0007\0\0\0\0\0\0\0/gnu/store/wf774mzvfjpw306y5x06wid80d9k90qq-import-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0(protocol-error
>1 \"getting status of `/etc/guix/signing-key.sec': Aucun fichier ou
>dossier de ce "..., 32768) = 352
>>
>> Again the error should be reported…
>
>Yes, this error was totally wrong, thanks for pointing it out. The
>actual error was the 192.168.1.105 offload machine not finding the key
>at /etc/guix/singning-key.sec (since it using the prefix
>/usr/local/etc/guix for some reason).
>
>I just did:
>
>--8<---------------cut here---------------start------------->8---
>sudo cp /usr/local/etc/guix/signing* /etc/guix/
>--8<---------------cut here---------------end--------------->8---
>
>And it is now working. Ouf!
>
>Summarizing this adventure:
>
>0) Make sure your .bashrc doesn't exit early when it is executed in
>non-interactive mode (as is the case in Ubuntu).
>
>1) Make sure the guix-authenticate program is available on the host as
>well as the offload machines, by installing guix (guix package -i guix)
>in the corresponding user profiles and sourcing
>$HOME/guix.profile/etc/profile in the ~/.bashrc.
>
>2) Make sure all your guix-daemons are configured to use /etc/guix as
>their sysconfdir, as Guix offload currently seems hardcoded to only
>look
>things under /etc/guix.
>
>3) Don't trust any errors output by guix offload ;)
>
>It'd be nice if this was as simple as setting up a Jenkins node... You
>tell Guix which machine you want to use and give it SSH access, and it
>does the required setup without having the user messing around with
>keys
>and what not.
>
>But I'm seeing far ahead. For now, we could start by adding some points
>to the `guix offload` info manual. Then we can try to modify the code
>to
>better capture the error messages. 
>
>I'll start with the documentation.
>
>Thank you,
>
>Maxim

Good job hunting the bug. 😀
Are you going to report a new bug about incorrect or insufficient error messages? 
Thanks for the summary. 
-- 
Cheers Swedebugia 

[Message part 2 (text/html, inline)]

Message #38 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 31825 <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Wed, 20 Jun 2018 16:06:11 +0200

Hello!

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> I just did:
>
> sudo cp /usr/local/etc/guix/signing* /etc/guix/
>
> And it is now working. Ouf!

Woohoo!

> Summarizing this adventure:
>
> 0) Make sure your .bashrc doesn't exit early when it is executed in
> non-interactive mode (as is the case in Ubuntu).
>
> 1) Make sure the guix-authenticate program is available on the host as
> well as the offload machines, by installing guix (guix package -i guix)
> in the corresponding user profiles and sourcing
> $HOME/guix.profile/etc/profile in the ~/.bashrc.
>
> 2) Make sure all your guix-daemons are configured to use /etc/guix as
> their sysconfdir, as Guix offload currently seems hardcoded to only look
> things under /etc/guix.

Hmm nothing’s hard-coded; it’s the daemon on the remote host that knows
where to look for keys etc.

I suspect there was a mixture of Guix with --sysconfdir=/etc and with
--sysconfdir=/usr/local/etc, perhaps due to an earlier installation
built from source or something, and that this is what led to the mess.

I’m afraid there’s not much Guix itself can do, but if you investigate
and manage to determine how we ended up with this confusion, perhaps
we’ll have ideas on how to avoid it.

> 3) Don't trust any errors output by guix offload ;)

Yeah we can definitely do better.  :-)

> It'd be nice if this was as simple as setting up a Jenkins node... You
> tell Guix which machine you want to use and give it SSH access, and it
> does the required setup without having the user messing around with keys
> and what not.

The security implications of authorizing each other’s keys are serious,
and have to be made by root on both machines.  So I’m not sure we could
easily automate it.  It’s quite common for SSH daemons to disallow root
logins by default, for instance, which prevents automation in this case.

> But I'm seeing far ahead. For now, we could start by adding some points
> to the `guix offload` info manual. Then we can try to modify the code to
> better capture the error messages. 

Yes, I’d say improving ‘guix offload status’ should be the priority.  To
be honest, I’m not sure anything important is missing in the manual,
looking at the items above.

Thanks,
Ludo’.

Message #41 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: swedebugia <swedebugia <at> riseup.net>
Cc: 31825 <at> debbugs.gnu.org, ludo <at> gnu.org, bug-guix <at> gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Thu, 21 Jun 2018 22:13:08 -0400

Hello!

swedebugia <swedebugia <at> riseup.net> writes:

> Good job hunting the bug. 😀

Thanks; credits to Ludovic for providing me with the right debugging
tool (and incantation).

> Are you going to report a new bug about incorrect or insufficient error messages? 
> Thanks for the summary. 

I think we can track it here for now. If it becomes a bigger task it
might be better to create an issue for itself.

Maxim

Message #49 received at 31825 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 31825 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Sat, 22 Feb 2020 00:18:39 -0500

Just as a follow-up; I've managed to fall into this trap again,
attempting to authorize the keys by adding them to the 'authorize-keys'
field of guix-configuration record.

On the local machine:

--8<---------------cut here---------------start------------->8---
guix offload test /etc/guix/machines.scm 127.0.0.1
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: Guix is usable on '127.0.0.1' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: '127.0.0.1' is running GNU Guile 3.0.0
sending 1 store item (0 MiB) to '127.0.0.1'...
exporting path `/gnu/store/l9mph3k5l26nm8mb50imsklbsz0bji0b-export-test'
guix offload: error: program `/gnu/store/amjsgks2n05k9lkck78z64nphad1dkqr-guix-1.0.1-13.50299ad/bin/guix' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

On the remote machine:

--8<---------------cut here---------------start------------->8---
sudo strace -p 15683 -p 15716 -f -s345 -o /tmp/log
--8<---------------cut here---------------end--------------->8---

And found within /tmp/log:

16120 write(2, "guix authenticate: error: error: unauthorized public key: (public-key \n (ecc \n  (curve Ed25519)\n  (q #MY-PUBLIC-KEY#)\n  )\n )\n", 176) = 176

So, still actual :-)

Maxim

Message #54 received at 31825-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 31825-done <at> debbugs.gnu.org
Subject: Re: bug#31825: guix offload fails with guix-authenticate error
Date: Sun, 08 Aug 2021 00:09:20 -0400

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Just as a follow-up; I've managed to fall into this trap again,
> attempting to authorize the keys by adding them to the 'authorize-keys'
> field of guix-configuration record.
>
> On the local machine:
>
> guix offload test /etc/guix/machines.scm 127.0.0.1
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: Guix is usable on '127.0.0.1' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> guix offload: '127.0.0.1' is running GNU Guile 3.0.0
> sending 1 store item (0 MiB) to '127.0.0.1'...
> exporting path `/gnu/store/l9mph3k5l26nm8mb50imsklbsz0bji0b-export-test'
> guix offload: error: program `/gnu/store/amjsgks2n05k9lkck78z64nphad1dkqr-guix-1.0.1-13.50299ad/bin/guix' failed with exit code 1
>
>
> On the remote machine:
>
> sudo strace -p 15683 -p 15716 -f -s345 -o /tmp/log
>
> And found within /tmp/log:
>
> 16120 write(2, "guix authenticate: error: error: unauthorized public
> key: (public-key \n (ecc \n (curve Ed25519)\n (q #MY-PUBLIC-KEY#)\n
> )\n )\n", 176) = 176
>
> So, still actual :-)
>
> Maxim

I think many things have been improved in the diagnostics of guix
offload since the original report.  The last gotcha I had hit described
above appears to had been caused by the keys added to the
'authorized-keys' field of the 'guix-configuration' record not being
taken into account when a /etc/guix/acl file was already populated (it
used to not be declarative).

Closing this forgotten issue.

Thanks,

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.