GNU bug report logs - #35662
Really relocatable binaries crash with Permission denied

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35662 in the body.
You can then email your comments to 35662 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: bug-guix <at> gnu.org
Subject: Really relocatable binaries crash with Permission denied
Date: Fri, 10 May 2019 00:01:36 +0200

The manual gives the following example of guix pack -RR:

      guix pack -RR -S /mybin=bin bash
      tar xf pack.tar.gz
      ./mybin/sh

This fails on my university’s server for students which uses Linux
container “VMs” with Ubuntu and has no user namespace support and Guix
is not installed.  This single line is all output:

$ ./mybin/sh
sh: run.c:162: bind_mount: Unexpected error: Permission denied.

Note that

PROOT_NO_SECCOMP=1 ~/gnu/store/iyd2ikxadcp89j5919pwja6swnx00493-proot-static-5.1.0/bin/proot -w $(pwd | sed 's/${HOME}//') -r ${HOME} -b /proc /mybin/sh

works just fine (inspired by
<https://guix-hpc.bordeaux.inria.fr/blog/2017/10/using-guix-without-being-root/>).

For testing purposes, I compile the wrapper
gnu/packages/aux-files/run-in-namespace.c:

sed -i 's|@STORE_DIRECTORY@|/gnu/store|g' run-in-namespace.c
sed -i 's|@WRAPPED_PROGRAM@|/mybin/sh|g' run-in-namespace.c
gcc -std=gnu99 -static -O0 -g -Wall run-in-namespace.c
scp run-in-namespace.c a.out … # upload it to the university server
ssh …
gdb a.out
[…]
(gdb) break main
Breakpoint 1 at 0x401ea1: file run-in-namespace.c, line 260.
(gdb) run
Starting program: /home/f_pelz12/a.out 

Breakpoint 1, main (argc=1, argv=0x7fffffffe818) at run-in-namespace.c:260
260	  size = readlink ("/proc/self/exe", self, sizeof self - 1);
(gdb) next
261	  assert (size > 0);
(gdb) 
265	  size_t index = strlen (self)
(gdb) 
268	  char *store = strdup (self);
(gdb) 
269	  store[index] = '\0';
(gdb) 
277	  if (strcmp (store, "/gnu/store") != 0
(gdb) 
278	      && lstat ("/mybin/sh", &statbuf) != 0)
(gdb) 
283	      char *new_root = mkdtemp (strdup ("/tmp/guix-exec-XXXXXX"));
(gdb) 
284	      char *new_store = concat (new_root, "/gnu/store");
(gdb) 
285	      char *cwd = get_current_dir_name ();
(gdb) 
292	      pid_t child = syscall (SYS_clone, SIGCHLD | CLONE_NEWNS | CLONE_NEWUSER,
(gdb) 
[Detaching after fork from child process 12748]
294	      switch (child)
(gdb) a.out: run-in-namespace.c:162: bind_mount: Unexpected error: Permission denied.

337		    disallow_setgroups (child);
(gdb) 
a.out: run-in-namespace.c:205: disallow_setgroups: Unexpected error: Permission denied.

Program received signal SIGABRT, Aborted.
0x000000000040796f in raise ()

I do not know how to break into the detached child’s bind_mount call,
so I am unable to give details on this bind_mount error (I do not know
if the bind_mount really is the cause of the crash; it is futile
anyway and the binary should just try proot after all and not crash
before).  A breakpoint from `break bind_mount` is ignored.  Can I get
more information out of this somehow?

For completeness:
$ uname -a
Linux tux6 4.15.18-14-pve #1 SMP PVE 4.15.18-38 (Tue, 30 Apr 2019 10:51:33 +0200) x86_64 x86_64 x86_64 GNU/Linux

Regards,
Florian

Message #8 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Fri, 10 May 2019 07:54:41 +0200

On Fri, May 10, 2019 at 12:01:36AM +0200, pelzflorian (Florian Pelz) wrote:
> sed -i 's|@STORE_DIRECTORY@|/gnu/store|g' run-in-namespace.c
> sed -i 's|@WRAPPED_PROGRAM@|/mybin/sh|g' run-in-namespace.c
> gcc -std=gnu99 -static -O0 -g -Wall run-in-namespace.c

I think it should have been

sed -i 's|@STORE_DIRECTORY@|/gnu/store|g' run-in-namespace.c
sed -i 's|@WRAPPED_PROGRAM@|/gnu/store/qn1ax1fkj16x280m1rv7mcimfmn9l2pf-bash-4.4.23/bin/sh|g' run-in-namespace.c
echo '#define PROOT_PROGRAM "iyd2ikxadcp89j5919pwja6swnx00493-proot-static-5.1.0/bin/proot"' > new
cat run-in-namespace.c >> new
mv new run-in-namespace.c
gcc -std=gnu99 -static -O0 -g -Wall run-in-namespace.c

but it does not make a difference to the gdb output except the line

> 278	      && lstat ("/mybin/sh", &statbuf) != 0)

Regards,
Florian

Message #11 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Fri, 10 May 2019 23:50:19 +0200

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> The manual gives the following example of guix pack -RR:
>
>       guix pack -RR -S /mybin=bin bash
>       tar xf pack.tar.gz
>       ./mybin/sh
>
> This fails on my university’s server for students which uses Linux
> container “VMs” with Ubuntu and has no user namespace support and Guix
> is not installed.  This single line is all output:
>
> $ ./mybin/sh
> sh: run.c:162: bind_mount: Unexpected error: Permission denied.

That suggests the wrapper chose the user namespace method (not PRoot),
but that didn’t quite work.

Could you post the output of:

  strace ./mybin/sh

?

TIA!

Ludo’.

Message #14 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Sat, 11 May 2019 07:05:18 +0200

On Fri, May 10, 2019 at 11:50:19PM +0200, Ludovic Courtès wrote:
> That suggests the wrapper chose the user namespace method (not PRoot),
> but that didn’t quite work.
> 
> Could you post the output of:
> 
>   strace ./mybin/sh
> 
> ?
> 

[f_pelz12 <at> tux6 ~]$ strace ./mybin/sh 
execve("./mybin/sh", ["./mybin/sh"], 0x7fffcdf87290 /* 39 vars */) = 0
brk(NULL)                               = 0x2301000
brk(0x23021c0)                          = 0x23021c0
arch_prctl(ARCH_SET_FS, 0x2301880)      = 0
uname({sysname="Linux", nodename="tux6", ...}) = 0
readlink("/proc/self/exe", "/home/f_pelz12/gnu/store/wl2l59l"..., 4096) = 77
brk(0x23231c0)                          = 0x23231c0
brk(0x2324000)                          = 0x2324000
readlink("/proc/self/exe", "/home/f_pelz12/gnu/store/wl2l59l"..., 4095) = 77
lstat("/gnu/store/qn1ax1fkj16x280m1rv7mcimfmn9l2pf-bash-4.4.23/bin/sh", 0x7ffd9741c980) = -1 ENOENT (No such file or directory)
gettimeofday({tv_sec=1557550876, tv_usec=116037}, NULL) = 0
getpid()                                = 28923
mkdir("/tmp/guix-exec-ABt7cT", 0700)    = 0
stat(".", {st_mode=S_IFDIR|0700, st_size=113, ...}) = 0
stat("/home/f_pelz12", {st_mode=S_IFDIR|0700, st_size=113, ...}) = 0
clone(child_stack=NULL, flags=CLONE_NEWNS|CLONE_NEWUSER|SIGCHLD) = 28924
openat(AT_FDCWD, "/proc/28924/setgroups", O_WRONLY) = 3
write(3, "deny\0", 5)                   = 5
close(3)                                = 0
getuid()                                = 24038
openat(AT_FDCWD, "/proc/28924/uid_map", O_WRONLY) = 3
write(3, "24038 24038 1\n", 14)         = 14
close(3)                                = 0
getgid()                                = 10004
openat(AT_FDCWD, "/proc/28924/gid_map", O_WRONLY) = 3
write(3, "10004 10004 1\n", 14)         = 14
close(3)                                = 0
wait4(28924, sh: run.c:162: bind_mount: Unexpected error: Permission denied.
[{WIFSIGNALED(s) && WTERMSIG(s) == SIGABRT}], 0, NULL) = 28924
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=28924, si_uid=24038, si_status=SIGABRT, si_utime=0, si_stime=0} ---
chdir("/")                              = 0
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0700, st_size=25, ...}) = 0
getdents64(3, /* 25 entries */, 131072) = 632
unlink("/tmp/guix-exec-ABt7cT/home")    = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/home", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4e71c68000
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
munmap(0x7f4e71c68000, 135168)          = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/home")     = 0
unlink("/tmp/guix-exec-ABt7cT/tmp")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/tmp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
brk(0x2363000)                          = 0x2363000
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/tmp")      = 0
unlink("/tmp/guix-exec-ABt7cT/mnt")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/mnt", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/mnt")      = 0
unlink("/tmp/guix-exec-ABt7cT/sys")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/sys", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/sys")      = 0
unlink("/tmp/guix-exec-ABt7cT/libx32")  = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/libx32", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/libx32")   = 0
unlink("/tmp/guix-exec-ABt7cT/opt")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/opt", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/opt")      = 0
unlink("/tmp/guix-exec-ABt7cT/srv")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/srv", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/srv")      = 0
unlink("/tmp/guix-exec-ABt7cT/dev")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/dev", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/dev")      = 0
unlink("/tmp/guix-exec-ABt7cT/var")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/var", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/var")      = 0
unlink("/tmp/guix-exec-ABt7cT/sbin")    = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/sbin", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/sbin")     = 0
unlink("/tmp/guix-exec-ABt7cT/lib64")   = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/lib64", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/lib64")    = 0
unlink("/tmp/guix-exec-ABt7cT/lib32")   = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/lib32", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/lib32")    = 0
unlink("/tmp/guix-exec-ABt7cT/media")   = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/media", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/media")    = 0
unlink("/tmp/guix-exec-ABt7cT/usr")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/usr", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/usr")      = 0
unlink("/tmp/guix-exec-ABt7cT/bin")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/bin", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/bin")      = 0
unlink("/tmp/guix-exec-ABt7cT/boot")    = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/boot", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/boot")     = 0
unlink("/tmp/guix-exec-ABt7cT/etc")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/etc", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/etc")      = 0
unlink("/tmp/guix-exec-ABt7cT/run")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/run", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/run")      = 0
unlink("/tmp/guix-exec-ABt7cT/core")    = 0
unlink("/tmp/guix-exec-ABt7cT/snap")    = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/snap", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/snap")     = 0
unlink("/tmp/guix-exec-ABt7cT/lib")     = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/lib", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/lib")      = 0
unlink("/tmp/guix-exec-ABt7cT/proc")    = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/proc", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/proc")     = 0
unlink("/tmp/guix-exec-ABt7cT/root")    = -1 EISDIR (Is a directory)
openat(AT_FDCWD, "/tmp/guix-exec-ABt7cT/root", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0
getdents64(4, /* 2 entries */, 131072)  = 48
getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
rmdir("/tmp/guix-exec-ABt7cT/root")     = 0
getdents64(3, /* 0 entries */, 131072)  = 0
close(3)                                = 0
rmdir("/tmp/guix-exec-ABt7cT")          = 0
exit_group(6)                           = ?
+++ exited with 6 +++

Thank you for looking into it!

Regards,
Florian

Message #17 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Mon, 13 May 2019 09:49:40 +0200

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> On Fri, May 10, 2019 at 11:50:19PM +0200, Ludovic Courtès wrote:
>> That suggests the wrapper chose the user namespace method (not PRoot),
>> but that didn’t quite work.
>> 
>> Could you post the output of:
>> 
>>   strace ./mybin/sh
>> 
>> ?

My bad, this should be:

  strace -f -o log ./mybin/sh

and then post the ‘log’ file (we need ‘-f’ because the problem happens
in the child process.)

Thanks in advance,
Ludo’.

Message #20 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Mon, 13 May 2019 12:34:40 +0200

[Message part 1 (text/plain, inline)]

On Mon, May 13, 2019 at 09:49:40AM +0200, Ludovic Courtès wrote:
> Hi Florian,
> 
> "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:
> 
> > On Fri, May 10, 2019 at 11:50:19PM +0200, Ludovic Courtès wrote:
> >> That suggests the wrapper chose the user namespace method (not PRoot),
> >> but that didn’t quite work.
> >> 
> >> Could you post the output of:
> >> 
> >>   strace ./mybin/sh
> >> 
> >> ?
> 
> My bad, this should be:
> 
>   strace -f -o log ./mybin/sh
> 
> and then post the ‘log’ file (we need ‘-f’ because the problem happens
> in the child process.)
> 
> Thanks in advance,
> Ludo’.

Oh I did not know there is -f.

[f_pelz12 <at> tux6 ~]$ strace -f -o log ./mybin/sh
sh: run.c:162: bind_mount: Unexpected error: Permission denied.

The log file is attached.

When I do not use -o log, the unexpected error is here:

[pid 36622] mount("//sys", "/tmp/guix-exec-85li6j/sys", 0x47e0c5, MS_RDONLY|MS_BIND|MS_REC, NULL) = -1 EACCES (Permission denied)
[pid 36622] openat(AT_FDCWD, "/tmp/guix-exec-85li6j/core", O_WRONLY|O_CREAT, 056306) = 4
[pid 36622] close(4)                    = 0
[pid 36622] mount("//core", "/tmp/guix-exec-85li6j/core", 0x47e0c5, MS_RDONLY|MS_BIND|MS_REC, NULL) = -1 EACCES (Permission denied)
[pid 36622] write(2, "sh: run.c:162: bind_mount: Unexp"..., 64sh: run.c:162: bind_mount: Unexpected error: Permission denied.
) = 64
[pid 36622] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4b024f4000
[pid 36622] rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
[pid 36622] rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
[pid 36622] getpid()                    = 36622
[pid 36622] gettid()                    = 36622
[pid 36622] tgkill(36622, 36622, SIGABRT) = 0
[pid 36622] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 36622] --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=36622, si_uid=24038} ---
[pid 36622] +++ killed by SIGABRT +++


Regards,
Florian

[log (text/plain, attachment)]

Message #23 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Mon, 13 May 2019 15:54:11 +0200

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> 32476 clone(child_stack=NULL, flags=CLONE_NEWNS|CLONE_NEWUSER|SIGCHLD) = 32477

[...]

> 32477 mount("//lib", "/tmp/guix-exec-eqHoYA/lib", 0x47e0c5, MS_RDONLY|MS_BIND|MS_REC, NULL) = -1 EACCES (Permission denied)
> 32477 mkdir("/tmp/guix-exec-eqHoYA/home", 0700) = 0
> 32477 mount("//home", "/tmp/guix-exec-eqHoYA/home", 0x47e0c5, MS_RDONLY|MS_BIND|MS_REC, NULL) = -1 EACCES (Permission denied)

This is weird.  On a machine without Guix and with “proper” user
namespace support, I see:

--8<---------------cut here---------------start------------->8---
4519  clone(child_stack=0, flags=CLONE_NEWNS|CLONE_NEWUSER|SIGCHLD) = 4520

[...]

4520  mkdir("/tmp/guix-exec-4lVNRO/tmp", 0700) = 0
4520  mount("//tmp", "/tmp/guix-exec-4lVNRO/tmp", 0x47e0cc, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
4520  mkdir("/tmp/guix-exec-4lVNRO/boot", 0700) = 0
4520  mount("//boot", "/tmp/guix-exec-4lVNRO/boot", 0x47e0cc, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
--8<---------------cut here---------------end--------------->8---

That is, all bind-mount operations in the child process, which lives in
a separate namespace, succeed.

Can you show the mount options of you root file system?

  mount | grep 'on / '

What’s the exit code of this command:

  guile -c '((@@ (guix scripts environment) assert-container-features))'

?

Thanks for helping out!

Ludo’.

Message #26 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Mon, 13 May 2019 17:17:36 +0200

On Mon, May 13, 2019 at 03:54:11PM +0200, Ludovic Courtès wrote:
> Can you show the mount options of you root file system?
> 
>   mount | grep 'on / '
> 

[f_pelz12 <at> tux6 ~]$ mount | grep 'on / '
rpool/data/subvol-161199-disk-0 on / type zfs (rw,noatime,xattr,posixacl)


> What’s the exit code of this command:
> 
>   guile -c '((@@ (guix scripts environment) assert-container-features))'
> 
> ?
>

Guix is not installed.  Using a Guix git repository in ~/guix:

[f_pelz12 <at> tux6 guix]$ guile -c '((@@ (guix scripts environment) assert-container-features))'
[…]
;;; In procedure scm_lreadr: guix/packages.scm:534:11: Unknown # object: #\~
ERROR: In procedure primitive-load-path:
In procedure scm_lreadr: guix/packages.scm:534:11: Unknown # object: #\~


The line in question is:

        #~(begin
            (use-modules (ice-9 ftw)

I do not see how to make it recognize gexps.

If I wanted to compile Guix myself, the configure script reports
various missing dependencies (guile-gnutls is among them).  I could
ask the admin tomorrow if they could set up guix on a test “virtual
machine”/container.

I will instead now try this from gnu/build/linux-container.scm:

scheme@(guile-user)> (define (user-namespace-supported?)
  "Return #t if user namespaces are supported on this system."
  (file-exists? "/proc/self/ns/user"))

(define (unprivileged-user-namespace-supported?)
  "Return #t if user namespaces can be created by unprivileged users."
  (let ((userns-file "/proc/sys/kernel/unprivileged_userns_clone"))
    (if (file-exists? userns-file)
        (eqv? #\1 (call-with-input-file userns-file read-char))
        #t)))

(define (setgroups-supported?)
  "Return #t if the setgroups proc file, introduced in Linux-libre 3.19,
exists."
  (file-exists? "/proc/self/setgroups"))

scheme@(guile-user)> (user-namespace-supported?)
$1 = #t
scheme@(guile-user)> (unprivileged-user-namespace-supported?)
$2 = #t
scheme@(guile-user)> (setgroups-supported?)
$3 = #t

Regards,
Florian

Message #29 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Mon, 13 May 2019 22:39:21 +0200

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> On Mon, May 13, 2019 at 03:54:11PM +0200, Ludovic Courtès wrote:
>> Can you show the mount options of you root file system?
>> 
>>   mount | grep 'on / '
>> 
>
> [f_pelz12 <at> tux6 ~]$ mount | grep 'on / '
> rpool/data/subvol-161199-disk-0 on / type zfs (rw,noatime,xattr,posixacl)

I suspect ZFS-on-Linux (right?) is doing something unusual here:
mount(2) specifies the following reasons for EACCESS, and I don’t see
anything that would apply:

--8<---------------cut here---------------start------------->8---
EACCES A component of a path was not searchable.  (See also path_resolution(7).)

EACCES Mounting a read-only filesystem was attempted without giving the
       MS_RDONLY flag.

       The file system may be read-only for various reasons, including:
       it resides on a read-only optical disk; it is resides on a device
       with a physical switch that has been set to mark the device read-
       only; the filesystem implementation was compiled with read-only
       support; or errors were detected when initially mounting the
       filesystem, so that it was marked read-only and can't be
       remounted as read-write (until the errors are fixed).

       Some filesystems instead return the error EROFS on an attempt to
       mount a read-only filesystem.

EACCES The block device source is located on a filesystem mounted with
       the MS_NODEV option.
--8<---------------cut here---------------end--------------->8---

What do the following commands do on this system?

--8<---------------cut here---------------start------------->8---
$ mkdir -p /tmp/test/lib
$ unshare -mrf mount /lib /tmp/test/lib -o bind,readonly
--8<---------------cut here---------------end--------------->8---

Thanks,
Ludo’.

Message #32 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Mon, 13 May 2019 22:45:24 +0200

On Mon, May 13, 2019 at 10:39:21PM +0200, Ludovic Courtès wrote:
> I suspect ZFS-on-Linux (right?) is doing something unusual here:

I suppose it is ZFS on Linux; it is Linux, I can ask the admins if it
could be something else.



> What do the following commands do on this system?
> 
> --8<---------------cut here---------------start------------->8---
> $ mkdir -p /tmp/test/lib
> $ unshare -mrf mount /lib /tmp/test/lib -o bind,readonly
> --8<---------------cut here---------------end--------------->8---
> 

[f_pelz12 <at> tux6 ~]$ mkdir -p /tmp/test/lib
[f_pelz12 <at> tux6 ~]$ unshare -mrf mount /lib /tmp/test/lib -o bind,readonly
unshare: cannot change root filesystem propagation: Permission denied

Thank *you*, Ludo!  A working guix pack would be helpful for me.

Regards,
Florian

Message #35 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Tue, 14 May 2019 10:05:25 +0200

On Mon, May 13, 2019 at 10:45:24PM +0200, pelzflorian (Florian Pelz) wrote:
> On Mon, May 13, 2019 at 10:39:21PM +0200, Ludovic Courtès wrote:
> > I suspect ZFS-on-Linux (right?) is doing something unusual here:
> 
> I suppose it is ZFS on Linux; it is Linux, I can ask the admins if it
> could be something else.
> 

The admins have confirmed that they use “Proxmox on ZFS” (judging from
<https://pve.proxmox.com/wiki/ZFS_on_Linux> it is ZFS on Linux) and
they have confirmed that they have disabled user namespaces in their
Proxmox settings.

Regards,
Florian

Message #38 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Tue, 14 May 2019 22:43:56 +0200

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> On Mon, May 13, 2019 at 10:45:24PM +0200, pelzflorian (Florian Pelz) wrote:
>> On Mon, May 13, 2019 at 10:39:21PM +0200, Ludovic Courtès wrote:
>> > I suspect ZFS-on-Linux (right?) is doing something unusual here:
>> 
>> I suppose it is ZFS on Linux; it is Linux, I can ask the admins if it
>> could be something else.
>> 
>
> The admins have confirmed that they use “Proxmox on ZFS” (judging from
> <https://pve.proxmox.com/wiki/ZFS_on_Linux> it is ZFS on Linux) and
> they have confirmed that they have disabled user namespaces in their
> Proxmox settings.

User namespaces are orthogonal to file systems, but anyway it looks like
ZFS is refusing to let us do these things.

I don’t have any great option to offer.  You could perhaps modify
run-in-namespace.c so that it doesn’t even try user namespaces and
instead goes directly to the PRoot option?

However working around this behavior of ZFS it not completely trivial
and I’m not sure we should put much energy to paper over non-standard
file system behavior.

Thoughts?

Ludo’.

Message #41 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Tue, 14 May 2019 23:04:53 +0200

On Tue, May 14, 2019 at 10:43:56PM +0200, Ludovic Courtès wrote:
> "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:
> 
> > On Mon, May 13, 2019 at 10:45:24PM +0200, pelzflorian (Florian Pelz) wrote:
> >> On Mon, May 13, 2019 at 10:39:21PM +0200, Ludovic Courtès wrote:
> >> > I suspect ZFS-on-Linux (right?) is doing something unusual here:
> >> 
> >> I suppose it is ZFS on Linux; it is Linux, I can ask the admins if it
> >> could be something else.
> >> 
> >
> > The admins have confirmed that they use “Proxmox on ZFS” (judging from
> > <https://pve.proxmox.com/wiki/ZFS_on_Linux> it is ZFS on Linux) and
> > they have confirmed that they have disabled user namespaces in their
> > Proxmox settings.
> 
> User namespaces are orthogonal to file systems, but anyway it looks like
> ZFS is refusing to let us do these things.
> 

Do I understand correctly that user namespaces are not really disabled
(?) but fail on ZFS?  This seems strange, but a Web search for “zfs
user namespaces” shows other people having trouble with this
combination.  The admins told me they had to disable user namespaces
because it caused some kind of trouble.

> I don’t have any great option to offer.  You could perhaps modify
> run-in-namespace.c so that it doesn’t even try user namespaces and
> instead goes directly to the PRoot option?
>
> However working around this behavior of ZFS it not completely trivial
> and I’m not sure we should put much energy to paper over non-standard
> file system behavior.
> 
> Thoughts?
>

If ZFS makes user namespaces fail, then could run-un-namespace.c fall
back to PRoot when detecting ZFS, somehow?

Regards,
Florian

Message #44 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Giovanni Biscuolo <g <at> xelera.eu>
To: Ludovic Courtès <ludo <at> gnu.org>, "pelzflorian (Florian
 Pelz)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Wed, 15 May 2019 17:20:25 +0200

[Message part 1 (text/plain, inline)]

Hello Ludovic and Florian,

I cannot help here, just some thoughts

as you probably already know, Florian, ZFS is not supported in Linux for
various reasons, above all for a controversial licensing problem [1]

so using zfsonlinux (the ZFS Linux unofficial kernel module) is
basically calling for problems

Ludovic Courtès <ludo <at> gnu.org> writes:

> "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

[...]

>> The admins have confirmed that they use “Proxmox on ZFS” (judging from
>> <https://pve.proxmox.com/wiki/ZFS_on_Linux> it is ZFS on Linux)

it's not clearly stated there, I guess it's
https://github.com/zfsonlinux/zfs

>> and
>> they have confirmed that they have disabled user namespaces in their
>> Proxmox settings.

I do not understand what this means: if namespaces are disabled **in
kernel** that whould be detected and guix relocatable binaries should
use PRoot by default: am I wrong?

If "disabled user namespace in Promox settings" means it have something
to do with ZFS filesystem settings, well: it's unorthodox at least :-)

> User namespaces are orthogonal to file systems, but anyway it looks like
> ZFS is refusing to let us do these things.

I don't know if this have something to do with this bug, but:

ZFS is confused by user namespaces (uid/gid mapping) when used with acltype=posixacl
https://github.com/zfsonlinux/zfs/issues/4177

Florian: it should be solved but AFAIU it depends on the
kernel/zfsonlinux combination

> I don’t have any great option to offer.  You could perhaps modify
> run-in-namespace.c so that it doesn’t even try user namespaces and
> instead goes directly to the PRoot option?

Ludovic (and others): is it possible to add an option to "guix pack -RR"
(-RRF?!?) to force the use of PRoot for resulting relocated binaries?

> However working around this behavior of ZFS it not completely trivial
> and I’m not sure we should put much energy to paper over non-standard
> file system behavior.

I agree, this seems a zfsonlinux bug: Florian please can you report it
upstream to zfsonlinux?

[...]

HTH! Gio'


[1] https://www.fsf.org/licensing/zfs-and-linux
    https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[signature.asc (application/pgp-signature, inline)]

Message #47 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Wed, 15 May 2019 18:15:19 +0200

Hi,

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> Do I understand correctly that user namespaces are not really disabled
> (?) but fail on ZFS?

Correct.  Specifically, read-only bind mounts of (and to?) files that
reside on ZFS fail with EACCESS, which is normally “impossible.”

It would be great if you could ask the admins specifically what they did
in relation to user namespaces.

>> I don’t have any great option to offer.  You could perhaps modify
>> run-in-namespace.c so that it doesn’t even try user namespaces and
>> instead goes directly to the PRoot option?
>>
>> However working around this behavior of ZFS it not completely trivial
>> and I’m not sure we should put much energy to paper over non-standard
>> file system behavior.
>> 
>> Thoughts?
>>
>
> If ZFS makes user namespaces fail, then could run-un-namespace.c fall
> back to PRoot when detecting ZFS, somehow?

It’s code, so everything is possible :-), but like I wrote it’s a bit of
work, and it’s something that cannot happen (AFAIK) with file systems
that are part of Linux.

Thanks,
Ludo’.

Message #50 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Giovanni Biscuolo <g <at> xelera.eu>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Thu, 16 May 2019 13:02:57 +0200

On Wed, May 15, 2019 at 05:20:25PM +0200, Giovanni Biscuolo wrote:
> Hello Ludovic and Florian,
> 
> I cannot help here, just some thoughts
> 
> as you probably already know, Florian, ZFS is not supported in Linux for
> various reasons, above all for a controversial licensing problem [1]
> 

I had forgotten.  I remember now that I heard about this.


From a Guix point of view, I believe this maybe should be a
WONT-FIX/NOT-OUR-BUG.  I will try and set up current ZFS 0.7.13 and
test if guix pack -RR works there in a week.

Feel free to skip this unless you are interested:

I asked the admins again.  They are using Proxmox 5.4.  They say they
have disabled user namespaces by commenting the corresponding line in
the Proxmox config file (but I am unsure if this just disables Linux
Container use of user namespaces or something).  They use the ZFS from
Proxmox.  I looked and found confirmation that this Proxmox uses
current ZFS 0.7.13 at:

https://pve.proxmox.com/wiki/Roadmap

http://download.proxmox.com/debian/pve/dists/stretch/pve-no-subscription/binary-amd64/


> I agree, this seems a zfsonlinux bug: Florian please can you report it
> upstream to zfsonlinux?
> 

I will try to reproduce on a private PC in a week, then I can report.

Regards,
Florian

Message #53 received at 35662 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "pelzflorian \(Florian Pelz\)" <pelzflorian <at> pelzflorian.de>
Cc: Giovanni Biscuolo <g <at> xelera.eu>, 35662 <at> debbugs.gnu.org
Subject: Re: bug#35662: Really relocatable binaries crash with Permission
 denied
Date: Thu, 16 May 2019 13:10:19 +0200

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> skribis:

> From a Guix point of view, I believe this maybe should be a
> WONT-FIX/NOT-OUR-BUG.

Sounds good to me.  :-)

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.