Package: guix;
Reported by: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org>
Date: Wed, 1 Jul 2020 14:30:01 UTC
Severity: normal
Tags: patch
Done: Jan Nieuwenhuizen <janneke <at> gnu.org>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 42151 in the body.
You can then email your comments to 42151 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Wed, 01 Jul 2020 14:30:01 GMT) Full text and rfc822 format available."Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org>
:bug-guix <at> gnu.org
.
(Wed, 01 Jul 2020 14:30:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: bug-guix <at> gnu.org Subject: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Wed, 1 Jul 2020 16:29:29 +0200
Hello Guix! I configured my system with a childhurd service and added --8<---------------cut here---------------start------------->8--- (build-machine (name "childhurd") (system "i586-gnu") (host-key "ssh-ed25519 ... root <at> childhurd") (user "root") (private-key "/home/janneke/.ssh/id_rsa_childhurd")) --8<---------------cut here---------------end--------------->8--- to /etc/guix/machines. Sadly, running guix offload test gives --8<---------------cut here---------------start------------->8--- $ guix offload test guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'... guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test") guix offload: 'childhurd' is running GNU Guile 3.0.4 guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>) --8<---------------cut here---------------end--------------->8--- So, I looked into the Debian/Hurd sqlite3 package for inspiration and found debian/patches/20-hurd-locking-style.patch...well, that should fix it, right? Sadly, this patch seems to break sqlite3 altogether. --8<---------------cut here---------------start------------->8--- $ guix offload test guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'... Backtrace: In ice-9/boot-9.scm: 1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) In unknown file: 10 (apply-smob/0 #<thunk 7f3c99b1f480>) In ice-9/boot-9.scm: 718:2 9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>) In ice-9/eval.scm: 619:8 8 (_ #(#(#<directory (guile-user) 7f3c99743f00>))) In guix/ui.scm: 1949:12 7 (run-guix-command _ . _) 663:2 6 (call-with-error-handling _) In ice-9/boot-9.scm: 1736:10 5 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) 1731:15 4 (with-exception-handler #<procedure 7f3c9546d3f0 at ice-9/boot-9.scm:1815:7 (exn)> _ #:unwind? _ # …) In guix/scripts/offload.scm: 647:6 3 (check-machine-availability _ _) In srfi/srfi-1.scm: 650:11 2 (for-each #<procedure assert-node-has-guix (node name)> (#<<inferior> pid: pipe socket: #<input…>) …) In guix/scripts/offload.scm: 582:2 1 (assert-node-has-guix #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500…> …) In ice-9/boot-9.scm: 1669:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1669:16: In procedure raise-exception: ERROR: 1. &inferior-exception: arguments: (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: unable to open database file" status: 1>>) inferior: #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500> close: #<procedure close-port (_)> version: (0 1 1) packages: #<promise #<procedure 7f3c954a8000 at guix/inferior.scm:161:32 ()>> table: #<promise #<procedure 7f3c954a56f0 at guix/inferior.scm:162:32 ()>>> stack: ((#f ("ice-9/boot-9.scm" 1763 13)) (raise-exception ("ice-9/boot-9.scm" 1666 16)) (raise-exception ("ice-9/boot-9.scm" 1668 16)) (#f ("guix/store.scm" 582 42)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (open-connection ("guix/store.scm" 545 2)) (call-with-store ("guix/store.scm" 626 15)) (#f (#f #f #f)) (#f ("guix/repl.scm" 92 21)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (with-exception-handler ("ice-9/boot-9.scm" 1730 15)) (#f ("guix/repl.scm" 119 7))) --8<---------------cut here---------------end--------------->8--- Maybe we're missing some file_lock patch that Debian has? Where to look, glibc, hurd, ...? Ideas? Greetings, Janneke Jan (janneke) Nieuwenhuizen (3): gnu: Add sqlite/hurd with locking fix. gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd. gnu: guix: Use sqlite/hurd for locking on the Hurd. gnu/local.mk | 1 + gnu/packages/guile.scm | 2 +- gnu/packages/package-management.scm | 2 +- gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++ gnu/packages/sqlite.scm | 9 +++++ 5 files changed, 63 insertions(+), 2 deletions(-) create mode 100644 gnu/packages/patches/sqlite3-hurd.patch -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Wed, 01 Jul 2020 14:32:01 GMT) Full text and rfc822 format available.Message #8 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Subject: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd. Date: Wed, 1 Jul 2020 16:31:17 +0200
This fixes guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>) * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of sqlite. --- gnu/packages/guile.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm index a32bd33293..c651e2427a 100644 --- a/gnu/packages/guile.scm +++ b/gnu/packages/guile.scm @@ -665,7 +665,7 @@ Guile's foreign function interface.") ("pkg-config" ,pkg-config))) (inputs `(("guile" ,guile-3.0) - ("sqlite" ,sqlite))) + ("sqlite" ,sqlite/hurd))) (synopsis "Access SQLite databases from Guile") (description "This package provides Guile bindings to the SQLite database system.") -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Wed, 01 Jul 2020 14:32:02 GMT) Full text and rfc822 format available.Message #11 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Subject: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix. Date: Wed, 1 Jul 2020 16:31:16 +0200
* gnu/packages/patches/sqlite3-hurd.patch: New file. * gnu/local.mk (dist_patch_DATA): Add it. * gnu/packages/sqlite.scm (sqlite/hurd): New variable. --- gnu/local.mk | 1 + gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++ gnu/packages/sqlite.scm | 9 +++++ 3 files changed, 61 insertions(+) create mode 100644 gnu/packages/patches/sqlite3-hurd.patch diff --git a/gnu/local.mk b/gnu/local.mk index 5e9dba5ab7..d5451bf2d5 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -1358,6 +1358,7 @@ dist_patch_DATA = \ %D%/packages/patches/sdl-pango-header-guard.patch \ %D%/packages/patches/sdl-pango-matrix_declarations.patch \ %D%/packages/patches/sdl-pango-sans-serif.patch \ + %D%/packages/patches/sqlite3-hurd.patch \ %D%/packages/patches/patchutils-test-perms.patch \ %D%/packages/patches/patch-hurd-path-max.patch \ %D%/packages/patches/perl-autosplit-default-time.patch \ diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patch new file mode 100644 index 0000000000..de87a30cb1 --- /dev/null +++ b/gnu/packages/patches/sqlite3-hurd.patch @@ -0,0 +1,51 @@ +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch +Upstream status: Not offered upstream. + +diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c +--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100 ++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200 +@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode + # include <sys/mman.h> + #endif + +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + /* # include <sys/ioctl.h> */ + # include <sys/file.h> + # include <sys/param.h> +@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id + ** + ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off + */ +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + + /* + ** Retry flock() calls that fail with EINTR +@@ -38586,7 +38586,7 @@ IOMETHODS( + 0 /* xShmMap method */ + ) + +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + IOMETHODS( + flockIoFinder, /* Finder function name */ + flockIoMethods, /* sqlite3_io_methods object name */ +@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){ + UNIXVFS("unix", autolockIoFinder ), + #elif OS_VXWORKS + UNIXVFS("unix", vxworksIoFinder ), ++#elif defined(__GNU__) ++ UNIXVFS("unix", flockIoFinder ), + #else + UNIXVFS("unix", posixIoFinder ), + #endif +@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){ + #if OS_VXWORKS + UNIXVFS("unix-namedsem", semIoFinder ), + #endif +-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS ++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__) + UNIXVFS("unix-posix", posixIoFinder ), + #endif + #if SQLITE_ENABLE_LOCKING_STYLE diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm index 8468131ddf..d5c44b0d0f 100644 --- a/gnu/packages/sqlite.scm +++ b/gnu/packages/sqlite.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2017 Jelle Licht <jlicht <at> fsfe.org> ;;; Copyright © 2018 Tobias Geerinckx-Rice <me <at> tobias.gr> ;;; Copyright © 2018 Alex Vong <alexvong1995 <at> gmail.com> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -107,3 +108,11 @@ is in the public domain.") ;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7. (define-public sqlite-with-column-metadata (deprecated-package "sqlite-with-column-metadata" sqlite)) + +(define-public sqlite/hurd + ;; TODO move into sqlite on the next rebuild cycle. + (package + (inherit sqlite) + (name "sqlite-for-hurd") + (source (origin (inherit (package-source sqlite)) + (patches (search-patches "sqlite3-hurd.patch")))))) -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Wed, 01 Jul 2020 14:32:02 GMT) Full text and rfc822 format available.Message #14 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Subject: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd. Date: Wed, 1 Jul 2020 16:31:18 +0200
* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd. --- gnu/packages/package-management.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 068d46566c..52c5a1e60d 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n"))) ("zlib" ,zlib) ;for 'guix publish' ("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute' - ("sqlite" ,sqlite) + ("sqlite" ,sqlite/hurd) ("libgcrypt" ,libgcrypt) ("guile" ,guile-3.0-latest) -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Wed, 01 Jul 2020 17:02:01 GMT) Full text and rfc822 format available.Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: bug-guix <at> gnu.org Subject: Re: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Wed, 01 Jul 2020 19:01:08 +0200
Jan (janneke) Nieuwenhuizen writes: > Maybe we're missing some file_lock patch that Debian has? Where to look, > glibc, hurd, ...? Ideas? So...I found a way to reproduce the feature/bug: run "pragma synchronous = normal;" in two instances. I created a db.sqlite using sh -x tests/guix-pack-localstatedir.sh (after removing the cleanup trap). Then, doing --8<---------------cut here---------------start------------->8--- $ sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54 Enter ".help" for usage hints. sqlite> SELECT * FROM ValidPaths; 1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||2664 2|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||1351880 3|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||12718808 4|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||432 5|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960 sqlite> pragma synchronous = normal; sqlite> C-c C-z^Z [2]+ Stopped sqlite3 db.sqlite [148]18:55:06 janneke <at> dundal:~/src/guix/master [env] $ sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54 Enter ".help" for usage hints. sqlite> pragma synchronous = normal; sqlite> --8<---------------cut here---------------end--------------->8--- works nice on GNU/Linux and on current master Childhurd gives --8<---------------cut here---------------start------------->8--- $ ssh childhurd Last login: Wed Jul 1 12:08:17 2020 from 10.0.2.2 This is the GNU Hurd. Welcome. root <at> childhurd ~# sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54 Enter ".help" for usage hints. sqlite> SELECT * FROM ValidPaths; 1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||2664 2|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||1351880 3|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||12718808 4|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||432 5|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960 sqlite> pragma synchronous = normal; sqlite> C-c C-z^Z [1]+ Stopped sqlite3 db.sqlite root <at> childhurd ~# sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54 Enter ".help" for usage hints. sqlite> pragma synchronous = normal; Error: locking protocol sqlite> --8<---------------cut here---------------end--------------->8--- On Debian/Hurd (which I hoped had a patched, fixed sqlite3), it doesn't work at all --8<---------------cut here---------------start------------->8--- $ ssh hurd GNU debian 0.9 GNU-Mach 1.8+git20191117-486/Hurd-0.9 i686-AT386 This is the GNU Hurd. Welcome. The Hurd is not Linux. Make sure to read http://www.debian.org/ports/hurd/hurd-install to check out the few things you _need_ to know. Also check out the FAQ http://www.gnu.org/software/hurd/faq.html or its latest version on http://darnassus.sceen.net/~hurd-web/faq/ To read a short intro on some nice features of the Hurd, just have a look at the translator_primer file, for example via 'nano translator_primer' Last login: Wed Jul 1 12:58:44 2020 from 10.0.2.2 janneke <at> debian:~$ sqlite3 db.sqlite SQLite version 3.32.3 2020-06-18 14:00:33 Enter ".help" for usage hints. sqlite> SELECT * FROM ValidPaths; Error: unable to open database file sqlite> --8<---------------cut here---------------end--------------->8--- So, still a similar question...is the patch simply broken/bitrotted, or did some file locking change on the Hurd? Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Thu, 02 Jul 2020 12:50:02 GMT) Full text and rfc822 format available.Message #20 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: "Jan \(janneke\) Nieuwenhuizen" <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Thu, 02 Jul 2020 14:49:32 +0200
Hi! "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > $ guix offload test > guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'... > guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test") > guix offload: 'childhurd' is running GNU Guile 3.0.4 > guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>) Does sqlite pass its tests on GNU/Hurd? Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon? As a stop-gap, we could add a command-line option if that helps. The “synchronous = normal” mode translates to ‘fsync’ calls, right? If you rpctrace sqlite3, do you see ‘file_sync’ calls failing? My 2¢! Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 03 Jul 2020 10:04:01 GMT) Full text and rfc822 format available.Message #23 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Fri, 03 Jul 2020 12:03:26 +0200
Ludovic Courtès writes: Hi! > "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > >> $ guix offload test >> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'... >> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test") >> guix offload: 'childhurd' is running GNU Guile 3.0.4 >> guix offload: error: exception occurred on remote host 'localhost': >> (%exception #<inferior-object #<&store-protocol-error message: >> "setting synchronous mode: locking protocol" status: 1>>) > Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon? > As a stop-gap, we could add a command-line option if that helps. Tried that, thanks. No, it does not help. (But that's good news, see below!) > The “synchronous = normal” mode translates to ‘fsync’ calls, right? If > you rpctrace sqlite3, do you see ‘file_sync’ calls failing? Tried that before, rpctrace hangs before I see something useful. > Does sqlite pass its tests on GNU/Hurd? That's the (or at least a) right question: YES! > My 2¢! Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;) It turns out that Debian's patch (and thus this patch series) is probably OK: It fixes the locking problem on the Hurd, while exposing another bug, apparently: "unable to open database file". It seems there is a compatibility bug/problem/thing with the db.sqlite that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd can read it, and work with it, the unpatched sqlite has locking problems. I found a workaround, though: dumping and loading the database file. Look... --8<---------------cut here---------------start------------->8--- $ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig db.sqlite 100% 144KB 5.8MB/s 00:00 11:30:37 janneke <at> dundal:~/tmp [env] $ sqlite3 db.sqlite-orig .dump > db.dump 11:30:45 janneke <at> dundal:~/tmp [env] $ sqlite3 -init db.dump db.sqlite-init .quit -- Loading resources from db.dump 11:30:49 janneke <at> dundal:~/tmp [env] $ cmp db.sqlite-orig db.sqlite-init db.sqlite-orig db.sqlite-init differ: byte 19, line 1 [1]11:31:11 janneke <at> dundal:~/tmp [env] $ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite db.sqlite-init 100% 144KB 7.3MB/s 00:00 11:31:21 janneke <at> dundal:~/tmp [env] $ guix offload test guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'... guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test") guix offload: 'localhost' is running GNU Guile 3.0.4 sending 1 store item (0 MiB) to 'localhost'... exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test' guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test' retrieving 1 store item from 'localhost'... guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost' 11:31:29 janneke <at> dundal:~/tmp [env] --8<---------------cut here---------------end--------------->8--- So...about the compatibility problem. I tried to diff the db.sqlite-orig db.sqlite-init binary files: they look completely different. Not sure how to handle this workaround, maybe we can insert a two system* calls somewhere when building the disk image? Greetings, Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 03 Jul 2020 13:52:02 GMT) Full text and rfc822 format available.Message #26 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: Jan Nieuwenhuizen <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Fri, 03 Jul 2020 15:51:29 +0200
Hi! Jan Nieuwenhuizen <janneke <at> gnu.org> skribis: > Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;) Heheh. :-) > It turns out that Debian's patch (and thus this patch series) is > probably OK: It fixes the locking problem on the Hurd, while exposing > another bug, apparently: "unable to open database file". > > It seems there is a compatibility bug/problem/thing with the db.sqlite > that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd > can read it, and work with it, the unpatched sqlite has locking > problems. I found a workaround, though: dumping and loading the > database file. > > Look... > > $ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig > db.sqlite 100% 144KB 5.8MB/s 00:00 > 11:30:37 janneke <at> dundal:~/tmp [env] > $ sqlite3 db.sqlite-orig .dump > db.dump > 11:30:45 janneke <at> dundal:~/tmp [env] > $ sqlite3 -init db.dump db.sqlite-init .quit > -- Loading resources from db.dump > 11:30:49 janneke <at> dundal:~/tmp [env] > $ cmp db.sqlite-orig db.sqlite-init > db.sqlite-orig db.sqlite-init differ: byte 19, line 1 > [1]11:31:11 janneke <at> dundal:~/tmp [env] > $ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite > db.sqlite-init 100% 144KB 7.3MB/s 00:00 > 11:31:21 janneke <at> dundal:~/tmp [env] > $ guix offload test > guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'... > guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test") > guix offload: 'localhost' is running GNU Guile 3.0.4 > sending 1 store item (0 MiB) to 'localhost'... > exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test' > guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test' > retrieving 1 store item from 'localhost'... > guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost' > 11:31:29 janneke <at> dundal:~/tmp [env] > > So...about the compatibility problem. I tried to diff the db.sqlite-orig > db.sqlite-init binary files: they look completely different. Not sure > how to handle this workaround, maybe we can insert a two system* calls > somewhere when building the disk image? Weird, weird! Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created from the same ‘sqlite3 -init’ command on GNU/Linux? (Perhaps loading the dump reorders entries or something.) I think the binary format of the database is supposed to be architecture-independent and filling it is supposed to be deterministic. Congrats for getting this far anyway! Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 03 Jul 2020 15:28:01 GMT) Full text and rfc822 format available.Message #29 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Fri, 03 Jul 2020 17:27:40 +0200
Ludovic Courtès writes: Hi! >> It seems there is a compatibility bug/problem/thing with the db.sqlite >> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd >> can read it, and work with it, the unpatched sqlite has locking >> problems. I found a workaround, though: dumping and loading the >> database file. [..] >> So...about the compatibility problem. I tried to diff the db.sqlite-orig >> db.sqlite-init binary files: they look completely different. Not sure >> how to handle this workaround, maybe we can insert a two system* calls >> somewhere when building the disk image? > > Weird, weird! > > Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created > from the same ‘sqlite3 -init’ command on GNU/Linux? Yeah, they are identical. The initial dump can only be created atm on GNU/Linux; the dump can be loaded (obviously) anywhere like so >> $ sqlite3 -init db.dump db.sqlite-init .quit and the resulting initial db.sqlite is the same. Guess we can do that by hand for now... > (Perhaps loading the dump reorders entries or something.) Yes, "or something" certainly! :) I have no clue... If/when we decide to pinpoint this bug, what could be a first step? Who is creating the database right now, is that the C++ daemon or guile-sqlite3. IWBN to have that code create/save a smaller version and see when reading fails. > I think the binary format of the database is supposed to be > architecture-independent and filling it is supposed to be deterministic. Yes, that works. > Congrats for getting this far anyway! Yeah... \o/ Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Sat, 04 Jul 2020 08:14:01 GMT) Full text and rfc822 format available.Message #32 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. Date: Sat, 04 Jul 2020 10:12:47 +0200
[Message part 1 (text/plain, inline)]
Jan Nieuwenhuizen writes: Adding one additional patch to this series, hence 4/3 (see attached) fixes the sqlite problem. With this over-complete series we can now do offload builds to a childhurd: --8<---------------cut here---------------start------------->8--- $ guix build -e '(@@ (gnu packages commencement) rsync-boot0)' sending 74 store items (365 MiB) to 'localhost'... exporting path `/gnu/store/1kf05mhh31wl89r3mlx52x5sxrjivi7i-module-import-compiled-guile-builder' exporting path `/gnu/store/7jxamp3g8wdr6vwrisqfmrncxh8nbfb4-mirrors' [...] exporting path `/gnu/store/x8wnf0h0ipibgzadlbmll8bcnhvv1yaq-diffutils-boot0-3.7' offloading '/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' to 'localhost'... offloading build of /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv to 'localhost' [...] @ build-succeeded /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv - retrieving 1 store item from 'localhost'... importing file or directory '/gnu/store/haf6mlm8xa6s2q918s05pijl6ql17mnq-rsync-3.1.3'... guix offload: error: corrupt input while restoring archive from #<input: string 7ff4794cee00> guix build: error: build of `/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' failed [1]10:04:55 janneke <at> dundal:~/src/guix/master [env] --8<---------------cut here---------------end--------------->8--- After the build succeeds, the download fails but lets first get this patch series done... Janneke
[0004-database-Do-not-use-journal_model-WAL-for-the-Hurd.patch (text/x-patch, inline)]
From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001 From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> Date: Fri, 3 Jul 2020 23:45:20 +0200 Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 This fixes <https://bugs.gnu.org/42151>. * guix/store/database.scm (call-with-database): When building for the Hurd, do not set journal_model=WAL. --- guix/store/database.scm | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index a38e4d7e52..da46b0abce 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt <at> cune.org> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo <at> gnu.org> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -20,6 +21,7 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config) + #:use-module (guix gexp) #:use-module (guix serialization) #:use-module (guix store deduplication) #:use-module (guix base16) @@ -27,6 +29,7 @@ #:use-module (guix build syscalls) #:use-module ((guix build utils) #:select (mkdir-p executable-file?)) + #:use-module (guix utils) #:use-module (guix build store-copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) @@ -105,9 +108,12 @@ create it and initialize it as a new database." (mkdir-p (dirname file)) #t))) (db (sqlite-open file))) - ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED - ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. - (sqlite-exec db "PRAGMA journal_mode=WAL;") + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>. + (unless (let-system (system target) + (equal? target "i586-pc-gnu")) + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. + (sqlite-exec db "PRAGMA journal_mode=WAL;")) ;; Install a busy handler such that, when the database is locked, sqlite ;; retries until 30 seconds have passed, at which point it gives up and -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
[Message part 3 (text/plain, inline)]
-- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Tue, 07 Jul 2020 05:05:01 GMT) Full text and rfc822 format available.Message #35 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. Date: Tue, 07 Jul 2020 07:04:31 +0200
Jan Nieuwenhuizen writes: > Jan Nieuwenhuizen writes: > > Adding one additional patch to this series, hence 4/3 (see attached) > fixes the sqlite problem. For clarity -- patches 1-4 really do fix offloading to the Hurd and this... > After the build succeeds, the download fails but lets first get this > patch series done... ...is not true; you gat that when you (I) forgot to add keys in /etc/guix/ As seen on IRC, possibly Ricardo will an update for the QEMU command-line (at least for berlin), adding "--cpu base". Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 12:04:01 GMT) Full text and rfc822 format available.Message #38 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: Jan Nieuwenhuizen <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. Date: Fri, 10 Jul 2020 14:03:08 +0200
Hi! Jan Nieuwenhuizen <janneke <at> gnu.org> skribis: >>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001 > From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> > Date: Fri, 3 Jul 2020 23:45:20 +0200 > Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. > Content-Transfer-Encoding: 8bit > Content-Type: text/plain; charset=UTF-8 > > This fixes <https://bugs.gnu.org/42151>. > > * guix/store/database.scm (call-with-database): When building for the Hurd, > do not set journal_model=WAL. > --- > guix/store/database.scm | 12 +++++++++--- > 1 file changed, 9 insertions(+), 3 deletions(-) > > diff --git a/guix/store/database.scm b/guix/store/database.scm > index a38e4d7e52..da46b0abce 100644 > --- a/guix/store/database.scm > +++ b/guix/store/database.scm > @@ -1,6 +1,7 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt <at> cune.org> > ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo <at> gnu.org> > +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -20,6 +21,7 @@ > (define-module (guix store database) > #:use-module (sqlite3) > #:use-module (guix config) > + #:use-module (guix gexp) > #:use-module (guix serialization) > #:use-module (guix store deduplication) > #:use-module (guix base16) > @@ -27,6 +29,7 @@ > #:use-module (guix build syscalls) > #:use-module ((guix build utils) > #:select (mkdir-p executable-file?)) > + #:use-module (guix utils) > #:use-module (guix build store-copy) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-11) > @@ -105,9 +108,12 @@ create it and initialize it as a new database." > (mkdir-p (dirname file)) > #t))) > (db (sqlite-open file))) > - ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED > - ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. > - (sqlite-exec db "PRAGMA journal_mode=WAL;") > + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>. > + (unless (let-system (system target) > + (equal? target "i586-pc-gnu")) > + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED > + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. > + (sqlite-exec db "PRAGMA journal_mode=WAL;")) I think this has the same effect as: (unless #t (sqlite-exec db "PRAGMA journal_mode=WAL;")) because this code is not in a gexp, so ‘let-system’ evaluates to an object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call is never made. You want to disable WAL mode not just when running this code natively on GNU/Hurd, but also when building a database that will eventually be used on GNU/Hurd, right? In that case, I think you’ll have to add, say, a #:wal-mode? parameter to ‘call-with-database’ (defaulting to #true), and change the caller to set it appropriately. The caller, directly or indirectly, is in a gexp, where you can use the ‘let-system’ expression above. Does that make sense? Thanks, and apologies for the delay! Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 12:07:02 GMT) Full text and rfc822 format available.Message #41 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: "Jan \(janneke\) Nieuwenhuizen" <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix. Date: Fri, 10 Jul 2020 14:05:47 +0200
"Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > * gnu/packages/patches/sqlite3-hurd.patch: New file. > * gnu/local.mk (dist_patch_DATA): Add it. > * gnu/packages/sqlite.scm (sqlite/hurd): New variable. […] > +++ b/gnu/packages/patches/sqlite3-hurd.patch > @@ -0,0 +1,51 @@ > +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch > +Upstream status: Not offered upstream. The URL is 404, could you fix it? It would also be great if you could add a sentence or two explaining what this works around, so that our future selves know how to deal with this patch. :-) Otherwise LGTM, you can push with changes along these lines! Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 12:07:02 GMT) Full text and rfc822 format available.Message #44 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: "Jan \(janneke\) Nieuwenhuizen" <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd. Date: Fri, 10 Jul 2020 14:06:28 +0200
"Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > This fixes > > guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>) > > * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of > sqlite. > --- > gnu/packages/guile.scm | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm > index a32bd33293..c651e2427a 100644 > --- a/gnu/packages/guile.scm > +++ b/gnu/packages/guile.scm > @@ -665,7 +665,7 @@ Guile's foreign function interface.") > ("pkg-config" ,pkg-config))) > (inputs > `(("guile" ,guile-3.0) > - ("sqlite" ,sqlite))) > + ("sqlite" ,sqlite/hurd))) This should be guarded by ‘if (hurd-target?)’, right? Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 12:08:02 GMT) Full text and rfc822 format available.Message #47 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: "Jan \(janneke\) Nieuwenhuizen" <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd. Date: Fri, 10 Jul 2020 14:07:33 +0200
"Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd. Same here. Which makes me thing: how about applying the sqlite/hurd patch in a build phase of sqlite that would be Hurd-specific? That way, we’d still have a single sqlite package. Thanks, Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 12:35:01 GMT) Full text and rfc822 format available.Message #50 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: Jan Nieuwenhuizen <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol Date: Fri, 10 Jul 2020 14:34:28 +0200
Hi, Jan Nieuwenhuizen <janneke <at> gnu.org> skribis: > Ludovic Courtès writes: > > Hi! > >>> It seems there is a compatibility bug/problem/thing with the db.sqlite >>> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd >>> can read it, and work with it, the unpatched sqlite has locking >>> problems. I found a workaround, though: dumping and loading the >>> database file. > > [..] > >>> So...about the compatibility problem. I tried to diff the db.sqlite-orig >>> db.sqlite-init binary files: they look completely different. Not sure >>> how to handle this workaround, maybe we can insert a two system* calls >>> somewhere when building the disk image? >> >> Weird, weird! >> >> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created >> from the same ‘sqlite3 -init’ command on GNU/Linux? > > Yeah, they are identical. The initial dump can only be created atm on > GNU/Linux; the dump can be loaded (obviously) anywhere like so > >>> $ sqlite3 -init db.dump db.sqlite-init .quit > > and the resulting initial db.sqlite is the same. Guess we can do that > by hand for now... > >> (Perhaps loading the dump reorders entries or something.) > > Yes, "or something" certainly! :) I have no clue... Sorry for catching up days later but… did disabling WAL mode fix this discrepancy? IOW, without WAL mode, is the file produced on GNU/Linux bit-identical to that produced on GNU/Hurd? > If/when we decide to pinpoint this bug, what could be a first step? Who > is creating the database right now, is that the C++ daemon or > guile-sqlite3. IWBN to have that code create/save a smaller version and > see when reading fails. As you found out it’s created by (guix store database) in this case. Thanks, Ludo’.
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 15:41:01 GMT) Full text and rfc822 format available.Message #53 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. Date: Fri, 10 Jul 2020 17:40:23 +0200
Ludovic Courtès writes: Hi! > Jan Nieuwenhuizen <janneke <at> gnu.org> skribis: > >>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001 >> From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> >> Date: Fri, 3 Jul 2020 23:45:20 +0200 >> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd. >> Content-Transfer-Encoding: 8bit >> Content-Type: text/plain; charset=UTF-8 >> >> This fixes <https://bugs.gnu.org/42151>. [...] >> diff --git a/guix/store/database.scm b/guix/store/database.scm >> index a38e4d7e52..da46b0abce 100644 >> --- a/guix/store/database.scm >> +++ b/guix/store/database.scm [...] >> + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>. >> + (unless (let-system (system target) >> + (equal? target "i586-pc-gnu")) >> + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED >> + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. >> + (sqlite-exec db "PRAGMA journal_mode=WAL;")) > > I think this has the same effect as: > > (unless #t > (sqlite-exec db "PRAGMA journal_mode=WAL;")) > > because this code is not in a gexp, so ‘let-system’ evaluates to an > object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call > is never made. Oops. Great catch, thanks! > You want to disable WAL mode not just when running this code natively on > GNU/Hurd, but also when building a database that will eventually be used > on GNU/Hurd, right? Exactly. > In that case, I think you’ll have to add, say, a #:wal-mode? parameter > to ‘call-with-database’ (defaulting to #true), and change the caller to > set it appropriately. The caller, directly or indirectly, is in a gexp, > where you can use the ‘let-system’ expression above. > > Does that make sense? Yes, very much so. [...] After some typing, it turns out that the whole let-system is gone, we can set use a hard-coded #:wal-mode? #t in hurd-initialize-root-partition. > Thanks, and apologies for the delay! Sure, I'm happy I didn't "just push" ;-) I'm sending a v2 patch set in a minute. Greetings, Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 15:43:01 GMT) Full text and rfc822 format available.Message #56 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Cc: Ludovic Courtès <ludo <at> gnu.org> Subject: [PATCH v2 2/4] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd. Date: Fri, 10 Jul 2020 17:42:03 +0200
This fixes guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>) * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of sqlite. --- gnu/packages/guile.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm index a32bd33293..c651e2427a 100644 --- a/gnu/packages/guile.scm +++ b/gnu/packages/guile.scm @@ -665,7 +665,7 @@ Guile's foreign function interface.") ("pkg-config" ,pkg-config))) (inputs `(("guile" ,guile-3.0) - ("sqlite" ,sqlite))) + ("sqlite" ,sqlite/hurd))) (synopsis "Access SQLite databases from Guile") (description "This package provides Guile bindings to the SQLite database system.") -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 15:43:02 GMT) Full text and rfc822 format available.Message #59 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Cc: Ludovic Courtès <ludo <at> gnu.org> Subject: [PATCH v2 3/4] gnu: guix: Use sqlite/hurd for locking on the Hurd. Date: Fri, 10 Jul 2020 17:42:04 +0200
* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd. --- gnu/packages/package-management.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 9986976cc6..64c6931011 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n"))) ("zlib" ,zlib) ;for 'guix publish' ("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute' - ("sqlite" ,sqlite) + ("sqlite" ,sqlite/hurd) ("libgcrypt" ,libgcrypt) ("guile" ,guile-3.0-latest) -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 15:43:02 GMT) Full text and rfc822 format available.Message #62 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Cc: Ludovic Courtès <ludo <at> gnu.org> Subject: [PATCH v2 1/4] gnu: Add sqlite/hurd with locking fix. Date: Fri, 10 Jul 2020 17:42:02 +0200
* gnu/packages/patches/sqlite3-hurd.patch: New file. * gnu/local.mk (dist_patch_DATA): Add it. * gnu/packages/sqlite.scm (sqlite/hurd): Use it in new variable. --- gnu/local.mk | 1 + gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++ gnu/packages/sqlite.scm | 9 +++++ 3 files changed, 61 insertions(+) create mode 100644 gnu/packages/patches/sqlite3-hurd.patch diff --git a/gnu/local.mk b/gnu/local.mk index 5c3b391960..e452004945 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -1370,6 +1370,7 @@ dist_patch_DATA = \ %D%/packages/patches/sdl-pango-header-guard.patch \ %D%/packages/patches/sdl-pango-matrix_declarations.patch \ %D%/packages/patches/sdl-pango-sans-serif.patch \ + %D%/packages/patches/sqlite3-hurd.patch \ %D%/packages/patches/patchutils-test-perms.patch \ %D%/packages/patches/patch-hurd-path-max.patch \ %D%/packages/patches/perl-autosplit-default-time.patch \ diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patch new file mode 100644 index 0000000000..de87a30cb1 --- /dev/null +++ b/gnu/packages/patches/sqlite3-hurd.patch @@ -0,0 +1,51 @@ +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch +Upstream status: Not offered upstream. + +diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c +--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100 ++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200 +@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode + # include <sys/mman.h> + #endif + +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + /* # include <sys/ioctl.h> */ + # include <sys/file.h> + # include <sys/param.h> +@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id + ** + ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off + */ +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + + /* + ** Retry flock() calls that fail with EINTR +@@ -38586,7 +38586,7 @@ IOMETHODS( + 0 /* xShmMap method */ + ) + +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + IOMETHODS( + flockIoFinder, /* Finder function name */ + flockIoMethods, /* sqlite3_io_methods object name */ +@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){ + UNIXVFS("unix", autolockIoFinder ), + #elif OS_VXWORKS + UNIXVFS("unix", vxworksIoFinder ), ++#elif defined(__GNU__) ++ UNIXVFS("unix", flockIoFinder ), + #else + UNIXVFS("unix", posixIoFinder ), + #endif +@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){ + #if OS_VXWORKS + UNIXVFS("unix-namedsem", semIoFinder ), + #endif +-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS ++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__) + UNIXVFS("unix-posix", posixIoFinder ), + #endif + #if SQLITE_ENABLE_LOCKING_STYLE diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm index 8468131ddf..d5c44b0d0f 100644 --- a/gnu/packages/sqlite.scm +++ b/gnu/packages/sqlite.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2017 Jelle Licht <jlicht <at> fsfe.org> ;;; Copyright © 2018 Tobias Geerinckx-Rice <me <at> tobias.gr> ;;; Copyright © 2018 Alex Vong <alexvong1995 <at> gmail.com> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -107,3 +108,11 @@ is in the public domain.") ;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7. (define-public sqlite-with-column-metadata (deprecated-package "sqlite-with-column-metadata" sqlite)) + +(define-public sqlite/hurd + ;; TODO move into sqlite on the next rebuild cycle. + (package + (inherit sqlite) + (name "sqlite-for-hurd") + (source (origin (inherit (package-source sqlite)) + (patches (search-patches "sqlite3-hurd.patch")))))) -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 15:43:03 GMT) Full text and rfc822 format available.Message #65 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Cc: Ludovic Courtès <ludo <at> gnu.org> Subject: [PATCH v2 4/4] image: Do not set journal_model=WAL for the Hurd. Date: Fri, 10 Jul 2020 17:42:05 +0200
This fixes <https://bugs.gnu.org/42151>. * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f in call to ... * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode? parameter, pass it to ... (register-closure): ... this, add #:wal-mode? parameter, pass it to ... * guix/store/database.scm (with-database): ... this, add #:wal-mode? parameter, pass it to ... (call-with-database): ... this, add #:wal-mode? parameter; when set to #f, do not set journal_model=WAL. --- gnu/build/image.scm | 26 ++++++++++++++++---------- gnu/system/images/hurd.scm | 4 ++-- guix/store/database.scm | 29 ++++++++++++++++++++--------- 3 files changed, 38 insertions(+), 21 deletions(-) diff --git a/gnu/build/image.scm b/gnu/build/image.scm index e7b0418182..d8efa73f16 100644 --- a/gnu/build/image.scm +++ b/gnu/build/image.scm @@ -131,20 +131,23 @@ given CONFIG file." (define* (register-closure prefix closure #:key (deduplicate? #t) (reset-timestamps? #t) - (schema (sql-schema))) + (schema (sql-schema)) + (wal-mode? #t)) "Register CLOSURE in PREFIX, where PREFIX is the directory name of the target store and CLOSURE is the name of a file containing a reference graph as produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS? is true, reset timestamps on store files and, if DEDUPLICATE? is true, -deduplicates files common to CLOSURE and the rest of PREFIX." +deduplicates files common to CLOSURE and the rest of PREFIX. Pass WAL-MODE? +to call-with-database." (let ((items (call-with-input-file closure read-reference-graph))) (parameterize ((sql-schema schema)) (with-database (store-database-file #:prefix prefix) db - (register-items db items - #:prefix prefix - #:deduplicate? deduplicate? - #:reset-timestamps? reset-timestamps? - #:registration-time %epoch))))) + #:wal-mode? wal-mode? + (register-items db items + #:prefix prefix + #:deduplicate? deduplicate? + #:reset-timestamps? reset-timestamps? + #:registration-time %epoch))))) (define* (initialize-efi-partition root #:key @@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX." (register-closures? #t) system-directory make-device-nodes + (wal-mode? #t) #:allow-other-keys) "Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to install the bootloader configuration. If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store. If DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the -rest of the store when registering the closures. SYSTEM-DIRECTORY is the name -of the directory of the 'system' derivation." +rest of the store when registering the closures. SYSTEM-DIRECTORY is the name +of the directory of the 'system' derivation. Pass WAL-MODE? to +register-closure." (populate-root-file-system system-directory root) (populate-store references-graphs root) @@ -184,7 +189,8 @@ of the directory of the 'system' derivation." (register-closure root closure #:reset-timestamps? #t - #:deduplicate? deduplicate?)) + #:deduplicate? deduplicate? + #:wal-mode? wal-mode?)) references-graphs)) (when bootloader-installer diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm index 31942e7386..4e1db37104 100644 --- a/gnu/system/images/hurd.scm +++ b/gnu/system/images/hurd.scm @@ -61,8 +61,8 @@ #~(lambda* (#:rest args) (apply initialize-root-partition (append args - (list #:make-device-nodes - make-hurd-device-nodes))))) + (list #:make-device-nodes make-hurd-device-nodes + #:wal-mode? #f))))) (define hurd-disk-image (image diff --git a/guix/store/database.scm b/guix/store/database.scm index a38e4d7e52..50b66ce282 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt <at> cune.org> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo <at> gnu.org> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -20,6 +21,7 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config) + #:use-module (guix gexp) #:use-module (guix serialization) #:use-module (guix store deduplication) #:use-module (guix base16) @@ -27,6 +29,7 @@ #:use-module (guix build syscalls) #:use-module ((guix build utils) #:select (mkdir-p executable-file?)) + #:use-module (guix utils) #:use-module (guix build store-copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) @@ -97,17 +100,20 @@ as specified by SQL-SCHEMA." (sqlite-exec db (call-with-input-file schema get-string-all))) -(define (call-with-database file proc) +(define* (call-with-database file proc #:key (wal-mode? #t)) "Pass PROC a database record corresponding to FILE. If FILE doesn't exist, -create it and initialize it as a new database." +create it and initialize it as a new database. Unless WAL-MODE? is set to #f, +set journal_mode=WAL." (let ((new? (and (not (file-exists? file)) (begin (mkdir-p (dirname file)) #t))) (db (sqlite-open file))) - ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED - ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. - (sqlite-exec db "PRAGMA journal_mode=WAL;") + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>. + (when wal-mode? + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. + (sqlite-exec db "PRAGMA journal_mode=WAL;")) ;; Install a busy handler such that, when the database is locked, sqlite ;; retries until 30 seconds have passed, at which point it gives up and @@ -200,10 +206,15 @@ prior to returning." ;; Default location of the store database. (string-append %store-database-directory "/db.sqlite")) -(define-syntax-rule (with-database file db exp ...) - "Open DB from FILE and close it when the dynamic extent of EXP... is left. -If FILE doesn't exist, create it and initialize it as a new database." - (call-with-database file (lambda (db) exp ...))) +(define-syntax with-database + (syntax-rules () + "Open DB from FILE and close it when the dynamic extent of EXP... is left. +If FILE doesn't exist, create it and initialize it as a new database. Pass +#:wal-mode? to call-with-database." + ((_ file db #:wal-mode? wal-mode? exp ...) + (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?)) + ((_ file db exp ...) + (call-with-database file (lambda (db) exp ...))))) (define (sqlite-finalize stmt) ;; As of guile-sqlite3 0.1.0, cached statements aren't reset when -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 16:26:01 GMT) Full text and rfc822 format available.Message #68 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd. Date: Fri, 10 Jul 2020 18:25:28 +0200
Ludovic Courtès writes: > "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: >> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd. > > Same here. > > Which makes me thing: how about applying the sqlite/hurd patch in a > build phase of sqlite that would be Hurd-specific? That way, we’d still > have a single sqlite package. Ah, yes that sounds better. We don't need sqlite/hurd "sqlite-for-hurd" and all that. I'll give that a go! Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 18:35:01 GMT) Full text and rfc822 format available.Message #71 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix. Date: Fri, 10 Jul 2020 20:34:07 +0200
Ludovic Courtès writes: > "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > >> * gnu/packages/patches/sqlite3-hurd.patch: New file. >> * gnu/local.mk (dist_patch_DATA): Add it. >> * gnu/packages/sqlite.scm (sqlite/hurd): New variable. > > […] > >> +++ b/gnu/packages/patches/sqlite3-hurd.patch >> @@ -0,0 +1,51 @@ >> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch >> +Upstream status: Not offered upstream. > > The URL is 404, could you fix it? That's weird...Changing it (in completely rewritten patch) to https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch/ > It would also be great if you could > add a sentence or two explaining what this works around, so that our > future selves know how to deal with this patch. :-) Ah, sure! > Otherwise LGTM, you can push with changes along these lines! As it's so much changed, I'll send it by one more time :-) Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 18:35:02 GMT) Full text and rfc822 format available.Message #74 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd. Date: Fri, 10 Jul 2020 20:34:42 +0200
Ludovic Courtès writes: > "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > >> This fixes >> >> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>) >> >> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of >> sqlite. >> --- >> gnu/packages/guile.scm | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm >> index a32bd33293..c651e2427a 100644 >> --- a/gnu/packages/guile.scm >> +++ b/gnu/packages/guile.scm >> @@ -665,7 +665,7 @@ Guile's foreign function interface.") >> ("pkg-config" ,pkg-config))) >> (inputs >> `(("guile" ,guile-3.0) >> - ("sqlite" ,sqlite))) >> + ("sqlite" ,sqlite/hurd))) > > This should be guarded by ‘if (hurd-target?)’, right? Yeah...but all changed now...look out for v3. Thanks! Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 18:37:01 GMT) Full text and rfc822 format available.Message #77 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Cc: Ludovic Courtès <ludo <at> gnu.org> Subject: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd. Date: Fri, 10 Jul 2020 20:35:51 +0200
This fixes <https://bugs.gnu.org/42151>. * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f in call to ... * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode? parameter, pass it to ... (register-closure): ... this, add #:wal-mode? parameter, pass it to ... * guix/store/database.scm (with-database): ... this, add #:wal-mode? parameter, pass it to ... (call-with-database): ... this, add #:wal-mode? parameter; when set to #f, do not set journal_model=WAL. --- gnu/build/image.scm | 26 ++++++++++++++++---------- gnu/system/images/hurd.scm | 4 ++-- guix/store/database.scm | 29 ++++++++++++++++++++--------- 3 files changed, 38 insertions(+), 21 deletions(-) diff --git a/gnu/build/image.scm b/gnu/build/image.scm index e7b0418182..d8efa73f16 100644 --- a/gnu/build/image.scm +++ b/gnu/build/image.scm @@ -131,20 +131,23 @@ given CONFIG file." (define* (register-closure prefix closure #:key (deduplicate? #t) (reset-timestamps? #t) - (schema (sql-schema))) + (schema (sql-schema)) + (wal-mode? #t)) "Register CLOSURE in PREFIX, where PREFIX is the directory name of the target store and CLOSURE is the name of a file containing a reference graph as produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS? is true, reset timestamps on store files and, if DEDUPLICATE? is true, -deduplicates files common to CLOSURE and the rest of PREFIX." +deduplicates files common to CLOSURE and the rest of PREFIX. Pass WAL-MODE? +to call-with-database." (let ((items (call-with-input-file closure read-reference-graph))) (parameterize ((sql-schema schema)) (with-database (store-database-file #:prefix prefix) db - (register-items db items - #:prefix prefix - #:deduplicate? deduplicate? - #:reset-timestamps? reset-timestamps? - #:registration-time %epoch))))) + #:wal-mode? wal-mode? + (register-items db items + #:prefix prefix + #:deduplicate? deduplicate? + #:reset-timestamps? reset-timestamps? + #:registration-time %epoch))))) (define* (initialize-efi-partition root #:key @@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX." (register-closures? #t) system-directory make-device-nodes + (wal-mode? #t) #:allow-other-keys) "Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to install the bootloader configuration. If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store. If DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the -rest of the store when registering the closures. SYSTEM-DIRECTORY is the name -of the directory of the 'system' derivation." +rest of the store when registering the closures. SYSTEM-DIRECTORY is the name +of the directory of the 'system' derivation. Pass WAL-MODE? to +register-closure." (populate-root-file-system system-directory root) (populate-store references-graphs root) @@ -184,7 +189,8 @@ of the directory of the 'system' derivation." (register-closure root closure #:reset-timestamps? #t - #:deduplicate? deduplicate?)) + #:deduplicate? deduplicate? + #:wal-mode? wal-mode?)) references-graphs)) (when bootloader-installer diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm index 31942e7386..4e1db37104 100644 --- a/gnu/system/images/hurd.scm +++ b/gnu/system/images/hurd.scm @@ -61,8 +61,8 @@ #~(lambda* (#:rest args) (apply initialize-root-partition (append args - (list #:make-device-nodes - make-hurd-device-nodes))))) + (list #:make-device-nodes make-hurd-device-nodes + #:wal-mode? #f))))) (define hurd-disk-image (image diff --git a/guix/store/database.scm b/guix/store/database.scm index a38e4d7e52..50b66ce282 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt <at> cune.org> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo <at> gnu.org> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -20,6 +21,7 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config) + #:use-module (guix gexp) #:use-module (guix serialization) #:use-module (guix store deduplication) #:use-module (guix base16) @@ -27,6 +29,7 @@ #:use-module (guix build syscalls) #:use-module ((guix build utils) #:select (mkdir-p executable-file?)) + #:use-module (guix utils) #:use-module (guix build store-copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) @@ -97,17 +100,20 @@ as specified by SQL-SCHEMA." (sqlite-exec db (call-with-input-file schema get-string-all))) -(define (call-with-database file proc) +(define* (call-with-database file proc #:key (wal-mode? #t)) "Pass PROC a database record corresponding to FILE. If FILE doesn't exist, -create it and initialize it as a new database." +create it and initialize it as a new database. Unless WAL-MODE? is set to #f, +set journal_mode=WAL." (let ((new? (and (not (file-exists? file)) (begin (mkdir-p (dirname file)) #t))) (db (sqlite-open file))) - ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED - ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. - (sqlite-exec db "PRAGMA journal_mode=WAL;") + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>. + (when wal-mode? + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>. + (sqlite-exec db "PRAGMA journal_mode=WAL;")) ;; Install a busy handler such that, when the database is locked, sqlite ;; retries until 30 seconds have passed, at which point it gives up and @@ -200,10 +206,15 @@ prior to returning." ;; Default location of the store database. (string-append %store-database-directory "/db.sqlite")) -(define-syntax-rule (with-database file db exp ...) - "Open DB from FILE and close it when the dynamic extent of EXP... is left. -If FILE doesn't exist, create it and initialize it as a new database." - (call-with-database file (lambda (db) exp ...))) +(define-syntax with-database + (syntax-rules () + "Open DB from FILE and close it when the dynamic extent of EXP... is left. +If FILE doesn't exist, create it and initialize it as a new database. Pass +#:wal-mode? to call-with-database." + ((_ file db #:wal-mode? wal-mode? exp ...) + (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?)) + ((_ file db exp ...) + (call-with-database file (lambda (db) exp ...))))) (define (sqlite-finalize stmt) ;; As of guile-sqlite3 0.1.0, cached statements aren't reset when -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Fri, 10 Jul 2020 18:37:02 GMT) Full text and rfc822 format available.Message #80 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> To: 42151 <at> debbugs.gnu.org Cc: Ludovic Courtès <ludo <at> gnu.org> Subject: [PATCH v3 1/2] gnu: sqlite: Add locking-mode fix for the Hurd. Date: Fri, 10 Jul 2020 20:35:50 +0200
* gnu/packages/patches/sqlite-hurd.patch: New file. * gnu/local.mk (dist_patch_DATA): Add it. * gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building for the Hurd. [arguments]: Apply it when building for the Hurd. --- gnu/local.mk | 1 + gnu/packages/patches/sqlite-hurd.patch | 58 ++++++++++++++++++++++++++ gnu/packages/sqlite.scm | 19 +++++++++ 3 files changed, 78 insertions(+) create mode 100644 gnu/packages/patches/sqlite-hurd.patch diff --git a/gnu/local.mk b/gnu/local.mk index 5c3b391960..42c9d0f379 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -1370,6 +1370,7 @@ dist_patch_DATA = \ %D%/packages/patches/sdl-pango-header-guard.patch \ %D%/packages/patches/sdl-pango-matrix_declarations.patch \ %D%/packages/patches/sdl-pango-sans-serif.patch \ + %D%/packages/patches/sqlite-hurd.patch \ %D%/packages/patches/patchutils-test-perms.patch \ %D%/packages/patches/patch-hurd-path-max.patch \ %D%/packages/patches/perl-autosplit-default-time.patch \ diff --git a/gnu/packages/patches/sqlite-hurd.patch b/gnu/packages/patches/sqlite-hurd.patch new file mode 100644 index 0000000000..d80a2c5be8 --- /dev/null +++ b/gnu/packages/patches/sqlite-hurd.patch @@ -0,0 +1,58 @@ +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch +Upstream status: Not upstreamed. + +This patch is needed to get offloading to work. + +Sqlite can use simple file locking mode, but that does not work for the Hurd; +a second sqlite process fails with a "locking protocol" error. + +See also: https://bugs.debian.org/529734. + +diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c +--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100 ++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200 +@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode + # include <sys/mman.h> + #endif + +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + /* # include <sys/ioctl.h> */ + # include <sys/file.h> + # include <sys/param.h> +@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id + ** + ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off + */ +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + + /* + ** Retry flock() calls that fail with EINTR +@@ -38586,7 +38586,7 @@ IOMETHODS( + 0 /* xShmMap method */ + ) + +-#if SQLITE_ENABLE_LOCKING_STYLE ++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__) + IOMETHODS( + flockIoFinder, /* Finder function name */ + flockIoMethods, /* sqlite3_io_methods object name */ +@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){ + UNIXVFS("unix", autolockIoFinder ), + #elif OS_VXWORKS + UNIXVFS("unix", vxworksIoFinder ), ++#elif defined(__GNU__) ++ UNIXVFS("unix", flockIoFinder ), + #else + UNIXVFS("unix", posixIoFinder ), + #endif +@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){ + #if OS_VXWORKS + UNIXVFS("unix-namedsem", semIoFinder ), + #endif +-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS ++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__) + UNIXVFS("unix-posix", posixIoFinder ), + #endif + #if SQLITE_ENABLE_LOCKING_STYLE diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm index 8468131ddf..eeb77749d8 100644 --- a/gnu/packages/sqlite.scm +++ b/gnu/packages/sqlite.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2017 Jelle Licht <jlicht <at> fsfe.org> ;;; Copyright © 2018 Tobias Geerinckx-Rice <me <at> tobias.gr> ;;; Copyright © 2018 Alex Vong <alexvong1995 <at> gmail.com> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke <at> gnu.org> ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (gnu packages sqlite) #:use-module (gnu packages) + #:use-module (gnu packages hurd) #:use-module (gnu packages readline) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages) @@ -65,6 +67,11 @@ "1bj936svd8i5g25xd1bj52hj4zca01fgl3sqkj86z9q5pkz4wa32")))) (build-system gnu-build-system) (inputs `(("readline" ,readline))) + (native-inputs (if (hurd-target?) + ;; TODO move into origin on the next rebuild cycle. + `(("hurd-locking-mode.patch" + ,@(search-patches "sqlite-hurd.patch"))) + '())) (outputs '("out" "static")) (arguments `(#:configure-flags @@ -79,6 +86,18 @@ ;; Column metadata is required by GNU Jami and Qt, et.al. "-DSQLITE_ENABLE_COLUMN_METADATA")) #:phases (modify-phases %standard-phases + ;; TODO: remove in the next rebuild cycle + ,@(if (hurd-target?) + `((add-after 'unpack 'patch-sqlite/hurd + (lambda* (#:key inputs native-inputs + #:allow-other-keys) + (let ((patch (assoc-ref + (if ,(%current-target-system) + native-inputs + inputs) + "hurd-locking-mode.patch"))) + (invoke "patch" "-p1" "--force" "-i" patch))))) + '()) (add-after 'install 'move-static-library (lambda* (#:key outputs #:allow-other-keys) (let* ((out (assoc-ref outputs "out")) -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Sat, 11 Jul 2020 14:19:02 GMT) Full text and rfc822 format available.Message #83 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: "Jan \(janneke\) Nieuwenhuizen" <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH v3 1/2] gnu: sqlite: Add locking-mode fix for the Hurd. Date: Sat, 11 Jul 2020 16:18:30 +0200
Hi! "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > * gnu/packages/patches/sqlite-hurd.patch: New file. > * gnu/local.mk (dist_patch_DATA): Add it. > * gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building > for the Hurd. > [arguments]: Apply it when building for the Hurd. LGTM, thank you!
bug-guix <at> gnu.org
:bug#42151
; Package guix
.
(Sat, 11 Jul 2020 14:23:02 GMT) Full text and rfc822 format available.Message #86 received at 42151 <at> debbugs.gnu.org (full text, mbox):
From: Ludovic Courtès <ludo <at> gnu.org> To: "Jan \(janneke\) Nieuwenhuizen" <janneke <at> gnu.org> Cc: 42151 <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd. Date: Sat, 11 Jul 2020 16:22:18 +0200
"Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > This fixes <https://bugs.gnu.org/42151>. > > * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f > in call to ... > * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode? > parameter, pass it to ... > (register-closure): ... this, add #:wal-mode? parameter, pass it to ... > * guix/store/database.scm (with-database): ... this, add #:wal-mode? > parameter, pass it to ... > (call-with-database): ... this, add #:wal-mode? parameter; when > set to #f, do not set journal_model=WAL. [...] > +++ b/gnu/system/images/hurd.scm > @@ -61,8 +61,8 @@ > #~(lambda* (#:rest args) > (apply initialize-root-partition > (append args > - (list #:make-device-nodes > - make-hurd-device-nodes))))) > + (list #:make-device-nodes make-hurd-device-nodes > + #:wal-mode? #f))))) IWBN to have an XXX comment here stating why we disable WAL mode on GNU/Hurd, possibly referencing this issue. That way, we can eventually revisit the issue and hopefully get rid of this workaround. Apart from that the patch LGTM, thanks! Ludo’.
Jan Nieuwenhuizen <janneke <at> gnu.org>
:"Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org>
:Message #91 received at 42151-done <at> debbugs.gnu.org (full text, mbox):
From: Jan Nieuwenhuizen <janneke <at> gnu.org> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 42151-done <at> debbugs.gnu.org Subject: Re: bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd. Date: Sat, 11 Jul 2020 16:36:26 +0200
Ludovic Courtès writes: > "Jan (janneke) Nieuwenhuizen" <janneke <at> gnu.org> skribis: > >> This fixes <https://bugs.gnu.org/42151>. >> >> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f >> in call to ... [...] >> + (list #:make-device-nodes make-hurd-device-nodes >> + #:wal-mode? #f))))) > > IWBN to have an XXX comment here stating why we disable WAL mode on > GNU/Hurd, possibly referencing this issue. That way, we can eventually > revisit the issue and hopefully get rid of this workaround. Sure, done! > Apart from that the patch LGTM, thanks! Yay! Pushed series to master as 4b9eecd322e566783369795ebea63a479b51f486. Let the offloadings commence! Janneke -- Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
Debbugs Internal Request <help-debbugs <at> gnu.org>
to internal_control <at> debbugs.gnu.org
.
(Sun, 09 Aug 2020 11:24:05 GMT) Full text and rfc822 format available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.