GNU bug report logs - #65177
udevd error with lvm-raid array leading to race condition with luks

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 65177 in the body.
You can then email your comments to 65177 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Adrien 'neox' Bourmault <neox <at> gnu.org>
To: bug-guix <at> gnu.org
Subject: udevd error with lvm-raid array leading to race condition with luks
Date: Wed, 9 Aug 2023 12:25:02 +0200

Hi there.
My setup is the following (LVM array containing a LUKS partition):

  (mapped-devices
   (list
    (mapped-device
      (source "HOMERAID")
      (target "HOMERAID-HOMERAID_data")
      (type lvm-device-mapping))
    (mapped-device
      (source "/dev/mapper/HOMERAID-HOMERAID_data")
      (target "luks-f0a72a6c-499b-4445-8d13-21dc99337752")
      (type luks-device-mapping))))

  (file-systems
    (cons*
      (file-system
        (mount-point "/")
        (device (uuid "2e44f3f7-bb6b-43ac-933a-e8992bf10d29" 'ext4))
        (type "ext4"))
      (file-system
        (mount-point "/home")
        (device "/dev/mapper/luks-f0a72a6c-499b-4445-8d13-21dc99337752")
        (type "ext4")
        (dependencies mapped-devices))
      (file-system
        (mount-point "/boot/efi")
        (device (uuid "DC58-946E" 'fat32))
        (type "vfat"))
     %base-file-systems)))

I use Guix System since 2022, and never had any problem booting with 
this configuration. But I did update my system (and reconfigure it) last 
week and now I can't boot. I don't have any older generation to restore 
(yes I'm dumb, I executed a delete-generations to show how it works to a 
friend), but I think the last working one was from July, 25th or just a 
bit older.

I have in my /var/log/messages :

  Aug  9 11:40:27 localhost vmunix: [    7.525877] udevd[515]: failed 
to execute '/usr/bin/systemd-run' '/usr/bin/systemd-run --no-block 
--property DefaultDependencies=no --unit lvm-activate-HOMERAID 
/gnu/store/hffkn63zx2zjadawrkxpnr486frc9n74-lvm2-2.03.21/sbin/lvm 
vgchange -aay --autoactivation event HOMERAID': No such file or directory

On the screen, during boot, I obtain a wall of messages like (not the 
exact message, sorry, can't find on syslog) "Device 
HOMERAID-HOMERAID_data could not be found: does not exist or access 
denied" and after that the system hangs.

However, I can boot successfully if and only if I press Scroll Lock just 
after modesetting and before this message ("Device ... not be found") 
appears and when I press it again (to deactivate its effect) after some 
seconds, the LUKS passphrase prompt appears and allows me to boot properly.

I can use my system but it requires multiple tries each time x) I have 
to be really synchronized with the modesetting and the message xD

Thank you very much.

Freely,
-- 
Adrien Bourmault
Co-maintainer, GNU Boot project
Elected member, XMPP Standards Foundation
Associate member, Free Software Foundation
Trésorier, Association Libre en Communs (https://www.a-lec.org)
GPG : 1DF1132CF1658A8559025C98AAD6B069819E6979

Message #8 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Josselin Poiret <dev <at> jpoiret.xyz>
To: Adrien 'neox' Bourmault <neox <at> gnu.org>, 65177 <at> debbugs.gnu.org
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
 condition with luks
Date: Thu, 10 Aug 2023 10:02:22 +0200

[Message part 1 (text/plain, inline)]

Hi Adrien,

Adrien 'neox' Bourmault <neox <at> gnu.org> writes:

>    Aug  9 11:40:27 localhost vmunix: [    7.525877] udevd[515]: failed 
> to execute '/usr/bin/systemd-run' '/usr/bin/systemd-run --no-block 
> --property DefaultDependencies=no --unit lvm-activate-HOMERAID 
> /gnu/store/hffkn63zx2zjadawrkxpnr486frc9n74-lvm2-2.03.21/sbin/lvm 
> vgchange -aay --autoactivation event HOMERAID': No such file or directory

Since lvm2 2.03.14, the included udev rules use systemd-run to run
vgchange and activate the volume group.  lvm2 was updated recently from
2.03.11 to 2.03.21, then 2.03.22 (cc'ing tobias), and probably started
exhibiting this behavior then.  I think we can probably remove the
indirection through systemd-run and directly run vgchange.

Best,
-- 
Josselin Poiret

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Yann Dupont <Yann.Dupont <at> univ-nantes.fr>
To: 65177 <at> debbugs.gnu.org
Subject: udevd error with lvm-raid array leading to race condition with luks
Date: Thu, 14 Sep 2023 11:25:38 +0200

hello everyone, we're also victims of this bug, in an even simpler use case.

[…]
(file-system
                                (device "/dev/mapper/VG0-DATA")
                                (mount-point "/VG0-DATA")
                                (type "ext4"))
[…]

The culprit seems to be 69-dm-lvm.rules

[ 18.226226] udevd[115]: failed to execute '/usr/bin/systemd-run' 
'/usr/bin/systemd-run --no-block --property DefaultDependencies=no 
--unit lvm-activate-VG0 
/gnu/store/0hndg947ywdl5izvy63ny38hyywci66k-lvm2-2.03.22/sbin/lvm vy

I can confirm that when using time-machine to revert to lvm2 2-03.11 
versions, the VM boots.

cheers,

Message #14 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Yann Dupont <Yann.Dupont <at> univ-nantes.fr>
To: 65177 <at> debbugs.gnu.org
Subject: Re: udevd error with lvm-raid array leading to race condition with
 luks
Date: Thu, 14 Sep 2023 18:23:08 +0200

[Message part 1 (text/plain, inline)]

Hi, as suggested by Josselin, I tested the following patch and it seems 
to do the job here.


Be careful, I'm not an udev or lvm2 specialist at all and basically, I 
don't really know if what I did is the right way to do it.


All I can say is that the VMs now boot.


Cheers,



diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 91109c41d9..28b3c1e0bf 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -4421,6 +4421,7 @@ (define-public lvm2
               (sha256
                (base32
"0z6w6bknhwh1n3qfkb5ij6x57q3wjf28lq3l8kh7rkhsplinjnjc"))
+              (patches (search-patches "lvm2-no-systemd.patch"))
               (modules '((guix build utils)))
               (snippet
                '(begin
diff --git a/gnu/packages/patches/lvm2-no-systemd.patch 
b/gnu/packages/patches/lvm2-no-systemd.patch
new file mode 100644
index 0000000000..7e8a37abcc
--- /dev/null
+++ b/gnu/packages/patches/lvm2-no-systemd.patch
@@ -0,0 +1,13 @@
+diff --git a/udev/69-dm-lvm.rules.in b/udev/69-dm-lvm.rules.in
+index ff1568145..8879a2ef9 100644
+--- a/udev/69-dm-lvm.rules.in
++++ b/udev/69-dm-lvm.rules.in
+@@ -76,7 +76,7 @@ LABEL="lvm_scan"
+ # it's better suited to appearing in the journal.
+
+ IMPORT{program}="(LVM_EXEC)/lvm pvscan --cache --listvg 
--checkcomplete --vgonline --autoactivation event --udevoutput 
--journal=output $env{DEVNAME}"
+-ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(SYSTEMDRUN) --no-block 
--property DefaultDependencies=no --unit 
lvm-activate-$env{LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay 
--autoactivation event $env{LVM_VG_NAME_COMPLETE}"
++ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(LVM_EXEC)/lvm vgchange -aay 
--autoactivation event $env{LVM_VG_NAME_COMPLETE}"
+ GOTO="lvm_end"
+
+ LABEL="lvm_end"

[Message part 2 (text/html, inline)]

Message #17 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Yann Dupont <Yann.Dupont <at> univ-nantes.fr>
Cc: 65177 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
 condition with luks
Date: Mon, 25 Sep 2023 09:35:54 +0200

Hi,

On Thu, 14 Sep 2023 at 18:23, Yann Dupont <Yann.Dupont <at> univ-nantes.fr> wrote:

> All I can say is that the VMs now boot.

WDYT about this patch?

> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
> index 91109c41d9..28b3c1e0bf 100644
> --- a/gnu/packages/linux.scm
> +++ b/gnu/packages/linux.scm
> @@ -4421,6 +4421,7 @@ (define-public lvm2
>                (sha256
>                 (base32
>                  "0z6w6bknhwh1n3qfkb5ij6x57q3wjf28lq3l8kh7rkhsplinjnjc"))
> +              (patches (search-patches "lvm2-no-systemd.patch"))
>                (modules '((guix build utils)))
>                (snippet
>                 '(begin
> diff --git a/gnu/packages/patches/lvm2-no-systemd.patch
> b/gnu/packages/patches/lvm2-no-systemd.patch
> new file mode 100644
> index 0000000000..7e8a37abcc
> --- /dev/null
> +++ b/gnu/packages/patches/lvm2-no-systemd.patch
> @@ -0,0 +1,13 @@
> +diff --git a/udev/69-dm-lvm.rules.in b/udev/69-dm-lvm.rules.in
> +index ff1568145..8879a2ef9 100644
> +--- a/udev/69-dm-lvm.rules.in
> ++++ b/udev/69-dm-lvm.rules.in
> +@@ -76,7 +76,7 @@ LABEL="lvm_scan"
> + # it's better suited to appearing in the journal.
> +
> + IMPORT{program}="(LVM_EXEC)/lvm pvscan --cache --listvg --checkcomplete
> --vgonline --autoactivation event --udevoutput --journal=output $env{DEVNAME}"
> +-ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(SYSTEMDRUN) --no-block
> --property DefaultDependencies=no --unit lvm-activate-$env
> {LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay --autoactivation event
> $env{LVM_VG_NAME_COMPLETE}"
> ++ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="(LVM_EXEC)/lvm vgchange -aay
> --autoactivation event $env{LVM_VG_NAME_COMPLETE}"
> + GOTO="lvm_end"
> +
> + LABEL="lvm_end"

Josselin, is it what you had in mind?

Cheers,
simon

Message #20 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Yann Dupont <Yann.Dupont <at> univ-nantes.fr>
Cc: 65177 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Tobias Geerinckx-Rice <me <at> tobias.gr>, Adrien 'neox' Bourmault <neox <at> gnu.org>
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
 condition with luks
Date: Tue, 17 Oct 2023 11:00:14 +0200

Hi,

On Mon, 25 Sep 2023 at 09:35, Simon Tournier <zimon.toutoune <at> gmail.com> wrote:

>> All I can say is that the VMs now boot.
>
> WDYT about this patch?

For easing the discussion, I extracted Yann’s diff and prepared a patch
ready to merge.  See:

    https://issues.guix.gnu.org/issue/66586

Cheers,
simon

Message #25 received at 65177-done <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Yann Dupont <Yann.Dupont <at> univ-nantes.fr>
Cc: Tobias Geerinckx-Rice <me <at> tobias.gr>, Josselin Poiret <dev <at> jpoiret.xyz>,
 65177-done <at> debbugs.gnu.org, Adrien 'neox' Bourmault <neox <at> gnu.org>
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
 condition with luks
Date: Tue, 31 Oct 2023 09:59:59 +0100

Hi,

>>> All I can say is that the VMs now boot.
>>
>> WDYT about this patch?
>
> For easing the discussion, I extracted Yann’s diff and prepared a patch
> ready to merge.  See:
>
>     https://issues.guix.gnu.org/issue/66586

Patch pushed as c0895371c5759c7d9edb330774e90f192cc4cf2c.

Closing.

Feel free to reopen if the patch does not fix the issue.

Cheers,
simon

Message #30 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <wolf <at> wolfsden.cz>
To: 65177 <at> debbugs.gnu.org
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
Date: Tue, 31 Oct 2023 19:38:09 +0100

[Message part 1 (text/plain, inline)]

Hi,

I think the patch is wrong.  No comment regarding the udev rule itself (I do not know much about it), it does spam the logs with:

Oct 31 18:31:35 localhost vmunix: [   60.013538] udevd[327]: invalid key/value pair in file /gnu/store/a3xvv1zxzyb53r5r752q8nh9v9vd150h-udev-rules/69-dm-lvm.rules on line 78, starting at character 1 ('I')
Oct 31 18:31:35 localhost vmunix: [   60.013544] udevd[327]: invalid key/value pair in file /gnu/store/a3xvv1zxzyb53r5r752q8nh9v9vd150h-udev-rules/69-dm-lvm.rules on line 79, starting at character 32 (',')

The fix[0] for that is fairly trivial.

0: https://git.sr.ht/~graywolf/guix/commit/207794679281c1e7e5f2051719fe5c705506eb23

Have a nice day,
W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[signature.asc (application/pgp-signature, inline)]

Message #33 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Tomas Volf <wolf <at> wolfsden.cz>, 65177 <at> debbugs.gnu.org
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
Date: Thu, 02 Nov 2023 13:16:45 +0100

Hi,

On Tue, 31 Oct 2023 at 19:38, Tomas Volf <wolf <at> wolfsden.cz> wrote:

> I think the patch is wrong.

Is patch v3 from

    https://issues.guix.gnu.org/issue/66586#10

fixing the issue?

Cheers,
simon

Message #38 received at 65177-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: Tomas Volf <wolf <at> wolfsden.cz>, 65177-done <at> debbugs.gnu.org
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
 condition with luks
Date: Mon, 03 Feb 2025 12:23:05 +0900

Hi,

Simon Tournier <zimon.toutoune <at> gmail.com> writes:

> Hi,
>
> On Tue, 31 Oct 2023 at 19:38, Tomas Volf <wolf <at> wolfsden.cz> wrote:
>
>> I think the patch is wrong.
>
> Is patch v3 from
>
>     https://issues.guix.gnu.org/issue/66586#10
>
> fixing the issue?

I'm assuming so.  Do open an issue if something problematic in the LVM
area persists.

-- 
Thanks,
Maxim

Message #41 received at 65177 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Adrien 'neox' Bourmault <neox <at> gnu.org>
Cc: 65177 <at> debbugs.gnu.org
Subject: Re: bug#65177: udevd error with lvm-raid array leading to race
 condition with luks
Date: Mon, 03 Feb 2025 12:24:57 +0900

Hi Adrien,

Adrien 'neox' Bourmault <neox <at> gnu.org> writes:

> Hi there.
> My setup is the following (LVM array containing a LUKS partition):
>
>   (mapped-devices
>    (list
>     (mapped-device
>       (source "HOMERAID")
>       (target "HOMERAID-HOMERAID_data")
>       (type lvm-device-mapping))
>     (mapped-device
>       (source "/dev/mapper/HOMERAID-HOMERAID_data")
>       (target "luks-f0a72a6c-499b-4445-8d13-21dc99337752")
>       (type luks-device-mapping))))
>
>   (file-systems
>     (cons*
>       (file-system
>         (mount-point "/")
>         (device (uuid "2e44f3f7-bb6b-43ac-933a-e8992bf10d29" 'ext4))
>         (type "ext4"))
>       (file-system
>         (mount-point "/home")
>         (device "/dev/mapper/luks-f0a72a6c-499b-4445-8d13-21dc99337752")
>         (type "ext4")
>         (dependencies mapped-devices))
>       (file-system
>         (mount-point "/boot/efi")
>         (device (uuid "DC58-946E" 'fat32))
>         (type "vfat"))
>      %base-file-systems)))
>
> I use Guix System since 2022, and never had any problem booting with
> this configuration. But I did update my system (and reconfigure it)
> last week and now I can't boot. I don't have any older generation to
> restore (yes I'm dumb, I executed a delete-generations to show how it
> works to a friend), but I think the last working one was from July,
> 25th or just a bit older.
>
> I have in my /var/log/messages :
>
>   Aug  9 11:40:27 localhost vmunix: [    7.525877] udevd[515]: failed
>   to execute '/usr/bin/systemd-run' '/usr/bin/systemd-run --no-block
>   --property DefaultDependencies=no --unit lvm-activate-HOMERAID
>   /gnu/store/hffkn63zx2zjadawrkxpnr486frc9n74-lvm2-2.03.21/sbin/lvm
>   vgchange -aay --autoactivation event HOMERAID': No such file or
>  directory

It seems there were other issues resolved that may have fixed that.
I've tentatively marked this one as resolved, but if it doesn't work for
you still, let us know and we can reopen/create a new issue.

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.