GNU bug report logs - #34176
X200 kernel panic on S3 resume (linux-libre > 4.18.9)

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 34176 in the body.
You can then email your comments to 34176 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: bug-guix <at> gnu.org
Subject: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Wed, 23 Jan 2019 00:58:57 -0500

[Message part 1 (text/plain, inline)]

Hello, Guix:

I've been sitting on this one for a little while since I didn't have
time to try debugging it.

Sometime after 2018-09-25 (linux-libre 4.18.9), my X200 locks up after
resuming from S3 (suspend to memory).  The system is unresponsive, but
Alt+Sysreq+B is recognized and reboots the system.

Unfortunately, I don't know when this happened---my next upgrade on this
box was 2018-12-01, kernel version 4.19.5.

I produced a minimal system configuration where I stripped out all but
base services (attached below) and the problem still occurs.  After
wasting a bunch of time on that, I finally added the
`no_console_suspend' boot param and, sure enough, I get a kernel
panic.  It does not write the log to disk, so I'm reproducing it here in
part manually:

--8<---------------cut here---------------start------------->8---
ACPI: Waking up from system sleep state S3
ACPI: EC: Interrupt unblocked
ACPI: EC: event unblocked
usb usb3: root hub lost power or was reset
general protection fault: 0000 [#1] SMP PTI
CPU: 1 PID: 441 Comm: kworker/u4:23 Tainted: 6     4.28-3gnu #1
Hardware name: LENOVO 7459VLP/7459VLP, BIOS CBET4000 3774c98 09/07/2016
Workqueue: events_unbound async_run_entry_fn
RIP: 0010:uhci_scan_schedule.part.38+0xa3/0xad0
Code: [...machine code...]
[...register values...]
Call trace:  [I'm omitting address offsets]
 ? __dev_printk
 uhci_hub_status_data
 usb_hcd_poll_rh_status
 uhci_pci_resume
 resume_common
 hcd_pci_resume
 pci_pm_resume
[...ata stuff, successful...]
uhci_hcd 0000:00:1d.0: host controller process error, something bad happened!
uhci_hcd 0000:00:1d.0: host controller halted, very bad!
uhci_hcd 0000:00:1d.0: HC died; cleaning up
--8<---------------cut here---------------end--------------->8---

I found this:

  https://01.org/linuxgraphics/gfx-docs/drm/driver-api/usb/persist.html

But it states for `persist':

   For hubs the feature is automatically and permanently enabled and the
   power/persist file doesn’t even exist, so you only have to worry
   about setting it for devices where it really matters.

(Which is indeed true for my hub 0000:00:1d.0, which is usb6, according
to dmesg.)

If someone with more experience with these types of issues offer me some
advice on my next steps (including the best place to report upstream
either for linux-libre or linux, if that's more appropriate), I'd
appreciate it.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[signature.asc (application/pgp-signature, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: bug-guix <at> gnu.org
Subject: Re: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Wed, 23 Jan 2019 01:08:39 -0500

[Message part 1 (text/plain, inline)]

Well, I'm not getting those hours of my life back.

On Wed, Jan 23, 2019 at 00:58:57 -0500, Mike Gerwitz wrote:
> usb usb3: root hub lost power or was reset

[...]

> uhci_hcd 0000:00:1d.0: host controller process error, something bad happened!
> uhci_hcd 0000:00:1d.0: host controller halted, very bad!
> uhci_hcd 0000:00:1d.0: HC died; cleaning up

[...]

> (Which is indeed true for my hub 0000:00:1d.0, which is usb6, according
> to dmesg.)

Right after sending this message, I recalled that I used powertop to do
"auto tuning".  Sure enough, one of the lines was this:

   Good          Autosuspend for USB device UHCI Host Controller [usb6]

By disabling it, it runs this command:

  echo on > /sys/bus/usb/devices/usb6/power/control

And it resolves my suspend issues on 4.20.3-gnu.

What sucks even more is that I had disabled those settings on the USB
controllers over a year ago (because it was putting my USB keyboard to
sleep, dropping the first keypress); don't know how it got re-enabled,
but the user is probably to blame.

Sorry for the noise, everyone.  I hope that my suffering provides some
value to others.

This bug can be closed.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 34176-done <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Mike Gerwitz <mtg <at> gnu.org>
Cc: 34176-done <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Wed, 23 Jan 2019 11:55:31 +0200

[Message part 1 (text/plain, inline)]

On Wed, Jan 23, 2019 at 01:08:39AM -0500, Mike Gerwitz wrote:
> Well, I'm not getting those hours of my life back.
> 
> On Wed, Jan 23, 2019 at 00:58:57 -0500, Mike Gerwitz wrote:
> > usb usb3: root hub lost power or was reset
> 
> [...]
> 
> > uhci_hcd 0000:00:1d.0: host controller process error, something bad happened!
> > uhci_hcd 0000:00:1d.0: host controller halted, very bad!
> > uhci_hcd 0000:00:1d.0: HC died; cleaning up
> 
> [...]
> 
> > (Which is indeed true for my hub 0000:00:1d.0, which is usb6, according
> > to dmesg.)
> 
> Right after sending this message, I recalled that I used powertop to do
> "auto tuning".  Sure enough, one of the lines was this:
> 
>    Good          Autosuspend for USB device UHCI Host Controller [usb6]
> 
> By disabling it, it runs this command:
> 
>   echo on > /sys/bus/usb/devices/usb6/power/control
> 
> And it resolves my suspend issues on 4.20.3-gnu.
> 
> What sucks even more is that I had disabled those settings on the USB
> controllers over a year ago (because it was putting my USB keyboard to
> sleep, dropping the first keypress); don't know how it got re-enabled,
> but the user is probably to blame.
> 
> Sorry for the noise, everyone.  I hope that my suffering provides some
> value to others.
> 
> This bug can be closed.

I'm glad you figured it out. Powertop is a fun beast, often with
unintended consequences.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #16 received at 34176-done <at> debbugs.gnu.org (full text, mbox):

From: Gábor Boskovits <boskovits <at> gmail.com>
To: Mike Gerwitz <mtg <at> gnu.org>
Cc: 34176-done <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Wed, 23 Jan 2019 10:56:31 +0100

Hello Mike,

Mike Gerwitz <mtg <at> gnu.org> ezt írta (időpont: 2019. jan. 23., Sze, 7:10):
>
> Well, I'm not getting those hours of my life back.
>

Sorry to hear that.

> On Wed, Jan 23, 2019 at 00:58:57 -0500, Mike Gerwitz wrote:
> > usb usb3: root hub lost power or was reset
>
> [...]
>
> > uhci_hcd 0000:00:1d.0: host controller process error, something bad happened!
> > uhci_hcd 0000:00:1d.0: host controller halted, very bad!
> > uhci_hcd 0000:00:1d.0: HC died; cleaning up
>
> [...]
>
> > (Which is indeed true for my hub 0000:00:1d.0, which is usb6, according
> > to dmesg.)
>
> Right after sending this message, I recalled that I used powertop to do
> "auto tuning".  Sure enough, one of the lines was this:
>
>    Good          Autosuspend for USB device UHCI Host Controller [usb6]
>
> By disabling it, it runs this command:
>
>   echo on > /sys/bus/usb/devices/usb6/power/control
>
> And it resolves my suspend issues on 4.20.3-gnu.
>

I am glad you finally found out.

> What sucks even more is that I had disabled those settings on the USB
> controllers over a year ago (because it was putting my USB keyboard to
> sleep, dropping the first keypress); don't know how it got re-enabled,
> but the user is probably to blame.
>
> Sorry for the noise, everyone.  I hope that my suffering provides some
> value to others.

No problem.

> This bug can be closed.
>

You can close bugs by sending a mail to
bugnumber-done.debbugs.gnu.org, like I did here.

> --
> Mike Gerwitz
> Free Software Hacker+Activist | GNU Maintainer & Volunteer
> GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
> https://mikegerwitz.com

Best regards,
g_bor

Message #19 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Sun, 27 Jan 2019 10:34:11 -0500

[Message part 1 (text/plain, inline)]

On Wed, Jan 23, 2019 at 01:08:39 -0500, Mike Gerwitz wrote:
> Right after sending this message, I recalled that I used powertop to do
> "auto tuning".  Sure enough, one of the lines was this:
>
>    Good          Autosuspend for USB device UHCI Host Controller [usb6]
>
> By disabling it, it runs this command:
>
>   echo on > /sys/bus/usb/devices/usb6/power/control
>
> And it resolves my suspend issues on 4.20.3-gnu.

[...]

> This bug can be closed.

Sorry, I take that back.  I could have sworn it did resolve the issue,
but that does not seem to be the case.

This seems unrelated to powertop; `auto' seems to be the default setting
of /sys/bus/usb/devices/*/power/control, which is reset on reboot.  In
any case, setting it to `on' does not solve my suspend issue.

It's not a hardware issue, because earlier kernel versions work just
fine.

I tried debugging this further, but even on a minimal Guix system, I
cannot seem to work around the problem.

Any advice, anyone?

-- 
Mike Gerwitz

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Mike Gerwitz <mtg <at> gnu.org>
Cc: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Sun, 27 Jan 2019 23:12:11 -0800

[Message part 1 (text/plain, inline)]

Mike Gerwitz <mtg <at> gnu.org> writes:

> Any advice, anyone?

I don't have any advice yet, but I can say that in the last few weeks
(months?), after updating my Lenovo x200 system, I've noticed similar
problems.  I can put the laptop to sleep by shutting the lid, for
example, but when I open it back up, it's frozen and I have to reboot.

I've been so busy I haven't investigated, but you're probably not alone.

-- 
Chris

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Lemmer Webber <cwebber <at> dustycloud.org>
To: Mike Gerwitz <mtg <at> gnu.org>
Cc: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Mon, 28 Jan 2019 04:53:40 -0500

Mike Gerwitz writes:

> On Wed, Jan 23, 2019 at 01:08:39 -0500, Mike Gerwitz wrote:
>> Right after sending this message, I recalled that I used powertop to do
>> "auto tuning".  Sure enough, one of the lines was this:
>>
>>    Good          Autosuspend for USB device UHCI Host Controller [usb6]
>>
>> By disabling it, it runs this command:
>>
>>   echo on > /sys/bus/usb/devices/usb6/power/control
>>
>> And it resolves my suspend issues on 4.20.3-gnu.
>
> [...]
>
>> This bug can be closed.
>
> Sorry, I take that back.  I could have sworn it did resolve the issue,
> but that does not seem to be the case.
>
> This seems unrelated to powertop; `auto' seems to be the default setting
> of /sys/bus/usb/devices/*/power/control, which is reset on reboot.  In
> any case, setting it to `on' does not solve my suspend issue.
>
> It's not a hardware issue, because earlier kernel versions work just
> fine.
>
> I tried debugging this further, but even on a minimal Guix system, I
> cannot seem to work around the problem.
>
> Any advice, anyone?

Recently I had to switch back to my x200 and found that it wouldn't
suspend anymore, and I wasn't sure why.  Then I somehow screwed up my
system and had to do a reinstall of guix and suspend worked again.  I
wasn't sure why... now I seem to have the answer.  When I reinstalled, I
must not have done so from the latest guix versions.

cwebber <at> jasmine:~$ uname -a
Linux jasmine 4.17.3-gnu #1 SMP 1 x86_64 GNU/Linux

So I guess if I did a system upgrade, I would have the problem again :)

Note that I am running this x200 with a (somewhat out of date) Libreboot
install.

Message #28 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Tue, 29 Jan 2019 07:14:15 -0500

[Message part 1 (text/plain, inline)]

reopen 34176
thanks

On Sun, Jan 27, 2019 at 10:34:11 -0500, Mike Gerwitz wrote:
> Sorry, I take that back.  I could have sworn it did resolve the issue,
> but that does not seem to be the case.
>
> This seems unrelated to powertop; `auto' seems to be the default setting
> of /sys/bus/usb/devices/*/power/control, which is reset on reboot.  In
> any case, setting it to `on' does not solve my suspend issue.

Okay, the value of `control' makes no difference.  I just accidentally
closed my laptop on 4.20.3-gnu and it resumed.  I'm _not_ on a minimal
system---I have all my usual services running.

I have to go to work shortly, so I'll play around with it a little more
later tonight and see if e.g. plugging in my Nitrokey (actually using
the USB hub) had an effect.  With that said, I use my Nitrokey every day
(but I did _not_ use it most of the times I was trying to debug this
issue) and I thought I've had suspend issues with it before.  I guess
we'll see.

-- 
Mike Gerwitz

[signature.asc (application/pgp-signature, inline)]

Message #31 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Wed, 30 Jan 2019 23:07:52 -0500

[Message part 1 (text/plain, inline)]

On Tue, Jan 29, 2019 at 07:14:15 -0500, Mike Gerwitz wrote:
> reopen 34176
> thanks

I guess I don't have permission to do this, since nothing happened.  Can
someone do this for me?

Alternatively, if this seems outside the scope of something we'd want to
track in Guix, just leave it closed.

> On Sun, Jan 27, 2019 at 10:34:11 -0500, Mike Gerwitz wrote:
> I have to go to work shortly, so I'll play around with it a little more
> later tonight and see if e.g. plugging in my Nitrokey (actually using
> the USB hub) had an effect.  With that said, I use my Nitrokey every day
> (but I did _not_ use it most of the times I was trying to debug this
> issue) and I thought I've had suspend issues with it before.  I guess
> we'll see.

I was not able to figure it out with the time I had available to spend
on this the past couple of nights.  It's once again locking up when
resuming, and I cannot get it to resume correctly.

-- 
Mike Gerwitz

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Mike Gerwitz <mtg <at> gnu.org>
Cc: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Thu, 31 Jan 2019 02:02:34 -0500

reopen 34176
thanks

Hi Mike,

Mike Gerwitz <mtg <at> gnu.org> writes:

> On Tue, Jan 29, 2019 at 07:14:15 -0500, Mike Gerwitz wrote:
>> reopen 34176
>> thanks
>
> I guess I don't have permission to do this, since nothing happened.  Can
> someone do this for me?

No permission is needed, but it must be sent to the correct address.  It
must be sent to <control <at> debbugs.gnu.org>.  I add that address to the
BCC header when including such commands.

> Alternatively, if this seems outside the scope of something we'd want to
> track in Guix, just leave it closed.

I'm glad to have it in our tracker.

>> On Sun, Jan 27, 2019 at 10:34:11 -0500, Mike Gerwitz wrote:
>> I have to go to work shortly, so I'll play around with it a little more
>> later tonight and see if e.g. plugging in my Nitrokey (actually using
>> the USB hub) had an effect.  With that said, I use my Nitrokey every day
>> (but I did _not_ use it most of the times I was trying to debug this
>> issue) and I thought I've had suspend issues with it before.  I guess
>> we'll see.
>
> I was not able to figure it out with the time I had available to spend
> on this the past couple of nights.  It's once again locking up when
> resuming, and I cannot get it to resume correctly.

FWIW, unless someone has a better idea, my recommendation would be to do
a binary search on the kernel versions, to find which version introduced
the problem.

You mentioned in your original report that 4.18.9 works and 4.19.5
fails.  The first thing I would check is whether 4.19 works.  If it
works, then you could find which update to the 4.19.y branch introduced
the problem.  If 4.19 fails, then you could check 4.18.20, which is the
final release of 4.18.y.  If it fails, you could find which update to
4.18.y introduced the problem.

The reason I suggest checking these first is because each stable update
is relatively small, which will simplify the task of finding the faulty
commit.  It might be possible to look at the changelog of the relevant
stable update and make some educated guesses about which commits to try
reverting.

If 4.18.20 works and 4.19 fails, then it will probably be necessary to
do a "git bisect" on the upstream kernel git repository between those
two versions, to find the commit that introduced the problem.  If it
comes to this, let me know and I'll help you find a way to do this
efficiently.

     Regards,
       Mark

Message #39 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Thu, 31 Jan 2019 22:56:26 -0500

[Message part 1 (text/plain, inline)]

On Thu, Jan 31, 2019 at 02:02:34 -0500, Mark H Weaver wrote:
> reopen 34176
> thanks
>
> Hi Mike,
>
> Mike Gerwitz <mtg <at> gnu.org> writes:
>
>> On Tue, Jan 29, 2019 at 07:14:15 -0500, Mike Gerwitz wrote:
>>> reopen 34176
>>> thanks
>>
>> I guess I don't have permission to do this, since nothing happened.  Can
>> someone do this for me?
>
> No permission is needed, but it must be sent to the correct address.  It
> must be sent to <control <at> debbugs.gnu.org>.  I add that address to the
> BCC header when including such commands.

Ah, that explains it.  I saw control <at> debbugs.gnu.org in the debbugs
documentation, but looking at past commands on this list (e.g. from
you), I didn't see it CC'd, so I thought maybe it wasn't necessary.

>> Alternatively, if this seems outside the scope of something we'd want to
>> track in Guix, just leave it closed.
>
> I'm glad to have it in our tracker.

Thanks.  Hopefully it won't persist for too long.

> FWIW, unless someone has a better idea, my recommendation would be to do
> a binary search on the kernel versions, to find which version introduced
> the problem.

That's what I was going to do originally, except I hadn't had the time
to research the best way of doing this in Guix.  I did see a kernel
system configuration option, but I thought that looked more like
specifying Linux or Hurd (the default value is `LINUX-LIBRE'), not the
specific kernel version.

What's the proper way to go about doing this on GuixSD?  Sorry if that's
clearly documented and I just missed it.

> If 4.18.20 works and 4.19 fails, then it will probably be necessary to
> do a "git bisect" on the upstream kernel git repository between those
> two versions, to find the commit that introduced the problem.  If it
> comes to this, let me know and I'll help you find a way to do this
> efficiently.

This is another situation where my unfamiliarity with Guix is the
difficult part.  I'm comfortable compiling Linux, but I'd need to figure
out how to actually boot to it.

-- 
Mike Gerwitz

[signature.asc (application/pgp-signature, inline)]

Message #42 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Sun, 12 May 2019 10:16:12 -0400

[Message part 1 (text/plain, inline)]

On Wed, Jan 23, 2019 at 00:58:57 -0500, Mike Gerwitz wrote:
> usb usb3: root hub lost power or was reset
> general protection fault: 0000 [#1] SMP PTI
> CPU: 1 PID: 441 Comm: kworker/u4:23 Tainted: 6     4.28-3gnu #1
> Hardware name: LENOVO 7459VLP/7459VLP, BIOS CBET4000 3774c98 09/07/2016
> Workqueue: events_unbound async_run_entry_fn
> RIP: 0010:uhci_scan_schedule.part.38+0xa3/0xad0
> Code: [...machine code...]
> [...register values...]
> Call trace:  [I'm omitting address offsets]
>  ? __dev_printk
>  uhci_hub_status_data
>  usb_hcd_poll_rh_status
>  uhci_pci_resume
>  resume_common
>  hcd_pci_resume
>  pci_pm_resume
> [...ata stuff, successful...]
> uhci_hcd 0000:00:1d.0: host controller process error, something bad happened!
> uhci_hcd 0000:00:1d.0: host controller halted, very bad!
> uhci_hcd 0000:00:1d.0: HC died; cleaning up

In 5.x this doesn't appear to be a problem anymore---the first message
still happens, but it no longer causes a panic.  I don't know precisely
when it was fixed, since it's been a little while since I've
upgraded.  I haven't had the chance [yet] to look at the linux sources
to see any technical details.

I'm going to give it a week or so, and if the problem has indeed been
resolved, I'll close this.

-- 
Mike Gerwitz

[signature.asc (application/pgp-signature, inline)]

Message #45 received at 34176 <at> debbugs.gnu.org (full text, mbox):

From: Mike Gerwitz <mtg <at> gnu.org>
To: 34176 <at> debbugs.gnu.org
Subject: Re: bug#34176: X200 kernel panic on S3 resume (linux-libre > 4.18.9)
Date: Wed, 15 May 2019 20:57:46 -0400

[Message part 1 (text/plain, inline)]

close 34176
thanks

This is no longer a problem on my X200 with Libreboot.

On Sun, May 12, 2019 at 10:16:12 -0400, Mike Gerwitz wrote:
> In 5.x this doesn't appear to be a problem anymore---the first message
> still happens, but it no longer causes a panic.  I don't know precisely
> when it was fixed, since it's been a little while since I've
> upgraded.  I haven't had the chance [yet] to look at the linux sources
> to see any technical details.
>
> I'm going to give it a week or so, and if the problem has indeed been
> resolved, I'll close this.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.