GNU bug report logs - #67629
[Cuirass] ‘remote-server’ wrecks havoc when misbehaving clients connect

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 67629 in the body.
You can then email your comments to 67629 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: bug-guix <at> gnu.org
Subject: [Cuirass] ‘remote-server’ wrecks havoc
 when misbehaving clients connect
Date: Mon, 04 Dec 2023 22:46:21 +0100

The ‘cuirass remote-server’ process goes awry when misbehaved clients
connect; specifically, some of its fibers crash, leaving it running but
inactive:

--8<---------------cut here---------------start------------->8---
2023-12-04 10:13:14 periodic update: 0 resumable, 0 failed builds
2023-12-04 10:13:28 error: Connection reset by peer when replying to xx.xx.xx.254.
2023-12-04 10:13:42 error: invalid log received from xx.xx.xx.254
2023-12-04 10:13:42 error: EOF while receiving log from xx.xx.xx.254
2023-12-04 10:13:48 Uncaught exception in task:
2023-12-04 10:13:48 In fibers.scm:
2023-12-04 10:13:48    186:20  8 (_)
2023-12-04 10:13:48    145:21  7 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48   1752:10  6 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
2023-12-04 10:13:48    695:11  5 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48   1747:15  4 (with-exception-handler #<procedure 7f982c2abd50 at ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
2023-12-04 10:13:48    437:23  3 (serve-build-requests _ #<<channel> getq: #<atomic-box 7f982e764390 value: (() #(#<atomic-box 7f98283753a0 value: W> #<procedure resume (thunk)>))> getq-gc-co
unter: #<atomic-box 7f982e764320 value: 16> putq: #<atomic-box 7f982e764240 value: (())> putq-gc-counter: #<atomic-box 7f982e764230 value: 31>>)
2023-12-04 10:13:48 In cuirass/remote.scm:
2023-12-04 10:13:48     466:6  2 (receive-message _ #:router? _)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48   1685:16  1 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48   1685:16  0 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:13:48 Throw to key `match-error' with args `("match" "no matching pattern" (#<zmq-message 7f981881d9a0> #<zmq-message 7f981881d900> #<zmq-message 7f981881d860>))'.
2023-12-04 10:13:48 Uncaught exception in task:
2023-12-04 10:13:48 In fibers.scm:
2023-12-04 10:13:48     172:8  2 (_)
2023-12-04 10:13:48 In cuirass/notification.scm:
2023-12-04 10:13:48    162:15  1 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48   1685:16  0 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:13:48 Throw to key `psql-query-error' with args `(#f #f "no connection to the server\n")'.
2023-12-04 10:14:06 0 items queued for eventual download
2023-12-04 10:14:07 Uncaught exception in task:
2023-12-04 10:14:07 In fibers.scm:
2023-12-04 10:14:07     172:8  2 (_)
2023-12-04 10:14:07 In cuirass/scripts/remote-server.scm:
2023-12-04 10:14:07    430:10  1 (_)
2023-12-04 10:14:07 In ice-9/boot-9.scm:
2023-12-04 10:14:07   1685:16  0 (raise-exception _ #:continuable? _)
2023-12-04 10:14:07 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:14:07 Throw to key `psql-query-error' with args `(#f #f "no connection to the server\n")'.
2023-12-04 10:14:30 error: invalid log received from xx.xx.xx.254
2023-12-04 10:14:30 error: invalid log received from xx.xx.xx.254
2023-12-04 10:14:35 error: invalid log received from xx.xx.xx.254
--8<---------------cut here---------------end--------------->8---

Ludo’.

Message #8 received at 67629 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 67629 <at> debbugs.gnu.org
Subject: Re: bug#67629: [Cuirass] ‘remote-server’ wrecks havoc when misbehaving clients connect
Date: Tue, 12 Dec 2023 12:03:51 -0500

Hi,

Ludovic Courtès <ludovic.courtes <at> inria.fr> writes:

> The ‘cuirass remote-server’ process goes awry when misbehaved clients
> connect; specifically, some of its fibers crash, leaving it running but
> inactive:

I guess what I reported in bug#67633 is a symptom of the above problem,
perhaps?

-- 
Thanks,
Maxim

Message #11 received at 67629 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 67629 <at> debbugs.gnu.org
Subject: Re: bug#67629: [Cuirass] ‘remote-server’ wrecks havoc when misbehaving clients connect
Date: Wed, 03 Apr 2024 11:09:43 +0200

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> Ludovic Courtès <ludovic.courtes <at> inria.fr> writes:
>
>> The ‘cuirass remote-server’ process goes awry when misbehaved clients
>> connect; specifically, some of its fibers crash, leaving it running but
>> inactive:
>
> I guess what I reported in bug#67633 is a symptom of the above problem,
> perhaps?

It could be yes, if ‘remote-server’ crashed right at that moment.

Ludo’.

Message #16 received at 67629-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 67629-done <at> debbugs.gnu.org
Subject: Re: bug#67629: [Cuirass] ‘remote-server’ wrecks havoc when misbehaving clients connect
Date: Wed, 03 Apr 2024 11:13:10 +0200

Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:

> The ‘cuirass remote-server’ process goes awry when misbehaved clients
> connect; specifically, some of its fibers crash, leaving it running but
> inactive:
>
> 2023-12-04 10:13:14 periodic update: 0 resumable, 0 failed builds
> 2023-12-04 10:13:28 error: Connection reset by peer when replying to xx.xx.xx.254.
> 2023-12-04 10:13:42 error: invalid log received from xx.xx.xx.254
> 2023-12-04 10:13:42 error: EOF while receiving log from xx.xx.xx.254
> 2023-12-04 10:13:48 Uncaught exception in task:
> 2023-12-04 10:13:48 In fibers.scm:
> 2023-12-04 10:13:48    186:20  8 (_)
> 2023-12-04 10:13:48    145:21  7 (_)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48   1752:10  6 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
> 2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
> 2023-12-04 10:13:48    695:11  5 (_)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48   1747:15  4 (with-exception-handler #<procedure 7f982c2abd50 at ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
> 2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
> 2023-12-04 10:13:48    437:23  3 (serve-build-requests _ #<<channel> getq: #<atomic-box 7f982e764390 value: (() #(#<atomic-box 7f98283753a0 value: W> #<procedure resume (thunk)>))> getq-gc-co
> unter: #<atomic-box 7f982e764320 value: 16> putq: #<atomic-box 7f982e764240 value: (())> putq-gc-counter: #<atomic-box 7f982e764230 value: 31>>)
> 2023-12-04 10:13:48 In cuirass/remote.scm:
> 2023-12-04 10:13:48     466:6  2 (receive-message _ #:router? _)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48   1685:16  1 (raise-exception _ #:continuable? _)
> 2023-12-04 10:13:48   1685:16  0 (raise-exception _ #:continuable? _)
> 2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> 2023-12-04 10:13:48 Throw to key `match-error' with args `("match" "no matching pattern" (#<zmq-message 7f981881d9a0> #<zmq-message 7f981881d900> #<zmq-message 7f981881d860>))'.

With commit 9a1452ee021c9f773424961cfeef47ca0b7c5c5a, ‘cuirass
remote-server’ handles this situation gracefully.

I’m still curious about who’s sending us garbage: it has to be valid
zmq, just with the wrong number of parts.  From the logs, it seemed like
a port scanner or something on the MDC network was sending garbage on
all the open ports, but that would need to be a sophisticated one.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.