GNU bug report logs - #69291
[PATCH 0/5] Start making substitute code less coupled

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix-patches; Reported by: Christopher Baines <mail@HIDDEN>; Keywords: patch; dated Tue, 20 Feb 2024 19:21:02 UTC; Maintainer for guix-patches is guix-patches@HIDDEN.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 3 Apr 2024 17:30:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Apr 03 13:30:46 2024
Received: from localhost ([127.0.0.1]:59448 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rs4Rh-0007dE-So
	for submit <at> debbugs.gnu.org; Wed, 03 Apr 2024 13:30:46 -0400
Received: from mira.cbaines.net
 ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:43921)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rs4Rf-0007cb-II
 for 69291 <at> debbugs.gnu.org; Wed, 03 Apr 2024 13:30:44 -0400
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id 57F6B27BBE2;
 Wed,  3 Apr 2024 18:30:38 +0100 (BST)
Received: from felis (localhost.lan [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id ec37e971;
 Wed, 3 Apr 2024 17:30:38 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#69291] [PATCH 5/5] scripts: substitute: Extract script
 specific output from download-nar.
In-Reply-To: <87o7c7f8nb.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Fri, 23 Feb 2024 17:27:36 +0100")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <f4af19b037826cad90bbcfe400ad864f028cc7d8.1708458147.git.mail@HIDDEN>
 <87o7c7f8nb.fsf@HIDDEN>
User-Agent: mu4e 1.12.2; emacs 29.3
Date: Wed, 03 Apr 2024 18:30:37 +0100
Message-ID: <87frw2jqv6.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 69291
Cc: 69291 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Christopher Baines <mail@HIDDEN> skribis:
>
>> As this moves download-nar in a direction where it could be used outside=
 the
>> substitute script.
>>
>> * guix/scripts/substitute.scm (download-nar): Return more information an=
d move
>> status-port output to=E2=80=A6
>> (guix-substitute): here.
>>
>> Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
>
> LGTM.

Looking at this patch some more, I missed the failure case in
process-substitution/fallback, so I'll send an updated patch (for this
and the previous patch).

The other patches (1 to 3) have been pushed to master as
ecbab97f0732d6979642078a7164d4032b2102b8.

Chris

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNkj1fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xf4DQ//aHxQVHxASQ+8EM31F4gEfI7+gfcdGRrU
Ba0kGYO5KlMTHI07MBU4UD5ptiAO4kkAcMxnhKKSzleNrsSEddevtuoT1y/+rRmQ
zDwjB2caAxFiQngL1YdTZJ0cl/b8Wey9JF5xuGZQ94KHjkQv0Dp/G/DdiGgeE++h
ph/rKclaXyzmHyao9lQgcjnhMYA/Bct8QLHsVaWLu+T7y/Iv4vaYbel0zvhiwvnL
wfrR64L6TlzWA9lDhdPHW77P9X23MkEdVqbKXSujxvQCgJAzP3HDd1EmZi1XP2z8
H9nXcuAfZ4uaDqHvlzhaUeuLMZlkGzq1S2+zSkUanF/HH/ECZgHyFWSC1d/tITFM
MPe5bK3iMqN/8dfqfz/HmQenacOqLXHEHibSyYwfOo2GeCzLKTAjLbFfu64/QIFW
HbEvPWaB6rN4fgQKT24SXqZbu5CnqsOL0kUGdVL9ZCUXJzDwHTNundVcgWJrETkm
1IRw/6w1YivStZ/d71/cyDAu1CJ3jWaJZolfptr5m+8JnZvgW8vH0A89cqABietG
MbRfBraDG3q9qHJ5hhVM0LrxUaBfM8HBQXRqe2Ozjrif/pseo38Kr7EiXOSuEXmJ
Y3U+E2B16ZBXiDjhhuEmXqKR/aEhsR5/SnI5MOkdlAH++aO7r1mYsGYWD/BLCQhW
OtPqWEOt2cE=
=n8G9
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 3 Apr 2024 17:28:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Apr 03 13:28:23 2024
Received: from localhost ([127.0.0.1]:59441 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rs4PO-0007Ot-Q6
	for submit <at> debbugs.gnu.org; Wed, 03 Apr 2024 13:28:23 -0400
Received: from mira.cbaines.net ([212.71.252.8]:43298)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rs4PN-0007Ol-Bj
 for 69291 <at> debbugs.gnu.org; Wed, 03 Apr 2024 13:28:22 -0400
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id 630B727BBE2;
 Wed,  3 Apr 2024 18:28:16 +0100 (BST)
Received: from felis (localhost.lan [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id 3e8971ef;
 Wed, 3 Apr 2024 17:28:16 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting
 fast vs small compressions.
In-Reply-To: <87sf1jf8os.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Fri, 23 Feb 2024 17:26:43 +0100")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@HIDDEN>
 <87sf1jf8os.fsf@HIDDEN>
User-Agent: mu4e 1.12.2; emacs 29.3
Date: Wed, 03 Apr 2024 18:28:15 +0100
Message-ID: <87o7aqjqz4.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 69291
Cc: 69291 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Christopher Baines <mail@HIDDEN> skribis:
>
>> Pulling the logic up to the script makes this code more portable and not
>> reliant on setting a global variable.
>>
>> * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to=
=E2=80=A6
>> (%default-prefer-fast-decompression?): this.
>> (display-narinfo-data): Update accordingly.
>> (download-nar): Add prefer-fast-decompression? as a keyword argument, re=
move
>> code to set! it and return the cpu-usage recorded.
>> (process-substitution, process-substitution/fallback): Accept and pass t=
hrough
>> prefer-fast-decompression? to download-nar.
>> (guix-substitute): Move the prefer fast decompression switching logic he=
re.
>>
>> Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb
>
> [...]
>
>> -(define %prefer-fast-decompression?
>> -  ;; Whether to prefer fast decompression over good compression ratios.=
  This
>> -  ;; serves in particular to choose between lzip (high compression rati=
o but
>> -  ;; low decompression throughput) and zstd (lower compression ratio bu=
t high
>> -  ;; decompression throughput).
>> -  #f)
>> +(define %default-prefer-fast-decompression? #f)
>
> I would either remove this variable or add a comment describing it (we
> should do that for all top-level variables).

I've added a comment now, and I'll sent an updated patch.

>> @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination
>>              (format status-port "hash-mismatch ~a ~a ~a~%"
>>                      (hash-algorithm-name algorithm)
>>                      (bytevector->nix-base32-string expected)
>> -                    (bytevector->nix-base32-string actual)))))))
>> +                    (bytevector->nix-base32-string actual))))
>> +
>> +      cpu-usage)))
>
> [...]
>
>> +               (let ((cpu-usage
>> +                      (process-substitution reply-port store-path desti=
nation
>> +                                            #:cache-urls (substitute-ur=
ls)
>> +                                            #:acl (current-acl)
>> +                                            #:deduplicate? deduplicate?
>> +                                            #:print-build-trace?
>> +                                            print-build-trace?
>> +                                            #:prefer-fast-decompression?
>> +                                            prefer-fast-decompression?)=
))
>> +
>> +                 ;; Create a hysteresis: depending on CPU usage, favor
>> +                 ;; compression methods with faster decompression (like=
 ztsd)
>> +                 ;; or methods with better compression ratios (like lzi=
p).
>> +                 ;; This stems from the observation that substitution c=
an be
>> +                 ;; CPU-bound when high-speed networks are used:
>> +                 ;; <https://lists.gnu.org/archive/html/guix-devel/2020=
-12/msg00177.html>.
>> +                 ;; To simulate "slow" networking or changing condition=
s, run:
>> +                 ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit la=
tency
>> +                 ;; 50ms burst 1540 and then cancel with: sudo tc qdisc=
 del
>> +                 ;; dev eno1 root
>> +                 (loop (cond
>> +                        ;; Whether to prefer fast decompression over go=
od
>> +                        ;; compression ratios.  This serves in particul=
ar to
>> +                        ;; choose between lzip (high compression ratio =
but low
>> +                        ;; decompression throughput) and zstd (lower
>> +                        ;; compression ratio but high decompression
>> +                        ;; throughput).
>> +                        ((> cpu-usage .8) #t)
>> +                        ((< cpu-usage .2) #f)
>> +                        (else prefer-fast-decompression?)))))))))
>
>
> Instead of having =E2=80=98download-nar=E2=80=99 return its CPU usage, wh=
ich is
> surprising, maybe should wrap the =E2=80=98process-substitution=E2=80=99 =
call in
> =E2=80=98guix-substitute=E2=80=99 in =E2=80=98with-cpu-usage-monitoring=
=E2=80=99 and keep all the logic
> in =E2=80=98guix-substitute=E2=80=99?

Yeah, that makes sense. I'll send an updated patch shortly.

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNka9fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdEWA//XHWY7mjjHWzznRmKks7rlBVFWjgjW60g
2/UVJp6YGz2DPunTKMcVPPF3+PF47JbuPZbPf4fPli17oh861QKJ30LFXtFRFomo
hmOAdW52yGjIZOBBE77wDmxgOTQctzuTDXl+xfdVMHptDt7HmsMmeFWs6mvGnxI+
1pUM4UjN0rDoEnjgX0pw3SUMlxao2Hfh77oOvgmb6AkVArTctcyOiAKQ6FojVqt8
QKb19eCNPudJhNR0fgnX1CEBQkif8S8xgHzj3nG+YKQ3RRxtexebh9oTBA1Uc9xE
apNpwRwOcLaJE2IblAM2CJ5EMHnLDFSh6jyKW2M6R/G7Of8MtWX9xvQQy3hdAc6L
KWjQhdyQYWazoOVOmxq/AnAQnAJo+ZaOVraM07ZTPpjBlCnG1vCYrlTRbrFQKosP
qSDqz3jn7lPNTXNWP3RNtNBZU94eXf3ODlHmeHv6v12jWVj91qG3Xm4I7kXQTvS9
2/ux3InIitJJs3YopC1TtjJkTSCPSyl28kVEvAeHn6nNQWd0hVJG5Ik8VKi67ePK
qjDD923+8WoC5P9lrYfuF6NFmzhxTX2UHIUEtqDauE4m3FQsidrbK8x3/fLR6hGc
qPDcWKvNwL8dNv7a2dOY1SZDh6dK5q+/QMSM9jNSCeS6WeyKazVdFtnP4uj3T9xd
AuUqofni73E=
=iiwd
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 3 Apr 2024 17:26:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Apr 03 13:26:21 2024
Received: from localhost ([127.0.0.1]:59435 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rs4NQ-0007DS-Pn
	for submit <at> debbugs.gnu.org; Wed, 03 Apr 2024 13:26:21 -0400
Received: from mira.cbaines.net ([212.71.252.8]:43296)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rs4NM-0007DF-3f
 for 69291 <at> debbugs.gnu.org; Wed, 03 Apr 2024 13:26:19 -0400
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id D7AA427BBE2;
 Wed,  3 Apr 2024 18:26:10 +0100 (BST)
Received: from felis (localhost.lan [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id d718ea08;
 Wed, 3 Apr 2024 17:26:10 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#69291] [PATCH 2/5] scripts: substitute: Allow not using
 with-timeout in download-nar.
In-Reply-To: <8734tjgnkt.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Fri, 23 Feb 2024 17:19:46 +0100")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <2bcc2278b5aefcd81fbc53b791d08622b6d93c7c.1708458147.git.mail@HIDDEN>
 <8734tjgnkt.fsf@HIDDEN>
User-Agent: mu4e 1.12.2; emacs 29.3
Date: Wed, 03 Apr 2024 18:26:08 +0100
Message-ID: <87wmpejr2n.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 69291
Cc: 69291 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Christopher Baines <mail@HIDDEN> skribis:
>
>> I don't think the approach of using SIGALARM here for the timeout will w=
ork
>> well in all cases (e.g. when using Guile Fibers), so make it possible to=
 avoid
>> this.
>>
>> * guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in =
as an
>> option.
>>
>> Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6
>
> The patch LGTM.
>
> That said, maybe we should just pass #:timeout to =E2=80=98http-fetch=E2=
=80=99?  It=E2=80=99s
> not strictly equivalent because it only controls the timeout on
> connection establishment, but in practice it should have the same
> effect.

I haven't done that yet, but longer term I do want to make more changes
here.

In particular, I think the way to go regarding timeouts is to use Guile
suspendable ports and have the read/write waiters handle the
timeout. The build coordinator does this [1], it's quite similar to what
is happening with the http-fetch timeout in connect*, but it's
compatible with fibers.

1: https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/tree/guix-b=
uild-coordinator/utils/fibers.scm#n473

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNkTBfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xe5Jg//fpPob6caE6VA3e0war4+32itKvSmN/vj
ze7PjwiJLK/5t1ZnpUHCrbGLbYux4uKhYET2eKDaCeshF/Wwccd160/UC5OuJ0vr
Dxp49jR37AzC2sOGNTasWF14VItVpXLS6+fRGV1SOYuZYmX64rEkQo812VHHZgiJ
bgHwtdSgchkjgiQRjq63xOjEY8smePVCDwdRZPA1XRv1OAT8tgcK3Q4xyvQbN65m
KItU+rfz7ObcAn4Xo6oe/qU0dSlcozRGAsxCaEU3RSI7YHAElDG2qcOyhHogYRQ2
UyyhvxTDSrG4F/AKpPsnx6NM64jAAzbH0GEIe0nmeaOnPKLQMxfiBhlp1y/x3z3o
/MwokDXQJaXEq/v/D9eYraVr1vkVcqzWSyc11KGFZsyGXw2viLvX9td1AiXZ/+VI
wDHGJQ4X0gJRdIS4v7Ou9RfFv9ORZERVt95vL4gqtST2KHlf8419hn7ehURDIQIV
8nSPl6JU5A1VREG7aETEBX2QQQsRKdj7Fhkln/uPJsMPupNGsXh91S3j723nXLWB
xKyQJr7K88jN1ODfiUbTM8DoxlZkpBIaJ4qKiymZjKgbKvHAvbjxVgjeZcZ7jVLN
3EGgwTGSgo0+y4OiHu3S2CrKHWEj7i/P59AUPVUbLI/YHXUZqPILQVWt13g9GqhW
QEpePhJmp0k=
=Ug3X
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 23 Feb 2024 16:55:23 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 23 11:55:23 2024
Received: from localhost ([127.0.0.1]:53693 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rdYpT-0005Pr-J9
	for submit <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:55:23 -0500
Received: from eggs.gnu.org ([209.51.188.92]:34142)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1rdYbs-0004c2-TK
 for 69291 <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:41:17 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1rdYHV-0005Rq-NP; Fri, 23 Feb 2024 11:20:13 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=lxo//paOuNYkPAtNk4/irjTUO4yrBqxWb6qDnnN0cB0=; b=CR/Qlhdi6vYwtSdvloVU
 ZGEE6XrwU7tLO2itm9hrNNcTMBgm3z8wpP9E3u502NPWY51X3J1JmdpV21D1uAqY5M99tcpERqOkz
 f9Qj/Wh+GItrghfq7rd9jreIVhOuQaH2ZoxIeoc56LKJgaBTBjwsDHMxuEopK2RayHALW3+r+jgtl
 kEJUAqa0jI9JNIdw1fVWr9A40ZFJ7tRiyYfySY3oUwNJoewaQ9I1sBygkwGprCrNy5qIf8PO9a4A6
 iSDhJ3XRLnj1m2mSEz9r4hVVd1glMT8e0emYpqcqBZHf7+5otGQqZzUvNkv8HR0d/kt71U8tow14V
 ZUz9D/5M7ZMBBg==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Christopher Baines <mail@HIDDEN>
Subject: Re: [bug#69291] [PATCH 3/5] scripts: substitute: Replace some leave
 calls with raise.
In-Reply-To: <3a88dc8a8369278f86f3a51a99aa62d991ba997d.1708458147.git.mail@HIDDEN>
 (Christopher Baines's message of "Tue, 20 Feb 2024 19:42:25 +0000")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <3a88dc8a8369278f86f3a51a99aa62d991ba997d.1708458147.git.mail@HIDDEN>
Date: Fri, 23 Feb 2024 17:20:07 +0100
Message-ID: <87y1bbf8zs.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 69291
Cc: Josselin Poiret <dev@HIDDEN>,
 Simon Tournier <zimon.toutoune@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>,
 Tobias Geerinckx-Rice <me@HIDDEN>, 69291 <at> debbugs.gnu.org,
 Ricardo Wurmus <rekado@HIDDEN>, Christopher Baines <guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Christopher Baines <mail@HIDDEN> skribis:

> These calls happen inside of with-error-handling, so the effect should be the
> same, but this opens up the possibility of using this code in a program that
> doesn't want to exit when one of these error conditions is met.
>
> Change-Id: I15d963615d85d419559fa0f4333fa4dc1dfbfd3b
>
> * guix/scripts/substitute.scm (download-nar, process-substitution): Use raise
> formatted-message rather than leave.
>
> Change-Id: Idd0880206b69e3903e19e0536b87d65a52c200d5

LGTM.




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 23 Feb 2024 16:55:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 23 11:55:19 2024
Received: from localhost ([127.0.0.1]:53690 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rdYpT-0005Pj-0H
	for submit <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:55:19 -0500
Received: from eggs.gnu.org ([209.51.188.92]:34142)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1rdYbr-0004c2-2g
 for 69291 <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:41:16 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1rdYNv-0006y6-RV; Fri, 23 Feb 2024 11:26:51 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=+s4shKIpRGvto9YZ7Yc35Q27K1Cb15aOsObxLoN+bs4=; b=PfnZUX/7w5mn0BmCUHbr
 rVxFYTQDKXaas76MCe1Vv8A9At820i4EUEVT9WTFYRmzW13/lIPP6BHE4agkbOhjlPW2Nl5iQAfOy
 pdeeQZrHj/+fzpWYVgNp9tm6tsXlswqYhSJ+GRRDksyeWnGjvf5W3Y5EO8S0zi0o7KaUwG3oBPC30
 1JlceB/NnQgT38TCkb9qrNajhaqFIbTmSdYzWF/44pekAP2Rqn/TgT4CW3IluYbavHlrPIeuLgB0+
 0E4/BKKPqwMue5+oSE/C5TyARDmzESk7C/1N1EqbLE1DOQGEVnuiW/ik9JtqNtLuL6ynostymqwRP
 ir/EQ4aWdbgbuA==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Christopher Baines <mail@HIDDEN>
Subject: Re: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting
 fast vs small compressions.
In-Reply-To: <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@HIDDEN>
 (Christopher Baines's message of "Tue, 20 Feb 2024 19:42:26 +0000")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@HIDDEN>
Date: Fri, 23 Feb 2024 17:26:43 +0100
Message-ID: <87sf1jf8os.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 69291
Cc: Josselin Poiret <dev@HIDDEN>,
 Simon Tournier <zimon.toutoune@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>,
 Tobias Geerinckx-Rice <me@HIDDEN>, 69291 <at> debbugs.gnu.org,
 Ricardo Wurmus <rekado@HIDDEN>, Christopher Baines <guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Christopher Baines <mail@HIDDEN> skribis:

> Pulling the logic up to the script makes this code more portable and not
> reliant on setting a global variable.
>
> * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to=E2=
=80=A6
> (%default-prefer-fast-decompression?): this.
> (display-narinfo-data): Update accordingly.
> (download-nar): Add prefer-fast-decompression? as a keyword argument, rem=
ove
> code to set! it and return the cpu-usage recorded.
> (process-substitution, process-substitution/fallback): Accept and pass th=
rough
> prefer-fast-decompression? to download-nar.
> (guix-substitute): Move the prefer fast decompression switching logic her=
e.
>
> Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb

[...]

> -(define %prefer-fast-decompression?
> -  ;; Whether to prefer fast decompression over good compression ratios. =
 This
> -  ;; serves in particular to choose between lzip (high compression ratio=
 but
> -  ;; low decompression throughput) and zstd (lower compression ratio but=
 high
> -  ;; decompression throughput).
> -  #f)
> +(define %default-prefer-fast-decompression? #f)

I would either remove this variable or add a comment describing it (we
should do that for all top-level variables).

> @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination
>              (format status-port "hash-mismatch ~a ~a ~a~%"
>                      (hash-algorithm-name algorithm)
>                      (bytevector->nix-base32-string expected)
> -                    (bytevector->nix-base32-string actual)))))))
> +                    (bytevector->nix-base32-string actual))))
> +
> +      cpu-usage)))

[...]

> +               (let ((cpu-usage
> +                      (process-substitution reply-port store-path destin=
ation
> +                                            #:cache-urls (substitute-url=
s)
> +                                            #:acl (current-acl)
> +                                            #:deduplicate? deduplicate?
> +                                            #:print-build-trace?
> +                                            print-build-trace?
> +                                            #:prefer-fast-decompression?
> +                                            prefer-fast-decompression?)))
> +
> +                 ;; Create a hysteresis: depending on CPU usage, favor
> +                 ;; compression methods with faster decompression (like =
ztsd)
> +                 ;; or methods with better compression ratios (like lzip=
).
> +                 ;; This stems from the observation that substitution ca=
n be
> +                 ;; CPU-bound when high-speed networks are used:
> +                 ;; <https://lists.gnu.org/archive/html/guix-devel/2020-=
12/msg00177.html>.
> +                 ;; To simulate "slow" networking or changing conditions=
, run:
> +                 ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit lat=
ency
> +                 ;; 50ms burst 1540 and then cancel with: sudo tc qdisc =
del
> +                 ;; dev eno1 root
> +                 (loop (cond
> +                        ;; Whether to prefer fast decompression over good
> +                        ;; compression ratios.  This serves in particula=
r to
> +                        ;; choose between lzip (high compression ratio b=
ut low
> +                        ;; decompression throughput) and zstd (lower
> +                        ;; compression ratio but high decompression
> +                        ;; throughput).
> +                        ((> cpu-usage .8) #t)
> +                        ((< cpu-usage .2) #f)
> +                        (else prefer-fast-decompression?)))))))))


Instead of having =E2=80=98download-nar=E2=80=99 return its CPU usage, whic=
h is
surprising, maybe should wrap the =E2=80=98process-substitution=E2=80=99 ca=
ll in
=E2=80=98guix-substitute=E2=80=99 in =E2=80=98with-cpu-usage-monitoring=E2=
=80=99 and keep all the logic
in =E2=80=98guix-substitute=E2=80=99?

Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 23 Feb 2024 16:41:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 23 11:41:16 2024
Received: from localhost ([127.0.0.1]:52604 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rdYbs-0004dK-9l
	for submit <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:41:16 -0500
Received: from eggs.gnu.org ([209.51.188.92]:34142)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1rdYbp-0004c2-QV
 for 69291 <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:41:14 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1rdYOr-0007B6-SB; Fri, 23 Feb 2024 11:27:49 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=jqUHIdvwjzd18nMPhGRCbzKOfO/Gc5lByogNSWth8+Y=; b=DHZ8W1FPXvKEEdUMCWxr
 tS9Px5zj3s7TZQvU7Dv5kObpmwianwSCpWJ/gPs8WDFIOjk9kCyYY5jheTDnzuYNL//dPjKV+AUvQ
 RpSSvzqkkKHHZr+zkMa++9LW9pwkWNQZzfRSgNT+0txxoRQN6QJtV5HEW1vp5DKONehZUinimVQj1
 0eB26p0axTWaSH7iDz4hFOdEk8tEiURnFsbgrPTx/Ovl0FpzaEcmkWLWBDSEy2yO1l86PkY4Derae
 4hKZT+wjuNAQ8fOm9jkfPjdGIUdyQ5rUuKQQb6093ga03UnP/EWTxwKkquJknMxaMfN6ZTWHWPtb/
 JJiD1y3pETmgMw==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Christopher Baines <mail@HIDDEN>
Subject: Re: [bug#69291] [PATCH 5/5] scripts: substitute: Extract script
 specific output from download-nar.
In-Reply-To: <f4af19b037826cad90bbcfe400ad864f028cc7d8.1708458147.git.mail@HIDDEN>
 (Christopher Baines's message of "Tue, 20 Feb 2024 19:42:27 +0000")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <f4af19b037826cad90bbcfe400ad864f028cc7d8.1708458147.git.mail@HIDDEN>
Date: Fri, 23 Feb 2024 17:27:36 +0100
Message-ID: <87o7c7f8nb.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 69291
Cc: Josselin Poiret <dev@HIDDEN>,
 Simon Tournier <zimon.toutoune@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>,
 Tobias Geerinckx-Rice <me@HIDDEN>, 69291 <at> debbugs.gnu.org,
 Ricardo Wurmus <rekado@HIDDEN>, Christopher Baines <guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Christopher Baines <mail@HIDDEN> skribis:

> As this moves download-nar in a direction where it could be used outside =
the
> substitute script.
>
> * guix/scripts/substitute.scm (download-nar): Return more information and=
 move
> status-port output to=E2=80=A6
> (guix-substitute): here.
>
> Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0

LGTM.

Thanks for this series!




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 23 Feb 2024 16:20:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 23 11:20:21 2024
Received: from localhost ([127.0.0.1]:51313 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rdYHc-0003gu-Se
	for submit <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:20:21 -0500
Received: from eggs.gnu.org ([209.51.188.92]:37830)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1rdYHb-0003gX-IF
 for 69291 <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:20:20 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1rdYH7-00050K-9n; Fri, 23 Feb 2024 11:19:50 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=0mSxxpshL/EU+E+eJ2McFr7Bt88ket6Qn0W+MZ6c3i0=; b=UFWMdKCzkPFSlms5+AxE
 ZWJjQE4UqjKFQ64S0ja68hW1Pmiv95GOg24R4yrR+1JhaG/j0Q1bccv7fLA8kfNG/I4436//P2Wll
 9QDWHKL2TDtmpF9O4U361Hy9xeu/2Itclv7sxnNgdY13q53MirXW19DC5Rnnce/CSZQmC7p68NzWh
 Y3vHvIJckMU7xHvU8wa3j/hiEIaZQM7XpdrUXyN3p7N+EGV7zeog0zBtsWdhpYEee/Frstg8bjUJY
 AQWxovAmG/d4BZowt2klABXtJXHAiclo4K7OBEDwlwahW97MJ4PMIG+UJ7oaK162CLvMUQtf55DLv
 ZH2VQ6P/m8u+ig==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Christopher Baines <mail@HIDDEN>
Subject: Re: [bug#69291] [PATCH 2/5] scripts: substitute: Allow not using
 with-timeout in download-nar.
In-Reply-To: <2bcc2278b5aefcd81fbc53b791d08622b6d93c7c.1708458147.git.mail@HIDDEN>
 (Christopher Baines's message of "Tue, 20 Feb 2024 19:42:24 +0000")
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 <2bcc2278b5aefcd81fbc53b791d08622b6d93c7c.1708458147.git.mail@HIDDEN>
Date: Fri, 23 Feb 2024 17:19:46 +0100
Message-ID: <8734tjgnkt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 69291
Cc: Josselin Poiret <dev@HIDDEN>,
 Simon Tournier <zimon.toutoune@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>,
 Tobias Geerinckx-Rice <me@HIDDEN>, 69291 <at> debbugs.gnu.org,
 Ricardo Wurmus <rekado@HIDDEN>, Christopher Baines <guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Christopher Baines <mail@HIDDEN> skribis:

> I don't think the approach of using SIGALARM here for the timeout will wo=
rk
> well in all cases (e.g. when using Guile Fibers), so make it possible to =
avoid
> this.
>
> * guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in a=
s an
> option.
>
> Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6

The patch LGTM.

That said, maybe we should just pass #:timeout to =E2=80=98http-fetch=E2=80=
=99?  It=E2=80=99s
not strictly equivalent because it only controls the timeout on
connection establishment, but in practice it should have the same
effect.

Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 23 Feb 2024 16:16:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Feb 23 11:16:43 2024
Received: from localhost ([127.0.0.1]:51037 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rdYE7-0003WF-3b
	for submit <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:16:43 -0500
Received: from eggs.gnu.org ([209.51.188.92]:50060)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1rdYE4-0003Vp-2I
 for 69291 <at> debbugs.gnu.org; Fri, 23 Feb 2024 11:16:40 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1rdYDY-00049a-6i; Fri, 23 Feb 2024 11:16:08 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=pLBA1tP69zPajPM/4dHYcm9n4V8fnn7AEGCpdiphxzw=; b=sOe+9Mb05mdQU1IBUZ7w
 G3UFzYbcpqi07ieX7+34lTQERgADcc5QC8QaWdWib5fDkgVx389VjD0twpAiLvv67AhsSy3tac+Fm
 t4zaTHs5rol9VTWpeVVfdwGaP7cxu1iI9DocTAguBLs+4ZQc9bw+7IMfssxa2ABJ0X5MBQVgno1XF
 sZiVnEaVv3rQNF5RIc5hVv9hFseC6+J7NJA9qirw5KKnZlF83iZ/DikCWjueDNySOS2D2YCEdbwIO
 DZPFsprHKM8ogtTawrYE4+Jdm1VSLouF+gBUS4KpoPgWnVUqiVeP403ogfj9tpgey4qmMpZgsFDmj
 8D/fkqAjRz3JhA==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Christopher Baines <mail@HIDDEN>
Subject: Re: [bug#69291] [PATCH 1/5] scripts: substitute: Remove side effect
 warning from network-error?.
In-Reply-To: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
 (Christopher Baines's message of "Tue, 20 Feb 2024 19:42:23 +0000")
References: <87le7flz89.fsf@HIDDEN>
 <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
Date: Fri, 23 Feb 2024 17:16:00 +0100
Message-ID: <877civgnr3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 69291
Cc: Josselin Poiret <dev@HIDDEN>,
 Simon Tournier <zimon.toutoune@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>,
 Tobias Geerinckx-Rice <me@HIDDEN>, 69291 <at> debbugs.gnu.org,
 Ricardo Wurmus <rekado@HIDDEN>, Christopher Baines <guix@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Christopher Baines <mail@HIDDEN> skribis:

> Instead, display the warning from process-substitution and
> process-substitution/fallback in the relevant places.
>
> I'm looking at this because I want to make the substitute code less tied =
to
> the script and usable in the Guile guix-daemon.
>
> * guix/scripts/substitute.scm (network-error?): Move warning to=E2=80=A6
> (process-substitution/fallback, process-substitution): here.
>
> Change-Id: I082b482c0e6ec7e02a8d437ba22dcefca5c40787

LGTM.




Information forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:43:02 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 20 14:43:02 2024
Received: from localhost ([127.0.0.1]:46681 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rcW17-0002Yy-Kj
	for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:43:02 -0500
Received: from mira.cbaines.net ([212.71.252.8]:43162)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rcW0y-0002Y3-Db
 for 69291 <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:53 -0500
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id 1138A27BBEA
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:30 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 23ba8778
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:29 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: 69291 <at> debbugs.gnu.org
Subject: [PATCH 5/5] scripts: substitute: Extract script specific output from
 download-nar.
Date: Tue, 20 Feb 2024 19:42:27 +0000
Message-ID: <f4af19b037826cad90bbcfe400ad864f028cc7d8.1708458147.git.mail@HIDDEN>
X-Mailer: git-send-email 2.41.0
In-Reply-To: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
X-Debbugs-Cc: Christopher Baines <guix@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-Debbugs-Envelope-To: 69291
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.9 (--)

As this moves download-nar in a direction where it could be used outside the
substitute script.

* guix/scripts/substitute.scm (download-nar): Return more information and move
status-port output to…
(guix-substitute): here.

Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
---
 guix/scripts/substitute.scm | 49 +++++++++++++++++++++----------------
 1 file changed, 28 insertions(+), 21 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 61e16b22db..94eb6d2f71 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -568,26 +568,10 @@ (define* (download-nar narinfo destination
       ;; Wait for the reporter to finish.
       (every (compose zero? cdr waitpid) pids)
 
-      ;; Skip a line after what 'progress-reporter/file' printed, and another
-      ;; one to visually separate substitutions.  When PRINT-BUILD-TRACE? is
-      ;; true, leave it up to (guix status) to prettify things.
-      (newline (current-error-port))
-      (unless print-build-trace?
-        (newline (current-error-port)))
-
-      ;; Check whether we got the data announced in NARINFO.
-      (let ((actual (get-hash)))
-        (if (bytevector=? actual expected)
-            ;; Tell the daemon that we're done.
-            (format status-port "success ~a ~a~%"
-                    (narinfo-hash narinfo) (narinfo-size narinfo))
-            ;; The actual data has a different hash than that in NARINFO.
-            (format status-port "hash-mismatch ~a ~a ~a~%"
-                    (hash-algorithm-name algorithm)
-                    (bytevector->nix-base32-string expected)
-                    (bytevector->nix-base32-string actual))))
-
-      cpu-usage)))
+      (values narinfo
+              expected
+              (get-hash)
+              cpu-usage))))
 
 (define (system-error? exception)
   "Return true if EXCEPTION is a Guile 'system-error exception."
@@ -891,7 +875,10 @@ (define-command (guix-substitute . args)
               ((? eof-object?)
                #t)
               ((= string-tokenize ("substitute" store-path destination))
-               (let ((cpu-usage
+               (let ((narinfo
+                      expected-hash
+                      actual-hash
+                      cpu-usage
                       (process-substitution reply-port store-path destination
                                             #:cache-urls (substitute-urls)
                                             #:acl (current-acl)
@@ -901,6 +888,26 @@ (define-command (guix-substitute . args)
                                             #:prefer-fast-decompression?
                                             prefer-fast-decompression?)))
 
+                 ;; Skip a line after what 'progress-reporter/file' printed,
+                 ;; and another one to visually separate substitutions.  When
+                 ;; PRINT-BUILD-TRACE? is true, leave it up to (guix status)
+                 ;; to prettify things.
+                 (newline (current-error-port))
+                 (unless print-build-trace?
+                   (newline (current-error-port)))
+
+                 ;; Check whether we got the data announced in NARINFO.
+                 (if (bytevector=? actual-hash expected-hash)
+                     ;; Tell the daemon that we're done.
+                     (format reply-port "success ~a ~a~%"
+                             (narinfo-hash narinfo) (narinfo-size narinfo))
+                     ;; The actual data has a different hash than that in NARINFO.
+                     (format reply-port "hash-mismatch ~a ~a ~a~%"
+                             (hash-algorithm-name
+                              (narinfo-hash-algorithm+value narinfo))
+                             (bytevector->nix-base32-string expected-hash)
+                             (bytevector->nix-base32-string actual-hash)))
+
                  ;; Create a hysteresis: depending on CPU usage, favor
                  ;; compression methods with faster decompression (like ztsd)
                  ;; or methods with better compression ratios (like lzip).
-- 
2.41.0





Information forwarded to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:43:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 20 14:43:01 2024
Received: from localhost ([127.0.0.1]:46679 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rcW16-0002Yl-NQ
	for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:43:01 -0500
Received: from mira.cbaines.net ([212.71.252.8]:43160)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rcW0y-0002Y2-CU
 for 69291 <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:53 -0500
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id 0AC2827BBE9
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:30 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 606a0cb8
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:29 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: 69291 <at> debbugs.gnu.org
Subject: [PATCH 4/5] scripts: substitute: Untangle selecting fast vs small
 compressions.
Date: Tue, 20 Feb 2024 19:42:26 +0000
Message-ID: <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@HIDDEN>
X-Mailer: git-send-email 2.41.0
In-Reply-To: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
X-Debbugs-Cc: Christopher Baines <guix@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-Debbugs-Envelope-To: 69291
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.9 (--)

Pulling the logic up to the script makes this code more portable and not
reliant on setting a global variable.

* guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to…
(%default-prefer-fast-decompression?): this.
(display-narinfo-data): Update accordingly.
(download-nar): Add prefer-fast-decompression? as a keyword argument, remove
code to set! it and return the cpu-usage recorded.
(process-substitution, process-substitution/fallback): Accept and pass through
prefer-fast-decompression? to download-nar.
(guix-substitute): Move the prefer fast decompression switching logic here.

Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb
---
 guix/scripts/substitute.scm | 90 +++++++++++++++++++++----------------
 1 file changed, 52 insertions(+), 38 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 1875a4332d..61e16b22db 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -261,12 +261,7 @@ (define (show-help)
 ;;; Daemon/substituter protocol.
 ;;;
 
-(define %prefer-fast-decompression?
-  ;; Whether to prefer fast decompression over good compression ratios.  This
-  ;; serves in particular to choose between lzip (high compression ratio but
-  ;; low decompression throughput) and zstd (lower compression ratio but high
-  ;; decompression throughput).
-  #f)
+(define %default-prefer-fast-decompression? #f)
 
 (define (call-with-cpu-usage-monitoring proc)
   (let ((before (times)))
@@ -297,7 +292,7 @@ (define (display-narinfo-data port narinfo)
   (let ((uri compression file-size
              (narinfo-best-uri narinfo
                                #:fast-decompression?
-                               %prefer-fast-decompression?)))
+                               %default-prefer-fast-decompression?)))
     (format port "~a\n~a\n"
             (or file-size 0)
             (or (narinfo-size narinfo) 0))))
@@ -453,7 +448,8 @@ (define-syntax-rule (catch-system-error exp)
 (define* (download-nar narinfo destination
                        #:key status-port
                        deduplicate? print-build-trace?
-                       (fetch-timeout %fetch-timeout))
+                       (fetch-timeout %fetch-timeout)
+                       prefer-fast-decompression?)
   "Download the nar prescribed in NARINFO, which is assumed to be authentic
 and authorized, and write it to DESTINATION.  When DEDUPLICATE? is true, and
 if DESTINATION is in the store, deduplicate its files.  Print a status line to
@@ -525,7 +521,7 @@ (define* (download-nar narinfo destination
 
   (let ((choices (narinfo-preferred-uris narinfo
                                          #:fast-decompression?
-                                         %prefer-fast-decompression?)))
+                                         prefer-fast-decompression?)))
     ;; 'guix publish' without '--cache' doesn't specify a Content-Length, so
     ;; DOWNLOAD-SIZE is #f in this case.
     (let* ((raw uri compression download-size (try-fetch choices))
@@ -566,21 +562,6 @@ (define* (download-nar narinfo destination
                                             deduplicate?)
                                        dump-file/deduplicate*
                                        dump-file))))
-
-      ;; Create a hysteresis: depending on CPU usage, favor compression
-      ;; methods with faster decompression (like ztsd) or methods with better
-      ;; compression ratios (like lzip).  This stems from the observation that
-      ;; substitution can be CPU-bound when high-speed networks are used:
-      ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
-      ;; To simulate "slow" networking or changing conditions, run:
-      ;;   sudo tc qdisc add dev eno1 root tbf rate 512kbit latency 50ms burst 1540
-      ;; and then cancel with:
-      ;;   sudo tc qdisc del dev eno1 root
-      (when (> cpu-usage .8)
-        (set! %prefer-fast-decompression? #t))
-      (when (< cpu-usage .2)
-        (set! %prefer-fast-decompression? #f))
-
       (close-port hashed)
       (close-port input)
 
@@ -604,7 +585,9 @@ (define* (download-nar narinfo destination
             (format status-port "hash-mismatch ~a ~a ~a~%"
                     (hash-algorithm-name algorithm)
                     (bytevector->nix-base32-string expected)
-                    (bytevector->nix-base32-string actual)))))))
+                    (bytevector->nix-base32-string actual))))
+
+      cpu-usage)))
 
 (define (system-error? exception)
   "Return true if EXCEPTION is a Guile 'system-error exception."
@@ -628,7 +611,8 @@ (define network-error?
 
 (define* (process-substitution/fallback port narinfo destination
                                         #:key cache-urls acl
-                                        deduplicate? print-build-trace?)
+                                        deduplicate? print-build-trace?
+                                        prefer-fast-decompression?)
   "Attempt to substitute NARINFO, which is assumed to be authorized or
 equivalent, by trying to download its nar from each entry in CACHE-URLS.
 
@@ -662,14 +646,17 @@ (define* (process-substitution/fallback port narinfo destination
                 (download-nar alternate destination
                               #:status-port port
                               #:deduplicate? deduplicate?
-                              #:print-build-trace? print-build-trace?))
+                              #:print-build-trace? print-build-trace?
+                              #:prefer-fast-decompression?
+                              prefer-fast-decompression?))
               (loop rest)))
          (()
           (loop rest)))))))
 
 (define* (process-substitution port store-item destination
                                #:key cache-urls acl
-                               deduplicate? print-build-trace?)
+                               deduplicate? print-build-trace?
+                               prefer-fast-decompression?)
   "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to
 DESTINATION as a nar file.  Verify the substitute against ACL, and verify its
 hash against what appears in the narinfo.  When DEDUPLICATE? is true, and if
@@ -701,11 +688,14 @@ (define* (process-substitution port store-item destination
                                             #:acl acl
                                             #:deduplicate? deduplicate?
                                             #:print-build-trace?
-                                            print-build-trace?)))
+                                            print-build-trace?
+                                            #:prefer-fast-decompression?
+                                            prefer-fast-decompression?)))
     (download-nar narinfo destination
                   #:status-port port
                   #:deduplicate? deduplicate?
-                  #:print-build-trace? print-build-trace?)))
+                  #:print-build-trace? print-build-trace?
+                  #:prefer-fast-decompression? prefer-fast-decompression?)))
 
 
 ;;;
@@ -895,18 +885,42 @@ (define-command (guix-substitute . args)
         ;; Specify the number of columns of the terminal so the progress
         ;; report displays nicely.
         (parameterize ((current-terminal-columns (client-terminal-columns)))
-          (let loop ()
+          (let loop ((prefer-fast-decompression?
+                      %default-prefer-fast-decompression?))
             (match (read-line)
               ((? eof-object?)
                #t)
               ((= string-tokenize ("substitute" store-path destination))
-               (process-substitution reply-port store-path destination
-                                     #:cache-urls (substitute-urls)
-                                     #:acl (current-acl)
-                                     #:deduplicate? deduplicate?
-                                     #:print-build-trace?
-                                     print-build-trace?)
-               (loop))))))
+               (let ((cpu-usage
+                      (process-substitution reply-port store-path destination
+                                            #:cache-urls (substitute-urls)
+                                            #:acl (current-acl)
+                                            #:deduplicate? deduplicate?
+                                            #:print-build-trace?
+                                            print-build-trace?
+                                            #:prefer-fast-decompression?
+                                            prefer-fast-decompression?)))
+
+                 ;; Create a hysteresis: depending on CPU usage, favor
+                 ;; compression methods with faster decompression (like ztsd)
+                 ;; or methods with better compression ratios (like lzip).
+                 ;; This stems from the observation that substitution can be
+                 ;; CPU-bound when high-speed networks are used:
+                 ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
+                 ;; To simulate "slow" networking or changing conditions, run:
+                 ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency
+                 ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del
+                 ;; dev eno1 root
+                 (loop (cond
+                        ;; Whether to prefer fast decompression over good
+                        ;; compression ratios.  This serves in particular to
+                        ;; choose between lzip (high compression ratio but low
+                        ;; decompression throughput) and zstd (lower
+                        ;; compression ratio but high decompression
+                        ;; throughput).
+                        ((> cpu-usage .8) #t)
+                        ((< cpu-usage .2) #f)
+                        (else prefer-fast-decompression?)))))))))
        (opts
         (leave (G_ "~a: unrecognized options~%") opts))))))
 
-- 
2.41.0





Information forwarded to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:43:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 20 14:43:00 2024
Received: from localhost ([127.0.0.1]:46677 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rcW16-0002Yi-Ac
	for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:43:00 -0500
Received: from mira.cbaines.net ([212.71.252.8]:43158)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rcW0y-0002Y0-BZ
 for 69291 <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:52 -0500
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id 0407A27BBE2
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:30 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 17dacf27
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:29 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: 69291 <at> debbugs.gnu.org
Subject: [PATCH 3/5] scripts: substitute: Replace some leave calls with raise.
Date: Tue, 20 Feb 2024 19:42:25 +0000
Message-ID: <3a88dc8a8369278f86f3a51a99aa62d991ba997d.1708458147.git.mail@HIDDEN>
X-Mailer: git-send-email 2.41.0
In-Reply-To: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
MIME-Version: 1.0
X-Debbugs-Cc: Christopher Baines <guix@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-Debbugs-Envelope-To: 69291
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.9 (--)

These calls happen inside of with-error-handling, so the effect should be the
same, but this opens up the possibility of using this code in a program that
doesn't want to exit when one of these error conditions is met.

Change-Id: I15d963615d85d419559fa0f4333fa4dc1dfbfd3b

* guix/scripts/substitute.scm (download-nar, process-substitution): Use raise
formatted-message rather than leave.

Change-Id: Idd0880206b69e3903e19e0536b87d65a52c200d5
---
 guix/scripts/substitute.scm | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 575fa2a0b3..1875a4332d 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -495,8 +495,10 @@ (define* (download-nar narinfo destination
                          #:keep-alive? #t
                          #:buffered? #f))))
       (else
-       (leave (G_ "unsupported substitute URI scheme: ~a~%")
-              (uri->string uri)))))
+       (raise
+        (formatted-message
+         (G_ "unsupported substitute URI scheme: ~a~%")
+         (uri->string uri))))))
 
   (define (try-fetch choices)
     (match choices
@@ -511,9 +513,11 @@ (define* (download-nar narinfo destination
                      (G_ "Downloading ~a...~%") (uri->string uri)))
            (values port uri compression download-size))))
       (()
-       (leave (G_ "no valid nar URLs for ~a at ~a~%")
-              (narinfo-path narinfo)
-              (narinfo-uri-base narinfo)))))
+       (raise
+        (formatted-message
+         (G_ "no valid nar URLs for ~a at ~a~%")
+         (narinfo-path narinfo)
+         (narinfo-uri-base narinfo))))))
 
   ;; Delete DESTINATION first--necessary when starting over after a failed
   ;; download.
@@ -678,8 +682,10 @@ (define* (process-substitution port store-item destination
                         (cut valid-narinfo? <> acl))))
 
   (unless narinfo
-    (leave (G_ "no valid substitute for '~a'~%")
-           store-item))
+    (raise
+     (formatted-message
+      (G_ "no valid substitute for '~a'~%")
+      store-item)))
 
   (guard (c ((network-error? c)
              (when (http-get-error? c)
-- 
2.41.0





Information forwarded to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:42:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 20 14:42:56 2024
Received: from localhost ([127.0.0.1]:46675 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rcW11-0002YQ-Tc
	for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:56 -0500
Received: from mira.cbaines.net ([212.71.252.8]:43154)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rcW0x-0002Xo-AB
 for 69291 <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:51 -0500
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id EBE3527BBE2
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:28 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 3127bf43
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:28 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: 69291 <at> debbugs.gnu.org
Subject: [PATCH 1/5] scripts: substitute: Remove side effect warning from
 network-error?.
Date: Tue, 20 Feb 2024 19:42:23 +0000
Message-ID: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
X-Mailer: git-send-email 2.41.0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
X-Debbugs-Cc: Christopher Baines <guix@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-Debbugs-Envelope-To: 69291
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.9 (--)

Instead, display the warning from process-substitution and
process-substitution/fallback in the relevant places.

I'm looking at this because I want to make the substitute code less tied to
the script and usable in the Guile guix-daemon.

* guix/scripts/substitute.scm (network-error?): Move warning to…
(process-substitution/fallback, process-substitution): here.

Change-Id: I082b482c0e6ec7e02a8d437ba22dcefca5c40787
---
 guix/scripts/substitute.scm | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 37cd08e289..f3eed0eb44 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -613,13 +613,7 @@ (define network-error?
           (and (kind-and-args? exception)
                (memq (exception-kind exception)
                      '(gnutls-error getaddrinfo-error)))
-          (and (http-get-error? exception)
-               (begin
-                 (warning (G_ "download from '~a' failed: ~a, ~s~%")
-                          (uri->string (http-get-error-uri exception))
-                          (http-get-error-code exception)
-                          (http-get-error-reason exception))
-                 #t))))))
+          (http-get-error? exception)))))
 
 (define* (process-substitution/fallback port narinfo destination
                                         #:key cache-urls acl
@@ -647,7 +641,13 @@ (define* (process-substitution/fallback port narinfo destination
           (if (or (equivalent-narinfo? narinfo alternate)
                   (valid-narinfo? alternate acl)
                   (%allow-unauthenticated-substitutes?))
-              (guard (c ((network-error? c) (loop rest)))
+              (guard (c ((network-error? c)
+                         (when (http-get-error? c)
+                           (warning (G_ "download from '~a' failed: ~a, ~s~%")
+                                    (uri->string (http-get-error-uri c))
+                                    (http-get-error-code c)
+                                    (http-get-error-reason c)))
+                         (loop rest)))
                 (download-nar alternate destination
                               #:status-port port
                               #:deduplicate? deduplicate?
@@ -675,6 +675,11 @@ (define* (process-substitution port store-item destination
            store-item))
 
   (guard (c ((network-error? c)
+             (when (http-get-error? c)
+               (warning (G_ "download from '~a' failed: ~a, ~s~%")
+                        (uri->string (http-get-error-uri c))
+                        (http-get-error-code c)
+                        (http-get-error-reason c)))
              (format (current-error-port)
                      (G_ "retrying download of '~a' with other substitute URLs...~%")
                      store-item)

base-commit: 3d061d9677027be7651f8e5a3a02e19daacd9a85
-- 
2.41.0





Information forwarded to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at 69291 <at> debbugs.gnu.org:


Received: (at 69291) by debbugs.gnu.org; 20 Feb 2024 19:42:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 20 14:42:55 2024
Received: from localhost ([127.0.0.1]:46673 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rcW0z-0002YA-7l
	for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:55 -0500
Received: from mira.cbaines.net ([212.71.252.8]:43156)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rcW0x-0002Xp-As
 for 69291 <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:42:51 -0500
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id F204C27BBE9
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:28 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 6addb520
 for <69291 <at> debbugs.gnu.org>; Tue, 20 Feb 2024 19:42:28 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
To: 69291 <at> debbugs.gnu.org
Subject: [PATCH 2/5] scripts: substitute: Allow not using with-timeout in
 download-nar.
Date: Tue, 20 Feb 2024 19:42:24 +0000
Message-ID: <2bcc2278b5aefcd81fbc53b791d08622b6d93c7c.1708458147.git.mail@HIDDEN>
X-Mailer: git-send-email 2.41.0
In-Reply-To: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
References: <c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@HIDDEN>
MIME-Version: 1.0
X-Debbugs-Cc: Christopher Baines <guix@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-Debbugs-Envelope-To: 69291
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.9 (--)

I don't think the approach of using SIGALARM here for the timeout will work
well in all cases (e.g. when using Guile Fibers), so make it possible to avoid
this.

* guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in as an
option.

Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6
---
 guix/scripts/substitute.scm | 37 ++++++++++++++++++++++---------------
 1 file changed, 22 insertions(+), 15 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index f3eed0eb44..575fa2a0b3 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -452,7 +452,8 @@ (define-syntax-rule (catch-system-error exp)
 
 (define* (download-nar narinfo destination
                        #:key status-port
-                       deduplicate? print-build-trace?)
+                       deduplicate? print-build-trace?
+                       (fetch-timeout %fetch-timeout))
   "Download the nar prescribed in NARINFO, which is assumed to be authentic
 and authorized, and write it to DESTINATION.  When DEDUPLICATE? is true, and
 if DESTINATION is in the store, deduplicate its files.  Print a status line to
@@ -473,20 +474,26 @@ (define* (download-nar narinfo destination
        (let ((port (open-file (uri-path uri) "r0b")))
          (values port (stat:size (stat port)))))
       ((http https)
-       ;; Test this with:
-       ;;   sudo tc qdisc add dev eth0 root netem delay 1500ms
-       ;; and then cancel with:
-       ;;   sudo tc qdisc del dev eth0 root
-       (with-timeout %fetch-timeout
-         (begin
-           (warning (G_ "while fetching ~a: server is somewhat slow~%")
-                    (uri->string uri))
-           (warning (G_ "try `--no-substitutes' if the problem persists~%")))
-         (with-cached-connection uri port
-           (http-fetch uri #:text? #f
-                       #:port port
-                       #:keep-alive? #t
-                       #:buffered? #f))))
+       (if fetch-timeout
+           ;; Test this with:
+           ;;   sudo tc qdisc add dev eth0 root netem delay 1500ms
+           ;; and then cancel with:
+           ;;   sudo tc qdisc del dev eth0 root
+           (with-timeout %fetch-timeout
+             (begin
+               (warning (G_ "while fetching ~a: server is somewhat slow~%")
+                        (uri->string uri))
+               (warning (G_ "try `--no-substitutes' if the problem persists~%")))
+             (with-cached-connection uri port
+               (http-fetch uri #:text? #f
+                           #:port port
+                           #:keep-alive? #t
+                           #:buffered? #f)))
+           (with-cached-connection uri port
+             (http-fetch uri #:text? #f
+                         #:port port
+                         #:keep-alive? #t
+                         #:buffered? #f))))
       (else
        (leave (G_ "unsupported substitute URI scheme: ~a~%")
               (uri->string uri)))))
-- 
2.41.0





Information forwarded to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 20 Feb 2024 19:20:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Feb 20 14:20:40 2024
Received: from localhost ([127.0.0.1]:46575 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rcVfU-0001x6-Gl
	for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:20:40 -0500
Received: from lists.gnu.org ([209.51.188.17]:40012)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1rcVfR-0001ww-Ct
 for submit <at> debbugs.gnu.org; Tue, 20 Feb 2024 14:20:39 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mail@HIDDEN>) id 1rcVf4-0005R5-V8
 for guix-patches@HIDDEN; Tue, 20 Feb 2024 14:20:14 -0500
Received: from mira.cbaines.net ([212.71.252.8])
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <mail@HIDDEN>) id 1rcVf3-0005Hr-3t
 for guix-patches@HIDDEN; Tue, 20 Feb 2024 14:20:14 -0500
Received: from localhost (unknown [212.132.255.10])
 by mira.cbaines.net (Postfix) with ESMTPSA id 8F01C27BBE2
 for <guix-patches@HIDDEN>; Tue, 20 Feb 2024 19:20:09 +0000 (GMT)
Received: from felis (localhost.lan [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id ff12a901
 for <guix-patches@HIDDEN>; Tue, 20 Feb 2024 19:20:08 +0000 (UTC)
User-agent: mu4e 1.10.8; emacs 29.1
From: Christopher Baines <mail@HIDDEN>
To: guix-patches@HIDDEN
Subject: [PATCH 0/5] Start making substitute code less coupled
Date: Tue, 20 Feb 2024 19:05:07 +0000
Message-ID: <87le7flz89.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@HIDDEN;
 helo=mira.cbaines.net
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -4.2 (----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.2 (-----)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

These changes should help with using the substitute code in a Guile
implementation of the Guix daemon.


Christopher Baines (5):
  scripts: substitute: Remove side effect warning from network-error?.
  scripts: substitute: Allow not using with-timeout in download-nar.
  scripts: substitute: Replace some leave calls with raise.
  scripts: substitute: Untangle selecting fast vs small compressions.
  scripts: substitute: Extract script specific output from download-nar.

 guix/scripts/substitute.scm | 207 +++++++++++++++++++++---------------
 1 file changed, 123 insertions(+), 84 deletions(-)


base-commit: 3d061d9677027be7651f8e5a3a02e19daacd9a85
=2D-=20
2.41.0

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmXU+2ZfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xfoag//Uoq9twKRF3tAV9Iuv4PmvPQwLwLXUR++
pA1dqG1dQnu3KhMmi0QBdA2MZUIfTgbtuBvqQwPHdXtV7sQZW7smREY85Q/EUThb
qJs3GJGvVEnXjGe2N00w0zol1VMIz1itXcFb+JFLQNSaBnPYDMzgdn04RhO+WpqT
lLT4ccwKEzFCh3TAroUHx9m3lgZ4A1tg6MkH9kKlMEw+UQl6cgMZCNSgLqrJYRTx
kwIEMpcVbJvmSePx+rU0SHwIUS5JBLaAKMRMb53fr6iNstW4WrH9ophrHRJf2N9B
GTWmaKnj1uzYUQOSF2D0Vq7ZFEBeKNfUkwF0vyspWFBjTdZxkshHHRL2B/wDCLAz
WIwhKe4ydkN4hpDDz67o8hIAeDce3K3d9LjXN2GYdFeEFYQWq36cyGG1I5o8sEZv
yEH6WihzLvjC1Mlso4KhiB4ZQubqkCfRWRIOdxVEhw211FOFubtTGF1JHN7ZNKG2
LZnhuJa6vfnEgdmvG46Gx5O2udqgG1IyN0xSsXEl033B1ZnrNTTy/Wos1KuGjBFz
uQg+xoTMKgokm9nqMoXdFmRViR3WILiNMhuY8eG2z8fqQpxBM+q3iMZNVuWsI2Nq
QqiW7/jHcWKEiHj/TM3RHUVKLv4hOSXbzxJVyD9skZETG4yYNqpSOcqoNU/LInXO
qWtukYWSwIc=
=xIzK
-----END PGP SIGNATURE-----
--=-=-=--




Acknowledgement sent to Christopher Baines <mail@HIDDEN>:
New bug report received and forwarded. Copy sent to guix-patches@HIDDEN. Full text available.
Report forwarded to guix-patches@HIDDEN:
bug#69291; Package guix-patches. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 3 Apr 2024 17:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.