GNU bug report logs - #51216
[PATCH 0/1] swh: Allows token from Software Heritage authentication service.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 51216 in the body.
You can then email your comments to 51216 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: [PATCH 0/1] swh: Allows token from Software Heritage authentication
 service.
Date: Thu, 14 Oct 2021 23:32:38 +0200

Hi,

This patch allows to use a token from SWH and change the rate limit when
submitting via the archival checker.  For instance, it reads,

   GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival

and will use the ${TOKEN} associated to the SWH account.  Having an account
enable by default a rate limit to 1200 (instead of 120).  Then, for testing
purpose I get a lot of android packages scheduled after checking many admin.

It is interesting to note lines as:

  Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'

which shows that sources.json is maybe not fully working as expected.
Investigations required.

--8<---------------cut here---------------start------------->8---
sitour <at> jack-Precision-7820-Tower$  GUIX_SWH_TOKEN=$TOKEN  ./pre-inst-env guix lint -c archival
gnu/packages/accessibility.scm:100:5: brltty <at> 6.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:4330:5: pam-mount <at> 2.18: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:3902:12: launchmon <at> 1.0.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:933:12: net-base <at> 5.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2272:12: testdisk <at> 7.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2148:12: acpica <at> 20210930:
Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
gnu/packages/admin.scm:3029:12: nmap <at> 7.92: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:725:5: pies <at> 1.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2116:5: dmidecode <at> 3.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:1108:5: prips <at> 1.1.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2939:12: munge <at> 0.5.14: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:534:12: facter <at> 4.0.52: scheduled Software Heritage archival
gnu/packages/admin.scm:1137:12: alive <at> 2.0.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2677:5: debops <at> 1.1.0: scheduled Software Heritage archival
checking ansible <at> 4.4.0 [archival]...
gnu/packages/admin.scm:3571:12: neofetch <at> 7.1.0: scheduled Software Heritage archival
gnu/packages/aidc.scm:112:12: qrencode <at> 4.1.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/aidc.scm:174:5: zbar <at> 0.23: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:489:12: arb <at> 2.21.0: scheduled Software Heritage archival
gnu/packages/algebra.scm:1053:14: eigen-for-tensorflow <at> 3.3.5-1.fd6845384b86: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:1104:12: xtensor <at> 0.20.10: scheduled Software Heritage archival
gnu/packages/algebra.scm:1682:11: sollya <at> 7.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:1139:5: gap <at> 4.11.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:957:5: symengine <at> 0.6.0: scheduled Software Heritage archival
gnu/packages/algebra.scm:1605:5: iml <at> 1.0.5: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/android.scm:529:12: android-ext4-utils <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:659:12: fastboot <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:744:5: git-repo <at> 2.4.1: scheduled Software Heritage archival
gnu/packages/android.scm:323:12: android-libziparchive <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:581:12: android-f2fs-utils <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:405:12: mkbootimg <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:358:12: adb <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:302:12: android-libsparse <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:248:12: android-libcutils <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:458:12: android-bionic-uapi <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:619:12: android-libutils <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:101:5: android-googletest <at> 1.8.0: scheduled Software Heritage archival
gnu/packages/android.scm:195:12: android-liblog <at> 7.1.2_r36: scheduled Software Heritage archival
gnu/packages/animation.scm:67:5: rlottie <at> 0.2: scheduled Software Heritage archival
gnu/packages/antivirus.scm:48:12: clamav <at> 0.103.3:
Disarchive entry refers to non-existent SWH directory 'b25414c9864a270899ca1ff494e7ba4c437b166d'
gnu/packages/aspell.scm:115:12: aspell-dict-en <at> 2020.12.07-0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:898:5: indi <at> 1.9.1: scheduled Software Heritage archival
gnu/packages/astronomy.scm:239:5: wcslib <at> 7.5: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:298:5: gnuastro <at> 0.16: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:823:5: gpredict <at> 2.2.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:518:5: stuff <at> 1.26.0:
Disarchive entry refers to non-existent SWH directory '128bbe76a82dd0b38b725565ed703a7148257ae0'
gnu/packages/astronomy.scm:216:5: eye <at> 1.4.1:
Disarchive entry refers to non-existent SWH directory '92625e2c6dbe3ad7c4f44a061ada24ce00637087'
gnu/packages/astronomy.scm:377:5: skymaker <at> 3.10.5:
Disarchive entry refers to non-existent SWH directory '6000a273dfff9de62725b53e41562fff711069c1'
gnu/packages/astronomy.scm:278:5: weightwatcher <at> 1.12:
Disarchive entry refers to non-existent SWH directory 'c68ff8714c6fd360a38158f3d8f22e555c061452'
gnu/packages/audio.scm:2941:12: supercollider <at> 3.11.2: scheduled Software Heritage archival
gnu/packages/audio.scm:3131:12: rubberband <at> 1.8.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:5431:6: mda-lv2 <at> 1.2.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3906:12: zita-alsa-pcmi <at> 0.3.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2581:12: lilv <at> 0.24.10: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:309:12: libgme <at> 0.6.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4813:5: redkite <at> 1.3.0: scheduled Software Heritage archival
gnu/packages/audio.scm:1908:12: guitarix <at> 0.41.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4657:5: gnaural <at> 20110606: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:1962:2: guitarix-lv2 <at> 0.41.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:947:12: azr3 <at> 1.2.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4266:12: snd <at> 20.9:
Disarchive entry refers to non-existent SWH directory 'cb52aaa9500df2b674bf7922811deeea1b766139'
gnu/packages/audio.scm:2811:12: patchage <at> 1.0.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:5130:6: lsp-dsp-lib <at> 0.5.11:
Disarchive entry refers to non-existent SWH directory '3e574043a04d77dd7231d23210547c4fe065a40c'
gnu/packages/audio.scm:4114:5: gsm <at> 1.0.19:
Disarchive entry refers to non-existent SWH directory 'aa763150704fe06f34097b38e839409cee52366d'
gnu/packages/audio.scm:4956:6: libaudec <at> 0.2.4: scheduled Software Heritage archival
gnu/packages/audio.scm:3699:5: qsynth <at> 0.5.7:
Disarchive entry refers to non-existent SWH directory '127c0a03c7ccba74870aef7dac36019af35798cc'
gnu/packages/audio.scm:3425:5: wavpack <at> 5.4.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3286:12: timidity++@2.15.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2176:12: jalv <at> 1.6.4: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3802:12: zita-convolver <at> 4.0.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2541:6: audio-to-midi <at> 2020.7: scheduled Software Heritage archival
gnu/packages/audio.scm:563:12: aubio <at> 0.4.9: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3199:12: sratom <at> 0.6.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3226:12: suil <at> 0.10.8: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3853:12: zita-resampler <at> 1.6.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2627:12: lv2 <at> 1.18.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3170:12: rtmidi <at> 4.0.0:
Disarchive entry refers to non-existent SWH directory 'd9745f29da983c6ad674871e68ac96362c4f11cc'
gnu/packages/audio.scm:528:12: alsa-modular-synth <at> 2.2.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/augeas.scm:37:12: augeas <at> 1.12.0:
Disarchive entry refers to non-existent SWH directory '7d7ed9f88ee649a90493f54d3988a062c3ddeafb'
gnu/packages/autogen.scm:38:5: autogen <at> 5.18.16: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/autotools.scm:291:5: autoconf-archive <at> 2021.02.19: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/backup.scm:765:12: wimlib <at> 1.13.4: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/backup.scm:192:5: hdup <at> 2.0.14: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/base.scm:587:11: binutils <at> 2.33.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/batik.scm:204:6: java-xmlgraphics-commons <at> 2.6:
Disarchive entry refers to non-existent SWH directory 'f5bd0fe7450175196c57d6f6d5aca8905393e814'
gnu/packages/benchmark.scm:57:12: fio <at> 3.27: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/bioconductor.scm:779:12: r-txdb-hsapiens-ucsc-hg38-knowngene <at> 3.4.6:
Disarchive entry refers to non-existent SWH directory '92bd3b93caa9a4b0840c70ddb96ac75b0684d7ec'
gnu/packages/bioconductor.scm:10755:5: r-timeseriesexperiment <at> 1.10.1: Software Heritage rate limit reached; try again later
--8<---------------cut here---------------end--------------->8---

All the best,
simon


zimoun (1):
  swh: Allows token from Software Heritage authentication service.

 guix/swh.scm | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)


base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58
-- 
2.32.0

Message #8 received at 51216 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 51216 <at> debbugs.gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: [PATCH 1/1] swh: Allows token from Software Heritage authentication
 service.
Date: Thu, 14 Oct 2021 23:33:46 +0200

The token is provided using the environment variable GUIX_SWH_TOKEN.

* guix/swh.scm (swh-token): New variable.
(http-get*, http-post*): Use it.
---
 guix/swh.scm | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/guix/swh.scm b/guix/swh.scm
index 5c41685a24..38a4af723a 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
 ;;; Copyright © 2021 Xinglu Chen <public <at> yoctocell.xyz>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -153,12 +154,30 @@ (define url
       url
       (string-append url "/")))
 
+;; Token from an account to the Software Heritage Authentication service
+;; <https://archive.softwareheritage.org/api/>
+(define swh-token
+  (match (getenv "GUIX_SWH_TOKEN")
+    (#f #f)
+    ((? string-null? s) #f)
+    ((? string? s) (string->symbol s))))
+
 ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would
 ;; be ignored (<https://bugs.gnu.org/40486>).
 (define* (http-get* uri #:rest rest)
-  (apply http-request uri #:method 'GET rest))
+  (apply http-request uri #:method 'GET
+         #:headers
+         (if swh-token
+             `((authorization . (Bearer ,swh-token)))
+             '())
+         rest))
 (define* (http-post* uri #:rest rest)
-  (apply http-request uri #:method 'POST rest))
+  (apply http-request uri #:method 'POST
+         #:headers
+         (if swh-token
+             `((authorization . (Bearer ,swh-token)))
+             '())
+         rest))
 
 (define %date-regexp
   ;; Match strings like "2014-11-17T22:09:38+01:00" or
-- 
2.32.0

Message #11 received at 51216 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 51216 <at> debbugs.gnu.org
Subject: Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage
 authentication service.
Date: Thu, 14 Oct 2021 23:44:04 +0200

Hi,

zimoun <zimon.toutoune <at> gmail.com> skribis:

> This patch allows to use a token from SWH and change the rate limit when
> submitting via the archival checker.  For instance, it reads,
>
>    GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival
>
> and will use the ${TOKEN} associated to the SWH account.  Having an account
> enable by default a rate limit to 1200 (instead of 120).  Then, for testing

Ooh, neat!

> purpose I get a lot of android packages scheduled after checking many admin.

You mean the SWH admins manually scheduled them?

> It is interesting to note lines as:
>
>   Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
>
> which shows that sources.json is maybe not fully working as expected.
> Investigations required.

Weird.  I guess we should pick a few specific cases, make sure they’re
listed in sources.json and they’re not 404, and then ask the SWH team
whether they can check their logs to understand what happened.

Thanks!

Ludo’.

Message #14 received at 51216 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 51216 <at> debbugs.gnu.org
Subject: Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage
 authentication service.
Date: Thu, 14 Oct 2021 23:46:46 +0200

zimoun <zimon.toutoune <at> gmail.com> skribis:

> The token is provided using the environment variable GUIX_SWH_TOKEN.
>
> * guix/swh.scm (swh-token): New variable.
> (http-get*, http-post*): Use it.

[...]

> +;; Token from an account to the Software Heritage Authentication service
> +;; <https://archive.softwareheritage.org/api/>
> +(define swh-token
> +  (match (getenv "GUIX_SWH_TOKEN")
> +    (#f #f)
> +    ((? string-null? s) #f)
> +    ((? string? s) (string->symbol s))))

Could you turn it into a SRFI-39 parameter?

>  ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would
>  ;; be ignored (<https://bugs.gnu.org/40486>).
>  (define* (http-get* uri #:rest rest)
> -  (apply http-request uri #:method 'GET rest))
> +  (apply http-request uri #:method 'GET
> +         #:headers
> +         (if swh-token
> +             `((authorization . (Bearer ,swh-token)))
> +             '())
> +         rest))
>  (define* (http-post* uri #:rest rest)
> -  (apply http-request uri #:method 'POST rest))
> +  (apply http-request uri #:method 'POST
> +         #:headers
> +         (if swh-token
> +             `((authorization . (Bearer ,swh-token)))
> +             '())
> +         rest))

These two procedures are meant to go away as soon as Guile > 3.0.2 is
required.

Could you instead pass #:headers in the ‘call’ procedure, and maybe in
‘vault-fetch’ too?

Thanks!

Ludo’.

Message #17 received at 51216 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 51216 <at> debbugs.gnu.org
Subject: Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage
 authentication service.
Date: Thu, 14 Oct 2021 23:52:30 +0200

Re,

On Thu, 14 Oct 2021 at 23:44, Ludovic Courtès <ludo <at> gnu.org> wrote:

>> purpose I get a lot of android packages scheduled after checking many admin.
>
> You mean the SWH admins manually scheduled them?

No no, sorry my bad wording.  I mean “guix lint -c archival” checked all
the 260+ packages in gnu/packages/admin.scm then it checked
gnu/packages/android.scm, found some unarchived and scheduled them.


>> It is interesting to note lines as:
>>
>>   Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
>>
>> which shows that sources.json is maybe not fully working as expected.
>> Investigations required.
>
> Weird.  I guess we should pick a few specific cases, make sure they’re
> listed in sources.json and they’re not 404, and then ask the SWH team
> whether they can check their logs to understand what happened.

Yeah… investigations required. :-)


Cheers,
simon

Message #20 received at 51216 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 51216 <at> debbugs.gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: [PATCH v2] swh: Allows token from Software Heritage authentication
 service.
Date: Fri, 15 Oct 2021 00:14:54 +0200

The token is provided using the environment variable GUIX_SWH_TOKEN.

* guix/swh.scm (%swh-token): New variable.
(call): Use it.
---
 guix/swh.scm | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/guix/swh.scm b/guix/swh.scm
index 5c41685a24..c7c1c873a2 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
 ;;; Copyright © 2021 Xinglu Chen <public <at> yoctocell.xyz>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -136,6 +137,12 @@ (define %verify-swh-certificate?
   ;; Whether to verify the X.509 HTTPS certificate for %SWH-BASE-URL.
   (make-parameter #t))
 
+;; Token from an account to the Software Heritage Authentication service
+;; <https://archive.softwareheritage.org/api/>
+(define %swh-token
+  (make-parameter (and=> (getenv "GUIX_SWH_TOKEN")
+                         string->symbol)))
+
 (define (swh-url path . rest)
   ;; URLs returned by the API may be relative or absolute. This has changed
   ;; without notice before. Handle both cases by detecting whether the path
@@ -246,6 +253,10 @@ (define* (call url decode #:optional (method http-get*)
   (and ((%allow-request?) url method)
        (let*-values (((response port)
                       (method url #:streaming? #t
+                              #:headers
+                              (if (%swh-token)
+                                  `((authorization . (Bearer ,(%swh-token))))
+                                  '())
                               #:verify-certificate?
                               (%verify-swh-certificate?))))
          ;; See <https://archive.softwareheritage.org/api/#rate-limiting>.

base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58
-- 
2.32.0

Message #25 received at 51216-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 51216-done <at> debbugs.gnu.org
Subject: Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage
 authentication service.
Date: Fri, 15 Oct 2021 22:54:39 +0200

zimoun <zimon.toutoune <at> gmail.com> skribis:

> The token is provided using the environment variable GUIX_SWH_TOKEN.
>
> * guix/swh.scm (%swh-token): New variable.
> (call): Use it.

Applied, thanks!  :-)

Ludo'.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.