GNU bug report logs - #35010
Many CPAN download URLs are no longer available

Previous Next

Package: guix;

Reported by: Mark H Weaver <mhw <at> netris.org>

Date: Tue, 26 Mar 2019 21:21:02 UTC

Severity: normal

Done: Mark H Weaver <mhw <at> netris.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35010 in the body.
You can then email your comments to 35010 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Tue, 26 Mar 2019 21:21:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Mark H Weaver <mhw <at> netris.org>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Tue, 26 Mar 2019 21:21:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: bug-guix <at> gnu.org
Subject: Many CPAN download URLs are no longer available
Date: Tue, 26 Mar 2019 17:18:00 -0400
Many perl packages cannot be built on hydra.gnu.org because the source
code is no longer available at the source URLs known to Guix.  Moreover,
apparently they were gone by the time Hydra made its first build
attempt.

Here are some examples:

  mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz
  mirror://cpan/authors/id/S/SY/SYOHEX/Mouse-v2.5.6.tar.gz
  mirror://cpan/authors/id/S/SZ/SZABGAB/Class-Date-1.1.17.tar.gz
  mirror://cpan/authors/id/S/SZ/SZABGAB/SVG-2.84.tar.gz
  mirror://cpan/authors/id/A/AB/ABW/Template-Toolkit-2.28.tar.gz
  mirror://cpan/authors/id/D/DA/DAGOLDEN/Log-Any-1.707.tar.gz
  mirror://cpan/authors/id/K/KE/KENTNL/Carp-Clan-6.07.tar.gz
  mirror://cpan/authors/id/R/RJ/RJBS/Carp-1.50.tar.gz
  mirror://cpan/authors/id/D/DA/DAGOLDEN/File-Temp-0.2309.tar.gz
  mirror://cpan/authors/id/M/MA/MAKAMAKA/Text-CSV-1.99.tar.gz

and these are the corresponding download derivations:

  /gnu/store/c73chq3k1z1n2wnbh0sbi2vwip6dfzbk-URI-1.76.tar.gz.drv
  /gnu/store/550r5gdrvs5370rsa2zwp1bs0xwj5dnm-Mouse-v2.5.6.tar.gz.drv
  /gnu/store/qxkgrj9gz1kp9id2d7d3mzrfn5b56019-Class-Date-1.1.17.tar.gz.drv
  /gnu/store/7239snjzan94plk58hw2zgachq71rh64-SVG-2.84.tar.gz.drv
  /gnu/store/qn1w1wvy5x2wngq4845j60i3qvkwhwvi-Template-Toolkit-2.28.tar.gz.drv
  /gnu/store/w1cb4r6c0v37xcjav6z16fzbc2zp47ff-Log-Any-1.707.tar.gz.drv
  /gnu/store/7gdg0aqsc3i7iwnrajcv6skkjbnf6z4r-Carp-Clan-6.07.tar.gz.drv
  /gnu/store/2b088r47jay82d91w62hr35lszqbdrkk-Carp-1.50.tar.gz.drv
  /gnu/store/jsddlzd5aba8vnjz1adggcxmvkgv5qf6-File-Temp-0.2309.tar.gz.drv
  /gnu/store/y2yb19kf1pfblhqwxcll7rrh6pgpr9sh-Text-CSV-1.99.tar.gz.drv

At least some, and probably most, of these URIs were updated quite
recently.  For example, the 'perl-mouse', 'perl-carp-clan', and
'perl-file-temp' were all updated on March 23, and presumably the source
URIs worked at that point, but then all three URIs had to be updated two
days later to fix the broken download links.

The CPAN download URLs that we are using are apparently not long-lived.
Does anyone know if CPAN, or perhaps another site, offers more stable
download URIs for these packages?

      Thanks,
        Mark




Information forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Tue, 26 Mar 2019 23:32:01 GMT) Full text and rfc822 format available.

Message #8 received at 35010 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 35010 <at> debbugs.gnu.org
Subject: Re: bug#35010: Many CPAN download URLs are no longer available
Date: Wed, 27 Mar 2019 01:31:00 +0100
[Message part 1 (text/plain, inline)]
Mark,

[Quick stream-of consciousness reply on a train, whee.]

Mark H. Weaver wrote:
> At least some, and probably most, of these URIs were updated 
> quite
> recently.  For example, the 'perl-mouse', 'perl-carp-clan', and
> 'perl-file-temp' were all updated on March 23, and presumably 
> the source
> URIs worked at that point, but then all three URIs had to be 
> updated two
> days later to fix the broken download links.

- The wave of Perl updates this March was me, using ‘guix refresh 
 -u’ (and manually checking for breakages, of course).

I never use ‘guix refresh’, until after about a year I forget why, 
use ‘guix refresh’ once, and promptly remember.

The updater for CPAN packages is at best caveat-quality.  It 
helpfully downloads the updated tarball to the store, but doesn't 
update the URL.  Since the file is in the store, everything works 
fine on the updater's machine, then breaks everywhere else.

Background: the problem here is that CPAN URLs contain the 
uploader name, e.g. (Karen?) ETHER(idge) in the case of

 mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz

which doesn't change every VERSION (so the problem is somewhat 
hidden) but more frequently than the author/maintainer would.

I'm planning on finally taking care this problem after I get home 
(and after I finally get the Overdrives set up, cough), either by 
making the CPAN updater also rewrite URL fields (if possible?), or 
finding out whether there's a way to construct these URLs without 
using the uploader name, or… well, that's all I have for now.

- While fixing the remaining fallout from this, I did find a few 
 other broken CPAN links for packages that I hadn't recently 
 touched.  Updating them broke others, so I just left them alone. 
 I don't know if these are more broken refreshes from longer ago 
 that went unnoticed (so basically nobody uses these packages), 
 or if there's another unrelated problem.

Kind regards,

T G-R
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Wed, 27 Mar 2019 14:09:01 GMT) Full text and rfc822 format available.

Message #11 received at 35010 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 35010 <at> debbugs.gnu.org, Mark H Weaver <mhw <at> netris.org>
Subject: Re: bug#35010: Many CPAN download URLs are no longer available
Date: Wed, 27 Mar 2019 15:07:54 +0100
Hello,

Tobias Geerinckx-Rice <me <at> tobias.gr> skribis:

> The updater for CPAN packages is at best caveat-quality.  It helpfully
> downloads the updated tarball to the store, but doesn't update the
> URL.  Since the file is in the store, everything works fine on the
> updater's machine, then breaks everywhere else.
>
> Background: the problem here is that CPAN URLs contain the uploader
> name, e.g. (Karen?) ETHER(idge) in the case of
>
>  mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz
>
> which doesn't change every VERSION (so the problem is somewhat hidden)
> but more frequently than the author/maintainer would.

Ouch, I didn’t know that, this is terrible.

The good news is that the (guix upstream) framework gets to see the
correct URL:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
$6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
--8<---------------cut here---------------end--------------->8---

So with a bit of work we could change ‘package-update’ to automatically
update the URL.

(Time passes…)

Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
I should have let figure it out ;-), but somehow I ended up
investigating too much, bah!

Let me know what you think.

In the meantime, to fix the Perl packages, you could maybe run:

  guix lint -c source $(guix package -A ^perl-)

Then you could perhaps comment the ‘version>?’ test in (guix upstream)
to force ‘guix package -u’ to update these seemingly up-to-date
packages.

HTH,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Wed, 27 Mar 2019 17:41:02 GMT) Full text and rfc822 format available.

Message #14 received at 35010 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35010 <at> debbugs.gnu.org, Mark H Weaver <mhw <at> netris.org>
Subject: Re: bug#35010: Many CPAN download URLs are no longer available
Date: Wed, 27 Mar 2019 19:40:20 +0100
[Message part 1 (text/plain, inline)]
Ludo',

Ludovic Courtès wrote:
> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
> I should have let [you?] figure it out ;-), but somehow I ended
> up investigating too much, bah!

Now I had time to rant about package names.  Win-win!  …wait

> Let me know what you think.

[…]

> In the meantime, to fix the Perl packages, you could maybe run:
>
>   guix lint -c source $(guix package -A ^perl-)
>
> Then you could perhaps comment the ‘version>?’ test in (guix 
> upstream)
> to force ‘guix package -u’ to update these seemingly up-to-date
> packages.

I'll deffo give it a try.  Thanks!

Kind regards,

T G-R
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Thu, 28 Mar 2019 22:12:02 GMT) Full text and rfc822 format available.

Message #17 received at 35010 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 35010 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#35010: Many CPAN download URLs are no longer available
Date: Thu, 28 Mar 2019 18:10:03 -0400
Ludovic Courtès <ludo <at> gnu.org> writes:

> The good news is that the (guix upstream) framework gets to see the
> correct URL:
>
> scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
> $6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
>
> So with a bit of work we could change ‘package-update’ to automatically
> update the URL.
>
> (Time passes…)
>
> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.

Nice!  I haven't looked closely, but does this somehow update the code
that computes source URLs using 'string-append'?  If so, that's
impressive :)

In other news, here's another batch of bad source URLs that Hydra
recently failed to fetch:

  mirror://savannah/starfighter/1.7/project-starfighter-1.7-src.tar.gz
  mirror://cpan/authors/id/P/PI/PIJLL/DateTime-Calendar-Julian-0.100.tar.gz
  mirror://cpan/authors/id/A/AD/ADAMK/Object-Signature-1.08.tar.gz
  mirror://cpan/authors/id/J/JH/JHI/Time-HiRes-1.9760.tar.gz
  mirror://cpan/authors/id/F/FR/FREW/Class-C3-Componentised-1.001002.tar.gz
  mirror://cpan/authors/id/R/RG/RGARCIA/base-2.23.tar.gz

       Mark




Information forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Fri, 29 Mar 2019 15:35:01 GMT) Full text and rfc822 format available.

Message #20 received at 35010 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 35010 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#35010: Many CPAN download URLs are no longer available
Date: Fri, 29 Mar 2019 16:34:02 +0100
Mark H Weaver <mhw <at> netris.org> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> The good news is that the (guix upstream) framework gets to see the
>> correct URL:
>>
>> scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
>> $6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
>>
>> So with a bit of work we could change ‘package-update’ to automatically
>> update the URL.
>>
>> (Time passes…)
>>
>> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
>
> Nice!  I haven't looked closely, but does this somehow update the code
> that computes source URLs using 'string-append'?  If so, that's
> impressive :)

Well, yes and no.  :-)

It looks like that’s what it does, but in practice it simply does a
string substitution on the ‘dirname’ of the URI, which probably works in
most cases (it definitely works for CPAN packages.)

> In other news, here's another batch of bad source URLs that Hydra
> recently failed to fetch:
>
>   mirror://savannah/starfighter/1.7/project-starfighter-1.7-src.tar.gz
>   mirror://cpan/authors/id/P/PI/PIJLL/DateTime-Calendar-Julian-0.100.tar.gz
>   mirror://cpan/authors/id/A/AD/ADAMK/Object-Signature-1.08.tar.gz
>   mirror://cpan/authors/id/J/JH/JHI/Time-HiRes-1.9760.tar.gz
>   mirror://cpan/authors/id/F/FR/FREW/Class-C3-Componentised-1.001002.tar.gz
>   mirror://cpan/authors/id/R/RG/RGARCIA/base-2.23.tar.gz

Thanks!

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#35010; Package guix. (Sun, 07 Apr 2019 01:33:01 GMT) Full text and rfc822 format available.

Message #23 received at 35010 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 35010 <at> debbugs.gnu.org
Subject: Re: bug#35010: Many CPAN download URLs are no longer available
Date: Sat, 06 Apr 2019 21:30:22 -0400
Hi Tobias,

There are still at least two more bad URLs from this batch of perl
updates circa March 23rd:

  mirror://cpan/authors/id/A/AB/ABW/Template-Toolkit-2.28.tar.gz
  mirror://cpan/authors/id/M/MA/MAKAMAKA/Text-CSV-1.99.tar.gz

Could you take a look?

     Thanks,
       Mark




Reply sent to Mark H Weaver <mhw <at> netris.org>:
You have taken responsibility. (Wed, 01 May 2019 03:01:02 GMT) Full text and rfc822 format available.

Notification sent to Mark H Weaver <mhw <at> netris.org>:
bug acknowledged by developer. (Wed, 01 May 2019 03:01:02 GMT) Full text and rfc822 format available.

Message #28 received at 35010-done <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: 35010-done <at> debbugs.gnu.org
Subject: Many CPAN download URLs are no longer available
Date: Tue, 30 Apr 2019 22:58:14 -0400
I believe that these broken URLs are now fixed, so I'm closing this bug.

      Mark




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Wed, 29 May 2019 11:24:04 GMT) Full text and rfc822 format available.

This bug report was last modified 4 years and 305 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.