GNU bug report logs - #63831
fix bug with failed inflation of .el.gz files in src/decompress.c

Previous Next

Package: emacs;

Reported by: Amritpal Singh <icy.amrit <at> gmail.com>

Date: Fri, 2 Jun 2023 07:53:01 UTC

Severity: normal

Tags: patch

Merged with 63832, 63848

Done: Eli Zaretskii <eliz <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 63831 in the body.
You can then email your comments to 63831 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Fri, 02 Jun 2023 07:53:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Amritpal Singh <icy.amrit <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Fri, 02 Jun 2023 07:53:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Amritpal Singh <icy.amrit <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: fix bug with failed inflation of .el.gz files in src/decompress.c
Date: Fri, 2 Jun 2023 11:09:35 +0530
[Message part 1 (text/plain, inline)]
Bug Report:
Compile emacs with system's gzip program set to `pigz`.
Run emacs and then `M-x eww RET`

Expected behavior:
Enter URL prompt in mini-buffer

Actual behavior:
hashing failed '/usr/share/emacs/30.0.50/lisp/gnus/gnus.el.gz'

Report:
The bug has been reproduced on emacs version 29.0.91 and HEAD which
seems to be at 30.0.50.
Later, a copy of the aforementioned file was saved somewhere else and
the program was uninstalled. Then emacs was recompiled with system's
gzip program set to GNU gzip and the initial steps were repeated and
the expected behavior was the result.
This lead to believing either that there's a bug with how zlib's
`inflate()` handles archives or emacs code was having an issue with
archives files.

The hashes for gz archives generated with different programs were as follows
> md5sum gnus-gzip.el.gz
edb3d0ffba7f19ff1d4ec3f889609e8a  gnus-gzip.el.gz
> md5sum gnus.el.gz
985deaaec6a5845ac8d6bd9648957b50  gnus.el.gz

And when uncompressing these archives, the resulting file was the same
and the hash for the files was the same (omitted for brevity).

Now after logging some code in $EMACS_REPO/src/decompress.c, it was
learned that in the pigz specific case, `inflate()` was returning
Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
or `avail_out` fields are 0.

Observing the code in `$EMACS_REPO/src/decompress.c`
L154:
    } while (!stream.avail_out);
only checks stream.avail_out and not stream.avail_in which also might
have been set to 0. A special case here can be constructed where
`avail_in` is 0, and the code keeps looping even though our input
buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
for it fixes the bug in pigz's gz archives case and does not cause any
issue with gzip archives.

A simple patch with a fix is attached below, I would also like to
thank a friend of mine cortexauth whom also helped during my debug
sessions.
[0001-check-stream.avail_in-as-well-when-looping-to-inflat.patch (text/x-diff, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Fri, 02 Jun 2023 11:13:02 GMT) Full text and rfc822 format available.

Message #8 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Amritpal Singh <icy.amrit <at> gmail.com>
To: 63831 <at> debbugs.gnu.org
Subject: Accidentally submit two bug reports for the same topic
Date: Fri, 2 Jun 2023 13:35:38 +0530
I'd like to apologise for submitting two bug reports, as I'd sent
another after ~1.5 hour of submitting this bug report. I hadn't
received any acknowledgement from the automatic bug tracking system
and thought I'd submitted the report in a wrong method.




Merged 63831 63832. Request was from Eli Zaretskii <eliz <at> gnu.org> to control <at> debbugs.gnu.org. (Fri, 02 Jun 2023 12:05:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Fri, 02 Jun 2023 12:13:01 GMT) Full text and rfc822 format available.

Message #13 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Amritpal Singh <icy.amrit <at> gmail.com>
Cc: 63831 <at> debbugs.gnu.org
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Fri, 02 Jun 2023 15:13:39 +0300
> From: Amritpal Singh <icy.amrit <at> gmail.com>
> Date: Fri, 2 Jun 2023 13:35:38 +0530
> 
> I'd like to apologise for submitting two bug reports, as I'd sent
> another after ~1.5 hour of submitting this bug report. I hadn't
> received any acknowledgement from the automatic bug tracking system
> and thought I'd submitted the report in a wrong method.

Please be more patient.  Each bug report is approved by a human, to
avoid the catastrophic consequences of admitting spam to the bug list.
So it might take some time, especially if the person who approves the
bug reports and you live in different time zones.




Merged 63831 63832 63848. Request was from Eli Zaretskii <eliz <at> gnu.org> to control <at> debbugs.gnu.org. (Fri, 02 Jun 2023 15:35:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sat, 03 Jun 2023 07:15:02 GMT) Full text and rfc822 format available.

Message #18 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Amritpal Singh <icy.amrit <at> gmail.com>
To: 63831 <at> debbugs.gnu.org
Subject: [PATCH] Elaboration on the bugfix for failed inflation of gz archives
Date: Sat, 3 Jun 2023 08:38:18 +0530
> Now after logging some code in $EMACS_REPO/src/decompress.c, it was
> learned that in the pigz specific case, `inflate()` was returning
> Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
> or `avail_out` fields are 0.
>
> Observing the code in `$EMACS_REPO/src/decompress.c`
> L154:
>     } while (!stream.avail_out);
> only checks stream.avail_out and not stream.avail_in which also might
> have been set to 0. A special case here can be constructed where
> `avail_in` is 0, and the code keeps looping even though our input
> buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
> for it fixes the bug in pigz's gz archives case and does not cause any
> issue with gzip archives.

I'd like to elaborate a bit further on this part mentioned in the bug
report-cum-patch email. The code in question where the bug occurs is
as follows:

> do {
>    stream.avail_in = fread (in, 1, MD5_BLOCKSIZE, source);
>    if (ferror (source)) {
>      inflateEnd (&stream);
>      return -1;
>    }
>    if (stream.avail_in == 0)
>      break;
>    stream.next_in = in;
>
>    do {
>      stream.avail_out = MD5_BLOCKSIZE;
>      stream.next_out = out;
>      res = inflate (&stream, Z_NO_FLUSH);
>
>      if (res != Z_OK && res != Z_STREAM_END)
>    return -1;
>
>      accumulate_and_process_md5 (out, MD5_BLOCKSIZE - stream.avail_out, &ctx);
L154:
>    } while (!stream.avail_out);
L155:
>
>  } while (res != Z_STREAM_END);


From here, it is a bit more clear what the originally mentioned `L154`
was referring to. The archives compressed using pigz set as the systems'
gzip program either via symlink as `gzip` in $PATH or using $GZIP_PROG,
fail to hash in this specific instance which is quite strange,
nonetheless this patch (provided in the original bug report) fixes the
issue of calling inflate() repeatedly (in the inner do-while() loop)
when we do not have any more input to process.

This happens so, as mentioned before, because `infalte()` can set
`stream.avail_in` to 0 which is the available number of bytes to
process. And since we're only checking if stream.avail_out hasn't been
set to 0 in the inner loop, the hashing fails even though our archive is
correctly made and decompresses successfully externally.

ps. The patch is in the original bug report.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sat, 03 Jun 2023 21:28:02 GMT) Full text and rfc822 format available.

Message #21 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 63831 <at> debbugs.gnu.org, icy.amrit <at> gmail.com
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Sat, 03 Jun 2023 17:27:27 -0400
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Please be more patient.  Each bug report is approved by a human, to
  > avoid the catastrophic consequences of admitting spam to the bug list.
  > So it might take some time, especially if the person who approves the
  > bug reports and you live in different time zones.

There are good reasons for this, but can we do something to help
bug submitters understand the situation?

For instance, we could set up an immeidate auto-reply to inform them
of the delay to expect.

If we had two people to do the moderation, maybe they could work
at different times of day -- that would shorten the usual wait time
without incresaing the work.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)






Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sun, 04 Jun 2023 05:01:02 GMT) Full text and rfc822 format available.

Message #24 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rms <at> gnu.org, Michael Albinus <michael.albinus <at> gmx.de>
Cc: 63831 <at> debbugs.gnu.org, icy.amrit <at> gmail.com
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Sun, 04 Jun 2023 08:01:04 +0300
> From: Richard Stallman <rms <at> gnu.org>
> Cc: icy.amrit <at> gmail.com, 63831 <at> debbugs.gnu.org
> Date: Sat, 03 Jun 2023 17:27:27 -0400
> 
>   > Please be more patient.  Each bug report is approved by a human, to
>   > avoid the catastrophic consequences of admitting spam to the bug list.
>   > So it might take some time, especially if the person who approves the
>   > bug reports and you live in different time zones.
> 
> There are good reasons for this, but can we do something to help
> bug submitters understand the situation?
> 
> For instance, we could set up an immeidate auto-reply to inform them
> of the delay to expect.
> 
> If we had two people to do the moderation, maybe they could work
> at different times of day -- that would shorten the usual wait time
> without incresaing the work.

Michael, any comments?




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sun, 04 Jun 2023 05:38:02 GMT) Full text and rfc822 format available.

Message #27 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Amritpal Singh <icy.amrit <at> gmail.com>
Cc: 63831 <at> debbugs.gnu.org, rms <at> gnu.org
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Sun, 04 Jun 2023 08:38:00 +0300
> From: Amritpal Singh <icy.amrit <at> gmail.com>
> Date: Sun, 4 Jun 2023 09:16:49 +0530
> Cc: "eliz <at> gnu.org" <eliz <at> gnu.org>, 63831 <at> debbugs.gnu.org
> 
> While we're on the topic of bug report moderation, is it possible that the
> current human moderation might be a little ineffective?

Feel free to volunteer to help, in order to improve the quality and
the timeliness of the moderation.

> We've observed that me accidentally submitting a duplicate report
> made it through human filters, and I had notice another instance of
> a "spam" email, where no subject or body was present.  See:
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-06/msg00107.html

It isn't spam.  The sender is an Emacs contributor.

> I'm not trying to impugn someone else's ability of moderating but the current
> system seems a little ineffective. Also I'd like to apologize if my
> language/wording
> seems abrasive towards the moderation team as this is no way an insult to their
> work but rather a lack of implementation of better systems by management.

I'm not sure I understand what you are suggesting here.  What better
system did you have in mind?

(In any case, discussing this as part of a specific bug report is
wrong; this should be taken to emacs-devel instead.)

> I'd also like to draw attention towards the actual bug+patch report
> #63831, it's been
> close to two days now and such a small change could easily be merged into HEAD.
> But the bug does seem low priority (even though breaking) to me as well so
> I would not question the maintainers' decision to prioritize critical
> bugs over this.

That bug is indeed low priority, and the change will go to master,
which makes it even lower priority (since we are in the process of
pretesting Emacs 29.1).  The bug report is in my queue, and two days
is really way too short to ring the alarm bells about such a minor
issue.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sun, 04 Jun 2023 06:14:02 GMT) Full text and rfc822 format available.

Message #30 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 63831 <at> debbugs.gnu.org, icy.amrit <at> gmail.com, rms <at> gnu.org
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Sun, 04 Jun 2023 08:13:32 +0200
Eli Zaretskii <eliz <at> gnu.org> writes:

Hi Eli & Richard,

>>   > Please be more patient.  Each bug report is approved by a human, to
>>   > avoid the catastrophic consequences of admitting spam to the bug list.
>>   > So it might take some time, especially if the person who approves the
>>   > bug reports and you live in different time zones.
>>
>> There are good reasons for this, but can we do something to help
>> bug submitters understand the situation?
>>
>> For instance, we could set up an immeidate auto-reply to inform them
>> of the delay to expect.

No. This would send the information to spam factories that the given
email address on debbugs.gnu.org is valid. Meaning increasing the
amount of spam.

>> If we had two people to do the moderation, maybe they could work
>> at different times of day -- that would shorten the usual wait time
>> without incresaing the work.

AFAIK, Bob Proulx and me are moderating. Bob lives in the USA
(Colorado?), I live in Germany, so there are different time zones.

Usually, I check debbugs for hold messages several times a day. A delay
of 1.5 hours, as reported here, is completely normal. We cannot change
it unless we could recruit more moderators.

Best regrds, Michael.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sun, 04 Jun 2023 06:31:02 GMT) Full text and rfc822 format available.

Message #33 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 63831 <at> debbugs.gnu.org, Amritpal Singh <icy.amrit <at> gmail.com>, rms <at> gnu.org
Subject: Re: bug#63831: fix bug with failed inflation of .el.gz files in
 src/decompress.c
Date: Sun, 04 Jun 2023 08:30:18 +0200
Eli Zaretskii <eliz <at> gnu.org> writes:

Hi,

>> We've observed that me accidentally submitting a duplicate report
>> made it through human filters,

FTR, there are several moderators, so you cannot expect that both
reports have been seen by the same moderator (honestly, I don't
remember the case)

Furthermore, we don't decide on the contents. If something looks like a
valid bug report we approve, whatever it is about. No further checks on
duplicates or alike, that would exceed our limited resources.

>> I'm not trying to impugn someone else's ability of moderating but the current
>> system seems a little ineffective.

> I'm not sure I understand what you are suggesting here.  What better
> system did you have in mind?

I don't feel offended :-)

Anyway, what we do is manual, human work. Do you recommend to hand over
moderation to ChatGPT? :-)

Best regards, Michael.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sun, 04 Jun 2023 08:10:06 GMT) Full text and rfc822 format available.

Message #36 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Amritpal Singh <icy.amrit <at> gmail.com>
To: rms <at> gnu.org
Cc: 63831 <at> debbugs.gnu.org, "eliz <at> gnu.org" <eliz <at> gnu.org>
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Sun, 4 Jun 2023 09:16:49 +0530
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > Please be more patient.  Each bug report is approved by a human, to
>   > avoid the catastrophic consequences of admitting spam to the bug list.
>   > So it might take some time, especially if the person who approves the
>   > bug reports and you live in different time zones.

> There are good reasons for this, but can we do something to help
> bug submitters understand the situation?

> For instance, we could set up an immeidate auto-reply to inform them
> of the delay to expect.

> If we had two people to do the moderation, maybe they could work
> at different times of day -- that would shorten the usual wait time
> without incresaing the work.

While we're on the topic of bug report moderation, is it possible that the
current human moderation might be a little ineffective? We've observed
that me accidentally submitting a duplicate report made it through human
filters, and I had notice another instance of a "spam" email, where no
subject or body was present.
See: https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-06/msg00107.html

I'm not trying to impugn someone else's ability of moderating but the current
system seems a little ineffective. Also I'd like to apologize if my
language/wording
seems abrasive towards the moderation team as this is no way an insult to their
work but rather a lack of implementation of better systems by management.

I'd also like to draw attention towards the actual bug+patch report
#63831, it's been
close to two days now and such a small change could easily be merged into HEAD.
But the bug does seem low priority (even though breaking) to me as well so
I would not question the maintainers' decision to prioritize critical
bugs over this.

I wish you a good day/night and, again would like to apoligise for any
rude language.

Regards,
Amritpal Singh




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Sun, 04 Jun 2023 08:10:07 GMT) Full text and rfc822 format available.

Message #39 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Amritpal Singh <icy.amrit <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 63831 <at> debbugs.gnu.org
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Sun, 4 Jun 2023 11:17:52 +0530
> Feel free to volunteer to help, in order to improve the quality and
> the timeliness of the moderation.
Thank you! Please let me know how I can contribute to this.

> It isn't spam.  The sender is an Emacs contributor.
Apologies, I'm new to emacs mailing lists and unfamiliar with the
contributors, and I considered the empty email spam.

> I'm not sure I understand what you are suggesting here.  What better
> system did you have in mind?
>
> (In any case, discussing this as part of a specific bug report is
> wrong; this should be taken to emacs-devel instead.)
Like RMS suggested, perhaps an automated immediate response
system for reports would indeed be a good choice to indicate to
contributors that their report indeed has been received.

I agree with the latter part about moving to emacs-devel.

> That bug is indeed low priority, and the change will go to master,
> which makes it even lower priority (since we are in the process of
> pretesting Emacs 29.1).  The bug report is in my queue, and two days
> is really way too short to ring the alarm bells about such a minor
> issue.

I'm sorry for that as well, like I mentioned before I'm a newbie contributor and
do not have the know-how with how contributors and maintainers interact.
This is no excuse for being impatient but I thought I'd explain myself.

Thank you!




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Mon, 05 Jun 2023 07:36:01 GMT) Full text and rfc822 format available.

Message #42 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 63831 <at> debbugs.gnu.org, eliz <at> gnu.org, icy.amrit <at> gmail.com
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Mon, 05 Jun 2023 03:35:01 -0400
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No. This would send the information to spam factories that the given
  > email address on debbugs.gnu.org is valid. Meaning increasing the
  > amount of spam.

Is the idea that, under the current system, spam never gets any response?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)






Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#63831; Package emacs. (Mon, 05 Jun 2023 08:18:01 GMT) Full text and rfc822 format available.

Message #45 received at 63831 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Richard Stallman <rms <at> gnu.org>
Cc: 63831 <at> debbugs.gnu.org, eliz <at> gnu.org, icy.amrit <at> gmail.com
Subject: Re: bug#63831: Accidentally submit two bug reports for the same topic
Date: Mon, 05 Jun 2023 10:17:07 +0200
Richard Stallman <rms <at> gnu.org> writes:

Hi Richard,

>   > No. This would send the information to spam factories that the given
>   > email address on debbugs.gnu.org is valid. Meaning increasing the
>   > amount of spam.
>
> Is the idea that, under the current system, spam never gets any response?

As moderator, I can decide to "Reject" or "Discard" a message. Usually I
decide to discard, which means "no response", yes. The Help describes
the options as

--8<---------------cut here---------------start------------->8---
Reject -- Reject the message, sending a rejection notice to the sender,
and discarding the original message. For membership requests, reject the
change in membership status. In either case, you should add a reason for
the rejection in the accompanying text box.

Discard -- Throw away the original message, without sending a rejection
notice. For membership requests, this simply discards the request
without notice to the person making the request. This is usually the
action you want to take for known spam.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Thu, 06 Jul 2023 11:24:05 GMT) Full text and rfc822 format available.

bug unarchived. Request was from Ulrich Mueller <ulm <at> gentoo.org> to control <at> debbugs.gnu.org. (Tue, 01 Aug 2023 05:37:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Tue, 29 Aug 2023 11:24:05 GMT) Full text and rfc822 format available.

This bug report was last modified 212 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.