GNU bug report logs - #63832
fix failed inflation of .el.gz archives due to passing empty buffer to inflate()

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 63832 in the body.
You can then email your comments to 63832 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Amritpal Singh <icy.amrit <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: fix failed inflation of .el.gz archives due to passing empty buffer
 to inflate()
Date: Fri, 2 Jun 2023 12:21:07 +0530

[Message part 1 (text/plain, inline)]

Compile emacs with the system's gzip program set to `pigz`.
Run emacs and then `M-x eww RET`

Expected behavior:
Enter URL prompt in mini-buffer

Actual behavior:
hashing failed '/usr/share/emacs/30.0.50/lisp/gnus/gnus.el.gz'

Report:
The bug has been reproduced on emacs version 29.0.91 and HEAD which
seems to be at 30.0.50.
Later, a copy of the aforementioned file was saved somewhere else and
the program was uninstalled. Then emacs was recompiled with system's
gzip program set to GNU gzip and the initial steps were repeated and
the expected behavior was the result.
This lead to believing either that there's a bug with how zlib's
`inflate()` handles archives or emacs code was having an issue with
archives files.

The hashes for gz archives generated with different programs were as follows
> md5sum gnus-gzip.el.gz
edb3d0ffba7f19ff1d4ec3f889609e8a  gnus-gzip.el.gz
> md5sum gnus.el.gz
985deaaec6a5845ac8d6bd9648957b50  gnus.el.gz

And when uncompressing these archives, the resulting file was the same
and the hash for the files was the same (omitted for brevity).

Now after logging some code in $EMACS_REPO/src/decompress.c, it was
learned that in the pigz specific case, `inflate()` was returning
Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
or `avail_out` fields are 0.

Observing the code in `$EMACS_REPO/src/decompress.c`
L154:
    } while (!stream.avail_out);
only checks stream.avail_out and not stream.avail_in which also might
have been set to 0. A special case here can be constructed where
`avail_in` is 0, and the code keeps looping even though our input
buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
for it fixes the bug in pigz's gz archives case and does not cause any
issue with gzip archives.

A patch with a simple fix is attached below

============================

From ffa8e140ed8b093c60f1238bf76935e815e82a21 Mon Sep 17 00:00:00 2001
From: icebarf <sysgrammer <at> protonmail.com>
Date: Fri, 2 Jun 2023 10:51:21 +0530
Subject: [PATCH] check stream.avail_in as well when looping to inflate gz
 archive

---
 src/decompress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/decompress.c b/src/decompress.c
index 6ef17db..162f616 100644
--- a/src/decompress.c
+++ b/src/decompress.c
@@ -151,7 +151,7 @@ md5_gz_stream (FILE *source, void *resblock)
     return -1;

       accumulate_and_process_md5 (out, MD5_BLOCKSIZE - stream.avail_out, &ctx);
-    } while (!stream.avail_out);
+    } while (stream.avail_in && !stream.avail_out);

   } while (res != Z_STREAM_END);

-- 
2.40.1

==========

In GNU Emacs 29.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.37, cairo version 1.16.0) of 2023-06-02 built on box
System Description: KISS Linux

Configured using:
 'configure --prefix=/usr --enable-link-time-optimization --with-cairo
 --with-gnutls --with-imagemagick --with-modules
 --with-native-compilation --with-pgtk --with-jpeg --with-tiff
 --with-png --with-rsvg --with-webp --without-dbus --without-lcms2
 --without-libsystemd --without-gif --without-gconf --without-gsettings
 --without-m17n-flt --without-selinux --without-x --without-xft
 --without-xaw3d --without-xim --without-xdbe 'CFLAGS=-O2 -pipe
 -march=native -mtune=native''

[0001-check-stream.avail_in-as-well-when-looping-to-inflat.patch (text/x-diff, attachment)]

Message #8 received at 63832 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Amritpal Singh <icy.amrit <at> gmail.com>
Cc: 63832 <at> debbugs.gnu.org
Subject: Re: bug#63832: fix failed inflation of .el.gz archives due to passing
 empty buffer to inflate()
Date: Fri, 02 Jun 2023 15:04:50 +0300

merge 63832 63831
thanks

> From: Amritpal Singh <icy.amrit <at> gmail.com>
> Date: Fri, 2 Jun 2023 12:21:07 +0530
> 
> Compile emacs with the system's gzip program set to `pigz`.
> Run emacs and then `M-x eww RET`
> 
> Expected behavior:
> Enter URL prompt in mini-buffer
> 
> Actual behavior:
> hashing failed '/usr/share/emacs/30.0.50/lisp/gnus/gnus.el.gz'
> 
> Report:
> The bug has been reproduced on emacs version 29.0.91 and HEAD which
> seems to be at 30.0.50.
> Later, a copy of the aforementioned file was saved somewhere else and
> the program was uninstalled. Then emacs was recompiled with system's
> gzip program set to GNU gzip and the initial steps were repeated and
> the expected behavior was the result.
> This lead to believing either that there's a bug with how zlib's
> `inflate()` handles archives or emacs code was having an issue with
> archives files.
> 
> The hashes for gz archives generated with different programs were as follows
> > md5sum gnus-gzip.el.gz
> edb3d0ffba7f19ff1d4ec3f889609e8a  gnus-gzip.el.gz
> > md5sum gnus.el.gz
> 985deaaec6a5845ac8d6bd9648957b50  gnus.el.gz
> 
> And when uncompressing these archives, the resulting file was the same
> and the hash for the files was the same (omitted for brevity).
> 
> Now after logging some code in $EMACS_REPO/src/decompress.c, it was
> learned that in the pigz specific case, `inflate()` was returning
> Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
> or `avail_out` fields are 0.
> 
> Observing the code in `$EMACS_REPO/src/decompress.c`
> L154:
>     } while (!stream.avail_out);
> only checks stream.avail_out and not stream.avail_in which also might
> have been set to 0. A special case here can be constructed where
> `avail_in` is 0, and the code keeps looping even though our input
> buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
> for it fixes the bug in pigz's gz archives case and does not cause any
> issue with gzip archives.
> 
> A patch with a simple fix is attached below
> 
> ============================
> 
> >From ffa8e140ed8b093c60f1238bf76935e815e82a21 Mon Sep 17 00:00:00 2001
> From: icebarf <sysgrammer <at> protonmail.com>
> Date: Fri, 2 Jun 2023 10:51:21 +0530
> Subject: [PATCH] check stream.avail_in as well when looping to inflate gz
>  archive
> 
> ---
>  src/decompress.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/decompress.c b/src/decompress.c
> index 6ef17db..162f616 100644
> --- a/src/decompress.c
> +++ b/src/decompress.c
> @@ -151,7 +151,7 @@ md5_gz_stream (FILE *source, void *resblock)
>      return -1;
> 
>        accumulate_and_process_md5 (out, MD5_BLOCKSIZE - stream.avail_out, &ctx);
> -    } while (!stream.avail_out);
> +    } while (stream.avail_in && !stream.avail_out);
> 
>    } while (res != Z_STREAM_END);
> 
> -- 
> 2.40.1
> 
> ==========
> 
> In GNU Emacs 29.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version
>  3.24.37, cairo version 1.16.0) of 2023-06-02 built on box
> System Description: KISS Linux
> 
> Configured using:
>  'configure --prefix=/usr --enable-link-time-optimization --with-cairo
>  --with-gnutls --with-imagemagick --with-modules
>  --with-native-compilation --with-pgtk --with-jpeg --with-tiff
>  --with-png --with-rsvg --with-webp --without-dbus --without-lcms2
>  --without-libsystemd --without-gif --without-gconf --without-gsettings
>  --without-m17n-flt --without-selinux --without-x --without-xft
>  --without-xaw3d --without-xim --without-xdbe 'CFLAGS=-O2 -pipe
>  -march=native -mtune=native''

This is a duplicate of bug#63831; merged.

Message #17 received at 63832-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Amritpal Singh <icy.amrit <at> gmail.com>
Cc: 63832-done <at> debbugs.gnu.org
Subject: Re: bug#63832: fix failed inflation of .el.gz archives due to passing
 empty buffer to inflate()
Date: Thu, 08 Jun 2023 12:43:11 +0300

> From: Amritpal Singh <icy.amrit <at> gmail.com>
> Date: Fri, 2 Jun 2023 12:21:07 +0530
> 
> Compile emacs with the system's gzip program set to `pigz`.
> Run emacs and then `M-x eww RET`
> 
> Expected behavior:
> Enter URL prompt in mini-buffer
> 
> Actual behavior:
> hashing failed '/usr/share/emacs/30.0.50/lisp/gnus/gnus.el.gz'
> 
> Report:
> The bug has been reproduced on emacs version 29.0.91 and HEAD which
> seems to be at 30.0.50.
> Later, a copy of the aforementioned file was saved somewhere else and
> the program was uninstalled. Then emacs was recompiled with system's
> gzip program set to GNU gzip and the initial steps were repeated and
> the expected behavior was the result.
> This lead to believing either that there's a bug with how zlib's
> `inflate()` handles archives or emacs code was having an issue with
> archives files.
> 
> The hashes for gz archives generated with different programs were as follows
> > md5sum gnus-gzip.el.gz
> edb3d0ffba7f19ff1d4ec3f889609e8a  gnus-gzip.el.gz
> > md5sum gnus.el.gz
> 985deaaec6a5845ac8d6bd9648957b50  gnus.el.gz
> 
> And when uncompressing these archives, the resulting file was the same
> and the hash for the files was the same (omitted for brevity).
> 
> Now after logging some code in $EMACS_REPO/src/decompress.c, it was
> learned that in the pigz specific case, `inflate()` was returning
> Z_BUF_ERROR(-5) which is an indicator for zstream's either `avail_in`
> or `avail_out` fields are 0.
> 
> Observing the code in `$EMACS_REPO/src/decompress.c`
> L154:
>     } while (!stream.avail_out);
> only checks stream.avail_out and not stream.avail_in which also might
> have been set to 0. A special case here can be constructed where
> `avail_in` is 0, and the code keeps looping even though our input
> buffer is empty and thus causing a Z_BUF_ERROR. Placing a simple check
> for it fixes the bug in pigz's gz archives case and does not cause any
> issue with gzip archives.
> 
> A patch with a simple fix is attached below

Thanks, installed on the master branch, and closing the bug.

Message #34 received at 63832 <at> debbugs.gnu.org (full text, mbox):

From: Ulrich Mueller <ulm <at> gentoo.org>
To: 63832 <at> debbugs.gnu.org
Subject: Re: bug#63832: fix failed inflation of .el.gz archives due to passing
Date: Tue, 01 Aug 2023 07:41:49 +0200

I wonder if commit 46b6d175054e could be cherry-picked to the emacs-29
branch? The fix looks unintrusive.

Users see the problem with Emacs 29.1 downstream, see Gentoo bug 911539:
https://bugs.gentoo.org/911539

Message #37 received at 63832 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ulrich Mueller <ulm <at> gentoo.org>
Cc: 63832 <at> debbugs.gnu.org
Subject: Re: bug#63832: fix failed inflation of .el.gz archives due to passing
Date: Tue, 01 Aug 2023 14:11:54 +0300

> From: Ulrich Mueller <ulm <at> gentoo.org>
> Date: Tue, 01 Aug 2023 07:41:49 +0200
> 
> I wonder if commit 46b6d175054e could be cherry-picked to the emacs-29
> branch? The fix looks unintrusive.

That's fine by me, thanks.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.