GNU logs - #21665, boring messages

Message sent to bug-diffutils@HIDDEN:

Subject: bug#21665: Use of mmap for large files
Resent-From: Maurice van der Pot <griffon26@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Sun, 11 Oct 2015 17:09:01 +0000
Resent-Message-ID: <handler.21665.B.144458329416432 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 21665 <at> debbugs.gnu.org
Date: Sun, 11 Oct 2015 15:42:03 +0200
From: Maurice van der Pot <griffon26@HIDDEN>
Message-ID: <20151011134203.GA1315@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
 protocol="application/pgp-signature"; boundary="jRHKVT23PllUwdXP"
Content-Disposition: inline
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>


--jRHKVT23PllUwdXP
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I am working on an application suitable for visually merging large files.
This application delegates determination of differences to GNU diff.

Unfortunately I have found that diff reads the entire input files into
memory, leading to "/usr/bin/diff: memory exhausted" messages on the
types of files I'd like to support.

Would you be open to patches that enable diffing large files by using
mmap?

Kind regards,
Maurice.

--=20
Maurice van der Pot

Kdiff3 developer   griffon26@HIDDEN   http://kdiff3.sourceforge.net
Tdiff3 developer                            https://github.com/Griffon26/td=
iff3

--jRHKVT23PllUwdXP
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlYaZysACgkQMGnpIbeahxxSIQCgmUl58w1BR6+oOUQK7TUZkhYl
6EcAnjRMByA1+xXDEtfjFHXa/oCCAwxw
=Zs9+
-----END PGP SIGNATURE-----

--jRHKVT23PllUwdXP--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Maurice van der Pot <griffon26@HIDDEN>
Subject: bug#21665: Acknowledgement (Use of mmap for large files)
Message-ID: <handler.21665.B.144458329416432.ack <at> debbugs.gnu.org>
References: <20151011134203.GA1315@HIDDEN>
X-Gnu-PR-Message: ack 21665
X-Gnu-PR-Package: diffutils
Reply-To: 21665 <at> debbugs.gnu.org
Date: Sun, 11 Oct 2015 17:09:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-diffutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 21665 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
21665: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D21665
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-diffutils@HIDDEN:

Subject: bug#21665: [bug-diffutils] bug#21665: Use of mmap for large files
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Sun, 11 Oct 2015 21:16:02 +0000
Resent-Message-ID: <handler.21665.B.14445981049089 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 21665 <at> debbugs.gnu.org
References: <20151011134203.GA1315@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
Message-ID: <561AD14D.5080009@HIDDEN>
Date: Sun, 11 Oct 2015 14:14:53 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <20151011134203.GA1315@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Maurice van der Pot wrote:
> Would you be open to patches that enable diffing large files by using
> mmap?

I doubt whether that would help that much, as it still needs to construct 
information about each line, and that information consumes memory too.  Doing 
this in secondary storage would be a bear.  In practice when I've run into this 
problem, I've either gotten a bigger machine or made my input lines shorter. 
Preferably the former.

Message sent to bug-diffutils@HIDDEN:

Subject: bug#21665: [bug-diffutils] bug#21665: Use of mmap for large files
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-diffutils@HIDDEN
Resent-Date: Mon, 02 May 2016 01:28:01 +0000
Resent-Message-ID: <handler.21665.B21665.146215245010674 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Maurice van der Pot <griffon26@HIDDEN>
Cc: 21665 <at> debbugs.gnu.org
MIME-Version: 1.0
In-Reply-To: <20151011134203.GA1315@HIDDEN>
References: <20151011134203.GA1315@HIDDEN>
From: Jim Meyering <jim@HIDDEN>
Date: Sun, 1 May 2016 18:27:03 -0700
Message-ID: <CA+8g5KFEe0+YrXJQtVH4Tb6YUr2JEWrPz_xbEj5aE6-Rqvp1mQ@HIDDEN>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

tags 21665 notabug
close 21665
done

On Sun, Oct 11, 2015 at 6:42 AM, Maurice van der Pot
<griffon26@HIDDEN> wrote:
> I am working on an application suitable for visually merging large files.
> This application delegates determination of differences to GNU diff.
>
> Unfortunately I have found that diff reads the entire input files into
> memory, leading to "/usr/bin/diff: memory exhausted" messages on the
> types of files I'd like to support.
>
> Would you be open to patches that enable diffing large files by using
> mmap?

As Paul responded in http://bugs.gnu.org/21665#8, using mmap seems
unlikely to help much, but if you write the patch and demonstrate that
it does make a difference, we'll be very interested, and I will
happily reopen the issue.

For now, I'm marking this as notabug and closing it.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.