GNU bug report logs - #71529
Crash for large directory due to memory outage

Previous Next

To reply to this bug, email your comments to 71529 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Evan Greenup <evan_greenup <at> protonmail.com>
To: "bug-diffutils <at> gnu.org" <bug-diffutils <at> gnu.org>
Subject: Crash for large directory due to memory outage
Date: Wed, 12 Jun 2024 22:39:51 +0000

Hi,

The `diff` command in `diffutils` can compare the difference between  two directories. 

However, for large directories (the size for each file is small), when traversal through them, `diff` command will eat up all the system memory. My computer is 16GB RAM. And I believe this issue still exist for system with 32GB or 64GB RAM.

I would suggest that it can use more memory efficient algorithm to compare the difference between directories.

Regards!!!

Evan Greenup.

Message #8 received at 71529 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Evan Greenup <evan_greenup <at> protonmail.com>
Cc: 71529 <at> debbugs.gnu.org
Subject: Re: [bug-diffutils] bug#71529: Crash for large directory due to
 memory outage
Date: Thu, 13 Jun 2024 17:07:26 -0700

What do you mean by "crash"? Did you need to reboot the system, or did 
"diff" dump core, or did diff print a diagnostic "memory exhausxted"?

Can you send a recipe for reproducing the problem? I didn't have any 
trouble running diff 3.10 on a Fedora 40 machine with only 8 GiB RAM, in 
directories with 100,000 files. The following shell commands succeeded 
quietly:

  mkdir d e
  (cd d && touch $(seq 100000))
  (cd e && touch $(seq 100000))
  diff -r d e

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.