GNU bug report logs - #33275
27.0.50; Image cache pruning

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Lars Ingebrigtsen <larsi@HIDDEN>; dated Mon, 5 Nov 2018 14:09:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 9 Nov 2018 12:26:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Nov 09 07:26:54 2018
Received: from localhost ([127.0.0.1]:42289 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gL5sE-0001AP-BJ
	for submit <at> debbugs.gnu.org; Fri, 09 Nov 2018 07:26:54 -0500
Received: from hermes.netfonds.no ([80.91.224.195]:49478)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gL5sC-0001AI-VZ
 for 33275 <at> debbugs.gnu.org; Fri, 09 Nov 2018 07:26:53 -0500
Received: from cm-84.212.221.165.getinternet.no ([84.212.221.165] helo=stories)
 by hermes.netfonds.no with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <larsi@HIDDEN>)
 id 1gL5sA-0002IL-EV; Fri, 09 Nov 2018 13:26:52 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
 <83bm72ciyz.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAElBMVEWHhWtoTzJ8ZTuAdVa8
 tIZjKBaOPNBXAAACdklEQVQ4jW2Uy5LjIAxFZcPsI3D2PJw9D3tPbGXfdjf//ysjSDpTUxVV2RAd
 pCuhJFDos8HmXFI+KceGp3UHokUfJwbWOW19tAoVokPleEEGObkPNsPoP/kZbD61hfeZH2JJ1z4Y
 GJGXNDvny+aMIPZxcjCwPXl7FPs4wmpeWcM19c2PPYPgvWkdNODVU+wlW8ZXVa6XNQ+coBVubr1X
 8z5IozN904vcWsTb/G+rBlgjcbmoXh6rHP42OGRlw+AsXx2fQHt1jdkJxmA1FKWjispaRqc9lNUT
 7GEAKAPkIFBblSNiRgwL7MB+IbLIJcZ4WTKfiBkakLAnBjMIldYgvJCiAQEzDzJkKCX7KIvwKYtQ
 GEgGe8hZFsieZy0SQDYBSoswmcFQohlkCSMIIVgjLESpSd4hx5XA70GMHWRHhqtd7jOAJOBMIjWQ
 QY7aWjGAjF4SVwi5A9a/YooaAqQgnPf8gp5Kys3mqEcnioBtjNmkAB1weB6mPUKrfct+3CLnKUBU
 xqRZl2QgWgzsG/QIItTp4IoK8CEyQXbwuND+pSwDGKumhSLKPV9meBgaqnYXgOEquTFaVRgVTPD4
 dr5+Xz0DvUg10+pmd/E/8KjnZa3WMviRNMx0dy7OWwMHz/7WAN8mzrQhVtNBdZp/OgxgKBuDSxHy
 TwP2dvqhifM13xnwUGhlYNuXav16prpTB7KBg0GqLUI+gW2RXTy6W32lGjkNTvyam/jpkEG+ToUU
 +276DRCPWitqCSe5beVLUr3cAxV7ufzae6oLbbWD88BrxXpt4HbeTrk8nqA28M9wmNYXwPqf4XJ9
 gQ82AfFsIcP730gW2Td/AYait0ePzXUgAAAAAElFTkSuQmCC
Date: Fri, 09 Nov 2018 13:26:50 +0100
In-Reply-To: <83bm72ciyz.fsf@HIDDEN> (Eli Zaretskii's message of "Tue, 06 Nov
 2018 17:40:36 +0200")
Message-ID: <m3r2fu5tdh.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

> After thinking about this and looking at the code, I don't think I
> understand your proposal(s).  Here are the issues that I saw with what
> I think you proposed:
>
>  . The hash table used by the image cache is not a Lisp hash-table, so
>    weakness doesn't apply.  We cache pointers to C structs, so using a
>    Lisp hash-tables would need "some work".

Yeah, it'd have to be rewritten substantially to use Lisp hash tables.

>  . The code you show above is called from redisplay, so it will not be
>    executed as long as a Lisp program (such as the one you show at the
>    beginning of this discussion) runs, and you will still have the OOM
>    killer get you before that code get a chance to clear the cache.

That's true, but calling the cache pruning function from the function
that enters images into the cache would be trivial, I think.

>  . Running such code from GC could be tricky, because freeing images
>    needs to remove references of those images from the glyph matrices,
>    and that cannot be safely done from arbitrary places.

Ah, I see.  Yes, that's a major complication...

> I think if we want to prevent the OOM killer from killing Emacs due to
> many cached images, we should inspect inside cache_image the current
> VM size of the process (like system_process_attributes does), compare
> it to the memory limits, and display a warning and/or signal an error
> when we get dangerously close to the limit.

I think that sounds like a good idea, yes.  Having such a check is good
for Emacs sanity, but we should also have a way for programmers to avoid
ending up in that pathological situation to begin with.

So either a DONT-CACHE parameter to `image-size' or just a docstring
thing.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 9 Nov 2018 12:22:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Nov 09 07:22:16 2018
Received: from localhost ([127.0.0.1]:42284 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gL5nj-00012w-Ow
	for submit <at> debbugs.gnu.org; Fri, 09 Nov 2018 07:22:15 -0500
Received: from hermes.netfonds.no ([80.91.224.195]:49350)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gL5nh-00012k-9K
 for 33275 <at> debbugs.gnu.org; Fri, 09 Nov 2018 07:22:14 -0500
Received: from cm-84.212.221.165.getinternet.no ([84.212.221.165] helo=stories)
 by hermes.netfonds.no with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <larsi@HIDDEN>)
 id 1gL5nc-0001fe-Nv; Fri, 09 Nov 2018 13:22:10 +0100
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
 <83muqncry4.fsf@HIDDEN> <83k1lrcqza.fsf@HIDDEN>
 <m3bm73gwx4.fsf@HIDDEN> <83h8gvcoin.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAElBMVEWHhWtoTzJ8ZTuAdVa8
 tIZjKBaOPNBXAAACdklEQVQ4jW2Uy5LjIAxFZcPsI3D2PJw9D3tPbGXfdjf//ysjSDpTUxVV2RAd
 pCuhJFDos8HmXFI+KceGp3UHokUfJwbWOW19tAoVokPleEEGObkPNsPoP/kZbD61hfeZH2JJ1z4Y
 GJGXNDvny+aMIPZxcjCwPXl7FPs4wmpeWcM19c2PPYPgvWkdNODVU+wlW8ZXVa6XNQ+coBVubr1X
 8z5IozN904vcWsTb/G+rBlgjcbmoXh6rHP42OGRlw+AsXx2fQHt1jdkJxmA1FKWjispaRqc9lNUT
 7GEAKAPkIFBblSNiRgwL7MB+IbLIJcZ4WTKfiBkakLAnBjMIldYgvJCiAQEzDzJkKCX7KIvwKYtQ
 GEgGe8hZFsieZy0SQDYBSoswmcFQohlkCSMIIVgjLESpSd4hx5XA70GMHWRHhqtd7jOAJOBMIjWQ
 QY7aWjGAjF4SVwi5A9a/YooaAqQgnPf8gp5Kys3mqEcnioBtjNmkAB1weB6mPUKrfct+3CLnKUBU
 xqRZl2QgWgzsG/QIItTp4IoK8CEyQXbwuND+pSwDGKumhSLKPV9meBgaqnYXgOEquTFaVRgVTPD4
 dr5+Xz0DvUg10+pmd/E/8KjnZa3WMviRNMx0dy7OWwMHz/7WAN8mzrQhVtNBdZp/OgxgKBuDSxHy
 TwP2dvqhifM13xnwUGhlYNuXav16prpTB7KBg0GqLUI+gW2RXTy6W32lGjkNTvyam/jpkEG+ToUU
 +276DRCPWitqCSe5beVLUr3cAxV7ufzae6oLbbWD88BrxXpt4HbeTrk8nqA28M9wmNYXwPqf4XJ9
 gQ82AfFsIcP730gW2Td/AYait0ePzXUgAAAAAElFTkSuQmCC
Date: Fri, 09 Nov 2018 13:22:08 +0100
In-Reply-To: <83h8gvcoin.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 05 Nov
 2018 21:28:32 +0200")
Message-ID: <m3va565tlb.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

(Sorry for the late response; things got in the way as usual...)

Eli Zaretskii <eliz@HIDDEN> writes:

>> `image-size' didn't use to cache images, but was introduced as an
>> optimisation.
>
> And that optimization had a reason, right?

Yeah, the normal use case when you're doing images is that you're going
to end up displaying it.  And `image-size' has to render the image to
see how big it is, so during normal usage, having `image-size' cache the
image is a big win.

Hm...  Perhaps `image-size' should just have a DONT-CACHE optional
parameter?

> Btw, why doesn't xmalloc return NULL in your case before invoking the
> OOM killer?  Is that expected behavior on GNU/Linux systems?

No, you've got overcommit on Linux, so you often don't get a NULL when
you need it...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 6 Nov 2018 15:40:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Nov 06 10:40:57 2018
Received: from localhost ([127.0.0.1]:37478 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gK3TJ-0003WL-V3
	for submit <at> debbugs.gnu.org; Tue, 06 Nov 2018 10:40:57 -0500
Received: from eggs.gnu.org ([208.118.235.92]:34173)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gK3TE-0003Vt-QU
 for 33275 <at> debbugs.gnu.org; Tue, 06 Nov 2018 10:40:52 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gK3T4-0002L5-Vn
 for 33275 <at> debbugs.gnu.org; Tue, 06 Nov 2018 10:40:43 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:55586)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gK3T4-0002KU-R1; Tue, 06 Nov 2018 10:40:38 -0500
Received: from [176.228.60.248] (port=1731 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gK3T4-0006dl-A5; Tue, 06 Nov 2018 10:40:38 -0500
Date: Tue, 06 Nov 2018 17:40:36 +0200
Message-Id: <83bm72ciyz.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3pnvjh0um.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 18:48:33 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 18:48:33 +0100
> 
> At present, the image cache is purely based on timestamps, if I recall
> correctly?  If we wish to fix this unexpected behaviour, we could also
> look at the size of the image cache and perhaps try to limit it, or if
> we wanted to be really fancy, we could somehow check whether there's any
> references to the images anywhere (beyond the image cache itself).
> 
> I think the latter would mean adding some... fun... to the gc system?
> 
> Or does Emacs have weak hashes?  In which case the image cache could be
> a weak hash and the problem would resolve itself?
> [...]
> Emacs does have weak hashes...  Hm...
> 
> Anyway, there's code that tries to keep the cache small:
> 
>           /* If the number of cached images has grown unusually large,
>              decrease the cache eviction delay (Bug#6230).  */
>           delay = XFIXNUM (Vimage_cache_eviction_delay);
>           if (nimages > 40)
>             delay = 1600 * delay / nimages / nimages;
> 
> But it works solely on the number of images in the cache, and not the
> size of the images.  So it's a heuristic that could be tweaked pretty
> easily, I think?

After thinking about this and looking at the code, I don't think I
understand your proposal(s).  Here are the issues that I saw with what
I think you proposed:

 . The hash table used by the image cache is not a Lisp hash-table, so
   weakness doesn't apply.  We cache pointers to C structs, so using a
   Lisp hash-tables would need "some work".
 . The code you show above is called from redisplay, so it will not be
   executed as long as a Lisp program (such as the one you show at the
   beginning of this discussion) runs, and you will still have the OOM
   killer get you before that code get a chance to clear the cache.
 . Running such code from GC could be tricky, because freeing images
   needs to remove references of those images from the glyph matrices,
   and that cannot be safely done from arbitrary places.
 . I don't think you can ensure GC will run during your program
   anyway, because most of the memory allocated by your program is not
   used to cons Lisp objects, so maybe_gc called by the Lisp
   interpreter will most probably decide there's no need for GC and do
   nothing.  Therefore, I don't think that avoiding to cache the
   images would prevent the OOM situation in your case.

I think if we want to prevent the OOM killer from killing Emacs due to
many cached images, we should inspect inside cache_image the current
VM size of the process (like system_process_attributes does), compare
it to the memory limits, and display a warning and/or signal an error
when we get dangerously close to the limit.  If we want to avoid
similar situations with other objects, we could do these tests inside
xmalloc and mmap_alloc/mmap_realloc.  (We used to do something similar
in the past, but that code needed a libc hook for memory allocation
from the OS, something I understand we cannot rely on now.)

IOW, I don't see how we could prevent being OOM-killed, except if we
monitor the committed memory at the point where it is allocated, as
otherwise the opportunistic memory allocation strategy can always trip
us.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 19:28:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 14:28:54 2018
Received: from localhost ([127.0.0.1]:35639 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJkYM-0000w1-V5
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:28:54 -0500
Received: from eggs.gnu.org ([208.118.235.92]:49725)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJkYI-0000vl-05
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:28:49 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJkY7-0006Og-Mt
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:28:40 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:34108)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJkY7-0006Ob-JF; Mon, 05 Nov 2018 14:28:35 -0500
Received: from [176.228.60.248] (port=2388 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJkY7-0004Dn-6x; Mon, 05 Nov 2018 14:28:35 -0500
Date: Mon, 05 Nov 2018 21:28:32 +0200
Message-Id: <83h8gvcoin.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3bm73gwx4.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 20:13:27 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
 <83muqncry4.fsf@HIDDEN> <83k1lrcqza.fsf@HIDDEN> <m3bm73gwx4.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 20:13:27 +0100
> 
> If I understand you correctly, you seem to favour introducing code that
> will break code that's working today (if you insert an image into a
> buffer today in a memory-pressure situation, that won't signal an error
> today) instead of tweaking a non-documented ad-hoc caching strategy (by
> taking the image cache size into consideration), or fixing the caching
> (by using weak hash tables), and I'm not quite sure why.

Because your use case is not important enough to change image caching
strategy that works well for many years.

> `image-size' didn't use to cache images, but was introduced as an
> optimisation.

And that optimization had a reason, right?

Btw, why doesn't xmalloc return NULL in your case before invoking the
OOM killer?  Is that expected behavior on GNU/Linux systems?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 19:25:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 14:25:08 2018
Received: from localhost ([127.0.0.1]:35635 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJkUj-0000r5-0d
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:25:08 -0500
Received: from eggs.gnu.org ([208.118.235.92]:48610)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJkUe-0000qS-2L
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:25:03 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJkUQ-0002bZ-U2
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:24:53 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:34047)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJkUQ-0002bU-Pl; Mon, 05 Nov 2018 14:24:46 -0500
Received: from [176.228.60.248] (port=2142 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJkUQ-0008Ru-CO; Mon, 05 Nov 2018 14:24:46 -0500
Date: Mon, 05 Nov 2018 21:24:42 +0200
Message-Id: <83in1bcop1.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3ftwfgx8e.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 20:06:41 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <83lg67crrq.fsf@HIDDEN> <m3ftwfgx8e.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 20:06:41 +0100
> 
> In general, if you don't have a reference to an object, you can be
> pretty sure that Emacs is going to garbage-collect it.

Of course.  But you cannot control when that GC will happen.

> > That it doesn't is just sheer luck: the way we manage buffer memory is
> > special.  With any other Lisp object, it could well grow
> > uncontrollably.
> 
> What other non-referenced Lisp object can realistically make Emacs grow
> in this way?

Theoretically, anything.  In practice, I've seen such situations with
fonts.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 19:13:38 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 14:13:38 2018
Received: from localhost ([127.0.0.1]:35621 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJkJe-0000Xb-7A
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:13:38 -0500
Received: from ryder.getmail.no ([84.210.184.14]:42825)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJkJc-0000XN-Jp
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:13:37 -0500
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id E2E9562C86;
 Mon,  5 Nov 2018 20:13:28 +0100 (CET)
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id bhxOkkO24Vop; Mon,  5 Nov 2018 20:13:28 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id 77E6262C87;
 Mon,  5 Nov 2018 20:13:28 +0100 (CET)
X-Virus-Scanned: amavisd-new at ryder.getmail.no
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id II4H5AMbQaO7; Mon,  5 Nov 2018 20:13:28 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by ryder.getmail.no (Postfix) with ESMTPSA id 3065862C86;
 Mon,  5 Nov 2018 20:13:28 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
 <83muqncry4.fsf@HIDDEN> <83k1lrcqza.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAIVBMVEX+/thGAw9eUVv+/93+
 //fq3LH58cKsqZr+/dP9+s7+/99hXvwTAAACQklEQVQ4jW3TMY/aMBQHcEup2pQtLJ1tiYvUCSlt
 VWV2qyoTQnLCdUI6Ac1NlVVjcVOWqmGrSgtnpui4tJw/ZZ8dc+QQb0rej/d37ATE0ulJMVsobd+n
 x2vkLieSBEGXyIYmDtZqygNTXdJzv7OQKzXDFgJChAszoNS6GWjJBNmBDT5Al4TNCMqhr4ZBcByZ
 O1DtJDuilAHoqz/4BNYONi2ArEszYqD88RQW0DNQnoO1nTiJWiyhiWDgCLiBEgTdlo9RXUKwXXx5
 gO/YPSjjBkbQUqhsQS9vg4/KJXbnlw1N3heAJUCBLEBHMolfkfC29C0gVLgJIjnu8QvoAxQIijfQ
 DTme86+lXcPW0PZlKGZkNLSwNG0P/bIPK8R1HQ6yn27C63voBYalRaIpFUWGfFjERHme95aEnA+S
 e0oZypEPZaK8OI64yOTnj78pzaFxiDIgxUxe1hVN2hBF8W4mplJkCw0AJ2H3AUk61n9ZmonxTfJp
 FSFY3E7EsYaC7zlPt3q/6ts+ACQZmLCMkHDwfAxH0WzQ8yzUjHfJXNfjl+MDRHEDmUy3u3rcP4EK
 /iu53r3L+4V/OCsbpQGuk11HuX08rgFZ+TZ56BTwGgoXdYB0FOW6qOx78M3EzsJGMjFd6Y6unlVm
 j8glVRIqvdKR3tGkDTMDbKv7+g2lVFelOSoownGIxfD1v4uHkJp6jyJzVnv4OOFEhuIu/MAtJE3U
 Psu45DK70d+otEBRBI9rLmrGxoN7+lgAcev2CZztU8SuzsN/F61wm8Zh404AAAAASUVORK5CYII=
Date: Mon, 05 Nov 2018 20:13:27 +0100
In-Reply-To: <83k1lrcqza.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 05 Nov
 2018 20:35:21 +0200")
Message-ID: <m3bm73gwx4.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

> We still could use the memory limit calculated by get_lim_data in
> strategic places, like cache_image, to catch situations where the
> cache size exceeds some fraction of what ret_lim_data returns.  Would
> that work?

If I understand you correctly, you seem to favour introducing code that
will break code that's working today (if you insert an image into a
buffer today in a memory-pressure situation, that won't signal an error
today) instead of tweaking a non-documented ad-hoc caching strategy (by
taking the image cache size into consideration), or fixing the caching
(by using weak hash tables), and I'm not quite sure why.

`image-size' didn't use to cache images, but was introduced as an
optimisation.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 19:06:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 14:06:55 2018
Received: from localhost ([127.0.0.1]:35616 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJkD9-0000OG-Db
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:06:55 -0500
Received: from ryder.getmail.no ([84.210.184.14]:42278)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJkD5-0000O1-TY
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 14:06:54 -0500
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id 02FB862C86;
 Mon,  5 Nov 2018 20:06:43 +0100 (CET)
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id gdRvsPZ94HYt; Mon,  5 Nov 2018 20:06:41 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id 9EF9062C87;
 Mon,  5 Nov 2018 20:06:41 +0100 (CET)
X-Virus-Scanned: amavisd-new at ryder.getmail.no
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id ub5QvtFJto0G; Mon,  5 Nov 2018 20:06:41 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by ryder.getmail.no (Postfix) with ESMTPSA id 6C3A762C83;
 Mon,  5 Nov 2018 20:06:41 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <83lg67crrq.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAIVBMVEX+/thGAw9eUVv+/93+
 //fq3LH58cKsqZr+/dP9+s7+/99hXvwTAAACQklEQVQ4jW3TMY/aMBQHcEup2pQtLJ1tiYvUCSlt
 VWV2qyoTQnLCdUI6Ac1NlVVjcVOWqmGrSgtnpui4tJw/ZZ8dc+QQb0rej/d37ATE0ulJMVsobd+n
 x2vkLieSBEGXyIYmDtZqygNTXdJzv7OQKzXDFgJChAszoNS6GWjJBNmBDT5Al4TNCMqhr4ZBcByZ
 O1DtJDuilAHoqz/4BNYONi2ArEszYqD88RQW0DNQnoO1nTiJWiyhiWDgCLiBEgTdlo9RXUKwXXx5
 gO/YPSjjBkbQUqhsQS9vg4/KJXbnlw1N3heAJUCBLEBHMolfkfC29C0gVLgJIjnu8QvoAxQIijfQ
 DTme86+lXcPW0PZlKGZkNLSwNG0P/bIPK8R1HQ6yn27C63voBYalRaIpFUWGfFjERHme95aEnA+S
 e0oZypEPZaK8OI64yOTnj78pzaFxiDIgxUxe1hVN2hBF8W4mplJkCw0AJ2H3AUk61n9ZmonxTfJp
 FSFY3E7EsYaC7zlPt3q/6ts+ACQZmLCMkHDwfAxH0WzQ8yzUjHfJXNfjl+MDRHEDmUy3u3rcP4EK
 /iu53r3L+4V/OCsbpQGuk11HuX08rgFZ+TZ56BTwGgoXdYB0FOW6qOx78M3EzsJGMjFd6Y6unlVm
 j8glVRIqvdKR3tGkDTMDbKv7+g2lVFelOSoownGIxfD1v4uHkJp6jyJzVnv4OOFEhuIu/MAtJE3U
 Psu45DK70d+otEBRBI9rLmrGxoN7+lgAcev2CZztU8SuzsN/F61wm8Zh404AAAAASUVORK5CYII=
Date: Mon, 05 Nov 2018 20:06:41 +0100
In-Reply-To: <83lg67crrq.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 05 Nov
 2018 20:18:17 +0200")
Message-ID: <m3ftwfgx8e.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> It would be reasonable if the code had asked for those images to be kept
>> in memory, but it hasn't.  Emacs decides that on its own without saying
>> that it's doing that.
>
> You cannot assume anything about memory management while a Lisp
> program runs.

In general, if you don't have a reference to an object, you can be
pretty sure that Emacs is going to garbage-collect it.  If you can't
assume that, then programming in Emacs Lisp becomes impossible.

Fortunately that's not the case, and programming in Emacs Lisp is nice
and easy.

>> It'd be similarly surprising if
>> 
>> (dolist (file (directory-files "/directory/with/many/images" t "png$"))
>>   (with-temp-buffer
>>     (insert-file-contents-literally file)))
>> 
>> were to lead to Emacs growing uncontrollably.
>
> That it doesn't is just sheer luck: the way we manage buffer memory is
> special.  With any other Lisp object, it could well grow
> uncontrollably.

What other non-referenced Lisp object can realistically make Emacs grow
in this way?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 18:35:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 13:35:54 2018
Received: from localhost ([127.0.0.1]:35588 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJjj7-0007vL-V8
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:35:54 -0500
Received: from eggs.gnu.org ([208.118.235.92]:57795)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJjj5-0007v6-LC
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:35:51 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJjiu-00046C-M7
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:35:42 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:32857)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJjiq-0003yn-Pk; Mon, 05 Nov 2018 13:35:38 -0500
Received: from [176.228.60.248] (port=3075 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJjij-0006OB-8t; Mon, 05 Nov 2018 13:35:33 -0500
Date: Mon, 05 Nov 2018 20:35:21 +0200
Message-Id: <83k1lrcqza.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: larsi@HIDDEN
In-reply-to: <83muqncry4.fsf@HIDDEN> (message from Eli Zaretskii on Mon, 05
 Nov 2018 20:14:27 +0200)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
 <83muqncry4.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Mon, 05 Nov 2018 20:14:27 +0200
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> 
> I don't see why we should tweak our cache management for the benefit
> of exceptional use case.  Last time we did it (with fonts) we are
> still leaking the wounds.
        ^^^^^^^
I meant "licking", of course ;-)

> I think it should be enough to catch excess memory usage and signal an
> error.  I hope this is possible.

I guess we lost the ability to do that automatically when we stopped
using vm-limit.c on modern platforms?

We still could use the memory limit calculated by get_lim_data in
strategic places, like cache_image, to catch situations where the
cache size exceeds some fraction of what ret_lim_data returns.  Would
that work?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 18:18:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 13:18:55 2018
Received: from localhost ([127.0.0.1]:35577 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJjSe-0007WL-6U
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:18:55 -0500
Received: from eggs.gnu.org ([208.118.235.92]:51677)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJjSY-0007W7-TL
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:18:50 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJjSN-0005ZX-TX
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:18:41 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:60729)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJjSK-0005X0-Ia; Mon, 05 Nov 2018 13:18:33 -0500
Received: from [176.228.60.248] (port=2004 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJjSA-0007X6-OJ; Mon, 05 Nov 2018 13:18:24 -0500
Date: Mon, 05 Nov 2018 20:18:17 +0200
Message-Id: <83lg67crrq.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3tvkvh1eg.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 18:36:39 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN> <m3tvkvh1eg.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 18:36:39 +0100
> 
> It would be reasonable if the code had asked for those images to be kept
> in memory, but it hasn't.  Emacs decides that on its own without saying
> that it's doing that.

You cannot assume anything about memory management while a Lisp
program runs.

> It'd be similarly surprising if
> 
> (dolist (file (directory-files "/directory/with/many/images" t "png$"))
>   (with-temp-buffer
>     (insert-file-contents-literally file)))
> 
> were to lead to Emacs growing uncontrollably.

That it doesn't is just sheer luck: the way we manage buffer memory is
special.  With any other Lisp object, it could well grow
uncontrollably.  Our way of keeping that in check is to signal
memory-full error when we are about to run out of memory.  Not sure
yet why this doesn't happen in your original recipe.

> > I think we should try to add some code that would call
> > display_malloc_warning and/or memory_full, before the system runs out
> > of memory.  That should be enough to prevent OOM in such cases.  Can
> > you spot why we never called memory_full in this case?
> 
> Sorry, I'm not familiar with the memory_full stuff...

OK, I will try to look into this soon.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 18:14:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 13:14:57 2018
Received: from localhost ([127.0.0.1]:35572 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJjOm-0007QI-Li
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:14:55 -0500
Received: from eggs.gnu.org ([208.118.235.92]:50176)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJjOh-0007Q1-AN
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:14:50 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJjOX-0000nS-OO
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 13:14:42 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:60674)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJjOX-0000n8-J6; Mon, 05 Nov 2018 13:14:37 -0500
Received: from [176.228.60.248] (port=1767 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJjOW-0007DH-Ki; Mon, 05 Nov 2018 13:14:37 -0500
Date: Mon, 05 Nov 2018 20:14:27 +0200
Message-Id: <83muqncry4.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3pnvjh0um.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 18:48:33 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN> <m3pnvjh0um.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 18:48:33 +0100
> 
> Anyway, there's code that tries to keep the cache small:
> 
> 	  /* If the number of cached images has grown unusually large,
> 	     decrease the cache eviction delay (Bug#6230).  */
> 	  delay = XFIXNUM (Vimage_cache_eviction_delay);
> 	  if (nimages > 40)
> 	    delay = 1600 * delay / nimages / nimages;
> 
> But it works solely on the number of images in the cache, and not the
> size of the images.  So it's a heuristic that could be tweaked pretty
> easily, I think?

I don't see why we should tweak our cache management for the benefit
of exceptional use case.  Last time we did it (with fonts) we are
still leaking the wounds.

I think it should be enough to catch excess memory usage and signal an
error.  I hope this is possible.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 17:48:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 12:48:53 2018
Received: from localhost ([127.0.0.1]:35555 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJizZ-0006pU-DN
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:48:53 -0500
Received: from ryder.getmail.no ([84.210.184.14]:35254)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJizU-0006oy-Dc
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:48:48 -0500
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id 49F5262C86;
 Mon,  5 Nov 2018 18:48:34 +0100 (CET)
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id 4LTxwgA4gwGJ; Mon,  5 Nov 2018 18:48:33 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id CDA8F62C87;
 Mon,  5 Nov 2018 18:48:33 +0100 (CET)
X-Virus-Scanned: amavisd-new at ryder.getmail.no
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id YDlDPCmlSzp3; Mon,  5 Nov 2018 18:48:33 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by ryder.getmail.no (Postfix) with ESMTPSA id A2BF862C86;
 Mon,  5 Nov 2018 18:48:33 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
 <m3tvkvh1eg.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEW6gTlVGIFdHox+SDZD
 EmhXGoJRFnz8xUXqvZ6yAAACO0lEQVQ4jW2UzY+DIBDFaa3bM0a8S2rPJG7iVbvUO0Q8S4NzbtKD
 //4OoK2bLP1I01/ePN4wQsBo/t8iBdX/IjIyzTUH6Yquu1wenF+rWwDK6IpzJ9fFr73rV0WGlVzv
 4NYNQbGWUoYpzuUPea/vewBGzYbry5Scl+d5we8ljR6mmtF9mGqRvAR5EdGc+hVYzpVr67p+hXdz
 vFUeaMN0VSBAxQZ4BHNmrAf4r2fNocNgHlhuQEbwRwFQMcgnEsyfQmzAWKqYzdvlvb56ju4EwDJr
 5Q6gwnuABcOyy/QtMdtCPIjmwGBmWd4KQYSoSf1qTjJ4OABgmUOQkAZB0qS9YbEU2McwiXYDXz3N
 gjnM5q1IEJzkCizzyT8g7SieKilgBkvzCYGIAEtRHRRWKa+oxVZqxFreA8ZZtqL2wJsfu5JtwObT
 spxD7ldzkAVl5q1I6giei/fIqhAQGJ7HD4kdWdJebR4Acbt4JqG7JTUbQPN2A6mk2BNiLX0rsI8e
 YMBQCoHNA8DsCLAlJgQsEMjoQWIpFZOz0Vo3fcCxi8DSwlLnzUUEviU67spirwg5n9czxxzrdmcz
 tFtLfKmg8Mkt24/PSZZMmwBmjaVEnSTE9yS9lVT7Xo3A2OBz15hkNz6Yg9KhfYYOPkOpyqxgVvtJ
 TPtKr9t9mGH6gEPPowIKzfPdw/kVJ9EDRsfHHS7j7X65Ph7XfgNWfy4AvyKwOKJMDX3nXH7v3ICf
 YlMA0yX+rP5eMgEYpXc3UIWv8hctdBMsfua7RgAAAABJRU5ErkJggg==
Date: Mon, 05 Nov 2018 18:48:33 +0100
In-Reply-To: <m3tvkvh1eg.fsf@HIDDEN> (Lars Ingebrigtsen's message of "Mon,
 05 Nov 2018 18:36:39 +0100")
Message-ID: <m3pnvjh0um.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Lars Ingebrigtsen <larsi@HIDDEN> writes:

> Or does Emacs have weak hashes?  In which case the image cache could be
> a weak hash and the problem would resolve itself?

Emacs does have weak hashes...  Hm...

Anyway, there's code that tries to keep the cache small:

	  /* If the number of cached images has grown unusually large,
	     decrease the cache eviction delay (Bug#6230).  */
	  delay = XFIXNUM (Vimage_cache_eviction_delay);
	  if (nimages > 40)
	    delay = 1600 * delay / nimages / nimages;

But it works solely on the number of images in the cache, and not the
size of the images.  So it's a heuristic that could be tweaked pretty
easily, I think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 17:36:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 12:36:48 2018
Received: from localhost ([127.0.0.1]:35530 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJinw-0006Xj-Fp
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:36:48 -0500
Received: from lamora.getmail.no ([84.210.184.7]:60571)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJinu-0006XW-NH
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:36:47 -0500
Received: from localhost (localhost [127.0.0.1])
 by lamora.getmail.no (Postfix) with ESMTP id 18CD56083C;
 Mon,  5 Nov 2018 18:36:40 +0100 (CET)
Received: from lamora.getmail.no ([127.0.0.1])
 by localhost (lamora.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id OcUNj5dXngkp; Mon,  5 Nov 2018 18:36:39 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by lamora.getmail.no (Postfix) with ESMTP id 9507C6083E;
 Mon,  5 Nov 2018 18:36:39 +0100 (CET)
X-Virus-Scanned: amavisd-new at lamora.get.c.bitbit.net
Received: from lamora.getmail.no ([127.0.0.1])
 by localhost (lamora.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id 4mkZH3iNi6IM; Mon,  5 Nov 2018 18:36:39 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by lamora.getmail.no (Postfix) with ESMTPSA id 649906083C;
 Mon,  5 Nov 2018 18:36:39 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
 <m3y3a7h47s.fsf@HIDDEN> <83o9b3cu4f.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEW6gTlVGIFdHox+SDZD
 EmhXGoJRFnz8xUXqvZ6yAAACO0lEQVQ4jW2UzY+DIBDFaa3bM0a8S2rPJG7iVbvUO0Q8S4NzbtKD
 //4OoK2bLP1I01/ePN4wQsBo/t8iBdX/IjIyzTUH6Yquu1wenF+rWwDK6IpzJ9fFr73rV0WGlVzv
 4NYNQbGWUoYpzuUPea/vewBGzYbry5Scl+d5we8ljR6mmtF9mGqRvAR5EdGc+hVYzpVr67p+hXdz
 vFUeaMN0VSBAxQZ4BHNmrAf4r2fNocNgHlhuQEbwRwFQMcgnEsyfQmzAWKqYzdvlvb56ju4EwDJr
 5Q6gwnuABcOyy/QtMdtCPIjmwGBmWd4KQYSoSf1qTjJ4OABgmUOQkAZB0qS9YbEU2McwiXYDXz3N
 gjnM5q1IEJzkCizzyT8g7SieKilgBkvzCYGIAEtRHRRWKa+oxVZqxFreA8ZZtqL2wJsfu5JtwObT
 spxD7ldzkAVl5q1I6giei/fIqhAQGJ7HD4kdWdJebR4Acbt4JqG7JTUbQPN2A6mk2BNiLX0rsI8e
 YMBQCoHNA8DsCLAlJgQsEMjoQWIpFZOz0Vo3fcCxi8DSwlLnzUUEviU67spirwg5n9czxxzrdmcz
 tFtLfKmg8Mkt24/PSZZMmwBmjaVEnSTE9yS9lVT7Xo3A2OBz15hkNz6Yg9KhfYYOPkOpyqxgVvtJ
 TPtKr9t9mGH6gEPPowIKzfPdw/kVJ9EDRsfHHS7j7X65Ph7XfgNWfy4AvyKwOKJMDX3nXH7v3ICf
 YlMA0yX+rP5eMgEYpXc3UIWv8hctdBMsfua7RgAAAABJRU5ErkJggg==
Date: Mon, 05 Nov 2018 18:36:39 +0100
In-Reply-To: <83o9b3cu4f.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 05 Nov
 2018 19:27:28 +0200")
Message-ID: <m3tvkvh1eg.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

> I think it's reasonable to expect 2GB worth of image files to take
> several times that in memory.  (Btw, visiting a 16GB file usually
> takes twice that much during insert-file-contents.)

It would be reasonable if the code had asked for those images to be kept
in memory, but it hasn't.  Emacs decides that on its own without saying
that it's doing that.

It'd be similarly surprising if

(dolist (file (directory-files "/directory/with/many/images" t "png$"))
  (with-temp-buffer
    (insert-file-contents-literally file)))

were to lead to Emacs growing uncontrollably.

At present, the image cache is purely based on timestamps, if I recall
correctly?  If we wish to fix this unexpected behaviour, we could also
look at the size of the image cache and perhaps try to limit it, or if
we wanted to be really fancy, we could somehow check whether there's any
references to the images anywhere (beyond the image cache itself).

I think the latter would mean adding some... fun... to the gc system?

Or does Emacs have weak hashes?  In which case the image cache could be
a weak hash and the problem would resolve itself?

> I think we should try to add some code that would call
> display_malloc_warning and/or memory_full, before the system runs out
> of memory.  That should be enough to prevent OOM in such cases.  Can
> you spot why we never called memory_full in this case?

Sorry, I'm not familiar with the memory_full stuff...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 17:28:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 12:28:01 2018
Received: from localhost ([127.0.0.1]:35520 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJifM-0006Ju-0K
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:28:01 -0500
Received: from eggs.gnu.org ([208.118.235.92]:60105)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJifH-0006Je-Bu
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:27:54 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJif4-0003wA-Bt
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 12:27:43 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:59541)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJif1-0003nj-1T; Mon, 05 Nov 2018 12:27:36 -0500
Received: from [176.228.60.248] (port=2737 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJiey-0000qH-Rn; Mon, 05 Nov 2018 12:27:34 -0500
Date: Mon, 05 Nov 2018 19:27:28 +0200
Message-Id: <83o9b3cu4f.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3y3a7h47s.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 17:35:51 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN> <m3y3a7h47s.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 17:35:51 +0100
> 
> >> If you visit a, say, 16GB file, then, indeed, it's expected that Emacs
> >> grows to 16GB (or more).  But if you're just calling `image-size' on a
> >> bunch of files (each of which is a lot smaller than that), it's rather
> >> unexpected that that's going to blow up Emacs.
> >
> > What was the sum of sizes of all the files you used to reproduce the
> > problem?
> 
> I think it's sufficient to run Emacs over 2GB worth of .png files to
> make Emacs grow to 16GB (which will kill it on this laptop).
> 
> But it'll vary on how well-compressed the .png files are, I guess,
> because I think Emacs caches the, er, pixmaps and not the files?  I
> don't remember.

I think it's reasonable to expect 2GB worth of image files to take
several times that in memory.  (Btw, visiting a 16GB file usually
takes twice that much during insert-file-contents.)

I think we should try to add some code that would call
display_malloc_warning and/or memory_full, before the system runs out
of memory.  That should be enough to prevent OOM in such cases.  Can
you spot why we never called memory_full in this case?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 16:36:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 11:36:10 2018
Received: from localhost ([127.0.0.1]:35506 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJhrC-0005BD-GD
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:36:10 -0500
Received: from bouvier.getmail.no ([84.210.184.8]:57510)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJhr8-0005Af-1U
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:36:05 -0500
Received: from localhost (localhost [127.0.0.1])
 by bouvier.getmail.no (Postfix) with ESMTP id 3D7F24002D;
 Mon,  5 Nov 2018 17:35:52 +0100 (CET)
Received: from bouvier.getmail.no ([127.0.0.1])
 by localhost (bouvier.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id V3dYAjJcGtdJ; Mon,  5 Nov 2018 17:35:51 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by bouvier.getmail.no (Postfix) with ESMTP id BF5A74002E;
 Mon,  5 Nov 2018 17:35:51 +0100 (CET)
X-Virus-Scanned: amavisd-new at bouvier.get.c.bitbit.net
Received: from bouvier.getmail.no ([127.0.0.1])
 by localhost (bouvier.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id Ftv9EFfbjJik; Mon,  5 Nov 2018 17:35:51 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by bouvier.getmail.no (Postfix) with ESMTPSA id 8BC2D4002D;
 Mon,  5 Nov 2018 17:35:51 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN> <83sh0fcx2r.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAFVBMVEVYQhQsJQN4TimCbkLl
 66uAWDOunm1MQAQZAAACX0lEQVQ4jW2TTXObMBCGVyL2WaSTnAkMOWNv0bmdaHR2pCpnB+H9/z+h
 rwTJtE2XGZt5n/0WIre6/xp5kYq8OHGrX/HvBD8EbXX/CSvA+46aLwCZ5HwxdPcvQFIhyl9CECFL
 9+jcj79kX8Gk0MnPP1T7LBWcCF2+f+qykC015OX0za/u+um/ZrsV96cjXl42b+8t81qB87VVDxXv
 4uPQyAdAUy5HHhg20YGSXR295Vk18NRxbmbIHVEoPnRp+AnD+cPMfI5TRyN1zDHAgxWKvzfTpWfu
 wtTrmpJIxzL4iYapF01gTdFHIsSiq4n6Xsuko5VqTMrQULo9d30jUyO7ZTKGfpXRFuoauX4CIUOH
 Cm6BjpJ3kAFaGlMFke5kMVW3IKT0mN4A5tgdZbnv2X6k0iml7+5ljop4MsE64Vr8vgdIawZo29Yc
 cW6HLPmG9gtIMcUhBaJjpgERFmCT5xR1WYQ2LdHB4hTPfQCYUxpMPxOyUbHbleYQxgLCyAOqGFJ4
 Wuy2TyFFLvkkK+iQSamSL5VU0FkWAAOdVKsVHUoNRo0o15KqZGtNDxBCyRPGKJcySTFVKun+UgcZ
 rVxqbQCNIERUEHpeSq94Si4UekhdBZGX6m1a1Ddo6yE9jdtSsrCq87UKD+Z43QALz+hXGV2YIZ3i
 DuypLqTUMTjw5QPES+kUo0MuK1jSZnN8qjOYtmtLfbpt7vwJJnnFF2Ipl10BML9uYz9K7kwjJDuY
 4w7ucx6SLWBLldJpj7C+S9bvIHHiHWjnzsy4UTecd1m638Ezbhe+NMInuF/X9w3gAma27jc2rfja
 fW4ALQAAAABJRU5ErkJggg==
Date: Mon, 05 Nov 2018 17:35:51 +0100
In-Reply-To: <83sh0fcx2r.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 05 Nov
 2018 18:23:40 +0200")
Message-ID: <m3y3a7h47s.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Lars Ingebrigtsen <larsi@HIDDEN>
>> Cc: 33275 <at> debbugs.gnu.org
>> Date: Mon, 05 Nov 2018 17:18:47 +0100
>> 
>> If you visit a, say, 16GB file, then, indeed, it's expected that Emacs
>> grows to 16GB (or more).  But if you're just calling `image-size' on a
>> bunch of files (each of which is a lot smaller than that), it's rather
>> unexpected that that's going to blow up Emacs.
>
> What was the sum of sizes of all the files you used to reproduce the
> problem?

I think it's sufficient to run Emacs over 2GB worth of .png files to
make Emacs grow to 16GB (which will kill it on this laptop).

But it'll vary on how well-compressed the .png files are, I guess,
because I think Emacs caches the, er, pixmaps and not the files?  I
don't remember.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 16:24:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 11:24:05 2018
Received: from localhost ([127.0.0.1]:35488 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJhfU-0004sT-6H
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:24:05 -0500
Received: from eggs.gnu.org ([208.118.235.92]:34764)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJhfP-0004sE-2H
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:23:58 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJhfF-0003M2-Sv
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:23:49 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:57918)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJhfD-0003H3-HS; Mon, 05 Nov 2018 11:23:44 -0500
Received: from [176.228.60.248] (port=2762 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJhfD-0006wH-7u; Mon, 05 Nov 2018 11:23:43 -0500
Date: Mon, 05 Nov 2018 18:23:40 +0200
Message-Id: <83sh0fcx2r.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m336sfijko.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 17:18:47 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
 <m336sfijko.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: 33275 <at> debbugs.gnu.org
> Date: Mon, 05 Nov 2018 17:18:47 +0100
> 
> If you visit a, say, 16GB file, then, indeed, it's expected that Emacs
> grows to 16GB (or more).  But if you're just calling `image-size' on a
> bunch of files (each of which is a lot smaller than that), it's rather
> unexpected that that's going to blow up Emacs.

What was the sum of sizes of all the files you used to reproduce the
problem?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 16:19:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 11:19:01 2018
Received: from localhost ([127.0.0.1]:35476 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJhae-0004kH-SH
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:19:01 -0500
Received: from ryder.getmail.no ([84.210.184.14]:53694)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJhac-0004k0-5q
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:18:59 -0500
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id 7996362C8A;
 Mon,  5 Nov 2018 17:18:48 +0100 (CET)
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id oGvfj7Lv1MK3; Mon,  5 Nov 2018 17:18:48 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by ryder.getmail.no (Postfix) with ESMTP id E386562C89;
 Mon,  5 Nov 2018 17:18:47 +0100 (CET)
X-Virus-Scanned: amavisd-new at ryder.getmail.no
Received: from ryder.getmail.no ([127.0.0.1])
 by localhost (ryder.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id BinrjQqDg4EU; Mon,  5 Nov 2018 17:18:47 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by ryder.getmail.no (Postfix) with ESMTPSA id B931062C8A;
 Mon,  5 Nov 2018 17:18:47 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN> <83va5bcxxu.fsf@HIDDEN>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAFVBMVEVYQhQsJQN4TimCbkLl
 66uAWDOunm1MQAQZAAACX0lEQVQ4jW2TTXObMBCGVyL2WaSTnAkMOWNv0bmdaHR2pCpnB+H9/z+h
 rwTJtE2XGZt5n/0WIre6/xp5kYq8OHGrX/HvBD8EbXX/CSvA+46aLwCZ5HwxdPcvQFIhyl9CECFL
 9+jcj79kX8Gk0MnPP1T7LBWcCF2+f+qykC015OX0za/u+um/ZrsV96cjXl42b+8t81qB87VVDxXv
 4uPQyAdAUy5HHhg20YGSXR295Vk18NRxbmbIHVEoPnRp+AnD+cPMfI5TRyN1zDHAgxWKvzfTpWfu
 wtTrmpJIxzL4iYapF01gTdFHIsSiq4n6Xsuko5VqTMrQULo9d30jUyO7ZTKGfpXRFuoauX4CIUOH
 Cm6BjpJ3kAFaGlMFke5kMVW3IKT0mN4A5tgdZbnv2X6k0iml7+5ljop4MsE64Vr8vgdIawZo29Yc
 cW6HLPmG9gtIMcUhBaJjpgERFmCT5xR1WYQ2LdHB4hTPfQCYUxpMPxOyUbHbleYQxgLCyAOqGFJ4
 Wuy2TyFFLvkkK+iQSamSL5VU0FkWAAOdVKsVHUoNRo0o15KqZGtNDxBCyRPGKJcySTFVKun+UgcZ
 rVxqbQCNIERUEHpeSq94Si4UekhdBZGX6m1a1Ddo6yE9jdtSsrCq87UKD+Z43QALz+hXGV2YIZ3i
 DuypLqTUMTjw5QPES+kUo0MuK1jSZnN8qjOYtmtLfbpt7vwJJnnFF2Ipl10BML9uYz9K7kwjJDuY
 4w7ucx6SLWBLldJpj7C+S9bvIHHiHWjnzsy4UTecd1m638Ezbhe+NMInuF/X9w3gAma27jc2rfja
 fW4ALQAAAABJRU5ErkJggg==
Date: Mon, 05 Nov 2018 17:18:47 +0100
In-Reply-To: <83va5bcxxu.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 05 Nov
 2018 18:05:01 +0200")
Message-ID: <m336sfijko.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

> I'm okay with saying something about this in the docs, but frankly,
> you could do the same in any number of ways, like by visiting a file
> whose size is larger than the VM you allow your system to use, right?
> So there's a potential for causing such problems in many different
> ways, and warning about that only in the doc string of image-size
> seems insufficient, to say the least...

If you visit a, say, 16GB file, then, indeed, it's expected that Emacs
grows to 16GB (or more).  But if you're just calling `image-size' on a
bunch of files (each of which is a lot smaller than that), it's rather
unexpected that that's going to blow up Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at 33275 <at> debbugs.gnu.org:


Received: (at 33275) by debbugs.gnu.org; 5 Nov 2018 16:05:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 11:05:49 2018
Received: from localhost ([127.0.0.1]:35468 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJhNp-0004Q4-Nn
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:05:49 -0500
Received: from eggs.gnu.org ([208.118.235.92]:55629)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gJhNk-0004Po-Nx
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:05:44 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gJhNG-0008Ob-ET
 for 33275 <at> debbugs.gnu.org; Mon, 05 Nov 2018 11:05:22 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:57387)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gJhNC-0008Is-Kb; Mon, 05 Nov 2018 11:05:08 -0500
Received: from [176.228.60.248] (port=1457 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gJhNA-00058R-TM; Mon, 05 Nov 2018 11:05:06 -0500
Date: Mon, 05 Nov 2018 18:05:01 +0200
Message-Id: <83va5bcxxu.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <m3bm73ipnh.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 05 Nov 2018 15:07:30 +0100)
Subject: Re: bug#33275: 27.0.50; Image cache pruning
References: <m3bm73ipnh.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: 33275
Cc: 33275 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Date: Mon, 05 Nov 2018 15:07:30 +0100
> 
> So after trying various things, it turned out that you can kill Emacs by
> doing this:
> 
> (dolist (file (directory-files "/directory/with/many/images" t "png$"))
>   (image-size (create-image file) t))
> 
> and this is because `image-size' puts the image into the image cache.
> Calling `clear-image-cache' releases the memory.
> 
> I think putting it into the cache is the right thing to do, because if
> you're calling `image-size' on a file, then nine out of ten times you
> want to display it, and my use case here is atypical (batch-processing
> of a large number of huge images), but it's still not optimal that you
> can crash Emacs in this way.

Doesn't this mean that your system's VM is misconfigured?  How is it
Emacs's fault that you've run a program that gobs so much memory as to
trigger the OS's OOM killer?

> So perhaps the cache should be pruned more aggressively according to
> some smart algorithm or other?  Or, at the very least, an
> innocuous-looking function like `image-size' should note in the doc
> string should note what's going on and recommend calling `image-flush'
> afterwards if we don't want to cache the image?

I'm okay with saying something about this in the docs, but frankly,
you could do the same in any number of ways, like by visiting a file
whose size is larger than the VM you allow your system to use, right?
So there's a potential for causing such problems in many different
ways, and warning about that only in the doc string of image-size
seems insufficient, to say the least...




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 5 Nov 2018 14:08:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Nov 05 09:08:05 2018
Received: from localhost ([127.0.0.1]:34702 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gJfXx-0001R7-1R
	for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 09:08:05 -0500
Received: from eggs.gnu.org ([208.118.235.92]:48689)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1gJfXv-0001Qe-JI
 for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 09:08:04 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <larsi@HIDDEN>) id 1gJfXp-0002BQ-7V
 for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 09:07:58 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:43099)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <larsi@HIDDEN>) id 1gJfXp-0002BD-2U
 for submit <at> debbugs.gnu.org; Mon, 05 Nov 2018 09:07:57 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:58286)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <larsi@HIDDEN>) id 1gJfXo-0002Nk-2Z
 for bug-gnu-emacs@HIDDEN; Mon, 05 Nov 2018 09:07:56 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <larsi@HIDDEN>) id 1gJfXk-00025e-Nf
 for bug-gnu-emacs@HIDDEN; Mon, 05 Nov 2018 09:07:56 -0500
Received: from lamora.getmail.no ([84.210.184.7]:50622)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <larsi@HIDDEN>) id 1gJfXk-0001lD-G9
 for bug-gnu-emacs@HIDDEN; Mon, 05 Nov 2018 09:07:52 -0500
Received: from localhost (localhost [127.0.0.1])
 by lamora.getmail.no (Postfix) with ESMTP id F1D6860840
 for <bug-gnu-emacs@HIDDEN>; Mon,  5 Nov 2018 15:07:30 +0100 (CET)
Received: from lamora.getmail.no ([127.0.0.1])
 by localhost (lamora.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id r3Ad1Lq2lR-M for <bug-gnu-emacs@HIDDEN>;
 Mon,  5 Nov 2018 15:07:30 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by lamora.getmail.no (Postfix) with ESMTP id 781E960842
 for <bug-gnu-emacs@HIDDEN>; Mon,  5 Nov 2018 15:07:30 +0100 (CET)
X-Virus-Scanned: amavisd-new at lamora.get.c.bitbit.net
Received: from lamora.getmail.no ([127.0.0.1])
 by localhost (lamora.get.c.bitbit.net [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id z3IaSEm1gCTC for <bug-gnu-emacs@HIDDEN>;
 Mon,  5 Nov 2018 15:07:30 +0100 (CET)
Received: from stories (cm-84.212.221.165.getinternet.no [84.212.221.165])
 by lamora.getmail.no (Postfix) with ESMTPSA id 4C5C960840
 for <bug-gnu-emacs@HIDDEN>; Mon,  5 Nov 2018 15:07:30 +0100 (CET)
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 27.0.50; Image cache pruning
Date: Mon, 05 Nov 2018 15:07:30 +0100
Message-ID: <m3bm73ipnh.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)


I was writing a function to do some image processing, and after it had
run a few minutes, the Emacs window disappeared.

After poking around a bit, it turned out that Emacs was killed by the
out-of-memory task.  I though that Emacs must have a serious memory leak
somewhere, but `garbage-collect' didn't really seem to have anything
mysterious in it.

So after trying various things, it turned out that you can kill Emacs by
doing this:

(dolist (file (directory-files "/directory/with/many/images" t "png$"))
  (image-size (create-image file) t))

and this is because `image-size' puts the image into the image cache.
Calling `clear-image-cache' releases the memory.

I think putting it into the cache is the right thing to do, because if
you're calling `image-size' on a file, then nine out of ten times you
want to display it, and my use case here is atypical (batch-processing
of a large number of huge images), but it's still not optimal that you
can crash Emacs in this way.

So perhaps the cache should be pruned more aggressively according to
some smart algorithm or other?  Or, at the very least, an
innocuous-looking function like `image-size' should note in the doc
string should note what's going on and recommend calling `image-flush'
afterwards if we don't want to cache the image?


In GNU Emacs 27.0.50 (build 50, x86_64-pc-linux-gnu, GTK+ Version 3.22.11)
 of 2018-08-26 built on stories
Repository revision: 5c642b2dc1b666ae488225b76251750a8cf331be
Windowing system distributor 'The X.Org Foundation', version 11.0.11902000
System Description: Debian GNU/Linux 9 (stretch)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





Acknowledgement sent to Lars Ingebrigtsen <larsi@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#33275; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Fri, 9 Nov 2018 12:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.