GNU bug report logs - #45705
[feature/native-comp] Excessive memory consumption on windows 10

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Édouard Debry <edouard.debry@HIDDEN>; merged with #45751; dated Wed, 6 Jan 2021 20:49:01 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.
Merged 45705 45751. Request was from Andrea Corallo <akrl@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 19:41:15 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 09 14:41:15 2021
Received: from localhost ([127.0.0.1]:52629 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kyK6t-0006KI-GQ
	for submit <at> debbugs.gnu.org; Sat, 09 Jan 2021 14:41:15 -0500
Received: from mx.sdf.org ([205.166.94.24]:63188)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kyK6o-0006K7-Fx
 for 45705 <at> debbugs.gnu.org; Sat, 09 Jan 2021 14:41:14 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 109Jf9oZ014968;
 Sat, 9 Jan 2021 19:41:09 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: =?utf-8?Q?K=C3=A9vin?= Le Gouguec <kevin.legouguec@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN> <xjfa6tjaxz5.fsf@HIDDEN>
 <83lfd2ilwf.fsf@HIDDEN> <xjf5z46bcro.fsf@HIDDEN>
 <83y2h2gw9v.fsf@HIDDEN> <xjfy2h29tgd.fsf@HIDDEN>
 <87im86c97t.fsf@HIDDEN>
Date: Sat, 09 Jan 2021 19:41:09 +0000
In-Reply-To: <87im86c97t.fsf@HIDDEN> (=?utf-8?Q?=22K=C3=A9vin?= Le
 Gouguec"'s message of "Sat, 09 Jan 2021 18:26:46 +0100")
Message-ID: <xjfturpaofe.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: Eli Zaretskii <eliz@HIDDEN>, edouard.debry@HIDDEN,
 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

K=C3=A9vin Le Gouguec <kevin.legouguec@HIDDEN> writes:

> Andrea Corallo <akrl@HIDDEN> writes:
>
>> Eli Zaretskii <eliz@HIDDEN> writes:
>>
>>>> From: Andrea Corallo <akrl@HIDDEN>
>>>>=20
>>>> In June we changed the way we store immediate objects in the shared and
>>>> this makes the compilation way lighter on the GCC side (both in time a=
nd
>>>> memory).  I've no precise data on this other than the experimental
>>>> observation that compiling all Elisp files in Emacs on 32bit systems is
>>>> not anymore an issue.  This IIUC implies that the memory footprint for
>>>> each compilation is always < 2GB.
>>>
>>> You assume that the compilations are all done serially?  AFAIK, most
>>> people build Emacs with "make -jN", so parallel compilation is an
>>> important use case.
>>
>>> I guess we will have to collect the information about that, if you say
>>> we don't have it now.
>>
>> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
>> produce quite accurate results on memory footprint during the
>> compilation process.  Perhaps he has time and he's so kind to gather
>> some data on the current state, that would be extremely helpful.
>
> See also bug#41194#20 and bug#41194#28 where I outlined how the
> improvements reduced compilation time and memory usage.
>
> I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's
> in need of=E2=80=A6 something (probably exorcism, given the noise).

At the moment I think we are interested more in the magnitude order and
also a measure on 64bit would be very interesting.

But you are right I see in bug#41194#20 bug#41194#28 you've already
measured the footprint with the new immediate handling!

IIRC the worstcase was org.el under 600MB.  I think till we get a more
recent measure we should assume this as number (I don't expect huge
changes tho).

Many thanks

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 17:26:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 09 12:26:59 2021
Received: from localhost ([127.0.0.1]:52536 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kyI0x-00034t-8z
	for submit <at> debbugs.gnu.org; Sat, 09 Jan 2021 12:26:59 -0500
Received: from mail-wm1-f43.google.com ([209.85.128.43]:55201)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kevin.legouguec@HIDDEN>) id 1kyI0s-00034S-Kg
 for 45705 <at> debbugs.gnu.org; Sat, 09 Jan 2021 12:26:58 -0500
Received: by mail-wm1-f43.google.com with SMTP id c133so10288370wme.4
 for <45705 <at> debbugs.gnu.org>; Sat, 09 Jan 2021 09:26:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=p3RbSEpgcVqcaRM+sho17m+rQzwp9qQmVHEaG0tV+8E=;
 b=LzikEmPfaMA8FZph1WMiCwbnuHdUccG9Osi4SAkQaLRiNPLYZh2cH7xdTZ6SR1gphP
 V/tU6jwvmHeqH6/cX/+5WMiwk9TNTHl63LPnHNuTcvRlnebfj8y0B5Ec/3as6i9td1Jp
 A21HlDgKmJ84whmWyiIzYCbOzsKHyS3tX8n6KS9sQAR9Odq9EPneVk/mqcbDemR9TEpe
 8DtYoVTS38WnmzWG3azYZQAzRRXFzzOR+lq/3M9DSew7ouXcu8uiqVYrnkjKGsAe01iE
 j13I+KJiuec66mrCDq1gZERByun7GbeN6jxbpoHDg6/ywN0aFh2x8ifX7BmU6uj1bhpP
 TuCg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=p3RbSEpgcVqcaRM+sho17m+rQzwp9qQmVHEaG0tV+8E=;
 b=eEwVXQuw5yhEzx0JpSva6VXy8/sooksTwjxHoaJBTdaSNozNlUHbW3q5ax/f9ZzwmN
 z3cOqL+W0ztDd/RFqbYH77qWhctvYqmvUOFHXZKnaMMYDfhkz/kLdXSw3ImTId5wRS9k
 fQyacal8GaaQLsyvcBd6QFcf8hkUxqKlUDB3Ss+fzCmeQgzhoST8Xp8vThwC1+QHh+fe
 EPEg7do45jvoLHvX+4XrWaP0lHbWhrmkEsQqL+gX28xeKPxXh1hO70N998gRN5chd3QM
 OsLsfB6bsoreQuzYiVUNfE/naddefQ3n4xV3nLEpIl43aP3NmAViOJlgHVDDZhBDZ4cY
 1dnQ==
X-Gm-Message-State: AOAM531REnwJZYkaG+h01g6qX1t0T5Ed1cjPlx9jio09aDSCy2r3wrka
 tKf8TpbZsmacO1198EyvTXRCQ0iDkPLFH8in
X-Google-Smtp-Source: ABdhPJz380Pp/cGH6Baq4VxLoEuRkjsiPsSYZEAXsYq2laUiz20L3FAwnY2rPzQ9X/U5s4Pa1m2jdw==
X-Received: by 2002:a1c:c254:: with SMTP id s81mr8048005wmf.132.1610213208414; 
 Sat, 09 Jan 2021 09:26:48 -0800 (PST)
Received: from my-little-tumbleweed ([2a01:e0a:20e:d340:922b:34ff:fe95:9aed])
 by smtp.gmail.com with ESMTPSA id
 m21sm15979328wml.13.2021.01.09.09.26.47
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 09 Jan 2021 09:26:47 -0800 (PST)
From: =?utf-8?Q?K=C3=A9vin_Le_Gouguec?= <kevin.legouguec@HIDDEN>
To: Andrea Corallo <akrl@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN> <xjfa6tjaxz5.fsf@HIDDEN>
 <83lfd2ilwf.fsf@HIDDEN> <xjf5z46bcro.fsf@HIDDEN>
 <83y2h2gw9v.fsf@HIDDEN> <xjfy2h29tgd.fsf@HIDDEN>
Date: Sat, 09 Jan 2021 18:26:46 +0100
In-Reply-To: <xjfy2h29tgd.fsf@HIDDEN> (Andrea Corallo's message of "Sat, 09
 Jan 2021 12:37:54 +0000")
Message-ID: <87im86c97t.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: Eli Zaretskii <eliz@HIDDEN>, edouard.debry@HIDDEN,
 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Andrea Corallo <akrl@HIDDEN> writes:

> Eli Zaretskii <eliz@HIDDEN> writes:
>
>>> From: Andrea Corallo <akrl@HIDDEN>
>>>=20
>>> In June we changed the way we store immediate objects in the shared and
>>> this makes the compilation way lighter on the GCC side (both in time and
>>> memory).  I've no precise data on this other than the experimental
>>> observation that compiling all Elisp files in Emacs on 32bit systems is
>>> not anymore an issue.  This IIUC implies that the memory footprint for
>>> each compilation is always < 2GB.
>>
>> You assume that the compilations are all done serially?  AFAIK, most
>> people build Emacs with "make -jN", so parallel compilation is an
>> important use case.
>
>> I guess we will have to collect the information about that, if you say
>> we don't have it now.
>
> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
> produce quite accurate results on memory footprint during the
> compilation process.  Perhaps he has time and he's so kind to gather
> some data on the current state, that would be extremely helpful.

See also bug#41194#20 and bug#41194#28 where I outlined how the
improvements reduced compilation time and memory usage.

I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's
in need of=E2=80=A6 something (probably exorcism, given the noise).

Until I figure that out, here are the (very hacky) scripts I used to
measure and plot the RAM usage, in case someone else wants to take some
measurements:

- ./monitor.sh $PID finds the most RAM-consuming process among $PID and
  its children, and logs its memory usage (VSZ and RSS) and its
  command-line.

  (Logs are collected every 10 seconds; this probably needs to be
  reduced for faster machines)

- ./plot.py uses matplotlib to make graphs out of these measurements; it
  attempts to replace the command line with the less-verbose diagnostics
  from "make".

- My workflow was to start an emacs session, run M-x compile RET make,
  then ./monitor.sh $PID_OF_EMACS_SESSION.

  (PARENT_RE in plot.py should match the command-line of this parent
   session; its RAM consumption is then labeled as "noise floor" on the
   graph.

   This serves no real purpose and should be removed; monitor.sh should
   be amended to filter the parent session out of monitored PIDs, with
   some error control to handle the lack of child processes when
   compilation is finished.)

- There are some hardcoded things to tweak at the bottom of plot.py,
  e.g. how long should a child process last for it to have a label on
  the graph.


--=-=-=
Content-Type: application/x-shellscript
Content-Disposition: attachment; filename=monitor.sh
Content-Transfer-Encoding: base64

IyEvYmluL2Jhc2gKCnNldCAtZXUKCm91dD0kKGRpcm5hbWUgJDApL21vbml0b3IubG9nCnBhcmVu
dD0kMQoKcmVjdXJzZSAoKQp7CiAgICBlY2hvICQxCiAgICBmb3IgY2hpbGQgaW4gJChwcyAtbyBw
aWQ9IC0tcHBpZCAkMSkKICAgIGRvCiAgICAgICAgcmVjdXJzZSAkY2hpbGQKICAgIGRvbmUKfQoK
d2hpbGUgc2xlZXAgMTAKZG8KICAgIGRhdGUgKyIlRi0lVCIgPj4gJHtvdXR9CiAgICBwcyBvIGNw
dXRpbWVzPSx2c3o9LHJzcz0sYXJncz0gLS1zb3J0PS12c3ogcCAkKHJlY3Vyc2UgJHtwYXJlbnR9
KSB8IGhlYWQgLTEgPj4gJHtvdXR9CiAgICBmcmVlID4+ICR7b3V0fQogICAgZWNobyA+PiAke291
dH0KZG9uZQo=
--=-=-=
Content-Type: text/x-python; charset=utf-8
Content-Disposition: attachment; filename=plot.py
Content-Transfer-Encoding: quoted-printable

#!/usr/bin/env python3

from datetime import datetime, timedelta
from pathlib import Path
import re

import matplotlib
from matplotlib import pyplot
from matplotlib.dates import DateFormatter, HourLocator, MinuteLocator
from matplotlib.ticker import EngFormatter


MONITOR_RE =3D re.compile('\n'.join((
    '(?P<time>.+)',
    r' *(?P<seconds>\d+) +(?P<vsz>\d+) +(?P<rss>\d+) +(?P<args>.+)',
    ' *(?P<memheader>.+)',
    'Mem: *(?P<memvalues>.+)',
    'Swap: *(?P<swapvalues>.+)',
    ''
)), flags=3Dre.MULTILINE)


def list_snapshots(monitor_log):
    snapshots =3D []

    for match in MONITOR_RE.finditer(monitor_log):
        md =3D match.groupdict()

        memkeys =3D md['memheader'].split()
        memvalues =3D md['memvalues'].split()
        swapvalues =3D md['swapvalues'].split()

        snapshot =3D {
            'time': datetime.strptime(md['time'], '%Y-%m-%d-%H:%M:%S'),
            'uptime': int(md['seconds']),
            'vsz': int(md['vsz'])*1024,
            'rss': int(md['rss'])*1024,
            'process': md['args'],
            'mem': {memkeys[i]: int(val)*1024 for i, val in enumerate(memva=
lues)},
            'swap': {memkeys[i]: int(val)*1024 for i, val in enumerate(swap=
values)}
        }

        snapshots.append(snapshot)

    return snapshots


LOADDEFS_RE =3D re.compile(
    r'--eval \(setq generated-autoload-file'
    r' \(expand-file-name \(unmsys--file-name "([^"]+)"\)\)\)'
    r' -f batch-update-autoloads'
)

SEMANTIC_RE =3D re.compile(
    r'-l semantic/(?:wisent|bovine)/grammar -f (?:wisent|bovine)-batch-make=
-parser'
    r' -o (.+) .+\.[wb]y'
)

ELCELN_RE =3D re.compile(
    r'\.\./src/(?:bootstrap-)?emacs -batch --no-site-file --no-site-lisp'
    r' --eval \(setq load-prefer-newer t\) -l comp'
    r'(?: -f byte-compile-refresh-preloaded)?'
    r' -f batch-byte-native-compile-for-bootstrap'
    r' (.+\.el)'
)

SHORTENED_NAMES =3D {
    LOADDEFS_RE: 'GEN',
    SEMANTIC_RE: 'GEN',
    ELCELN_RE: 'ELC+ELN'
}

QUAIL_TIT_RE =3D re.compile(
    r'-l titdic-cnv -f batch-titdic-convert'
    r' -dir \./\.\./lisp/leim/quail CXTERM-DIC/(.+)\.tit'
)

QUAIL_MISC_RE =3D re.compile(
    r'-l titdic-cnv -f batch-miscdic-convert'
    r' -dir \./\.\./lisp/leim/quail MISC-DIC/(.+\.(html|map|cin|cns|b5))'
)

QUAIL_JA_RE =3D re.compile(
    r'-l ja-dic-cnv -f batch-skkdic-convert'
)

PARENT_RE =3D re.compile(
    r'$^'                       # Adjust to match parent process.
)

TRANSFORMED_NAMES =3D {
    QUAIL_TIT_RE: lambda m: f'GEN ../lisp/leim/quail/{m.group(1)}.el',
    QUAIL_MISC_RE: lambda m: f'GEN from {m.group(1)}',
    QUAIL_JA_RE: lambda m: f'GEN ../lisp/leim/ja-dic/ja-dic.el',
    PARENT_RE: lambda _: '(noise floor)'
}

def shorten(process):
    for r, name in SHORTENED_NAMES.items():
        match =3D r.search(process)
        if match is not None:
            return f'{name} {match.group(1)}'

    for r, transform in TRANSFORMED_NAMES.items():
        match =3D r.search(process)
        if match is not None:
            return transform(match)

    if len(process) > 40:
        return f'{process[:20]}=E2=80=A6{process[-20:]}'
    return process


def list_processes(snapshots):
    t0 =3D snapshots[0]['time']
    current_process =3D snapshots[0]['process']
    current_process_start =3D t0

    processes =3D []

    for s in snapshots[1:]:
        if s['process'] =3D=3D current_process:
            continue

        s_start =3D s['time']
        processes.append((
            current_process, current_process_start, s_start-current_process=
_start
        ))

        current_process =3D s['process']
        current_process_start =3D s_start

    processes.append((
        current_process,
        current_process_start,
        snapshots[-1]['time']-current_process_start
    ))

    return processes


snapshots =3D list_snapshots(Path('monitor.log').read_text())

xs =3D tuple(s['time'] for s in snapshots)
vsz =3D tuple(s['vsz'] for s in snapshots)
rss =3D tuple(s['rss'] for s in snapshots)
memavail =3D tuple(s['mem']['available'] for s in snapshots)
swapused =3D tuple(s['swap']['used'] for s in snapshots)

matplotlib.use('TkAgg')
fig, axes =3D pyplot.subplots(figsize=3D(128, 9.6))
axes.plot(xs, vsz, label=3D'VSZ (process)')
axes.plot(xs, rss, label=3D'RSS (process)')
axes.plot(xs, memavail, label=3D'available memory (system)', linewidth=3D0.=
5)
axes.plot(xs, swapused, label=3D'used swap (system)')

axes.set_xlim(snapshots[0]['time'], snapshots[-1]['time'])
axes.xaxis.set_major_formatter(DateFormatter('%H:%M'))
axes.xaxis.set_major_locator(HourLocator())
axes.xaxis.set_minor_locator(MinuteLocator(tuple(5*i for i in range(1, 12))=
))
axes.xaxis.set_label_text('Hours')
axes.set_ylim(0)
axes.yaxis.set_major_formatter(EngFormatter(unit=3D'B'))
axes.legend()

for p, start, duration in list_processes(snapshots):
    if duration < timedelta(minutes=3D2):
        continue
    pyplot.text(start, 1e9, shorten(p), rotation=3D45)
    pyplot.plot(
        (start, start+duration), (1e9, 1e9),
        marker=3D'|', linewidth=3D0.5, linestyle=3D'--',
        color=3D'black', alpha=3D0.8
    )

pyplot.savefig('monitor.pdf')
pyplot.show()

--=-=-=--




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 12:37:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 09 07:37:59 2021
Received: from localhost ([127.0.0.1]:51348 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kyDVH-00047k-Fa
	for submit <at> debbugs.gnu.org; Sat, 09 Jan 2021 07:37:59 -0500
Received: from mx.sdf.org ([205.166.94.24]:52261)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kyDVE-00047a-9D
 for 45705 <at> debbugs.gnu.org; Sat, 09 Jan 2021 07:37:58 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 109CbsmP011267;
 Sat, 9 Jan 2021 12:37:54 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN> <xjfa6tjaxz5.fsf@HIDDEN>
 <83lfd2ilwf.fsf@HIDDEN> <xjf5z46bcro.fsf@HIDDEN>
 <83y2h2gw9v.fsf@HIDDEN>
Date: Sat, 09 Jan 2021 12:37:54 +0000
In-Reply-To: <83y2h2gw9v.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 09 Jan
 2021 13:55:08 +0200")
Message-ID: <xjfy2h29tgd.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org,
 =?utf-8?Q?K=C3=A9vin?= Le Gouguec <kevin.legouguec@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Andrea Corallo <akrl@HIDDEN>
>> Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
>> Date: Sat, 09 Jan 2021 10:55:23 +0000
>> 
>> > What about memory usage when there's a background compilation of Lisp
>> > going on?  GCC is known to be a memory hog in some cases, so I wonder
>> > what happens in this case with libgccjit.
>> 
>> In June we changed the way we store immediate objects in the shared and
>> this makes the compilation way lighter on the GCC side (both in time and
>> memory).  I've no precise data on this other than the experimental
>> observation that compiling all Elisp files in Emacs on 32bit systems is
>> not anymore an issue.  This IIUC implies that the memory footprint for
>> each compilation is always < 2GB.
>
> You assume that the compilations are all done serially?  AFAIK, most
> people build Emacs with "make -jN", so parallel compilation is an
> important use case.

Yeah, we can say this loose information is only a per compilation unit
upper-bound.

> I guess we will have to collect the information about that, if you say
> we don't have it now.

I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
produce quite accurate results on memory footprint during the
compilation process.  Perhaps he has time and he's so kind to gather
some data on the current state, that would be extremely helpful.

>> As a note: in all cases except bootstrap the final pass (the one driving
>> libgccjit) is executed as a sub-process, this to protect us from
>> eventual GCC leaks and not to generate unnecessary fragmentation.  In
>> async compilation we indeed run all the compilation (also the Lisp
>> computation) in the child process, so compiling should not have impact
>> on the memory footprint of the main Emacs session.
>
> That's fine, but the memory footprint of such a subprocess is also of
> interest, as it could be relevant to the overall memory pressure of
> the OS, and thus indirectly on the parent Emacs process as well.

Indeed, I thought was relevant to make you aware of this mechanism.

>> > (Do we allow multiple async compilations, btw? if so, how many
>> > concurrent compilations can be running, and how do we or the user
>> > control that?)
>> 
>> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>
>
> Thanks.  This needs further tuning, IMO, both per the FIXME
> (i.e. provide a primitive to return the number of execution units),
> and wrt the default value being half of the available units.  We
> should pay attention to the system's load average as well, I think.

Agree, I guess we'll be able to tune it better when we'll have more
data.  We could even think of using the memory pressure on the system to
make such a decision.

>> > Also, what are the numbers for a session that has been running for
>> > several days?  I understand that it would be hard for you to collect
>> > such numbers about all the configurations, but could you show the
>> > growth of the configuration you are routinely using, which I presume
>> > is --with-x --with-nativecomp and with your config?  As your numbers
>> > above show, it starts at 1.5 GiB, but what is the footprint after a
>> > day or a week?
>> 
>> ATM I can provide this number, this is an Aarch64 daemon compiled with
>> '--without-x' with an up-time of 25 days and is showing a footprint of
>> 765M.
>
> OK, thanks.
>
>> The hard part is to have a reference to compare against as the memory
>> footprint is strictly connected to the usage.  One with very regular
>> working habits should work like one week on vanilla and one week on
>> native-comp to make a comparison.  I've no regular working habits so I
>> fear I'm not the best fit for this comparison.
>
> I agree, these numbers still need to be collected.  Maybe we should
> ask on emacs-devel that people who use the branch report their
> numbers?

Is a good idea, my fear would be only to have very noisy or hard to
interpret results.  A priori I think would be nice to collect such data
also for master too.

Thanks

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 11:55:36 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 09 06:55:36 2021
Received: from localhost ([127.0.0.1]:51291 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kyCqG-0000yb-DD
	for submit <at> debbugs.gnu.org; Sat, 09 Jan 2021 06:55:36 -0500
Received: from eggs.gnu.org ([209.51.188.92]:50274)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1kyCqE-0000yO-VT
 for 45705 <at> debbugs.gnu.org; Sat, 09 Jan 2021 06:55:35 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:58564)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1kyCq8-0000bt-5g; Sat, 09 Jan 2021 06:55:28 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1912
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1kyCpk-0008Rr-8a; Sat, 09 Jan 2021 06:55:06 -0500
Date: Sat, 09 Jan 2021 13:55:08 +0200
Message-Id: <83y2h2gw9v.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Andrea Corallo <akrl@HIDDEN>
In-Reply-To: <xjf5z46bcro.fsf@HIDDEN> (message from Andrea Corallo on Sat, 09
 Jan 2021 10:55:23 +0000)
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN> <xjfa6tjaxz5.fsf@HIDDEN>
 <83lfd2ilwf.fsf@HIDDEN> <xjf5z46bcro.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Andrea Corallo <akrl@HIDDEN>
> Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
> Date: Sat, 09 Jan 2021 10:55:23 +0000
> 
> > What about memory usage when there's a background compilation of Lisp
> > going on?  GCC is known to be a memory hog in some cases, so I wonder
> > what happens in this case with libgccjit.
> 
> In June we changed the way we store immediate objects in the shared and
> this makes the compilation way lighter on the GCC side (both in time and
> memory).  I've no precise data on this other than the experimental
> observation that compiling all Elisp files in Emacs on 32bit systems is
> not anymore an issue.  This IIUC implies that the memory footprint for
> each compilation is always < 2GB.

You assume that the compilations are all done serially?  AFAIK, most
people build Emacs with "make -jN", so parallel compilation is an
important use case.

I guess we will have to collect the information about that, if you say
we don't have it now.

> As a note: in all cases except bootstrap the final pass (the one driving
> libgccjit) is executed as a sub-process, this to protect us from
> eventual GCC leaks and not to generate unnecessary fragmentation.  In
> async compilation we indeed run all the compilation (also the Lisp
> computation) in the child process, so compiling should not have impact
> on the memory footprint of the main Emacs session.

That's fine, but the memory footprint of such a subprocess is also of
interest, as it could be relevant to the overall memory pressure of
the OS, and thus indirectly on the parent Emacs process as well.

> > (Do we allow multiple async compilations, btw? if so, how many
> > concurrent compilations can be running, and how do we or the user
> > control that?)
> 
> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>

Thanks.  This needs further tuning, IMO, both per the FIXME
(i.e. provide a primitive to return the number of execution units),
and wrt the default value being half of the available units.  We
should pay attention to the system's load average as well, I think.

> > Also, what are the numbers for a session that has been running for
> > several days?  I understand that it would be hard for you to collect
> > such numbers about all the configurations, but could you show the
> > growth of the configuration you are routinely using, which I presume
> > is --with-x --with-nativecomp and with your config?  As your numbers
> > above show, it starts at 1.5 GiB, but what is the footprint after a
> > day or a week?
> 
> ATM I can provide this number, this is an Aarch64 daemon compiled with
> '--without-x' with an up-time of 25 days and is showing a footprint of
> 765M.

OK, thanks.

> The hard part is to have a reference to compare against as the memory
> footprint is strictly connected to the usage.  One with very regular
> working habits should work like one week on vanilla and one week on
> native-comp to make a comparison.  I've no regular working habits so I
> fear I'm not the best fit for this comparison.

I agree, these numbers still need to be collected.  Maybe we should
ask on emacs-devel that people who use the branch report their
numbers?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 10:55:27 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 09 05:55:27 2021
Received: from localhost ([127.0.0.1]:51261 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kyBu3-0007gw-1S
	for submit <at> debbugs.gnu.org; Sat, 09 Jan 2021 05:55:27 -0500
Received: from mx.sdf.org ([205.166.94.24]:58233)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kyBu0-0007gm-Vt
 for 45705 <at> debbugs.gnu.org; Sat, 09 Jan 2021 05:55:26 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 109AtNS0024509;
 Sat, 9 Jan 2021 10:55:23 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN> <xjfa6tjaxz5.fsf@HIDDEN>
 <83lfd2ilwf.fsf@HIDDEN>
Date: Sat, 09 Jan 2021 10:55:23 +0000
In-Reply-To: <83lfd2ilwf.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 09 Jan
 2021 09:56:16 +0200")
Message-ID: <xjf5z46bcro.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Andrea Corallo <akrl@HIDDEN>
>> Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
>> Date: Fri, 08 Jan 2021 22:02:38 +0000
>> 
>> I've compiled current native-comp with and without --with-nativecomp
>> repeating the experiment with and without X.  These are the data-points:
>> 
>>   |         | --without-x | --without-x --with-nativecomp |      |
>>   |---------+-------------+-------------------------------+------|
>>   | -Q      | 49M         | 92M                           | 1.9x |
>>   | my-conf | 92M         | 179M                          | 1.9x |
>>   
>>   
>>   |         |      | --with-nativecomp |      |
>>   |---------+------+-------------------+------|
>>   | -Q      | 536M | 756M              | 1.4x |
>>   | my-conf | 585M | 1453M             | 2.4x |
>> 
>> So yes shared are using considerably more memory, I think this is
>> expected as also the file footprint suggests native code is less dense
>> that byte-code (actually with a quite similar relative results).
>
> Thanks for the data points.
>
> What about memory usage when there's a background compilation of Lisp
> going on?  GCC is known to be a memory hog in some cases, so I wonder
> what happens in this case with libgccjit.

In June we changed the way we store immediate objects in the shared and
this makes the compilation way lighter on the GCC side (both in time and
memory).  I've no precise data on this other than the experimental
observation that compiling all Elisp files in Emacs on 32bit systems is
not anymore an issue.  This IIUC implies that the memory footprint for
each compilation is always < 2GB.

As a note: in all cases except bootstrap the final pass (the one driving
libgccjit) is executed as a sub-process, this to protect us from
eventual GCC leaks and not to generate unnecessary fragmentation.  In
async compilation we indeed run all the compilation (also the Lisp
computation) in the child process, so compiling should not have impact
on the memory footprint of the main Emacs session.

> (Do we allow multiple async compilations, btw? if so, how many
> concurrent compilations can be running, and how do we or the user
> control that?)

Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>

> Also, what are the numbers for a session that has been running for
> several days?  I understand that it would be hard for you to collect
> such numbers about all the configurations, but could you show the
> growth of the configuration you are routinely using, which I presume
> is --with-x --with-nativecomp and with your config?  As your numbers
> above show, it starts at 1.5 GiB, but what is the footprint after a
> day or a week?

ATM I can provide this number, this is an Aarch64 daemon compiled with
'--without-x' with an up-time of 25 days and is showing a footprint of
765M.

>> Indeed *with use the delta should decay as data are the same and there's
>> no difference in its representation*, so this picture should be more on
>> the worst case side than on the average.
>
> That's why I asked to see the memory footprint after the session has
> run for some time, yes.

The hard part is to have a reference to compare against as the memory
footprint is strictly connected to the usage.  One with very regular
working habits should work like one week on vanilla and one week on
native-comp to make a comparison.  I've no regular working habits so I
fear I'm not the best fit for this comparison.

Thanks

  Andrea





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 07:56:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 09 02:56:25 2021
Received: from localhost ([127.0.0.1]:51174 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ky96n-0003JP-2v
	for submit <at> debbugs.gnu.org; Sat, 09 Jan 2021 02:56:25 -0500
Received: from eggs.gnu.org ([209.51.188.92]:46500)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1ky96g-0003J5-KX
 for 45705 <at> debbugs.gnu.org; Sat, 09 Jan 2021 02:56:23 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:56212)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1ky96a-0004vH-Ng; Sat, 09 Jan 2021 02:56:12 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2798
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1ky96a-0003Zq-3O; Sat, 09 Jan 2021 02:56:12 -0500
Date: Sat, 09 Jan 2021 09:56:16 +0200
Message-Id: <83lfd2ilwf.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Andrea Corallo <akrl@HIDDEN>
In-Reply-To: <xjfa6tjaxz5.fsf@HIDDEN> (message from Andrea Corallo on Fri, 08
 Jan 2021 22:02:38 +0000)
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN> <xjfa6tjaxz5.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Andrea Corallo <akrl@HIDDEN>
> Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
> Date: Fri, 08 Jan 2021 22:02:38 +0000
> 
> I've compiled current native-comp with and without --with-nativecomp
> repeating the experiment with and without X.  These are the data-points:
> 
>   |         | --without-x | --without-x --with-nativecomp |      |
>   |---------+-------------+-------------------------------+------|
>   | -Q      | 49M         | 92M                           | 1.9x |
>   | my-conf | 92M         | 179M                          | 1.9x |
>   
>   
>   |         |      | --with-nativecomp |      |
>   |---------+------+-------------------+------|
>   | -Q      | 536M | 756M              | 1.4x |
>   | my-conf | 585M | 1453M             | 2.4x |
> 
> So yes shared are using considerably more memory, I think this is
> expected as also the file footprint suggests native code is less dense
> that byte-code (actually with a quite similar relative results).

Thanks for the data points.

What about memory usage when there's a background compilation of Lisp
going on?  GCC is known to be a memory hog in some cases, so I wonder
what happens in this case with libgccjit.

(Do we allow multiple async compilations, btw? if so, how many
concurrent compilations can be running, and how do we or the user
control that?)

Also, what are the numbers for a session that has been running for
several days?  I understand that it would be hard for you to collect
such numbers about all the configurations, but could you show the
growth of the configuration you are routinely using, which I presume
is --with-x --with-nativecomp and with your config?  As your numbers
above show, it starts at 1.5 GiB, but what is the footprint after a
day or a week?

> Indeed *with use the delta should decay as data are the same and there's
> no difference in its representation*, so this picture should be more on
> the worst case side than on the average.

That's why I asked to see the memory footprint after the session has
run for some time, yes.

> Also if we want to see a positive side, multiple Emacs sessions will
> share the majority the pages allocated for the shared libraries.

Assuming those sessions run the same binary of Emacs, yes.  Otherwise,
only some of them will be shared, the ones that are in common public
directories, and even that only if the running Emacsen are of the same
version.  Most of the shared code is in the pdumper file, btw, which
is outside of the picture for the purposes of this discussion.

Thanks.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 22:02:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 08 17:02:43 2021
Received: from localhost ([127.0.0.1]:50840 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxzqE-0005bY-US
	for submit <at> debbugs.gnu.org; Fri, 08 Jan 2021 17:02:43 -0500
Received: from mx.sdf.org ([205.166.94.24]:62187)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kxzqD-0005bM-56
 for 45705 <at> debbugs.gnu.org; Fri, 08 Jan 2021 17:02:41 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 108M2cYP003945;
 Fri, 8 Jan 2021 22:02:38 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
 <83turrif53.fsf@HIDDEN>
Date: Fri, 08 Jan 2021 22:02:38 +0000
In-Reply-To: <83turrif53.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 08 Jan
 2021 18:10:00 +0200")
Message-ID: <xjfa6tjaxz5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Andrea Corallo <akrl@HIDDEN>
>> Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
>> Date: Fri, 08 Jan 2021 15:50:28 +0000
>>
>> consumptions of few GiB is something I've seen more then once for long
>> standing sessions.  You might be right in this being a memory leak,
>> indeed I've no prove of that (I think we have none for the other
>> direction either).
>
> I'm not yet worried about memory leaks, I'm more worried that there's
> no memory leaks, and instead using libgccjit indeed requires such
> large memory amounts.  Are you sure this is not the case?

I see,

if we are interested in comparing the memory footprint of using shared
libraries for functions vs stock byte-code I think we can "statically"
compare two sessions after the startup.

I've compiled current native-comp with and without --with-nativecomp
repeating the experiment with and without X.  These are the data-points:

  |         | --without-x | --without-x --with-nativecomp |      |
  |---------+-------------+-------------------------------+------|
  | -Q      | 49M         | 92M                           | 1.9x |
  | my-conf | 92M         | 179M                          | 1.9x |
  
  
  |         |      | --with-nativecomp |      |
  |---------+------+-------------------+------|
  | -Q      | 536M | 756M              | 1.4x |
  | my-conf | 585M | 1453M             | 2.4x |

So yes shared are using considerably more memory, I think this is
expected as also the file footprint suggests native code is less dense
that byte-code (actually with a quite similar relative results).

Indeed *with use the delta should decay as data are the same and there's
no difference in its representation*, so this picture should be more on
the worst case side than on the average.

Also if we want to see a positive side, multiple Emacs sessions will
share the majority the pages allocated for the shared libraries.

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 16:10:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 08 11:10:05 2021
Received: from localhost ([127.0.0.1]:50394 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxuKz-0003CN-Cx
	for submit <at> debbugs.gnu.org; Fri, 08 Jan 2021 11:10:05 -0500
Received: from eggs.gnu.org ([209.51.188.92]:35170)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1kxuKx-0003Bn-Q0
 for 45705 <at> debbugs.gnu.org; Fri, 08 Jan 2021 11:10:04 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:39308)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1kxuKs-0006th-By; Fri, 08 Jan 2021 11:09:58 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4158
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1kxuKr-00048v-IH; Fri, 08 Jan 2021 11:09:58 -0500
Date: Fri, 08 Jan 2021 18:10:00 +0200
Message-Id: <83turrif53.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Andrea Corallo <akrl@HIDDEN>
In-Reply-To: <xjfh7nrbf7f.fsf@HIDDEN> (message from Andrea Corallo on Fri, 08
 Jan 2021 15:50:28 +0000)
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN> <xjfh7nrbf7f.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Andrea Corallo <akrl@HIDDEN>
> Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
> Date: Fri, 08 Jan 2021 15:50:28 +0000
> 
> consumptions of few GiB is something I've seen more then once for long
> standing sessions.  You might be right in this being a memory leak,
> indeed I've no prove of that (I think we have none for the other
> direction either).

I'm not yet worried about memory leaks, I'm more worried that there's
no memory leaks, and instead using libgccjit indeed requires such
large memory amounts.  Are you sure this is not the case?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 15:50:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 08 10:50:33 2021
Received: from localhost ([127.0.0.1]:50371 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxu25-0002g1-79
	for submit <at> debbugs.gnu.org; Fri, 08 Jan 2021 10:50:33 -0500
Received: from mx.sdf.org ([205.166.94.24]:53012)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kxu23-0002fp-C8
 for 45705 <at> debbugs.gnu.org; Fri, 08 Jan 2021 10:50:32 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 108FoSdS020904;
 Fri, 8 Jan 2021 15:50:28 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
 <831revjyja.fsf@HIDDEN>
Date: Fri, 08 Jan 2021 15:50:28 +0000
In-Reply-To: <831revjyja.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 08 Jan
 2021 16:25:45 +0200")
Message-ID: <xjfh7nrbf7f.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Date: Wed, 06 Jan 2021 20:55:35 +0000
>> Cc: 45705 <at> debbugs.gnu.org
>> From: Andrea Corallo via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN>
>>=20
>> > On windows 10, I noticed that emacs could use a lot of memory, up to
>> > 6Go and
>> > sometimes more. The memory consumption goes up and down from time to
>> > time while
>> > I am not running any specific program in emacs, apart `lsp-java`
>> >
>> > As my personal computer has 16Go of RAM, I can afford
>> > it. Unfortunately, my work
>> > computer has much less and the whole compute completely freezes at one
>> > point
>> > when using emacs.
>> >
>> > I did not notice that behavior on linux. I do not know if the master
>> > branch has
>> > the same problem. What could be the problem ?
>>=20
>> Hi =C3=89douard,
>>=20
>> AFAIK was never proved recently Emacs garbage collector is failing to
>> recall memory, so I guess this is just some Lisp program that is
>> allocating a lot of memory keeping then those objects referenced.
>
> IME, 6 GiB is too much for any Lisp program to explain away.
>
> Also, the memory allocator we use on Windows is known to return memory
> to the OS more than glibc on GNU/Linux.  I have never seen my Emacs
> session get anywhere near 1 GiB, let alone more, and my sessions run
> for many weeks without restarting.
>
> Can you tell what kind of memory footprints you see on your system
> with the native-comp branch, after running the session for several
> days or more?

Hi Eli,

consumptions of few GiB is something I've seen more then once for long
standing sessions.  You might be right in this being a memory leak,
indeed I've no prove of that (I think we have none for the other
direction either).

Assuming it's a bug I don't see a priori why this should be a different
one respect the one reported for master.

Regards

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 14:25:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 08 09:25:57 2021
Received: from localhost ([127.0.0.1]:49532 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxsiC-0000Ha-UX
	for submit <at> debbugs.gnu.org; Fri, 08 Jan 2021 09:25:57 -0500
Received: from eggs.gnu.org ([209.51.188.92]:35948)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1kxsiA-0000HG-VY
 for 45705 <at> debbugs.gnu.org; Fri, 08 Jan 2021 09:25:55 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37370)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1kxsi5-00033n-H1; Fri, 08 Jan 2021 09:25:49 -0500
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1631
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1kxsi0-0007Lj-Iz; Fri, 08 Jan 2021 09:25:46 -0500
Date: Fri, 08 Jan 2021 16:25:45 +0200
Message-Id: <831revjyja.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Andrea Corallo <akrl@HIDDEN>
In-Reply-To: <xjfft3ddbug.fsf@HIDDEN> (bug-gnu-emacs@HIDDEN)
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on
 windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 45705
Cc: edouard.debry@HIDDEN, 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Wed, 06 Jan 2021 20:55:35 +0000
> Cc: 45705 <at> debbugs.gnu.org
> From: Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN>
> 
> > On windows 10, I noticed that emacs could use a lot of memory, up to
> > 6Go and
> > sometimes more. The memory consumption goes up and down from time to
> > time while
> > I am not running any specific program in emacs, apart `lsp-java`
> >
> > As my personal computer has 16Go of RAM, I can afford
> > it. Unfortunately, my work
> > computer has much less and the whole compute completely freezes at one
> > point
> > when using emacs.
> >
> > I did not notice that behavior on linux. I do not know if the master
> > branch has
> > the same problem. What could be the problem ?
> 
> Hi Édouard,
> 
> AFAIK was never proved recently Emacs garbage collector is failing to
> recall memory, so I guess this is just some Lisp program that is
> allocating a lot of memory keeping then those objects referenced.

IME, 6 GiB is too much for any Lisp program to explain away.

Also, the memory allocator we use on Windows is known to return memory
to the OS more than glibc on GNU/Linux.  I have never seen my Emacs
session get anywhere near 1 GiB, let alone more, and my sessions run
for many weeks without restarting.

Can you tell what kind of memory footprints you see on your system
with the native-comp branch, after running the session for several
days or more?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 7 Jan 2021 14:25:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 07 09:25:30 2021
Received: from localhost ([127.0.0.1]:46910 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxWED-00064I-KB
	for submit <at> debbugs.gnu.org; Thu, 07 Jan 2021 09:25:29 -0500
Received: from mx.sdf.org ([205.166.94.24]:53196)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kxWEC-00064A-2C
 for 45705 <at> debbugs.gnu.org; Thu, 07 Jan 2021 09:25:28 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 107EPQ0g026194;
 Thu, 7 Jan 2021 14:25:27 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
Date: Thu, 07 Jan 2021 14:25:26 +0000
In-Reply-To: <xjfft3ddbug.fsf@HIDDEN> (Andrea Corallo via's message of "Wed, 
 06 Jan 2021 20:55:35 +0000")
Message-ID: <xjfble0ddt5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: =?utf-8?Q?=C3=89douard?= Debry <edouard.debry@HIDDEN>,
 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@HIDDEN> writes:

> =C3=89douard Debry <edouard.debry@HIDDEN> writes:
>
>> Hello,
>>
>> On windows 10, I noticed that emacs could use a lot of memory, up to
>> 6Go and
>> sometimes more. The memory consumption goes up and down from time to
>> time while
>> I am not running any specific program in emacs, apart `lsp-java`
>>
>> As my personal computer has 16Go of RAM, I can afford
>> it. Unfortunately, my work
>> computer has much less and the whole compute completely freezes at one
>> point
>> when using emacs.
>>
>> I did not notice that behavior on linux. I do not know if the master
>> branch has
>> the same problem. What could be the problem ?
>
> Hi =C3=89douard,
>
> AFAIK was never proved recently Emacs garbage collector is failing to
> recall memory, so I guess this is just some Lisp program that is
> allocating a lot of memory keeping then those objects referenced.
>
> Others may have more detaliled info about.
>
>   Andrea
>

IOW I believe this is a duplicate of bug#43389.  Should we merge these?

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 7 Jan 2021 14:25:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 07 09:25:40 2021
Received: from localhost ([127.0.0.1]:46913 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxWEO-00064f-1F
	for submit <at> debbugs.gnu.org; Thu, 07 Jan 2021 09:25:40 -0500
Received: from lists.gnu.org ([209.51.188.17]:57372)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kxWEM-00064X-2t
 for submit <at> debbugs.gnu.org; Thu, 07 Jan 2021 09:25:38 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:51234)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <akrl@HIDDEN>) id 1kxWEL-0005OH-SS
 for bug-gnu-emacs@HIDDEN; Thu, 07 Jan 2021 09:25:37 -0500
Received: from mx.sdf.org ([205.166.94.24]:53194)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <akrl@HIDDEN>) id 1kxWEI-00029k-Vw
 for bug-gnu-emacs@HIDDEN; Thu, 07 Jan 2021 09:25:37 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 107EPQ0g026194;
 Thu, 7 Jan 2021 14:25:27 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN> <xjfft3ddbug.fsf@HIDDEN>
Date: Thu, 07 Jan 2021 14:25:26 +0000
In-Reply-To: <xjfft3ddbug.fsf@HIDDEN> (Andrea Corallo via's message of "Wed, 
 06 Jan 2021 20:55:35 +0000")
Message-ID: <xjfble0ddt5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=205.166.94.24; envelope-from=akrl@HIDDEN;
 helo=mx.sdf.org
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: submit
Cc: =?utf-8?Q?=C3=89douard?= Debry <edouard.debry@HIDDEN>,
 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@HIDDEN> writes:

> =C3=89douard Debry <edouard.debry@HIDDEN> writes:
>
>> Hello,
>>
>> On windows 10, I noticed that emacs could use a lot of memory, up to
>> 6Go and
>> sometimes more. The memory consumption goes up and down from time to
>> time while
>> I am not running any specific program in emacs, apart `lsp-java`
>>
>> As my personal computer has 16Go of RAM, I can afford
>> it. Unfortunately, my work
>> computer has much less and the whole compute completely freezes at one
>> point
>> when using emacs.
>>
>> I did not notice that behavior on linux. I do not know if the master
>> branch has
>> the same problem. What could be the problem ?
>
> Hi =C3=89douard,
>
> AFAIK was never proved recently Emacs garbage collector is failing to
> recall memory, so I guess this is just some Lisp program that is
> allocating a lot of memory keeping then those objects referenced.
>
> Others may have more detaliled info about.
>
>   Andrea
>

IOW I believe this is a duplicate of bug#43389.  Should we merge these?

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at 45705 <at> debbugs.gnu.org:


Received: (at 45705) by debbugs.gnu.org; 6 Jan 2021 20:55:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jan 06 15:55:39 2021
Received: from localhost ([127.0.0.1]:45724 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxFqF-00036n-LR
	for submit <at> debbugs.gnu.org; Wed, 06 Jan 2021 15:55:39 -0500
Received: from mx.sdf.org ([205.166.94.24]:61564)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <akrl@HIDDEN>) id 1kxFqD-00036e-7b
 for 45705 <at> debbugs.gnu.org; Wed, 06 Jan 2021 15:55:38 -0500
Received: from mab (ma.sdf.org [205.166.94.33])
 by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 106KtaUb005111;
 Wed, 6 Jan 2021 20:55:36 GMT
From: Andrea Corallo <akrl@HIDDEN>
To: =?utf-8?Q?=C3=89douard?= Debry <edouard.debry@HIDDEN>
Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption
 on windows 10
References: <87y2h5hjve.fsf@HIDDEN>
Date: Wed, 06 Jan 2021 20:55:35 +0000
In-Reply-To: <87y2h5hjve.fsf@HIDDEN> (=?utf-8?Q?=22=C3=89douard?=
 Debry"'s message of "Wed, 06 Jan 2021 21:48:37 +0100")
Message-ID: <xjfft3ddbug.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 45705
Cc: 45705 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

=C3=89douard Debry <edouard.debry@HIDDEN> writes:

> Hello,
>
> On windows 10, I noticed that emacs could use a lot of memory, up to
> 6Go and
> sometimes more. The memory consumption goes up and down from time to
> time while
> I am not running any specific program in emacs, apart `lsp-java`
>
> As my personal computer has 16Go of RAM, I can afford
> it. Unfortunately, my work
> computer has much less and the whole compute completely freezes at one
> point
> when using emacs.
>
> I did not notice that behavior on linux. I do not know if the master
> branch has
> the same problem. What could be the problem ?

Hi =C3=89douard,

AFAIK was never proved recently Emacs garbage collector is failing to
recall memory, so I guess this is just some Lisp program that is
allocating a lot of memory keeping then those objects referenced.

Others may have more detaliled info about.

  Andrea




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 6 Jan 2021 20:48:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jan 06 15:48:49 2021
Received: from localhost ([127.0.0.1]:45706 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kxFjd-0002vy-0E
	for submit <at> debbugs.gnu.org; Wed, 06 Jan 2021 15:48:49 -0500
Received: from lists.gnu.org ([209.51.188.17]:34184)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <edouard.debry@HIDDEN>) id 1kxFjb-0002vr-Ou
 for submit <at> debbugs.gnu.org; Wed, 06 Jan 2021 15:48:48 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:50698)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <edouard.debry@HIDDEN>)
 id 1kxFjb-0005DJ-HQ
 for bug-gnu-emacs@HIDDEN; Wed, 06 Jan 2021 15:48:47 -0500
Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]:39723)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <edouard.debry@HIDDEN>)
 id 1kxFja-0004O7-2v
 for bug-gnu-emacs@HIDDEN; Wed, 06 Jan 2021 15:48:47 -0500
Received: by mail-wm1-x32e.google.com with SMTP id 3so3737379wmg.4
 for <bug-gnu-emacs@HIDDEN>; Wed, 06 Jan 2021 12:48:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=user-agent:from:to:subject:date:message-id:mime-version;
 bh=tY/UZzscwIDfBI7wI1fWTW6rfdgzKGQS5zg4Z2Omtlw=;
 b=LOO50Lr2jFJpGg1XruVCP+KZ3rZlzm5Zd2vZ1U0pwTx9LXeqZU8CM4Gh4f4xIvqlhi
 WnbUX0ijLQqX7mfGnTZ1py02UgETkTk0Xv3vCPCJ9KR+2xaLKTHGqCTHhpRwQHLISgE6
 vWpxQ8VHxy1NlrwOUtKx4X4gHvc2OH4+kkR8Dfi8FexJbaij/blmJ3awhXOR0rBNB8CP
 TtIeKUrFq5bSOyIrnkCz+UZXajeL5f/65nVlkETZWiVO1f7+z1umydokwRSZqXkSnq0Z
 be7o202eSTl5MH1XBh3UYwhKvczFz7kKJYJalo/bKLfJhNRLqvhvPPy3VZFBh1i1+BE3
 ZdMw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:user-agent:from:to:subject:date:message-id
 :mime-version;
 bh=tY/UZzscwIDfBI7wI1fWTW6rfdgzKGQS5zg4Z2Omtlw=;
 b=d6jUWOM2ZdadDgzRck2BSTE7KRaWtOv2491bOn0uSi20qbCcFKVdpMjhMQkgaH4WX7
 fTM6BgKHhTXt3FLpjFrLzB7ziYYJQ/xG6laNZZsKp+Nl0H8wydQgQ40MK2JuqyhmuwCL
 YTl7FY5tjzyUD/SUy0+bHfn+m7waVxpcZ7Gu7lN/pmUCp9IJ1pb27hd6R89LSLNGrGEW
 LAs9413Ncv2+MhB8fHoB8HLlxEHC8MXQI6isk98O9uSUiOcQiJTeN7NInsM1QeC88a1s
 075hFBPLdG12QPGBbNWMM6p3VaqC71A1V/gj1X76wu3ckt01ZRIB1LqWvWTNLpUKRmFP
 AkPQ==
X-Gm-Message-State: AOAM531FxAlH6uuE/dhn6fgt7mDZIrWVmnP31QZfa22GgPJmptgq4Yc+
 TkVtxa0frCucCu9qkq8kxcxUO8+FLnk=
X-Google-Smtp-Source: ABdhPJxXGQf4cqKhiQ+xBqrTgSU3kTbjsnTE0KsoNRbxQGqVg2VrDPru4VLjwABGLGlrkK2Ocy+iew==
X-Received: by 2002:a1c:6208:: with SMTP id w8mr5235133wmb.96.1609966123769;
 Wed, 06 Jan 2021 12:48:43 -0800 (PST)
Received: from paquerette (176-142-29-14.abo.bbox.fr. [176.142.29.14])
 by smtp.gmail.com with ESMTPSA id z15sm4614422wrv.67.2021.01.06.12.48.43
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Wed, 06 Jan 2021 12:48:43 -0800 (PST)
User-agent: mu4e 1.0; emacs 28.0.50
From: =?utf-8?Q?=C3=89douard?= Debry <edouard.debry@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: [feature/native-comp] Excessive memory consumption on windows 10
Date: Wed, 06 Jan 2021 21:48:37 +0100
Message-ID: <87y2h5hjve.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed
Received-SPF: pass client-ip=2a00:1450:4864:20::32e;
 envelope-from=edouard.debry@HIDDEN; helo=mail-wm1-x32e.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)


Hello,

On windows 10, I noticed that emacs could use a lot of memory, up 
to 6Go and
sometimes more. The memory consumption goes up and down from time 
to time while
I am not running any specific program in emacs, apart `lsp-java`

As my personal computer has 16Go of RAM, I can afford 
it. Unfortunately, my work
computer has much less and the whole compute completely freezes at 
one point
when using emacs.

I did not notice that behavior on linux. I do not know if the 
master branch has
the same problem. What could be the problem ?




Acknowledgement sent to Édouard Debry <edouard.debry@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#45705; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Tue, 12 Jan 2021 11:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.