GNU bug report logs - #45926
Tabulated-list-mode wasteful properties

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 45926 in the body.
You can then email your comments to 45926 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Boruch Baum <boruch_baum <at> gmx.com>
To: Emacs Bug Reporting <bug-gnu-emacs <at> gnu.org>
Subject: Tabulated-list-mode wasteful properties
Date: Sun, 17 Jan 2021 02:32:52 -0500

Tabulated lists can grow quite large. In fact, the most common use for
the mode is the *Packages* buffer listing, which for me today clocks in
at 5211 lines * five columns. In the case of the *Packages* buffer, Performing

   (buffer-substring (point) (line-end-position))

on a sampling of lines yields 'PP eval' buffers of length ~500 - ~700
characters.

Most of this looks to me to be unnecessary duplicate data. In using the
mode for my own purpose, I see that the mode is putting all data of all
columns in an entry (row) in each and every single column, ie the size
grows exponentially by number of columns (in my personal case, ten
columns).

This isn't breaking anything, but it looks like bad design unnecessarily
hogging memory and filling buffers with cruft.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Message #8 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Sun, 17 Jan 2021 13:40:02 +0000

[Message part 1 (text/plain, inline)]

Boruch Baum <boruch_baum <at> gmx.com> writes:

> Tabulated lists can grow quite large. In fact, the most common use for
> the mode is the *Packages* buffer listing, which for me today clocks in
> at 5211 lines * five columns. In the case of the *Packages* buffer, Performing
>
>    (buffer-substring (point) (line-end-position))
>
> on a sampling of lines yields 'PP eval' buffers of length ~500 - ~700
> characters.
>
> Most of this looks to me to be unnecessary duplicate data. In using the
> mode for my own purpose, I see that the mode is putting all data of all
> columns in an entry (row) in each and every single column, ie the size
> grows exponentially by number of columns (in my personal case, ten
> columns).
>
> This isn't breaking anything, but it looks like bad design unnecessarily
> hogging memory and filling buffers with cruft.

Are you sure the data is actually duplicated and doesn't share
structure?  I evaluated that expression over the xref package, and got
the attached string.  Nothing looks obviously duplicated, except for the
shared #N= and #N# structures.  Also, some of the repetition of shared
structures is probably just due to how string intervals are
represented/printed.

Not that any of this is necessarily optimal, of course, but it's not
immediately alarming to me, either, unless I'm missing something.

-- 
Basil

[pp-eval.el (application/emacs-lisp, inline)]

Message #11 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Boruch Baum <boruch_baum <at> gmx.com>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Sun, 17 Jan 2021 10:11:28 -0500

On 2021-01-17 13:40, Basil L. Contovounesios wrote:
> Boruch Baum <boruch_baum <at> gmx.com> writes:
>
> Are you sure the data is actually duplicated and doesn't share
> structure?

Until you wrote, I definitely was. I think I still am, but I'm willing
to look again. It's easier for me since I just recently wrote a
ten-column tabulated list mode application. In my case I do see it
clearly because the text of each row in my tabulated list isn't
disturbed/confused by any text properties, and I do see an exact
duplication of each entire row entry in the property for each column of
that property.

I expect to publish the software within a day or two, and plan to
announce it on the emacs-devel and possibly other lists, at which time
you'll be able to see the internals for yourself. The package is planned
to be named 'crossword.el' and it brings to emacs the ability to
download and play crossword puzzles.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Message #14 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>, 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Tue, 19 Jan 2021 06:24:54 +0100

Boruch Baum <boruch_baum <at> gmx.com> writes:

> Until you wrote, I definitely was. I think I still am, but I'm willing
> to look again.

Like Basil, I looked at a tabulated list line, and I didn't see any
excessive duplication -- tabulated list mode puts the data on the entire
line, but it's the same data structure, so it's all shared data, I
think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #19 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Boruch Baum <boruch_baum <at> gmx.com>, 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Thu, 11 Mar 2021 18:52:11 -0600

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Boruch Baum <boruch_baum <at> gmx.com> writes:
>
>> Until you wrote, I definitely was. I think I still am, but I'm willing
>> to look again.

Borouch, did you have the time to look into if this data is indeed
shared or not?  Thanks.

> Like Basil, I looked at a tabulated list line, and I didn't see any
> excessive duplication -- tabulated list mode puts the data on the entire
> line, but it's the same data structure, so it's all shared data, I
> think?

Message #22 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Boruch Baum <boruch_baum <at> gmx.com>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Thu, 11 Mar 2021 21:57:09 -0500

On 2021-03-11 18:52, Stefan Kangas wrote:
> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
> > Boruch Baum <boruch_baum <at> gmx.com> writes:
> >
> >> Until you wrote, I definitely was. I think I still am, but I'm willing
> >> to look again.
>
> Borouch, did you have the time to look into if this data is indeed
> shared or not?  Thanks.

Oh. I thought I had done that right away ...

Doing it again now. Yes. It's very simple to check for yourself. Open a
buffer that uses the mode and evaluate from any table line ...

  (buffer-substring (line-beginning-position) (line-end-position))

You should be able to clearly see all the duplication. It's easiest when
you have pretty-print set up to display the output in a dedicated
buffer, but even without that you can isearch through the duplications.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Message #25 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Thu, 11 Mar 2021 21:39:39 -0600

Boruch Baum <boruch_baum <at> gmx.com> writes:

>> Borouch, did you have the time to look into if this data is indeed
>> shared or not?  Thanks.
>
> Oh. I thought I had done that right away ...
>
> Doing it again now. Yes. It's very simple to check for yourself. Open a
> buffer that uses the mode and evaluate from any table line ...
>
>   (buffer-substring (line-beginning-position) (line-end-position))
>
> You should be able to clearly see all the duplication. It's easiest when
> you have pretty-print set up to display the output in a dedicated
> buffer, but even without that you can isearch through the duplications.

I think the interesting question is if that data is duplicated in memory
or if these are just pointers to the same shared data.

I would try modifying this data with something like `setcdr' to see if
it updates everywhere -- if the answer yes, then this data is shared and
there is no problem here.

Message #28 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Boruch Baum <boruch_baum <at> gmx.com>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Fri, 12 Mar 2021 00:46:07 -0500

On 2021-03-11 21:39, Stefan Kangas wrote:
> I would try modifying this data with something like `setcdr' to see if
> it updates everywhere -- if the answer yes, then this data is shared and
> there is no problem here.

Why are you telling me this when it's just as easy for you to do it
yourself?

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Message #31 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, Stefan Kangas <stefan <at> marxist.se>,
 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Fri, 12 Mar 2021 10:51:34 +0000

Boruch Baum <boruch_baum <at> gmx.com> writes:

> Doing it again now. Yes. It's very simple to check for yourself. Open a
> buffer that uses the mode and evaluate from any table line ...
>
>   (buffer-substring (line-beginning-position) (line-end-position))
>
> You should be able to clearly see all the duplication. It's easiest when
> you have pretty-print set up to display the output in a dedicated
> buffer, but even without that you can isearch through the duplications.

Have you tried setting print-circle non-nil first?  This will help you
see what is actually duplicated and what is shared.

-- 
Basil

Message #34 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, Stefan Kangas <stefan <at> marxist.se>,
 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Fri, 12 Mar 2021 11:01:30 +0000

Boruch Baum <boruch_baum <at> gmx.com> writes:

> On 2021-03-11 21:39, Stefan Kangas wrote:
>> I would try modifying this data with something like `setcdr' to see if
>> it updates everywhere -- if the answer yes, then this data is shared and
>> there is no problem here.
>
> Why are you telling me this when it's just as easy for you to do it
> yourself?

The opinions voiced so far are that there does not seem to be excessive
duplication of data, since most of it is shared.

You can help the discussion by providing either evidence to the
contrary, or a patch that demonstrably lowers the memory usage of
tabulated-list-mode and the package list.

Stefan is being helpful by nudging a bug discussion that fell silent; it
is unfair to expect him to do all the work (and where's the fun in that
anyway ;).

-- 
Basil

Message #41 received at 45926-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, 45926-done <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Tue, 20 Apr 2021 22:35:53 -0500

Stefan Kangas <stefan <at> marxist.se> writes:

> Boruch Baum <boruch_baum <at> gmx.com> writes:
>
>>> Borouch, did you have the time to look into if this data is indeed
>>> shared or not?  Thanks.
>>
>> Oh. I thought I had done that right away ...
>>
>> Doing it again now. Yes. It's very simple to check for yourself. Open a
>> buffer that uses the mode and evaluate from any table line ...
>>
>>   (buffer-substring (line-beginning-position) (line-end-position))
>>
>> You should be able to clearly see all the duplication. It's easiest when
>> you have pretty-print set up to display the output in a dedicated
>> buffer, but even without that you can isearch through the duplications.
>
> I think the interesting question is if that data is duplicated in memory
> or if these are just pointers to the same shared data.
>
> I would try modifying this data with something like `setcdr' to see if
> it updates everywhere -- if the answer yes, then this data is shared and
> there is no problem here.

More information was requested, but none was given within 5 weeks, so
I'm closing this bug.

If anyone can provide further information on this issue to show that
there is a bug here, please reply to this email (use "Reply to all" in
your email client) and we can reopen the bug report.

Message #44 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Jean Louis <bugs <at> gnu.support>
To: Boruch Baum <boruch_baum <at> gmx.com>
Cc: "Basil L. Contovounesios" <contovob <at> tcd.ie>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, Stefan Kangas <stefan <at> marxist.se>,
 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Wed, 21 Apr 2021 07:31:24 +0300

* Boruch Baum <boruch_baum <at> gmx.com> [2021-03-12 05:58]:
> On 2021-03-11 18:52, Stefan Kangas wrote:
> > Lars Ingebrigtsen <larsi <at> gnus.org> writes:
> >
> > > Boruch Baum <boruch_baum <at> gmx.com> writes:
> > >
> > >> Until you wrote, I definitely was. I think I still am, but I'm willing
> > >> to look again.
> >
> > Borouch, did you have the time to look into if this data is indeed
> > shared or not?  Thanks.
> 
> Oh. I thought I had done that right away ...
> 
> Doing it again now. Yes. It's very simple to check for yourself. Open a
> buffer that uses the mode and evaluate from any table line ...
> 
>   (buffer-substring (line-beginning-position) (line-end-position))
> 
> You should be able to clearly see all the duplication. It's easiest when
> you have pretty-print set up to display the output in a dedicated
> buffer, but even without that you can isearch through the
> duplications.

I have done this excercise, as I have people management in the
database.

If tabulated-list-entries is ⇒ ((320295 ["320295" "Boruch  Baum" "Emacs Users"]))

Then this below is what I get when invoking: (buffer-substring (line-beginning-position) (line-end-position)):

#(" 320295   Boruch  Baum                             Emacs Users" 0 1 (tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 1 7 (help-echo "ID: 320295" tabulated-list-column-name "ID" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 7 8 (tabulated-list-column-name "ID" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 8 10 (display (space :align-to 10) tabulated-list-column-name "ID" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 10 22 (help-echo "Name: Boruch  Baum" tabulated-list-column-name "Name" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 22 23 (tabulated-list-column-name "Name" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 23 51 (display (space :align-to 51) tabulated-list-column-name "Name" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]) 51 62 (help-echo "Account: Emacs Users" tabulated-list-column-name "Account" tabulated-list-id 320295 tabulated-list-entry ["320295" "Boruch  Baum" "Emacs Users"]))

Message #47 received at 45926 <at> debbugs.gnu.org (full text, mbox):

From: Jean Louis <bugs <at> gnu.support>
To: "Basil L. Contovounesios" <contovob <at> tcd.ie>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, Stefan Kangas <stefan <at> marxist.se>,
 Boruch Baum <boruch_baum <at> gmx.com>, 45926 <at> debbugs.gnu.org
Subject: Re: bug#45926: Tabulated-list-mode wasteful properties
Date: Wed, 21 Apr 2021 07:36:14 +0300

* Basil L. Contovounesios <contovob <at> tcd.ie> [2021-03-12 13:52]:
> Boruch Baum <boruch_baum <at> gmx.com> writes:
> 
> > Doing it again now. Yes. It's very simple to check for yourself. Open a
> > buffer that uses the mode and evaluate from any table line ...
> >
> >   (buffer-substring (line-beginning-position) (line-end-position))
> >
> > You should be able to clearly see all the duplication. It's easiest when
> > you have pretty-print set up to display the output in a dedicated
> > buffer, but even without that you can isearch through the duplications.
> 
> Have you tried setting print-circle non-nil first?  This will help you
> see what is actually duplicated and what is shared.

I have done that exercise where my tabulated-list-entries is only
this:

⇒ ((320295 ["320295" "Boruch  Baum" "Emacs Users"]))

and

(buffer-substring (line-beginning-position) (line-end-position))

gives me this:

a ⇒ #(" 320295   Boruch  Baum                             Emacs Users" 0 1 (tabulated-list-id 320295 tabulated-list-entry #1=["320295" "Boruch  Baum" "Emacs Users"]) 1 7 (help-echo "ID: 320295" tabulated-list-column-name #3="ID" tabulated-list-id 320295 tabulated-list-entry #1#) 7 8 (tabulated-list-column-name #3# tabulated-list-id 320295 tabulated-list-entry #1#) 8 10 (display (space :align-to 10) tabulated-list-column-name #3# tabulated-list-id 320295 tabulated-list-entry #1#) 10 22 (help-echo "Name: Boruch  Baum" tabulated-list-column-name #2="Name" tabulated-list-id 320295 tabulated-list-entry #1#) 22 23 (tabulated-list-column-name #2# tabulated-list-id 320295 tabulated-list-entry #1#) 23 51 (display (space :align-to 51) tabulated-list-column-name #2# tabulated-list-id 320295 tabulated-list-entry #1#) 51 62 (help-echo "Account: Emacs Users" tabulated-list-column-name "Account" tabulated-list-id 320295 tabulated-list-entry #1#))

If non-nil, shared substructures anywhere in the structure are printed
with ‘#N=’ before the first occurrence (in the order of the print
representation) and ‘#N#’ in place of each subsequent occurrence,
where N is a positive decimal integer.

Thus I can see that those are shared structures mostly.

Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.