GNU bug report logs - #65697
numproc, add flag to get numer of available cores

Previous Next

To reply to this bug, email your comments to 65697 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Heimberg <simon.heimberg <at> heimberg-ea.ch>
To: bug-coreutils <at> gnu.org
Subject: numproc, add flag to get numer of available cores
Date: Sat, 02 Sep 2023 11:38:46 +0200

Hello

Depending on what a process is mainly limited by, the ideal number of
processes to run concurrently varies.
A frequent use case is to run one process on each _physical_ core
(which is available) than on each logical processing unit.
I propose to introduce a flag --core which returns the number of
physical cores available to the process (or of all cores with --all).

I made a draft in python using getaffinity(), assuming the threads of a
core are listed after each other:
https://github.com/giampaolo/psutil/issues/1968#issuecomment-1688687251

Looking at the code in gnulib/lib/nproc.c this could get hard to
implement for all cases. (Many cases. No list for physical cores exists
yet.)

Maybe it is better to extend the doc and explain what the number really
means (with and without --all). Or add a link to more details about
processors and cores and ... . Extended doc could improve:
* As #42044 complains that nproc does not include the cores which go
online automatically. (The current help text _could_ match this.)
* As I complain that nproc can not report (available) _physical_ cores.
* I just found out (partially) why --all shows more cpus than are in my
system. (It returns the _configured_ ones. And those get set ...:
https://unix.stackexchange.com/questions/609555/how-does-linux-detect-number-of-possible-cpus
) But the help writes "installed processors", not configured ones. (On
my system, processors which are configured only are not listed by ls -l
/sys/devices/system/cpu/cpu*)


About better using one process for each core, depending on ...:
https://github.com/giampaolo/psutil/issues/1968
https://stackoverflow.com/questions/27404532/optimal-number-of-processes
https://stackoverflow.com/questions/36958661/openmp-dont-use-hyperthreading-cores-half-num-threads-w-hyperthreading

Message #8 received at 65697 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: 65697 <at> debbugs.gnu.org, Simon Heimberg <simon.heimberg <at> heimberg-ea.ch>
Subject: Re: bug#65697: numproc, add flag to get number of available cores
Date: Sat, 2 Sep 2023 15:08:29 +0100

On 02/09/2023 10:38, Simon Heimberg wrote:
> Hello
> 
> Depending on what a process is mainly limited by, the ideal number of
> processes to run concurrently varies.
> A frequent use case is to run one process on each _physical_ core
> (which is available) than on each logical processing unit.
> I propose to introduce a flag --core which returns the number of
> physical cores available to the process (or of all cores with --all).

That option should probably be called --physical.
Moving your use cases for this from below...

> About better using one process for each core, depending on ...:
> https://github.com/giampaolo/psutil/issues/1968
> https://stackoverflow.com/questions/27404532/optimal-number-of-processes
> https://stackoverflow.com/questions/36958661/openmp-dont-use-hyperthreading-cores-half-num-threads-w-hyperthreading

> I made a draft in python using getaffinity(), assuming the threads of a
> core are listed after each other:
> https://github.com/giampaolo/psutil/issues/1968#issuecomment-1688687251
> 
> Looking at the code in gnulib/lib/nproc.c this could get hard to
> implement for all cases. (Many cases. No list for physical cores exists
> yet.)
> 
> Maybe it is better to extend the doc and explain what the number really
> means (with and without --all). Or add a link to more details about
> processors and cores and ... . Extended doc could improve:
> * As #42044 complains that nproc does not include the cores which go
> online automatically. (The current help text _could_ match this.)

Right https://bugs.gnu.org/42044 is an interesting case we should probably support

> * As I complain that nproc can not report (available) _physical_ cores.
> * I just found out (partially) why --all shows more cpus than are in my
> system. (It returns the _configured_ ones. And those get set ...:
> https://unix.stackexchange.com/questions/609555/how-does-linux-detect-number-of-possible-cpus
> ) But the help writes "installed processors", not configured ones. (On
> my system, processors which are configured only are not listed by ls -l
> /sys/devices/system/cpu/cpu*)

Right there does seem to be an issue here.
On my 4 core laptop, there are 2 physical cores, and 2 hyperthread cores per physical core.

nproc shows 4 which is correct
nproc --all shows 8 which is surprising

An `strace nproc --all` shows that this 8 value is read from /sys/devices/system/cpu/possible
Now nproc uses getconf, so `getconf _NPROCESSORS_CONF` is reading this proc value.

Digging into the proc values, on my system we have:

  $ grep - /sys/devices/system/cpu/* 2>/dev/null
  /sys/devices/system/cpu/offline:4-7
  /sys/devices/system/cpu/online:0-3
  /sys/devices/system/cpu/possible:0-7
  /sys/devices/system/cpu/present:0-3

So the "possible" values above represent statically allocated kernel structures.
I.e. the kernel could support 4 more hotplug CPUs.
But that's not the normal hardware setup so it is surprising.

This change in glibc happened recently in glibc-2.36:
https://sourceware.org/git/?p=glibc.git;a=commit;h=97a912f7
So I presume you're also on a recent glibc system.

The associated glibc discussion is at:
https://sourceware.org/bugzilla/show_bug.cgi?id=28991

Now it's debatable as to whether glibc should be using /sys/devices/system/cpu/present
or /sys/devices/system/cpu/possible for _NPROCESSORS_CONF.
I notice also that uclibc and musl use /proc enumeration and affinity mask respectively,
which would return 4 on my system, rather than 8.

If we don't change any logic, then at least we should tweak the docs like:
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 373a407ed..bd7a25219 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -17097,8 +17097,9 @@ The program accepts the following options.  Also see @ref{Common options}.

 @item --all
 @opindex --all
-Print the number of installed processors on the system, which may
-be greater than the number online or available to the current process.
+Print the number of potentially installed processors on the system, which may
+be greater than the number currently present or online in the system,
+or available to the current process.
 The @env{OMP_NUM_THREADS} or @env{OMP_THREAD_LIMIT} environment variables
 are not honored in this case.


There are various combinations of present, online, masked, physical CPUs
so I'm not sure if it's feasible to add other selection options to nproc.

cheers,
Pádraig

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.