GNU bug report logs - #18995
Error: Could not reserve dynamic heap area.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 18995 in the body.
You can then email your comments to 18995 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: bug-gnu-emacs <bug-gnu-emacs <at> gnu.org>
Subject: Error: Could not reserve dynamic heap area.
Date: Sat, 8 Nov 2014 16:22:56 +0100

[Message part 1 (text/plain, inline)]

The build enviroment is MinGW-w64 and MSYS2.

When I build `emacs-24', Emacs is being configured with

Should Emacs use the GNU version of malloc?  yes
Should Emacs use a relocating allocator for buffers? yes
Should Emacs use mmap(2) for buffer allocation?  no

With this configuration the x64 version is built just fine, but the x86
version build fails with

make[2]: Entering directory
> '/c/Users/Haroogan/Projects/GitHub/MINGW-packages/mingw-w64-emacs/src/.build/i686-w64-mingw32/lisp'
> Compiling
> /C/Users/Haroogan/Projects/GitHub/MINGW-packages/mingw-w64-emacs/src/emacs/lisp/emacs-lisp/macroexp.el
> Error: Could not reserve dynamic heap area.


 I've checked what is called in this case and it seems to be fine at the
first glance:

#else  /* USE_LSB_TAG */
static char *
allocate_heap (void)
{
#ifdef _WIN64
  size_t size = 0x4000000000ull; /* start by asking for 32GB */
#else
  /* We used to start with 2GB here, but on Windows 7 that would leave
     too little room in the address space for threads started by
     Windows on our behalf, e.g. when we pop up the file selection
     dialog.  */
  size_t size = 0x68000000; /* start by asking for 1.7GB */ <<< This one is
used.
#endif
  void *ptr = NULL;

  while (!ptr && size > 0x00100000)
    {
      reserved_heap_size = size;
      ptr = VirtualAlloc (NULL,
  get_reserved_heap_size (),
  MEM_RESERVE,
  PAGE_NOACCESS);
      size -= 0x00800000; /* if failed, decrease request by 8MB */
    }

  return ptr;
}
#endif /* USE_LSB_TAG */


Any ideas?

[Message part 2 (text/html, inline)]

Message #8 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 08 Nov 2014 19:29:08 +0200

> Date: Sat, 8 Nov 2014 16:22:56 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> 
> When I build `emacs-24', Emacs is being configured with
> 
> Should Emacs use the GNU version of malloc? yes
> Should Emacs use a relocating allocator for buffers? yes
> Should Emacs use mmap(2) for buffer allocation? no
> 
> With this configuration the x64 version is built just fine, but the x86 version
> build fails with
> 
>     make[2]: Entering directory
>     '/c/Users/Haroogan/Projects/GitHub/MINGW-packages/mingw-w64-emacs/src/.build/i686-w64-mingw32/lisp'
>    
>     Compiling
>     /C/Users/Haroogan/Projects/GitHub/MINGW-packages/mingw-w64-emacs/src/emacs/lisp/emacs-lisp/macroexp.el
>    
>     Error: Could not reserve dynamic heap area.
> 
> I've checked what is called in this case and it seems to be fine at the first
> glance:

Sorry, I don't understand: what part of the allocate_heap function is
being run and fails in your case?  Does the loop start with
0x4000000000 or with 0x68000000?  Does it go all the way through
0x00100000, and each call to VirtualAlloc fails?  If so, can you see
what error code does VirtualAlloc return?  You can accomplish the
latter in GDB like this:

  (gdb) p w32_last_error()

Also, does the same command succeed if you run it from the Bash
command line?  Does it succeed if you run it from the cmd.exe command
line?

Finally, did you run the build from the Bash command line outside
Emacs, or did you run it from inside another Emacs session?

Message #11 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 8 Nov 2014 19:17:55 +0100

[Message part 1 (text/plain, inline)]

>
> or did you run it from inside another Emacs session?


No.

Also, does the same command succeed if you run it from the Bash
> command line?  Does it succeed if you run it from the cmd.exe command
> line?


No, has nothing to do with it.

I've got something amazing/phenomenal here. Read carefully. With "-g -O0"
it can reserve memory just fine and everything builds successfully. With
"-O3" it fails with the first try, i.e. with the size of 0x68000000 and the
corresponding error is:

ERROR_NOT_ENOUGH_MEMORY
> 8 (0x8)
> Not enough storage is available to process this command.


 Since with "-g -O0" it succeeds, it cannot be debugged with GDB. For "-O3"
I used simple `printf' testing.

 I can test "-O2" if you are interested.

What does it smell like? A toolchain bug?

[Message part 2 (text/html, inline)]

Message #14 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 08 Nov 2014 20:30:44 +0200

> Date: Sat, 8 Nov 2014 19:17:55 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> Cc: 18995 <at> debbugs.gnu.org
> 
> I've got something amazing/phenomenal here. Read carefully. With "-g -O0" it
> can reserve memory just fine and everything builds successfully. With "-O3" it
> fails with the first try, i.e. with the size of 0x68000000 and the
> corresponding error is:
> 
>     ERROR_NOT_ENOUGH_MEMORY
>     8 (0x8)
>     Not enough storage is available to process this command.
> 
> Since with "-g -O0" it succeeds, it cannot be debugged with GDB. For "-O3" I
> used simple `printf' testing.
> 
> I can test "-O2" if you are interested.

Yes, please do.  I don't think I ever tried -O3 (and I don't think
it's a good idea in Emacs in general).

> What does it smell like? A toolchain bug?

Could be.  What GCC version are you using?

Also, when -O3 is used, does it help to declare reserved_heap_size
volatile, like this:

  size_t volatile reserved_heap_size;

(You might need to change the extern declaration on w32heap.h as
well.)

Message #17 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 8 Nov 2014 19:40:58 +0100

[Message part 1 (text/plain, inline)]

Both "-O2" and "-O1" don't work as well. Only "-O0" works.

Could be.  What GCC version are you using?


Latest:

gcc.exe (Rev2, Built by MSYS2 project) 4.9.2


Also, when -O3 is used, does it help to declare reserved_heap_size
> volatile, like this:
>


  size_t volatile reserved_heap_size;
>


(You might need to change the extern declaration on w32heap.h as
> well.)


Unfortunately, nope.

[Message part 2 (text/html, inline)]

Message #20 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 08 Nov 2014 20:57:13 +0200

> Date: Sat, 8 Nov 2014 19:40:58 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> Cc: 18995 <at> debbugs.gnu.org
> 
> Both "-O2" and "-O1" don't work as well. Only "-O0" works.

So what, the optimized code goes only once through the loop, and then
bails out?  If so, what is the value of 'size' when the loop ends?

Message #23 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 8 Nov 2014 20:16:19 +0100

[Message part 1 (text/plain, inline)]

>
> So what, the optimized code goes only once through the loop, and then
> bails out?  If so, what is the value of 'size' when the loop ends?
>

OK, so there was one more detail that I forgot to mention. It looks like I
also had "-funroll-loops". After removing it, "-O3" works fine too (without
"volatile"). I think this is still a toolchain bug because "-O3
-funroll-loops" combination works fine in the x64 build and what's more
important --- it *should* work fine. Anyway, maybe it's a good idea to
somehow prevent people from passing "-funroll-loops" by filtering it out
from "CFLAGS" in "Makefile" for example?

[Message part 2 (text/html, inline)]

Message #26 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 08 Nov 2014 21:50:13 +0200

> Date: Sat, 8 Nov 2014 20:16:19 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> Cc: 18995 <at> debbugs.gnu.org
> 
>     So what, the optimized code goes only once through the loop, and then
>     bails out? If so, what is the value of 'size' when the loop ends?
>     
> 
> OK, so there was one more detail that I forgot to mention. It looks like I also
> had "-funroll-loops". After removing it, "-O3" works fine too (without
> "volatile"). I think this is still a toolchain bug because "-O3 -funroll-loops"
> combination works fine in the x64 build and what's more important --- it
> *should* work fine. Anyway, maybe it's a good idea to somehow prevent people
> from passing "-funroll-loops" by filtering it out from "CFLAGS" in "Makefile"
> for example?

Do you understand why -funroll-loops causes the code to fail to work?
Because I cannot.  Unless I'm missing something, this sounds like a
GCC bug.

Message #29 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 8 Nov 2014 20:58:02 +0100

[Message part 1 (text/plain, inline)]

>
> Do you understand why -funroll-loops causes the code to fail to work?
> Because I cannot.  Unless I'm missing something, this sounds like a
> GCC bug.
>

I've no idea. For that we would have to look into generated assembly code,
but I'd rather leave that for compiler professionals to analyze. Would
forward it to GCC guys?

And how about filtering this flag for good as I asked earlier?

CFLAGS := $(filter-out -funroll-loops,$(CFLAGS))

[Message part 2 (text/html, inline)]

Message #32 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 08 Nov 2014 22:05:26 +0200

> Date: Sat, 8 Nov 2014 20:58:02 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> Cc: 18995 <at> debbugs.gnu.org
> 
>     Do you understand why -funroll-loops causes the code to fail to work?
>     
>     Because I cannot. Unless I'm missing something, this sounds like a
>     GCC bug.
>     
> 
> I've no idea. For that we would have to look into generated assembly code, but
> I'd rather leave that for compiler professionals to analyze. Would forward it
> to GCC guys?

It probably does warrant a GCC bug report.

> And how about filtering this flag for good as I asked earlier?
> 
>     CFLAGS := $(filter-out -funroll-loops,$(CFLAGS))

What for?  It's a perfectly legitimate compiler switch, and should
work here as well, I think.

Message #35 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: Eli Zaretskii <eliz <at> gnu.org>, Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sat, 08 Nov 2014 20:49:01 +0000

[Message part 1 (text/plain, inline)]

On 11/08/2014 08:05 PM, Eli Zaretskii wrote:
>> Date: Sat, 8 Nov 2014 20:58:02 +0100
>> From: Alexander Shukaev <haroogan <at> gmail.com>
>> Cc: 18995 <at> debbugs.gnu.org
>>
>>     Do you understand why -funroll-loops causes the code to fail to work?
>>     
>>     Because I cannot. Unless I'm missing something, this sounds like a
>>     GCC bug.
>>     
>>
>> I've no idea. For that we would have to look into generated assembly code, but
>> I'd rather leave that for compiler professionals to analyze. Would forward it
>> to GCC guys?
> 
> It probably does warrant a GCC bug report.
> 
>> And how about filtering this flag for good as I asked earlier?
>>
>>     CFLAGS := $(filter-out -funroll-loops,$(CFLAGS))
> 
> What for?  It's a perfectly legitimate compiler switch, and should
> work here as well, I think.

Agreed on not filtering it out.

Alexander, does it help to add explicit debug output (say, fprintf to
stderr) on each iteration of the loop? If not, can you compare the debug
output from the version with -funroll-loops to the one without, to see
whether we're making the same sequence of VirtualAlloc calls? If they're
the same, maybe it's not the loop that's broken, but the address space
layout.

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 18995-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Daniel Colascione <dancol <at> dancol.org>
Cc: haroogan <at> gmail.com, 18995-done <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 18:12:16 +0200

> Date: Sat, 08 Nov 2014 20:49:01 +0000
> From: Daniel Colascione <dancol <at> dancol.org>
> CC: 18995 <at> debbugs.gnu.org
> 
> Alexander, does it help to add explicit debug output (say, fprintf to
> stderr) on each iteration of the loop?

I'm not Alexander, but I can answer that question now: no, it doesn't
help.  The loop is still executed only once, even if it calls fprintf.

> If not, can you compare the debug output from the version with
> -funroll-loops to the one without, to see whether we're making the
> same sequence of VirtualAlloc calls? If they're the same, maybe it's
> not the loop that's broken, but the address space layout.

There's nothing wrong with the address space, and there's nothing
wrong with GCC, either.  What we have here is a genuine bug in our
code, albeit one that is subtle, and also very hard to actually
reproduce in real life.

It looked like a GCC bug at first, but as I tried to modify the source
and look at the effect of that on the generated code, it finally
dawned on me: GCC's loop-unrolling code simply correctly calculated
that with the initial value of 0x68000000 and decrement of 0x00800000,
the value of 'size' in the loop will never be less than 0x00100000,
due to wraparound in the subtraction of unsigned values.  So what we
have here is a potentially infinite loop, i.e. "undefined behavior".
Given that, it is justified for GCC to give us what we deserve, i.e. a
loop "unrolled" by executing its body only once.  I present the
disassembly below, for the curious, and it's clear that there's no
loop there, and also the value of 'size' is never tested at all, since
GCC decided that the condition 'size > 0x00100000' is always true.

Of course, in reality this "infinite" loop will always be finite,
since a real live system will always find more than 8MB of free
address space to reserve.  Unless someone deliberately reserved a lot
of the address space just to cause Emacs hit this problem, that is.

I installed a trivial fix on the emacs-24 branch, and I'm closing this
bug.

Here's the disassembly of the original code, as compiled with
"-funroll-loops" (it is for init_heap, because allocate_heap is
inlined in it):

  (gdb) disassemble init_heap
  Dump of assembler code for function init_heap:
     0x0122f1fc <+0>:     push   %ebx
     0x0122f1fd <+1>:     sub    $0x18,%esp
     0x0122f200 <+4>:     movl   $0x0,(%esp)
     0x0122f207 <+11>:    call   0x1277184 <GetModuleHandleA <at> 4>
     0x0122f20c <+16>:    sub    $0x4,%esp
     0x0122f20f <+19>:    add    0x3c(%eax),%eax
     0x0122f212 <+22>:    mov    %eax,0x4(%esp)
     0x0122f216 <+26>:    movl   $0x14c1cd0,(%esp)
     0x0122f21d <+33>:    call   0x11c4cb5 <find_section>
     0x0122f222 <+38>:    mov    %eax,0x154abd4
     0x0122f227 <+43>:    cmpl   $0x0,0x1533cf0
     0x0122f22e <+50>:    je     0x122f28b <init_heap+143>
     0x0122f230 <+52>:    mov    $0x68000000,%ecx
     0x0122f235 <+57>:    mov    %ecx,0x15422a0
     0x0122f23b <+63>:    movl   $0x1,0xc(%esp)
     0x0122f243 <+71>:    movl   $0x2000,0x8(%esp)
     0x0122f24b <+79>:    mov    %ecx,0x4(%esp)
     0x0122f24f <+83>:    movl   $0x0,(%esp)
     0x0122f256 <+90>:    call   0x12775cc <VirtualAlloc <at> 16>
     0x0122f25b <+95>:    sub    $0x10,%esp
     0x0122f25e <+98>:    mov    %eax,0x15422ac
     0x0122f263 <+103>:   test   %eax,%eax                  <<<<<<<<<<<
     0x0122f265 <+105>:   jne    0x122f27f <init_heap+131>  <<<<<<<<<<<
     0x0122f267 <+107>:   movl   $0x14c1cd8,(%esp)          <<<<<<<<<<<
     0x0122f26e <+114>:   call   0x122f0e0 <printf>         <<<<<<<<<<<
     0x0122f273 <+119>:   movl   $0x1,(%esp)                <<<<<<<<<<<
     0x0122f27a <+126>:   call   0x1276948 <exit>           <<<<<<<<<<<
     0x0122f27f <+131>:   mov    %eax,0x15422a8
     0x0122f284 <+136>:   mov    %eax,0x15422a4
     0x0122f289 <+141>:   jmp    0x122f2bc <init_heap+192>
     0x0122f28b <+143>:   mov    0xc(%eax),%ebx
     0x0122f28e <+146>:   movl   $0x0,(%esp)
     0x0122f295 <+153>:   call   0x1277184 <GetModuleHandleA <at> 4>
     0x0122f29a <+158>:   sub    $0x4,%esp
     0x0122f29d <+161>:   add    %ebx,%eax
     0x0122f29f <+163>:   mov    %eax,0x15422ac
     0x0122f2a4 <+168>:   mov    %eax,0x15422a8
     0x0122f2a9 <+173>:   mov    %eax,0x15422a4
     0x0122f2ae <+178>:   mov    0x154abd4,%eax
     0x0122f2b3 <+183>:   mov    0x8(%eax),%edx
     0x0122f2b6 <+186>:   mov    %edx,0x15422a0
     0x0122f2bc <+192>:   call   0x1201f1d <cache_system_info>
     0x0122f2c1 <+197>:   add    $0x18,%esp
     0x0122f2c4 <+200>:   pop    %ebx
     0x0122f2c5 <+201>:   ret
  End of assembler dump.

Look, ma: no loop!

Message #43 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: 18995 <at> debbugs.gnu.org
Cc: eliz <at> gnu.org, haroogan <at> gmail.com
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 17:30:21 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> Given that, it is justified for GCC to give us what we deserve, i.e. a
> loop "unrolled" by executing its body only once.  I present the
> disassembly below, for the curious, and it's clear that there's no
> loop there, and also the value of 'size' is never tested at all, since
> GCC decided that the condition 'size > 0x00100000' is always true.

But if you replace 'size > 0x00100000' with true, you get a loop that is
conditionalized by '!ptr', which depends on the execution of the loop
body.  Why would it be correct to execute the loop only once,
unconditionally?

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #46 received at 18995-done <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: Daniel Colascione <dancol <at> dancol.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 18995-done <at> debbugs.gnu.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 9 Nov 2014 17:30:38 +0100

[Message part 1 (text/plain, inline)]

>
> How can this be a bug in our code? Wraparound of unsigned values is
> well-defined.


True, it is well-defined, but it was not intended for sure, hence it's a
100% bug. Eli's term of "undefined behavior" was an exaggeration I think.

I don't think the compiler is justified in making this optimization.
> Since when is an infinite loop undefined behavior? GCC has no right to
> alter program semantics in this case, loop unrolling or no.


I would agree here, throwing out wrapped loop is just too much even though
such loop was unintended. GCC should have either left it alone since it is
infinite or make a warning. I think this case is still worth showing to GCC
developers.

[Message part 2 (text/html, inline)]

Message #49 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: 18995 <at> debbugs.gnu.org
Cc: eliz <at> gnu.org, haroogan <at> gmail.com
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 17:35:30 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> It looked like a GCC bug at first, but as I tried to modify the source
> and look at the effect of that on the generated code, it finally
> dawned on me: GCC's loop-unrolling code simply correctly calculated
> that with the initial value of 0x68000000 and decrement of 0x00800000,
> the value of 'size' in the loop will never be less than 0x00100000,
> due to wraparound in the subtraction of unsigned values.

Before wraping around, size will become zero, which is definitely less
than 0x00100000.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #52 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: 18995 <at> debbugs.gnu.org
Cc: Eli Zaretskii <eliz <at> gnu.org>, dancol <at> dancol.org,
 Andreas Schwab <schwab <at> linux-m68k.org>
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 9 Nov 2014 17:39:32 +0100

[Message part 1 (text/plain, inline)]

Guys, don't forget one important detail, that this happens only in 32-bit
build. 64-bit does not have this problem.

[Message part 2 (text/html, inline)]

Message #55 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: 18995 <at> debbugs.gnu.org
Cc: Eli Zaretskii <eliz <at> gnu.org>, haroogan <at> gmail.com
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 16:40:17 +0000

[Message part 1 (text/plain, inline)]

On 11/09/2014 04:33 PM, Eli Zaretskii wrote:
> Why off-list?

User error. Re-adding the list.

>> Date: Sun, 09 Nov 2014 16:23:00 +0000
>> From: Daniel Colascione <dancol <at> dancol.org>
>> CC: haroogan <at> gmail.com
>>
>>> There's nothing wrong with the address space, and there's nothing
>>> wrong with GCC, either.  What we have here is a genuine bug in our
>>> code, albeit one that is subtle, and also very hard to actually
>>> reproduce in real life.
>>
>> How can this be a bug in our code?
> 
> Because whoever wrote that loop intended it to stop and bail out at
> 0x00100000.  And that cannot happen with unsigned subtraction.

Sure. And the fix you installed is good, but the bug shouldn't have
caused GCC to reduce the loop to one iteration.

>>> It looked like a GCC bug at first, but as I tried to modify the source
>>> and look at the effect of that on the generated code, it finally
>>> dawned on me: GCC's loop-unrolling code simply correctly calculated
>>> that with the initial value of 0x68000000 and decrement of 0x00800000,
>>> the value of 'size' in the loop will never be less than 0x00100000,
>>> due to wraparound in the subtraction of unsigned values.  So what we
>>> have here is a potentially infinite loop, i.e. "undefined behavior".
>>
>> I don't think the compiler is justified in making this optimization.
>> Since when is an infinite loop undefined behavior? GCC has no right to
>> alter program semantics in this case, loop unrolling or no.
> 
> OK, I'm not a C Standard lawyer, so I won't argue here.  It's enough
> for me to know that the trivial fix I installed makes the code work
> even under -funroll-loops, and that it fixes a real problem in our
> code.

Sure, as well as exposing one in GCC.

[signature.asc (application/pgp-signature, attachment)]

Message #58 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 18995 <at> debbugs.gnu.org, haroogan <at> gmail.com
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 18:47:00 +0200

> From: Andreas Schwab <schwab <at> linux-m68k.org>
> Cc: eliz <at> gnu.org,  haroogan <at> gmail.com
> Date: Sun, 09 Nov 2014 17:30:21 +0100
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > Given that, it is justified for GCC to give us what we deserve, i.e. a
> > loop "unrolled" by executing its body only once.  I present the
> > disassembly below, for the curious, and it's clear that there's no
> > loop there, and also the value of 'size' is never tested at all, since
> > GCC decided that the condition 'size > 0x00100000' is always true.
> 
> But if you replace 'size > 0x00100000' with true, you get a loop that is
> conditionalized by '!ptr', which depends on the execution of the loop
> body.  Why would it be correct to execute the loop only once,
> unconditionally?

Feel free to report this as a GCC bug, if you think that's what it is.
All I can tell is that the same behavior is present in 3 different GCC
releases from 3 different major versions.

In any case, I think the original code had a subtle bug, in that the
comparison of 'size' didn't do what the author intended.

Message #61 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org, dancol <at> dancol.org, schwab <at> linux-m68k.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 18:49:44 +0200

> Date: Sun, 9 Nov 2014 17:39:32 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>, dancol <at> dancol.org, Andreas Schwab <schwab <at> linux-m68k.org>
> 
> Guys, don't forget one important detail, that this happens only in 32-bit
> build. 64-bit does not have this problem.

Sheer luck: the first call to VirtualAlloc succeeds because it asks
for an amount that is much smaller than the 64-bit address space.

It worked for me in 32 bits for the same reason.

The problem only rears its ugly head when the first call to
VirtualAlloc fails.  Then we need more iterations, which don't happen.

Message #64 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Alexander Shukaev <haroogan <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 18995 <at> debbugs.gnu.org, dancol <at> dancol.org, schwab <at> linux-m68k.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 9 Nov 2014 17:53:25 +0100

[Message part 1 (text/plain, inline)]

>
> Sheer luck: the first call to VirtualAlloc succeeds because it asks
> for an amount that is much smaller than the 64-bit address space.
>

32 GB is not that small. I have only 8 GB on my current car...

[Message part 2 (text/html, inline)]

Message #67 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 18995 <at> debbugs.gnu.org, haroogan <at> gmail.com
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 19:14:58 +0200

> From: Andreas Schwab <schwab <at> linux-m68k.org>
> Cc: eliz <at> gnu.org,  haroogan <at> gmail.com
> Date: Sun, 09 Nov 2014 17:35:30 +0100
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > It looked like a GCC bug at first, but as I tried to modify the source
> > and look at the effect of that on the generated code, it finally
> > dawned on me: GCC's loop-unrolling code simply correctly calculated
> > that with the initial value of 0x68000000 and decrement of 0x00800000,
> > the value of 'size' in the loop will never be less than 0x00100000,
> > due to wraparound in the subtraction of unsigned values.
> 
> Before wraping around, size will become zero, which is definitely less
> than 0x00100000.

Right, so it could be pure GCC bug after all.

For the record, the problem also goes away if 'size' is declared
ptrdiff_t, and nothing else is changed.  So whatever the reason, it
definitely is somehow related to unsigned arithmetics.

Message #70 received at 18995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexander Shukaev <haroogan <at> gmail.com>
Cc: 18995 <at> debbugs.gnu.org, dancol <at> dancol.org, schwab <at> linux-m68k.org
Subject: Re: bug#18995: Error: Could not reserve dynamic heap area.
Date: Sun, 09 Nov 2014 19:17:47 +0200

> Date: Sun, 9 Nov 2014 17:53:25 +0100
> From: Alexander Shukaev <haroogan <at> gmail.com>
> Cc: 18995 <at> debbugs.gnu.org, dancol <at> dancol.org, schwab <at> linux-m68k.org
> 
>     Sheer luck: the first call to VirtualAlloc succeeds because it asks
>     for an amount that is much smaller than the 64-bit address space.
>     
> 
> 32 GB is not that small. I have only 8 GB on my current car...

(It's 256GB, not 32: the comment was wrong.)

That call only _reserves_ the address space, it doesn't actually
allocate anything.  So the amount of memory actually available on your
system is not important at that spot.  What _is_ important is that in
the 32-bit executable we are trying to reserve a significant portion
of the 2GB address space of 32-bit programs, while in the 64-bit
executable we reserve a very small portion of the address space.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.