GNU bug report logs - #43999
sed: couldn't re-allocate memory

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: sed; Reported by: markfilipak.linux+gnuorg@HIDDEN; Keywords: moreinfo; dated Wed, 14 Oct 2020 21:54:02 UTC; Maintainer for sed is bug-sed@HIDDEN.
Added tag(s) moreinfo. Request was from Assaf Gordon <assafgordon@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 43999 <at> debbugs.gnu.org:


Received: (at 43999) by debbugs.gnu.org; 15 Oct 2020 06:35:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Oct 15 02:35:53 2020
Received: from localhost ([127.0.0.1]:53918 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kSwrh-0001T0-1n
	for submit <at> debbugs.gnu.org; Thu, 15 Oct 2020 02:35:53 -0400
Received: from mail-pj1-f66.google.com ([209.85.216.66]:38243)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <assafgordon@HIDDEN>)
 id 1kSwrf-0001Sj-G2; Thu, 15 Oct 2020 02:35:51 -0400
Received: by mail-pj1-f66.google.com with SMTP id u3so1347414pjr.3;
 Wed, 14 Oct 2020 23:35:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=subject:to:references:from:message-id:date:user-agent:mime-version
 :in-reply-to:content-language:content-transfer-encoding;
 bh=sL/SLZ/zlnVkcWQv/RgDU105ymznSfzvydWTg7J0TCk=;
 b=JEMmSjj8Igw3B6h3fhhx5VBJht6GmxOknQuFswctFw71d0Lf548HeElCWMegqyWjL0
 tw0z0Km3I4ToKa8IFJVdGgACkikD4jCarArEeeMb6n/rDrXM81i+DY60wmj9NnB2DSMu
 7eeKj+jxSrYlgYcDOWz2zhvcbH0FXVcOjvMR6Qde5k3fnHUakf7Xte3y1InF9njGpIyD
 9URMy2ctHH3rBym964oe0O6E0aaVAXZOFFZXBE3H6O2kzUUZJpwrani+zunvJ5poc5JC
 8+7lPfGNevgPrDN/IbkOhFYj4fR8lhiXhoDiyTx6YaFo9LIiOKdEG7dsbaDS0DyuZSHT
 rfIQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:subject:to:references:from:message-id:date
 :user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=sL/SLZ/zlnVkcWQv/RgDU105ymznSfzvydWTg7J0TCk=;
 b=O4wcjuQv3YEUQU9b7ZEl0FmbkOlH26DVUvJTG+jiVEyJTp/kuoSUErOvUXg0Td01Ey
 iWBSK/1krEN72MnEDIC2TTo4kCU6FKJ/7IXkH6KtYp/dHfEWCdayZKqwJVwSAZUQPqYx
 WccyV19RjPqkHM5IXx22YPgoiasjtvmeVCcr56Kwo/ZD4hsZJsqdSygT8oI3WYez8YPz
 5XgU+STeGsQwA7w+u56W3jwnCOcligLns4CPr8ySUSbFvGwKngZ50sBPPsnfpzMKixYX
 vtt7QrJh/HUU1pg0EF5UE/iH3R2okF87TzQBh6BQDGj5KoaO3pdxi1rRk39xUdfjKOKd
 xMpQ==
X-Gm-Message-State: AOAM531UqJ7FFctd4xig49PbvY0L4hoGCfxTFWkM/jc+JHj0IjGTNgv8
 hnbJBecXxxJumH1W/TZ1cRn1DOILKgA=
X-Google-Smtp-Source: ABdhPJwqIhyRu02Kblp84kUK/SWSU2YjGI4KoLdzMXG/fBJ9q+/Z8nn28tUoOnGbswDSvmYNOTkAoA==
X-Received: by 2002:a17:90a:248:: with SMTP id t8mr3146198pje.64.1602743744945; 
 Wed, 14 Oct 2020 23:35:44 -0700 (PDT)
Received: from tomato.moose.housegordon.com (moose.housegordon.com.
 [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id
 c127sm1800842pfc.115.2020.10.14.23.35.43
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Wed, 14 Oct 2020 23:35:43 -0700 (PDT)
Subject: Re: bug#43999: sed: couldn't re-allocate memory
To: markfilipak.linux+gnuorg@HIDDEN, 43999 <at> debbugs.gnu.org
References: <29ab0237-d59d-6ab9-efaa-fa01445e5799@HIDDEN>
From: Assaf Gordon <assafgordon@HIDDEN>
Message-ID: <f3c0380d-94cc-af0d-0c91-1eb18da3a7a3@HIDDEN>
Date: Thu, 15 Oct 2020 00:35:42 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.12.0
MIME-Version: 1.0
In-Reply-To: <29ab0237-d59d-6ab9-efaa-fa01445e5799@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 43999
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

tag 43999 moreinfo
stop

Hello,

On 2020-10-14 2:40 p.m., Mark Filipak wrote:
> The file "1,073,709,056 bytes" provokes an error (& zero output), but 
> only if piped from 'tr' and only for a particular pattern: /00000100/.
> the file "1,073,739,776 bytes" succeeds with identical parameters.
> The pipe through 'tr' appears not to be the problem.
> 
> $ sed --version
> sed (GNU sed) 4.2.2

First,
sed version 4.2.2 is 8 years old.
It is not supported any more.
Please try with the latest version of sed and report back if you still 
get the error.

Second,
please send some information about your environment, namely "uname -a",
are you using a 32bit or 64bit system,
and how much RAM and SWAP does your machine have.

> ...
> $ xxd -p -u "1,073,709,056 bytes" | tr -d '\n' | sed -r 
> 's/00000100/\x0D\x0A&/g' > foo.txt
> sed: couldn't re-allocate memory

Third,
You did not describe what your input file contains.
I implicitly assume from the file name that the file is large (~1GB?).
I did not try reproduce this error, but my hunch is that by deleting
input newlines you are asking sed to load a very large input string and
store it in memory - so a memory-related error is not unexpected.

regards,
  - assaf







Information forwarded to bug-sed@HIDDEN:
bug#43999; Package sed. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 14 Oct 2020 21:53:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Oct 14 17:53:03 2020
Received: from localhost ([127.0.0.1]:53276 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kSohj-0005au-6Q
	for submit <at> debbugs.gnu.org; Wed, 14 Oct 2020 17:53:03 -0400
Received: from lists.gnu.org ([209.51.188.17]:59134)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <markfilipak@HIDDEN>) id 1kSnZq-0007ru-KW
 for submit <at> debbugs.gnu.org; Wed, 14 Oct 2020 16:40:50 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:56952)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <markfilipak@HIDDEN>)
 id 1kSnZq-0007Qh-D6
 for bug-sed@HIDDEN; Wed, 14 Oct 2020 16:40:50 -0400
Received: from p-impout004aa.msg.pkvw.co.charter.net ([47.43.26.135]:51613
 helo=p-impout004.msg.pkvw.co.charter.net)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <markfilipak@HIDDEN>)
 id 1kSnZo-00038U-Jd
 for bug-sed@HIDDEN; Wed, 14 Oct 2020 16:40:50 -0400
Received: from [192.168.0.3] ([76.188.128.133]) by cmsmtp with ESMTPA
 id SnZiknHMHOboRSnZjkTm3U; Wed, 14 Oct 2020 20:40:44 +0000
X-Authority-Analysis: v=2.3 cv=MM4eZ/Rl c=1 sm=1 tr=0
 a=yKNMihaBE5Rt6XcMTKyqHg==:117 a=yKNMihaBE5Rt6XcMTKyqHg==:17
 a=4TstAJWtRUUA:10 a=IkcTkHD0fZMA:10 a=x7bEGLp0ZPQA:10 a=UApgNeDKAAAA:8
 a=3ItE9Zqe7OjNITQ7FNUA:9 a=ZBJl85-7Spx0tf-K:21 a=QEXdDO2ut3YA:10
 a=nmsX9mYhfO4A:10 a=OylmjzD9BNwA:10 a=fasyu4nNmrhVRyehPdzp:22
 a=aZ922TYaLRI4xedjMu-5:22
To: bug-sed@HIDDEN
From: Mark Filipak <markfilipak@HIDDEN>
Subject: sed: couldn't re-allocate memory
Message-ID: <29ab0237-d59d-6ab9-efaa-fa01445e5799@HIDDEN>
Date: Wed, 14 Oct 2020 16:40:42 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Firefox/52.0 Thunderbird/52.7.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-CMAE-Envelope: MS4wfOnPkzp89ZmtlYErYW2wmUoKe/QQho1P8oLKPLeOYc37gq/6QmdRBPrm3DH4Y7YEbeg0HU8/9Yv5R83S8PeVVwRUEjrfifQkU4c9J6ZtxssTqh2GPZcW
 MDLRdjl2NCpgHYAxIhabAuhAT1N0xKKw7SwpIFSK/3DgC6qq3XDkqZs3
Received-SPF: pass client-ip=47.43.26.135; envelope-from=markfilipak@HIDDEN;
 helo=p-impout004.msg.pkvw.co.charter.net
X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/14 16:40:45
X-ACL-Warn: Detected OS   = ???
X-Spam_score_int: 2
X-Spam_score: 0.2
X-Spam_bar: /
X-Spam_report: (0.2 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_REPLYTO=2.095,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.2 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: Kindly expedite this and contact me for any reason. The file
 "1,073,709,056 bytes" provokes an error (& zero output), but only if piped
 from 'tr' and only for a particular pattern: /00000100/. the file "1, 073, 739,
 776 bytes" succeeds with identical para [...] 
 Content analysis details:   (1.2 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 1.0 SPF_SOFTFAIL           SPF: sender does not match SPF record (softfail)
 -2.3 RCVD_IN_DNSWL_MED      RBL: Sender listed at https://www.dnswl.org/,
 medium trust [209.51.188.17 listed in list.dnswl.org]
 0.0 RCVD_IN_MSPIKE_H4      RBL: Very Good reputation (+4)
 [209.51.188.17 listed in wl.mailspike.net]
 0.0 RCVD_IN_MSPIKE_WL      Mailspike good senders
 2.5 FREEMAIL_FORGED_REPLYTO Freemail in Reply-To, but not From
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Wed, 14 Oct 2020 17:53:02 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: markfilipak.linux+gnuorg@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Kindly expedite this and contact me for any reason.

The file "1,073,709,056 bytes" provokes an error (& zero output), but only if piped from 'tr' and 
only for a particular pattern: /00000100/.
the file "1,073,739,776 bytes" succeeds with identical parameters.
The pipe through 'tr' appears not to be the problem.

$ sed --version
sed (GNU sed) 4.2.2
...
$ xxd -p -u "1,073,709,056 bytes" | tr -d '\n' | sed -r 's/00000100/\x0D\x0A&/g' > foo.txt
sed: couldn't re-allocate memory
$ xxd -p -u "1,073,709,056 bytes" | tr -d '\n' | sed -r 's/000001/\x0D\x0A&/g' > foo.txt
$ xxd -p -u "1,073,709,056 bytes" | sed -r 's/00000100/\x0D\x0A&/g' > foo.txt
$ xxd -p -u "1,073,739,776 bytes" | tr -d '\n' | sed -r 's/00000100/\x0D\x0A&/g' > foo.txt
$

You probably want the two source files, or perhaps only the source file that provokes the error. 
Kindly let me know how I can send it to you.

Regards,
Mark Filipak.

PS: For your sake, I posted a request for vetting this case
https://forums.linuxmint.com/viewtopic.php?f=47&t=332836
I got no responses, so it has not been replicated by someone else.




Acknowledgement sent to markfilipak.linux+gnuorg@HIDDEN:
New bug report received and forwarded. Copy sent to bug-sed@HIDDEN. Full text available.
Report forwarded to bug-sed@HIDDEN:
bug#43999; Package sed. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Thu, 15 Oct 2020 06:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.