GNU bug report logs - #60506
feature: parallel grep --recursive

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: grep; Reported by: Eike Dierks <foonlyboy@HIDDEN>; dated Tue, 3 Jan 2023 00:22:03 UTC; Maintainer for grep is bug-grep@HIDDEN.

Message received at 60506 <at> debbugs.gnu.org:


Received: (at 60506) by debbugs.gnu.org; 3 Jan 2023 22:33:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jan 03 17:33:28 2023
Received: from localhost ([127.0.0.1]:47011 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pCpqa-0001yV-Ex
	for submit <at> debbugs.gnu.org; Tue, 03 Jan 2023 17:33:28 -0500
Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:38735)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pj@HIDDEN>) id 1pCpqX-0001yI-PH
 for 60506 <at> debbugs.gnu.org; Tue, 03 Jan 2023 17:33:26 -0500
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.west.internal (Postfix) with ESMTP id 8A9833200945;
 Tue,  3 Jan 2023 17:33:19 -0500 (EST)
Received: from imap51 ([10.202.2.101])
 by compute6.internal (MEProxy); Tue, 03 Jan 2023 17:33:19 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:date:feedback-id
 :feedback-id:from:from:in-reply-to:in-reply-to:message-id
 :mime-version:references:reply-to:sender:subject:subject:to:to
 :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=
 fm2; t=1672785199; x=1672871599; bh=qIhnPwU5g44s+774c2EkmlFgQDeb
 eVuQ86+T0JVEh9k=; b=g7/YG8j/hyIz/xztkykoVeDytKTGMsCRg3RJXEqnDICC
 MiTE7nuy9FjtfYDQSSqLmr+0xl4iUkrNVqRmGBVDu3ENaWJUibCrIj/XkzK7nY1I
 pcvBWmRljTU115Yq2KC9VDsKsAh+lyHDHaGZrJIpSfQxt63GBBQrO+mRAXm7825+
 kTrynCD2bb2w+pKJhLsuut2mKPg0jWzZTYWpwaJM5hMiZX7L3n8FoXXbl3rq7gPQ
 quqMtDiEsHTYBleoCWnIRioRN4domgWKLslOMVw9aV805Xlxdt8XWK2p0pW7KAUS
 +oeUNsUV8tYt9YGmzfmgZALPqmGH6TTvNTowVwK7mQ==
X-ME-Sender: <xms:Lq20Y43glVr3piRv4WVOUY47PSf1XAybzZfNwmbS6gQrlA8Na2eGrQ>
 <xme:Lq20YzFmgclY_sYbxao_fBRQHzajMqdvjvIzfCj7DbgHpuVR29b8yJLUKuEFUVu5V
 Bft-EKSZfVC>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrjeeggdduiedvucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreertdenucfhrhhomhepfdfrrghu
 lhculfgrtghkshhonhdfuceophhjsehushgrrdhnvghtqeenucggtffrrghtthgvrhhnpe
 ejteetudfhffdvgeekieeiteefvdefteejteduvdejtefhueekgfeludeuleelieenucff
 ohhmrghinhepthhhvghphihthhhonhhitggtohifrdhushenucevlhhushhtvghrufhiii
 gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpjhesuhhsrgdrnhgvth
X-ME-Proxy: <xmx:Lq20Yw7hdLjS_26u5eFKAoZ8kLtqyJYCUcWLI0kPBdIidVg_GPoMNQ>
 <xmx:Lq20Yx3TwW1qyKYJvnUJhjy4lr2f-LLXen-pWwNZHWLNlJx7wPcRog>
 <xmx:Lq20Y7G1MuCw-rXBs6OrYuGh1hbsNMYprClzxWL_WB2cVxnoW3uooA>
 <xmx:L620Y-PRK8DD0CClum1ljrkYKE4bbZzjumi_0daN4ibF4jyR4oDalQ>
Feedback-ID: i047841af:Fastmail
Received: by mailuser.nyi.internal (Postfix, from userid 501)
 id B81EEB60086; Tue,  3 Jan 2023 17:33:18 -0500 (EST)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.7.0-alpha0-1185-g841157300a-fm-20221208.002-g84115730
Mime-Version: 1.0
Message-Id: <eaa1bb2a-bb8b-430c-8971-0fae539ce5ac@HIDDEN>
In-Reply-To: <786328389.5641240.1672780518915@HIDDEN>
References: <CAPuuP=7KCvt2-zgJAdY8CSEAmuubnbpnmznqEfbArf-LaO2q8w@HIDDEN>
 <04d86085-a044-4b9b-8451-b6e0c3586bb3@HIDDEN>
 <31e8cb06-606f-ea05-7d99-08e0311920a5@HIDDEN>
 <68afdacb-e8db-41c4-a3bd-1ce5ddd185ac@HIDDEN>
 <786328389.5641240.1672780518915@HIDDEN>
Date: Tue, 03 Jan 2023 16:32:01 -0600
From: "Paul Jackson" <pj@HIDDEN>
To: "David G. Pickett" <dgpickett@HIDDEN>,
 "eggert@HIDDEN" <eggert@HIDDEN>,
 "60506 <at> debbugs.gnu.org" <60506 <at> debbugs.gnu.org>
Subject: Re: bug#60506: feature: parallel grep --recursive
Content-Type: multipart/alternative; boundary=3e9d5f89839d42a281a1fa27514752b9
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 60506
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--3e9d5f89839d42a281a1fa27514752b9
Content-Type: text/plain

David Pickett wrote:
<< I also wrote a simpler, line oriented, faster xargs, fxargs!  >>

I've been quite pleased with an xargs wrapper I wrote that basically
converts newlines to nuls, and then invokes either "xargs" or, if asked
to run multiple threads, "parallel --xargs", passing all the "xargs" arguments
to "xargs --null".

I got all the exit status's and such just right, and preferred having all the
xargs options available, once this hack worked around the confused
space character handling of xargs without the --null option.

I call my wrapper "x", a short name since  I use it a lot, having been a regular
xargs user since it was added to Version 7 Unix, inside Bell Labs, back around
1978.

You can find my wrapper at:

http://thepythoniccow.us/x.c

By the way, even the original author of xargs, Herb Gellis, agrees that its
interface is somewhat borked.  See a note Gellis posted a decade after writing
xargs, which I include in the above "x.c" source.  An amusing bit of history ...

-- 
                Paul Jackson
                pj@HIDDEN


--3e9d5f89839d42a281a1fa27514752b9
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}
p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style=3D"f=
ont-family:Arial;">David Pickett wrote:</div><div style=3D"font-family:A=
rial;"><div>&lt;&lt; I also wrote a simpler, line oriented, faster xargs=
, fxargs!&nbsp; &gt;&gt;<br></div><div><br></div><div>I've been quite pl=
eased with an xargs wrapper I wrote that basically<br></div><div style=3D=
"font-family:Arial;">converts newlines to nuls, and then invokes either =
"xargs" or, if asked<br></div><div style=3D"font-family:Arial;">to run m=
ultiple threads, "parallel --xargs", passing all the "xargs" arguments<b=
r></div><div style=3D"font-family:Arial;">to "xargs --null".<br><br>I go=
t all the exit status's and such just right, and preferred having all th=
e</div><div style=3D"font-family:Arial;">xargs options available, once t=
his hack worked around the confused<br></div><div style=3D"font-family:A=
rial;">space character handling of xargs without the --null option.<br><=
/div><div style=3D"font-family:Arial;"><br>I call my wrapper "x", a shor=
t name since&nbsp; I use it a lot, having been a regular</div><div style=
=3D"font-family:Arial;">xargs user since it was added to Version 7 Unix,=
 inside Bell Labs, back around<br></div><div style=3D"font-family:Arial;=
"><div>1978.<br><br>You can find my wrapper at:</div><div><br></div><div=
><a href=3D"http://thepythoniccow.us/x.c" style=3D"outline-width:0px !im=
portant;user-select:auto !important;">http://thepythoniccow.us/x.c</a><b=
r></div><div><br></div><div>By the way, even the original author of xarg=
s,&nbsp;Herb Gellis, agrees that its<br></div><div style=3D"font-family:=
Arial;">interface is somewhat borked.&nbsp; See a note Gellis posted a d=
ecade after writing<br></div><div style=3D"font-family:Arial;">xargs, wh=
ich I include in the above "x.c" source.&nbsp; An amusing bit of history=
 ...<br><br></div></div></div><div id=3D"sig78241"><pre class=3D"signatu=
re">--=20
                Paul Jackson
                pj@HIDDEN
<br></pre></div><div style=3D"font-family:Arial;"><br></div></body></htm=
l>
--3e9d5f89839d42a281a1fa27514752b9--




Information forwarded to bug-grep@HIDDEN:
bug#60506; Package grep. Full text available.

Message received at 60506 <at> debbugs.gnu.org:


Received: (at 60506) by debbugs.gnu.org; 3 Jan 2023 02:59:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 02 21:59:25 2023
Received: from localhost ([127.0.0.1]:44426 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pCXWP-0004TW-0i
	for submit <at> debbugs.gnu.org; Mon, 02 Jan 2023 21:59:25 -0500
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:58123)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pj@HIDDEN>) id 1pCXWN-0004TK-My
 for 60506 <at> debbugs.gnu.org; Mon, 02 Jan 2023 21:59:24 -0500
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 5237E5C0103;
 Mon,  2 Jan 2023 21:59:18 -0500 (EST)
Received: from imap51 ([10.202.2.101])
 by compute6.internal (MEProxy); Mon, 02 Jan 2023 21:59:18 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:date:feedback-id
 :feedback-id:from:from:in-reply-to:in-reply-to:message-id
 :mime-version:references:reply-to:sender:subject:subject:to:to
 :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=
 fm2; t=1672714758; x=1672801158; bh=YreBbwxX1LGnJ9S7cKaJWSqM1fvm
 jLb+hcQ9OMtwFM0=; b=t67jRLJbvuRycBG7myxtiFv7MU+LF/5YQ1TVRW4as6xg
 NDsJpMTk9qBTJ3gREFZ8VTlFFA3oRqs47mmcJBbSGmbLt3UDokjx0VsaDOcLAk0K
 u8K8qkF7+oCaYSy9I+waEwc41JIag4/tMjr1oXQZJ/w+zs6pYrcQB032e+NtUFHi
 nIRDuTx7qO/buqlY5/N4veRIGhLKFT/NqTu+N4orIM2oc+MN4lT5CELofj7sweeN
 kn3oxXPk6s0y0rRyB4vYGsn6kFxiHG+7lT1o7jXO92fjNeMRx7oNEZfkWnVsjczw
 jPqNhNmCUUhlvXcratV8TU1m9TvqjU9mODZWTAaE/w==
X-ME-Sender: <xms:BpqzYzLUuLNRkdlpVKNfwrWZfjP25LMWeCC-xmNq6AAYj2BGllNIFg>
 <xme:BpqzY3JhZda02riB1v0y_mq3Of0qmI7mf8h89xncgNWRUU2p7rEpgXnChPaO2n173
 h2RS7kjx6hA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrjeefgdehhecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfrfgruhhl
 ucflrggtkhhsohhnfdcuoehpjhesuhhsrgdrnhgvtheqnecuggftrfgrthhtvghrnhephf
 evtdehjeethfegjedtiefgffdtlefgvdekueefheejkeevveehudffgfduleejnecuvehl
 uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhjsehushgrrd
 hnvght
X-ME-Proxy: <xmx:BpqzY7tBLwed7aSr11dhGTBWHaFKf5rWecw7N208dssnM9ytPNriOw>
 <xmx:BpqzY8ZqdIsnAkvc5DV-rwTY8u95arDqi5-veYK5WyyQVn8-fg3psQ>
 <xmx:BpqzY6YeS1fZR9XvRswc29VljJtyo4mu2haClGFy90snSTJ5P5VdKw>
 <xmx:BpqzY42Q1e0SawA6qtuj3S99GoZzd-6Nsq3y3VnwukWM8u7LBerS_A>
Feedback-ID: i047841af:Fastmail
Received: by mailuser.nyi.internal (Postfix, from userid 501)
 id 2932BB60086; Mon,  2 Jan 2023 21:59:18 -0500 (EST)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.7.0-alpha0-1185-g841157300a-fm-20221208.002-g84115730
Mime-Version: 1.0
Message-Id: <68afdacb-e8db-41c4-a3bd-1ce5ddd185ac@HIDDEN>
In-Reply-To: <31e8cb06-606f-ea05-7d99-08e0311920a5@HIDDEN>
References: <CAPuuP=7KCvt2-zgJAdY8CSEAmuubnbpnmznqEfbArf-LaO2q8w@HIDDEN>
 <04d86085-a044-4b9b-8451-b6e0c3586bb3@HIDDEN>
 <31e8cb06-606f-ea05-7d99-08e0311920a5@HIDDEN>
Date: Mon, 02 Jan 2023 20:56:23 -0600
From: "Paul Jackson" <pj@HIDDEN>
To: "Paul Eggert" <eggert@HIDDEN>, 60506 <at> debbugs.gnu.org
Subject: Re: bug#60506: feature: parallel grep --recursive
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 60506
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

<< a parallel grep to search a single large file >>

I'm but one user, and a rather idiosyncratic user at that,
but for my usage patterns, the specialized logic that it
would take to run a parallelized grep on a large file
would likely not shrink the elapsed time enough to justify
the coding, documentation, and maintenance effort.

I would expect the time to read the large file in from disk to
dominate the total elapsed time in any case.

(or maybe I am just jealous that I didn't think of that parallel
grep use case myself <grin>.)

-- 
                Paul Jackson
                pj@HIDDEN




Information forwarded to bug-grep@HIDDEN:
bug#60506; Package grep. Full text available.

Message received at 60506 <at> debbugs.gnu.org:


Received: (at 60506) by debbugs.gnu.org; 3 Jan 2023 02:49:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 02 21:49:04 2023
Received: from localhost ([127.0.0.1]:44407 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pCXMN-0004Dv-Pk
	for submit <at> debbugs.gnu.org; Mon, 02 Jan 2023 21:49:04 -0500
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:57358)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1pCXMI-0004DK-5L
 for 60506 <at> debbugs.gnu.org; Mon, 02 Jan 2023 21:49:02 -0500
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id DE8F2160044;
 Mon,  2 Jan 2023 18:48:50 -0800 (PST)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id yQj8tRKMvcRo; Mon,  2 Jan 2023 18:48:50 -0800 (PST)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 3776C160045;
 Mon,  2 Jan 2023 18:48:50 -0800 (PST)
DKIM-Filter: OpenDKIM Filter v2.9.2 zimbra.cs.ucla.edu 3776C160045
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu;
 s=78364E5A-2AF3-11ED-87FA-8298ECA2D365; t=1672714130;
 bh=a3Anb5xxlFFoRHMTQAYDQvA8yY+tyHdXS9UeQM35mRY=;
 h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type:
 Content-Transfer-Encoding;
 b=Ed4DuX8cR6YmF2ul/xclcOqyRW/trGujR68XcUD56lG9nh8CneFV8GTl+ez6SRwIG
 +Y3m12AxQNh4v0nR7IfCHkmqHnybmJHiQoMnpjmLpR9n5u46OECCqBEMCCAUDnttAq
 /Yb8j45JZJQ2cN0frIwg3W6CNPQn4mKASg1+9Ecw=
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id hjzqh-yIerpF; Mon,  2 Jan 2023 18:48:50 -0800 (PST)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 0C84C160044;
 Mon,  2 Jan 2023 18:48:50 -0800 (PST)
Message-ID: <31e8cb06-606f-ea05-7d99-08e0311920a5@HIDDEN>
Date: Mon, 2 Jan 2023 18:48:49 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: bug#60506: feature: parallel grep --recursive
Content-Language: en-US
To: Paul Jackson <pj@HIDDEN>, 60506 <at> debbugs.gnu.org
References: <CAPuuP=7KCvt2-zgJAdY8CSEAmuubnbpnmznqEfbArf-LaO2q8w@HIDDEN>
 <04d86085-a044-4b9b-8451-b6e0c3586bb3@HIDDEN>
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
In-Reply-To: <04d86085-a044-4b9b-8451-b6e0c3586bb3@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -3.4 (---)
X-Debbugs-Envelope-To: 60506
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.4 (----)

On 2023-01-02 18:34, Paul Jackson wrote:
> There's no need for special logic in grep to run parallel grep's.

There might be, if one wants to use a parallel grep to search a single 
large file.




Information forwarded to bug-grep@HIDDEN:
bug#60506; Package grep. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 3 Jan 2023 02:35:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 02 21:35:10 2023
Received: from localhost ([127.0.0.1]:44403 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pCX8w-0003qC-BI
	for submit <at> debbugs.gnu.org; Mon, 02 Jan 2023 21:35:10 -0500
Received: from lists.gnu.org ([209.51.188.17]:33340)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pj@HIDDEN>) id 1pCX8u-0003q4-3E
 for submit <at> debbugs.gnu.org; Mon, 02 Jan 2023 21:35:08 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <pj@HIDDEN>) id 1pCX8q-00062n-AX
 for bug-grep@HIDDEN; Mon, 02 Jan 2023 21:35:07 -0500
Received: from out1-smtp.messagingengine.com ([66.111.4.25])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <pj@HIDDEN>) id 1pCX8o-0007bm-8a
 for bug-grep@HIDDEN; Mon, 02 Jan 2023 21:35:03 -0500
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 43F4E5C00B3
 for <bug-grep@HIDDEN>; Mon,  2 Jan 2023 21:35:00 -0500 (EST)
Received: from imap51 ([10.202.2.101])
 by compute6.internal (MEProxy); Mon, 02 Jan 2023 21:35:00 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:date:feedback-id
 :feedback-id:from:from:in-reply-to:in-reply-to:message-id
 :mime-version:references:reply-to:sender:subject:subject:to:to
 :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=
 fm2; t=1672713300; x=1672799700; bh=oPNbA7dbSc8BH6V1Q+cCvZOMRR6T
 B3tJE0hao5Hqpdw=; b=j05cZ5E8qtyZiGZTduqTsO/iwGdtjdoRmfzFczvRHptf
 bbdQw2o4vQ0RCPEYLuM3b9Jmv62UpbIC/DQiufybJjpv9aYjaUqf4Ccf8kIg15B4
 VV4sl12TK6gNr/G78wPCvAT5PBKWUaT6zH735GtRwSBPE7b4HDZ252qgaJQf7Y3u
 JBdScSXwNSN7EKX5uKFgeqZw1jvfa36TioPkTUpwfAgOuOowBaISy2KSMDqyYHJR
 tq1pyVPsj4hc29f6lLqB3WP1RikkVOCNvJj4Bcti7gbQKLX9NrC+YAube/chz2Er
 y6d2XBefjf+8vw5aRehPihePVqoHMouEUZQjYpRe1Q==
X-ME-Sender: <xms:U5SzY2DPL4AUhfz8hzz0rmcerklhBTNFj3gu0tgDYm0Pk8aeGu9oag>
 <xme:U5SzYwioIT3sjlfUI1zzRTToWlPntMTJkqFpgSaAeV-mT_hA3Oa8iWZ8lid214vCt
 YUEpIX2loDe>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrjeefgdehtdcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgesthdtre
 dtreertdenucfhrhhomhepfdfrrghulhculfgrtghkshhonhdfuceophhjsehushgrrdhn
 vghtqeenucggtffrrghtthgvrhhnpefhvedtheejtefhgeejtdeigffftdelgfdvkeeufe
 ehjeekveevhedufffgudeljeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhep
 mhgrihhlfhhrohhmpehpjhesuhhsrgdrnhgvth
X-ME-Proxy: <xmx:U5SzY5kTDBMvvHHWisChbMmoTQshs_PLOS-ls1dhJukDI22o8FLWNg>
 <xmx:U5SzY0w6r_nqbe359Cv3ktdWwDTm-9X-8bESU4MYlKy-SbMOn3Uutg>
 <xmx:U5SzY7TeT6C0DQyg7Cwi70SMPC57FCxGsFjl2LQuMOsY8SC1z8894A>
 <xmx:VJSzYyfGJl2pzUlT65WZ7PD5QwEdUiX2OGtJYyTqOL_MVOUKAUDfnw>
Feedback-ID: i047841af:Fastmail
Received: by mailuser.nyi.internal (Postfix, from userid 501)
 id CBF30B60086; Mon,  2 Jan 2023 21:34:59 -0500 (EST)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.7.0-alpha0-1185-g841157300a-fm-20221208.002-g84115730
Mime-Version: 1.0
Message-Id: <04d86085-a044-4b9b-8451-b6e0c3586bb3@HIDDEN>
In-Reply-To: <CAPuuP=7KCvt2-zgJAdY8CSEAmuubnbpnmznqEfbArf-LaO2q8w@HIDDEN>
References: <CAPuuP=7KCvt2-zgJAdY8CSEAmuubnbpnmznqEfbArf-LaO2q8w@HIDDEN>
Date: Mon, 02 Jan 2023 20:34:39 -0600
From: "Paul Jackson" <pj@HIDDEN>
To: bug-grep@HIDDEN
Subject: Re: bug#60506: feature: parallel grep --recursive
Content-Type: text/plain
Received-SPF: neutral client-ip=66.111.4.25; envelope-from=pj@HIDDEN;
 helo=out1-smtp.messagingengine.com
X-Spam_score_int: -17
X-Spam_score: -1.8
X-Spam_bar: -
X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001,
 RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001,
 SPF_NEUTRAL=0.779 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.6 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.6 (--)


There's no need for special logic in grep to run parallel grep's.

The "parallel" command can handle that for you.

For example, on the 12 core, 24 thread Ryzen CPU that I am using:

find $HOME -xdev -type f -ctime -333  |  wc -l     ## counts 136126 files.

find $HOME -xdev -type f -ctime -333 |
    parallel -m grep -l foobar | wc -l                        ## takes about 13 seconds

find $HOME -xdev -type f -ctime -333 |
    xargs -d '\n' grep -l foobar | wc -l                      ## takes about 52 seconds

The above parallel invocation ran 24 grep commands in parallel, and took
about 1/4 the time, otherwise performing rather like xargs, which ran one grep
command at a time.

(Granted, reading either the 'parallel' or 'xargs' man pages is not easy <grin>.)

-- 
                Paul Jackson
                pj@HIDDEN




Information forwarded to bug-grep@HIDDEN:
bug#60506; Package grep. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 3 Jan 2023 00:21:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 02 19:21:18 2023
Received: from localhost ([127.0.0.1]:44339 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pCV3N-0000QW-Dq
	for submit <at> debbugs.gnu.org; Mon, 02 Jan 2023 19:21:18 -0500
Received: from lists.gnu.org ([209.51.188.17]:37968)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <foonlyboy@HIDDEN>) id 1pCRkV-0003O3-A5
 for submit <at> debbugs.gnu.org; Mon, 02 Jan 2023 15:49:37 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <foonlyboy@HIDDEN>)
 id 1pCRkN-0001Dv-GY
 for bug-grep@HIDDEN; Mon, 02 Jan 2023 15:49:28 -0500
Received: from mail-pj1-x102d.google.com ([2607:f8b0:4864:20::102d])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <foonlyboy@HIDDEN>)
 id 1pCRkL-0006ss-Sl
 for bug-grep@HIDDEN; Mon, 02 Jan 2023 15:49:27 -0500
Received: by mail-pj1-x102d.google.com with SMTP id
 60-20020a17090a0fc200b002264ebad204so7528461pjz.1
 for <bug-grep@HIDDEN>; Mon, 02 Jan 2023 12:49:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=to:subject:message-id:date:from:mime-version:from:to:cc:subject
 :date:message-id:reply-to;
 bh=iGsFLjN3sBQM/cCbEVpMF+Yjthf1zgCYgzgj6dGJPsQ=;
 b=Ez7mx8a6G5hbD2qAxXdVC61TUHVZQBRly2s2qBZrZb6DTmh3YFzOTvj5JLWTtrTotX
 KgQbGn7bU10I6CzvRUSax4aAbtjFEe0SVvKTxjZU3j2MW46xhKJ3qbIFmr8OpDRm7L20
 mZmyDpq45vRtCfZuYD8xZvHJMJ7ViMLWyTloOeEKsdePzCROEjIKAweOlcrYRKjW2ISE
 iVMiaYR0cbSsegaa8Zosd1EPqrU/KDTnEWha+qxoJoxm3UeGo0yRAdB/TwnIPuPS7t6s
 P/8U/VDhG7B/rgQQeaHoDqCt2hhwUmxL7jB0VTnp/EqHXU4bXDnjqU7Q0zHMeITNJmy9
 5ffg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=to:subject:message-id:date:from:mime-version:x-gm-message-state
 :from:to:cc:subject:date:message-id:reply-to;
 bh=iGsFLjN3sBQM/cCbEVpMF+Yjthf1zgCYgzgj6dGJPsQ=;
 b=Ck8FW46wvQdsDgastJeFEz+fhQvtwHOXPXjW6PT+48zRMauaooJKRn9MpZH14mIJKh
 3rCrYVSp6XehNcCI5BKAF4BNwbWsXK4EL5BIWhhsQKGW9J7cVhFLSKMb5GYDtMTaF1v2
 sXZXRejq6Dqt1Bq5Ab7zJJbVLeLU9RROi7m6zqzWb3+VzPZpwj1vaAvIicjopAUeRLRK
 /Bu1Od4th7ms3SYZIf0J1bQwM7y1YMH2Ag4FxCa86ALTaeM0UJz2/17gUr+Ag8TTtqVH
 FgRk9e4Bd/Y0T3TH1eGIiThrqJBwt9quyG64PduuEBnuEoNyGm/itodHHHFlM+6Fq6uN
 74Aw==
X-Gm-Message-State: AFqh2kroBDo3UCtPw2XITyLUtzpb2W3myL4ROIdt7Vk3PcQdhZrmei0q
 wrHgB+qDGe2s7zVtzU13ObyvJn6mI3DkkPDq0yfgZsoWJfo=
X-Google-Smtp-Source: AMrXdXtsqiUck7P9LxMZO67tu/h6209D4R4WxwfprNkUStLTf3CmVVvnZPENCAY/dJpk2UTKZiUmKBx5LWqzh4LEZow=
X-Received: by 2002:a17:90a:fe93:b0:223:4a7d:878d with SMTP id
 co19-20020a17090afe9300b002234a7d878dmr3898903pjb.29.1672692563881; Mon, 02
 Jan 2023 12:49:23 -0800 (PST)
MIME-Version: 1.0
From: Eike Dierks <foonlyboy@HIDDEN>
Date: Mon, 2 Jan 2023 21:49:13 +0100
Message-ID: <CAPuuP=7KCvt2-zgJAdY8CSEAmuubnbpnmznqEfbArf-LaO2q8w@HIDDEN>
Subject: feature: parallel grep --recursive
To: bug-grep@HIDDEN
Content-Type: text/plain; charset="UTF-8"
Received-SPF: pass client-ip=2607:f8b0:4864:20::102d;
 envelope-from=foonlyboy@HIDDEN; helo=mail-pj1-x102d.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Mon, 02 Jan 2023 19:21:14 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hi at the gnu grep development team

I'd like to suggest a new feature
for: grep --recursive

The grep --recursive should work in parallel.

Rational:
This could speed up the grep by the numbers of threads

Currently, the --recursive option works on every file in sequence.
Instead, I want to start some greps in parallel.

If we want to be good,
then we would parse the expression first (which might be expensive)
and then fork on the files.

The master grep process would then collect the results,
so that the results would be serialized
to be identical with the current implementation.

I'd like to suggest a --fast option,
where results show up, as soon as they are found.
....

I am fed up with all that precomputed indexes.
I want to grep it really fast now.

I expect  that the file access is fast now, but has latency.
I want the grep to saturate the machine.

// job card






.




Acknowledgement sent to Eike Dierks <foonlyboy@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-grep@HIDDEN. Full text available.
Report forwarded to bug-grep@HIDDEN:
bug#60506; Package grep. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Tue, 3 Jan 2023 22:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.