GNU logs - #9500, boring messages

Message sent to owner <at> debbugs.gnu.org, bug-coreutils@HIDDEN:

Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Kelly Anderson <kelly@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-To: owner <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 14 Sep 2011 06:47:02 +0000
Resent-Message-ID: <handler.9500.B.1315982777527 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 9500 <at> debbugs.gnu.org
Message-ID: <4E7041CE.4090305@HIDDEN>
Date: Tue, 13 Sep 2011 23:55:26 -0600
From: Kelly Anderson <kelly@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0.2) Gecko/20110911 Thunderbird/6.0.2
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org

Hi,

I put together a patch 2 or 3 years ago (back when posix_fallocate was 
first introduced in glibc).
I've been using coreutils ever since with that patch applied with no 
problems.  The only error
I ever encountered (I had my patch error when posix_fallocate failed at 
that time) was when
I tried to copy a 25Gig file to a vfat partition, that's what it should 
do with a file over
4Gigs on a fat32 partition.  Anyway, I changed my patch to silently 
ignore posix_fallocate
errors, so coreutils would error the same as it currently does.

I copy a lot of large media files around on my servers and I want their 
space/continuity
to be allocated as efficiently as possible.

This patch has been tested for 2 to 3 years by me, so it should be good 
to go.
The patch applies to coreutils 8.13.

--- ./configure.ac.orig    2011-08-19 13:40:11.000000000 -0600
+++ ./configure.ac    2011-09-13 23:29:57.277354329 -0600
@@ -242,6 +242,18 @@ AC_DEFUN([coreutils_DUMMY_1],
  ])
  coreutils_DUMMY_1

+dnl * Old glibcs have broken posix_fallocate(). Make sure not to use it.
+AC_TRY_COMPILE([
+  #define _XOPEN_SOURCE 600
+  #include <stdlib.h>
+  #if defined(__GLIBC__) && (__GLIBC__ < 2 || __GLIBC_MINOR__ < 7)
+    possibly broken posix_fallocate
+  #endif
+], [
+  posix_fallocate(0, 0, 0);
+], [
+  AC_DEFINE([HAVE_POSIX_FALLOCATE], [1], [Define if you have a working 
posix_fallocate()]) ])
+
  AC_MSG_CHECKING([ut_host in struct utmp])
  AC_CACHE_VAL([su_cv_func_ut_host_in_utmp],
  [AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <sys/types.h>
--- ./src/copy.c.orig    2011-07-28 04:38:27.000000000 -0600
+++ ./src/copy.c    2011-09-13 23:29:57.280354149 -0600
@@ -1026,6 +1026,16 @@ copy_reg (char const *src_name, char con
            size_t blcm = buffer_lcm (io_blksize (src_open_sb), buf_size,
                                      blcm_max);

+#ifdef HAVE_POSIX_FALLOCATE
+          if (S_ISREG(src_open_sb.st_mode)
+ && ! S_ISFIFO(sb.st_mode)
+ && src_open_sb.st_size >= buf_size)
+          {
+            /* ignore errors, some filesystems may error if filesize 
exceeds the filesystem's limit */
+            posix_fallocate (dest_desc, 0, src_open_sb.st_size);
+          }
+#endif
+
            /* Do not bother with a buffer larger than the input file, 
plus one
               byte to make sure the file has not grown while reading 
it.  */
            if (S_ISREG (src_open_sb.st_mode) && src_open_sb.st_size < 
buf_size)

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Kelly Anderson <kelly@HIDDEN>
Subject: bug#9500: Acknowledgement ([PATCH]: use posix_fallocate where
 supported)
Message-ID: <handler.9500.B.1315982777527.ack <at> debbugs.gnu.org>
References: <4E7041CE.4090305@HIDDEN>
X-Gnu-PR-Message: ack 9500
X-Gnu-PR-Package: coreutils
X-Gnu-PR-Keywords: patch
Reply-To: 9500 <at> debbugs.gnu.org
Date: Wed, 14 Sep 2011 06:47:03 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-coreutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 9500 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
9500: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D9500
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to owner <at> debbugs.gnu.org, bug-coreutils@HIDDEN:

Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Eric Blake <eblake@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-To: owner <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 14 Sep 2011 14:37:01 +0000
Resent-Message-ID: <handler.9500.B9500.13160109989970 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Kelly Anderson <kelly@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org
Message-ID: <4E70B4F9.6080303@HIDDEN>
Date: Wed, 14 Sep 2011 08:06:49 -0600
From: Eric Blake <eblake@HIDDEN>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.20) Gecko/20110831 Fedora/3.1.12-2.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.12
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
In-Reply-To: <4E7041CE.4090305@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org

On 09/13/2011 11:55 PM, Kelly Anderson wrote:
> Hi,
>
> I put together a patch 2 or 3 years ago (back when posix_fallocate was
> first introduced in glibc).

Thanks for the effort.  However, this has been discussed in the past, 
and the consensus was that we should first write a patch to gnulib that 
provides a posix_fallocate() stub for all platforms, so that coreutils 
can unconditionally call posix_fallocate, rather than making coreutils 
have to use #ifdef.  Among other things, a gnulib module would make it 
possible to emulate posix_fallocate() even on older glibc where it is 
missing or broken.

-- 
Eric Blake   eblake@HIDDEN    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Message sent to owner <at> debbugs.gnu.org, bug-coreutils@HIDDEN:

Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-To: owner <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 14 Sep 2011 14:52:01 +0000
Resent-Message-ID: <handler.9500.B9500.131601188814685 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Eric Blake <eblake@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>
Message-ID: <4E70BE57.6030800@HIDDEN>
Date: Wed, 14 Sep 2011 15:46:47 +0100
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:5.0) Gecko/20110707 Thunderbird/5.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN>
In-Reply-To: <4E70B4F9.6080303@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
Precedence: list
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org

On 09/14/2011 03:06 PM, Eric Blake wrote:
> On 09/13/2011 11:55 PM, Kelly Anderson wrote:
>> Hi,
>>
>> I put together a patch 2 or 3 years ago (back when posix_fallocate was
>> first introduced in glibc).
> 
> Thanks for the effort.  However, this has been discussed in the past, and the consensus was that we should first write a patch to gnulib that provides a posix_fallocate() stub for all platforms, so that coreutils can unconditionally call posix_fallocate, rather than making coreutils have to use #ifdef.  Among other things, a gnulib module would make it possible to emulate posix_fallocate() even on older glibc where it is missing or broken.
> 

Also we probably want fallocate() for this use case
rather than posix_fallocate() in any case,
as we don't want to fall back to writing zeros.

Also I had a whole lot of fallocate() things to try
once the fiemap() stuff landed, but unfortunately
that doesn't work reliably on all file systems
and is currently restricted to sparse files.
So I need to dig out my notes on how to apply
fallocate() to files with holes and "empty portions" again.

cheers,
P�draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 23 Nov 2011 00:51:01 +0000
Resent-Message-ID: <handler.9500.B9500.13220094417512 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Eric Blake <eblake@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13220094417512
          (code B ref 9500); Wed, 23 Nov 2011 00:51:01 +0000
Received: (at 9500) by debbugs.gnu.org; 23 Nov 2011 00:50:41 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RT12v-0001x7-74
	for submit <at> debbugs.gnu.org; Tue, 22 Nov 2011 19:50:41 -0500
Received: from mx1.redhat.com ([209.132.183.28])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <P@HIDDEN>) id 1RT12s-0001wy-4S
	for 9500 <at> debbugs.gnu.org; Tue, 22 Nov 2011 19:50:39 -0500
Received: from int-mx12.intmail.prod.int.phx2.redhat.com
	(int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pAN0nE12015783
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Tue, 22 Nov 2011 19:49:14 -0500
Received: from [10.36.116.16] (ovpn-116-16.ams2.redhat.com [10.36.116.16])
	by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id pAN0nCZ5008431
	(version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
	Tue, 22 Nov 2011 19:49:13 -0500
Message-ID: <4ECC4307.9040600@HIDDEN>
Date: Wed, 23 Nov 2011 00:49:11 +0000
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
In-Reply-To: <4E70BE57.6030800@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	pAN0nE12015783
X-Spam-Score: -10.5 (----------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -10.5 (----------)

On 09/14/2011 03:46 PM, P=E1draig Brady wrote:
> On 09/14/2011 03:06 PM, Eric Blake wrote:
>> On 09/13/2011 11:55 PM, Kelly Anderson wrote:
>>> Hi,
>>>
>>> I put together a patch 2 or 3 years ago (back when posix_fallocate wa=
s
>>> first introduced in glibc).
>>
>> Thanks for the effort.  However, this has been discussed in the past, =
and the consensus was that we should first write a patch to gnulib that p=
rovides a posix_fallocate() stub for all platforms, so that coreutils can=
 unconditionally call posix_fallocate, rather than making coreutils have =
to use #ifdef.  Among other things, a gnulib module would make it possibl=
e to emulate posix_fallocate() even on older glibc where it is missing or=
 broken.
>>
>=20
> Also we probably want fallocate() for this use case
> rather than posix_fallocate() in any case,
> as we don't want to fall back to writing zeros.
>=20
> Also I had a whole lot of fallocate() things to try
> once the fiemap() stuff landed, but unfortunately
> that doesn't work reliably on all file systems
> and is currently restricted to sparse files.
> So I need to dig out my notes on how to apply
> fallocate() to files with holes and "empty portions" again.

I thought a little about this today.

fallocate() is a feature to quickly allocate space in a file system.
It's useful for 3 things as far as I can see:

  1. Improved file layout for subsequent access
  2. Immediate indication of ENOSPC
  3. Efficient writing of NUL portions

Note 1. is somewhat moot with newer file systems that do "delayed allocat=
ion".
So what do we need to consider when using fallocate on the destination fi=
le?
Considering just cp for the moment, its inputs impacting this are the opt=
ions:

  --sparse=3D{auto,always,never}
  Note with no --sparse specified we behave with --sparse=3Dauto,
  where we try to detect holes based on st_size vs st_blocks

The other significant input is the construction of the source file.
Now data in a file can generally be classed into 4 types:

  Data:  normal data
  Zero:  normal data containing only NULs
  Hole:  unallocated data containing only NULs
  Empty: allocated data containing only NULs

  One can have any of the above types at any point in the file.
  Also 'Empty' is special in that it can extend beyond the apparent size.
  In fact this tail allocation is common on XFS for performance reasons.

An important factor is how well we can distinguish the above data classes.
There are currently three possible identification options:

  Heuristics
    This is used by default to see if holes might be present.
    The test is simply st_size >=3D the appropriate number of allocated s=
t_blocks.
    Note, this can fail for example in the case where there is
    a tail allocation not accounted for in the size like:

      +-----------+---+
      | D | E | H | E |
      +-----------+---+

    Traditionally when a sparse source is detected we check input blocks
    for all zeros and create a 'Hole' in the destination instead.
    This is inefficient as it requires reading all the NUL data
    and verifying that it is in fact NUL.

  SEEK_HOLE
    Available on linux since 3.1

    'Empty' is treated like a 'Hole' which at least
    allows 'Empty' portions to be processed quickly by `cp`.

    We lose the ability to copy the allocation from src to dst.

  fiemap
    Available on linux since around 2.6.39

    Gives greater control by distinguishing Hole and Empty,
    thus allowing us to both efficiently copy and maintain allocation.

    Requires sync on ext4, xfs

    Code already done and used (with sync) for sparse files

    Note by not being able to use fiemap with non sparse files,
    means that we need to read() the empty extents which is
    inefficient, especially in --sparse=3Dalways mode.


So given the above info, what functionality might the use
of fallocate() make available to cp?

Exact copy from source to dest:

  Copying the source layout would mean that one could for example,
  create a backup copy of a large db file, which could be then used
  without worrying about fragmentation or ENOSPC issues.

  There is the argument that this might be better as a higher level
  file operation anyway, and perhaps `cp --reflink` might cover
  this use case on some file systems at least.

  fiemap gives us most control, allowing us to copy even tail
  allocations from source to destination. But the sync issue
  makes it not usable in general at present, and is currently
  restricted to sparse files where it's used to avoid reading
  'Empty' and 'Hole' portions.

Copying sparse files

 It's worth noting again, the caveat mentioned above that we
 might not recognise some sparse files due to tail allocation.

 Given that we use fiemap (with sync) for sparse files at present,
 we can augment the fiemap copying code to use fallocate where appropriat=
e.
  So dependent on the options the operations would be:
    --sparse=3Dauto   =3D> 'Empty' -> 'Empty'
    --sparse=3Dalways =3D> 'Empty' -> 'Hole'  && discard tail allocation
    --sparse=3Dnever  =3D> 'Hole'  -> 'Empty'
 Perhaps the first case could be simplified to initially doing:
    fallocate(dest, blocks*blocksize))

Copying normal files

 Note using SEEK_HOLE for this case, would only help
 to avoid reading 'Hole' and more likely 'Empty' portions,
 and should not impact on the use of fallocate(dest).

 So assuming we initially did:

   if ! --sparse=3Dalways
     fallocate(dest, st_size)

 That would throw away any tail allocation in the source,
 which is probably OK as noted above. In fact we might always
 discard tail allocation for consistency, unless we can use fiemap
 for all cases.

I'll cook something up on this soon.

cheers,
P=E1draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 23 Nov 2011 09:49:01 +0000
Resent-Message-ID: <handler.9500.B9500.132204170526057 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Eric Blake <eblake@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132204170526057
          (code B ref 9500); Wed, 23 Nov 2011 09:49:01 +0000
Received: (at 9500) by debbugs.gnu.org; 23 Nov 2011 09:48:25 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RT9RI-0006mD-OO
	for submit <at> debbugs.gnu.org; Wed, 23 Nov 2011 04:48:24 -0500
Received: from mx.meyering.net ([88.168.87.75])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <jim@HIDDEN>) id 1RT9RF-0006m4-NA
	for 9500 <at> debbugs.gnu.org; Wed, 23 Nov 2011 04:48:22 -0500
Received: from rho.meyering.net (localhost.localdomain [127.0.0.1])
	by rho.meyering.net (Acme Bit-Twister) with ESMTP id 4F19C60076;
	Wed, 23 Nov 2011 10:46:56 +0100 (CET)
From: Jim Meyering <jim@HIDDEN>
In-Reply-To: <4ECC4307.9040600@HIDDEN> ("=?UTF-8?Q?P=C3=A1draig?= Brady"'s message of
	"Wed, 23 Nov 2011 00:49:11 +0000")
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN>
Date: Wed, 23 Nov 2011 10:46:56 +0100
Message-ID: <87sjlf5fxr.fsf@HIDDEN>
Lines: 60
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.7 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.7 (--)

P=E1draig Brady wrote:
...
> I thought a little about this today.

Nice description of the issues.

It's probably worth putting something like this somewhere in version
control, even if only as a long commit message on whatever change you make.

> fallocate() is a feature to quickly allocate space in a file system.
> It's useful for 3 things as far as I can see:
>
>   1. Improved file layout for subsequent access
>   2. Immediate indication of ENOSPC
>   3. Efficient writing of NUL portions
>
> Note 1. is somewhat moot with newer file systems that do "delayed allocat=
ion".
> So what do we need to consider when using fallocate on the destination fi=
le?
> Considering just cp for the moment, its inputs impacting this are the opt=
ions:
>
...
> Copying sparse files
>
>  It's worth noting again, the caveat mentioned above that we
>  might not recognise some sparse files due to tail allocation.

Yes, this is worth repeating ;-)
It is surprising, at least in part because significant tail
allocation is not common.

>  Given that we use fiemap (with sync) for sparse files at present,
>  we can augment the fiemap copying code to use fallocate where appropriat=
e.
>   So dependent on the options the operations would be:
>     --sparse=3Dauto   =3D> 'Empty' -> 'Empty'
>     --sparse=3Dalways =3D> 'Empty' -> 'Hole'  && discard tail allocation
>     --sparse=3Dnever  =3D> 'Hole'  -> 'Empty'
>  Perhaps the first case could be simplified to initially doing:
>     fallocate(dest, blocks*blocksize))
>
> Copying normal files
>
>  Note using SEEK_HOLE for this case, would only help
>  to avoid reading 'Hole' and more likely 'Empty' portions,
>  and should not impact on the use of fallocate(dest).
>
>  So assuming we initially did:
>
>    if ! --sparse=3Dalways
>      fallocate(dest, st_size)
>
>  That would throw away any tail allocation in the source,
>  which is probably OK as noted above. In fact we might always
>  discard tail allocation for consistency, unless we can use fiemap
>  for all cases.

All sounds reasonable.

> I'll cook something up on this soon.

Thanks.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: "Voelker, Bernhard" <bernhard.voelker@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 23 Nov 2011 13:59:01 +0000
Resent-Message-ID: <handler.9500.B9500.132205669419174 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Jim Meyering <jim@HIDDEN>, =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Cc: "9500 <at> debbugs.gnu.org" <9500 <at> debbugs.gnu.org>, Kelly Anderson <kelly@HIDDEN>, Eric Blake <eblake@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132205669419174
          (code B ref 9500); Wed, 23 Nov 2011 13:59:01 +0000
Received: (at 9500) by debbugs.gnu.org; 23 Nov 2011 13:58:14 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RTDL2-0004zC-Iq
	for submit <at> debbugs.gnu.org; Wed, 23 Nov 2011 08:58:13 -0500
Received: from senmx11-mx.siemens-enterprise.com ([62.134.46.9])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <bernhard.voelker@HIDDEN>)
	id 1RTDKz-0004z4-Iz
	for 9500 <at> debbugs.gnu.org; Wed, 23 Nov 2011 08:58:10 -0500
Received: from MCHP063A.global-ad.net (unknown [172.29.37.61])
	by senmx11-mx.siemens-enterprise.com (Server) with ESMTP id
	45AB61EB845A; Wed, 23 Nov 2011 14:56:43 +0100 (CET)
Received: from MCHP058A.global-ad.net ([172.29.37.55]) by
	MCHP063A.global-ad.net ([172.29.37.61]) with mapi;
	Wed, 23 Nov 2011 14:56:43 +0100
From: "Voelker, Bernhard" <bernhard.voelker@HIDDEN>
Date: Wed, 23 Nov 2011 14:56:41 +0100
Thread-Topic: bug#9500: [PATCH]: use posix_fallocate where supported
Thread-Index: AcypxPVAMoVPDWdwTWaZD/7dxdB8VAAIU0bQ
Message-ID: <7856072A9D04C24B82DFE2B1112FE38A0D86E97480@HIDDEN>
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <87sjlf5fxr.fsf@HIDDEN>
In-Reply-To: <87sjlf5fxr.fsf@HIDDEN>
Accept-Language: de-DE, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: de-DE, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Spam-Score: -3.2 (---)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.2 (---)

Jim Meyering wrote:

> P=E1draig Brady wrote:
> ...
> > I thought a little about this today.
>=20
> Nice description of the issues.

BTW: there was a discussion recently about the
fallocate utility of util-linux, e.g.
http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/5045
Maybe looking into fallocate.c can help.

Berny

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 25 Nov 2011 10:00:02 +0000
Resent-Message-ID: <handler.9500.B9500.13222151875989 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: 9500 <at> debbugs.gnu.org
Cc: Kelly Anderson <kelly@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13222151875989
          (code B ref 9500); Fri, 25 Nov 2011 10:00:02 +0000
Received: (at 9500) by debbugs.gnu.org; 25 Nov 2011 09:59:47 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RTsZP-0001YY-6G
	for submit <at> debbugs.gnu.org; Fri, 25 Nov 2011 04:59:47 -0500
Received: from mx1.redhat.com ([209.132.183.28])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <P@HIDDEN>) id 1RTsZN-0001YQ-1U
	for 9500 <at> debbugs.gnu.org; Fri, 25 Nov 2011 04:59:46 -0500
Received: from int-mx12.intmail.prod.int.phx2.redhat.com
	(int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pAP9w8aK000576
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 25 Nov 2011 04:58:08 -0500
Received: from [10.36.116.18] (ovpn-116-18.ams2.redhat.com [10.36.116.18])
	by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id pAP9w5kv018046
	(version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
	Fri, 25 Nov 2011 04:58:07 -0500
Message-ID: <4ECF66AD.9050204@HIDDEN>
Date: Fri, 25 Nov 2011 09:58:05 +0000
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN>
In-Reply-To: <4ECC4307.9040600@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	pAP9w8aK000576
X-Spam-Score: -10.5 (----------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -10.5 (----------)

-------- Original Message --------
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Date: Fri, 25 Nov 2011 10:35:34 +0100
From: Goswin von Brederlow <goswin-v-b@HIDDEN>
To: Padraig Brady <p@HIDDEN>

Hi,

On 09/14/2011 03:46 PM, P=E1draig Brady wrote:

> I thought a little about this today.
>
> fallocate() is a feature to quickly allocate space in a file system.
> It's useful for 3 things as far as I can see:
>
>   1. Improved file layout for subsequent access
>   2. Immediate indication of ENOSPC
>   3. Efficient writing of NUL portions
>
> Note 1. is somewhat moot with newer file systems that do "delayed alloc=
ation".
> So what do we need to consider when using fallocate on the destination =
file?
> Considering just cp for the moment, its inputs impacting this are the o=
ptions:

Not every filesystem does delayed allocation and delayed allocation only
works for file considerably smaller than the amount of cache.


Also a note on sparse files:

If you have fiemap then you can fallocate the used chunks in the
destination. But there might be zero filled blocks that would become
unallocated with current cp. Those would remain allocated (but not zero
filles, which might be a difference) in the destination. Same if the
file is fully allocated but has empty blocks. So a bit more has to be
done.

So here is what I would do for sparse files:

- get fiemap of file, if it fails the fiemap is just one chunk for the
  whole file

- For each chunk in the fiemap:
  1 read block from source
  2 if (block =3D=3D ZERO) goto 7
  3 fallocate(offset, end of chunk)
  4 write block
  5 read block from source
  6 if (block !=3D ZERO) goto 4
  7 fruncate(offset)
  8 read block from source
  9 if (block =3D=3D ZERO) goto 8
 10 goto 3

Failure of fallocate should probably be ignored to preserve current
behaviour and because if empty blocks are found less space will be used
and might actually fit.

The above would result in the same allocation for the file as current cp
creates for sparse files.

If you want to copy files while preserving the allocation of the source
then I think cp needs a new --sparse=3Dsource mode. In that mode setp 7
would change from ftruncate() to write(ZERO) to preserve the empty
blocks and you would only fallocate once per chunk for the full chunk.

Hope that helps.

MfG
        Goswin

PS: Feel free to forward to the mailinglist, gmane doesn't show its
address or I would cc.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 25 Nov 2011 10:16:01 +0000
Resent-Message-ID: <handler.9500.B9500.13222161217512 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: 9500 <at> debbugs.gnu.org
Cc: Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13222161217512
          (code B ref 9500); Fri, 25 Nov 2011 10:16:01 +0000
Received: (at 9500) by debbugs.gnu.org; 25 Nov 2011 10:15:21 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RTsoT-0001x6-1O
	for submit <at> debbugs.gnu.org; Fri, 25 Nov 2011 05:15:21 -0500
Received: from mx1.redhat.com ([209.132.183.28])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <P@HIDDEN>) id 1RTsoQ-0001wz-TN
	for 9500 <at> debbugs.gnu.org; Fri, 25 Nov 2011 05:15:19 -0500
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pAPADgs8004034
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 25 Nov 2011 05:13:42 -0500
Received: from [10.36.116.18] (ovpn-116-18.ams2.redhat.com [10.36.116.18])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id pAPADcQ3021378
	(version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
	Fri, 25 Nov 2011 05:13:40 -0500
Message-ID: <4ECF6A53.3060104@HIDDEN>
Date: Fri, 25 Nov 2011 10:13:39 +0000
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
In-Reply-To: <4ECF66AD.9050204@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	pAPADgs8004034
X-Spam-Score: -10.5 (----------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -10.5 (----------)

On 11/25/2011 09:58 AM, P=E1draig Brady wrote:
> -------- Original Message --------
> Subject: bug#9500: [PATCH]: use posix_fallocate where supported
> Date: Fri, 25 Nov 2011 10:35:34 +0100
> From: Goswin von Brederlow <goswin-v-b@HIDDEN>
> To: Padraig Brady <p@HIDDEN>
>=20
> Hi,
>=20
> On 09/14/2011 03:46 PM, P=E1draig Brady wrote:
>=20
>> I thought a little about this today.
>>
>> fallocate() is a feature to quickly allocate space in a file system.
>> It's useful for 3 things as far as I can see:
>>
>>   1. Improved file layout for subsequent access
>>   2. Immediate indication of ENOSPC
>>   3. Efficient writing of NUL portions
>>
>> Note 1. is somewhat moot with newer file systems that do "delayed allo=
cation".
>> So what do we need to consider when using fallocate on the destination=
 file?
>> Considering just cp for the moment, its inputs impacting this are the =
options:
>=20
> Not every filesystem does delayed allocation and delayed allocation onl=
y
> works for file considerably smaller than the amount of cache.

Right.

> Also a note on sparse files:
>=20
> If you have fiemap then you can fallocate the used chunks in the
> destination. But there might be zero filled blocks that would become
> unallocated with current cp. Those would remain allocated (but not zero
> filles, which might be a difference) in the destination. Same if the
> file is fully allocated but has empty blocks. So a bit more has to be
> done.
>=20
> So here is what I would do for sparse files:
>=20
> - get fiemap of file, if it fails the fiemap is just one chunk for the
>   whole file
>=20
> - For each chunk in the fiemap:
>   1 read block from source
>   2 if (block =3D=3D ZERO) goto 7
>   3 fallocate(offset, end of chunk)
>   4 write block
>   5 read block from source
>   6 if (block !=3D ZERO) goto 4
>   7 fruncate(offset)
>   8 read block from source
>   9 if (block =3D=3D ZERO) goto 8
>  10 goto 3

I didn't fully understand the above,
but I was going to do essentially that I think.
The adjustments to the fiemap copy code
should be fairly obvious I think.

> Failure of fallocate should probably be ignored to preserve current
> behaviour and because if empty blocks are found less space will be used
> and might actually fit.

> The above would result in the same allocation for the file as current c=
p
> creates for sparse files.
>=20
> If you want to copy files while preserving the allocation of the source
> then I think cp needs a new --sparse=3Dsource mode.

I think --spare=3Dauto (the default) should cover that case if needed.
However as previously stated, this might be outside cp's remit.

> In that mode setp 7
> would change from ftruncate() to write(ZERO) to preserve the empty
> blocks and you would only fallocate once per chunk for the full chunk.
>=20
> Hope that helps.
>=20
> MfG
>         Goswin
>=20
> PS: Feel free to forward to the mailinglist, gmane doesn't show its
> address or I would cc.

Hmm, one of your points above got me thinking.
Might fallocate() fail to allocate an extent with ENOSPC,
but there could be fragmented space available to write()?
That would scupper benefit (2) above :(
I'll ask linux-fsdevel@HIDDEN

cheers,
P=E1draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 25 Nov 2011 11:01:02 +0000
Resent-Message-ID: <handler.9500.B9500.132221885211796 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: 9500 <at> debbugs.gnu.org
Cc: Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132221885211796
          (code B ref 9500); Fri, 25 Nov 2011 11:01:02 +0000
Received: (at 9500) by debbugs.gnu.org; 25 Nov 2011 11:00:52 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RTtWW-00034D-GO
	for submit <at> debbugs.gnu.org; Fri, 25 Nov 2011 06:00:52 -0500
Received: from mx1.redhat.com ([209.132.183.28])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <P@HIDDEN>) id 1RTtWT-000344-Eh
	for 9500 <at> debbugs.gnu.org; Fri, 25 Nov 2011 06:00:50 -0500
Received: from int-mx01.intmail.prod.int.phx2.redhat.com
	(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pAPAxCvE001124
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 25 Nov 2011 05:59:12 -0500
Received: from [10.36.116.18] (ovpn-116-18.ams2.redhat.com [10.36.116.18])
	by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP
	id pAPAx99I019919
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 25 Nov 2011 05:59:11 -0500
Message-ID: <4ECF74FD.1010609@HIDDEN>
Date: Fri, 25 Nov 2011 10:59:09 +0000
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN>
In-Reply-To: <4ECF6A53.3060104@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	pAPAxCvE001124
X-Spam-Score: -10.5 (----------)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -10.5 (----------)

On 11/25/2011 10:13 AM, P=E1draig Brady wrote:

> Hmm, one of your points above got me thinking.
> Might fallocate() fail to allocate an extent with ENOSPC,
> but there could be fragmented space available to write()?
> That would scupper benefit (2) above :(
> I'll ask linux-fsdevel@HIDDEN

And the response from there is that fallocate() will
check the free blocks first, and then try to allocate
as a contiguous extent, but that part is not guaranteed.
So we still get benefit (2).

cheers,
P=E1draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Goswin von Brederlow <goswin-v-b@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Sat, 26 Nov 2011 08:17:01 +0000
Resent-Message-ID: <handler.9500.B9500.13222953767052 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Padraig Brady <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13222953767052
          (code B ref 9500); Sat, 26 Nov 2011 08:17:01 +0000
Received: (at 9500) by debbugs.gnu.org; 26 Nov 2011 08:16:16 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RUDQm-0001pg-2S
	for submit <at> debbugs.gnu.org; Sat, 26 Nov 2011 03:16:16 -0500
Received: from fmmailgate01.web.de ([217.72.192.221])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <goswin-v-b@HIDDEN>) id 1RU9HR-000496-8B
	for 9500 <at> debbugs.gnu.org; Fri, 25 Nov 2011 22:50:22 -0500
Received: from moweb002.kundenserver.de (moweb002.kundenserver.de
	[172.19.20.108])
	by fmmailgate01.web.de (Postfix) with ESMTP id E959B1A410787
	for <9500 <at> debbugs.gnu.org>; Sat, 26 Nov 2011 04:48:39 +0100 (CET)
Received: from frosties.localnet ([95.208.118.96]) by smtp.web.de (mrweb002)
	with ESMTPA (Nemesis) id 0LhvUI-1R01R93J6A-00nBfJ;
	Sat, 26 Nov 2011 04:48:38 +0100
Received: from mrvn by frosties.localnet with local (Exim 4.76)
	(envelope-from <goswin-v-b@HIDDEN>)
	id 1RU9Fk-0002Ou-SV; Sat, 26 Nov 2011 04:48:36 +0100
From: Goswin von Brederlow <goswin-v-b@HIDDEN>
In-Reply-To: <4ECF6A53.3060104@HIDDEN> ("Padraig Brady"'s message of
	"Fri, 25 Nov 2011 10:13:39 +0000")
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN>
User-Agent: Gnus/5.110009 (No Gnus v0.9) XEmacs/21.4.22 (linux, no MULE)
Date: Sat, 26 Nov 2011 04:48:36 +0100
Message-ID: <87ipm7mtm3.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
X-Provags-ID: V02:K0:X6XSB+mvKME9qTnl4YH9YSoNmO4g6G2sM3Z1BDC26B4
	qSOog0zdtgH3v09y5gYR3oMDLt80oVOyceWbUKCqGWCwkn+Vgf
	Z/OVaynlNVGVHoVqAHO8Y8+JW/KFvL+6nnggixMTGGHaYGSU+g
	bXkNjhSdiBC7oJdM5hd4tK91l09hnPWjqaQ1EL1/GGzpzl0Dek
	1kR/qRd1CHFK5GDPz1AnA==
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.6 (--)
X-Mailman-Approved-At: Sat, 26 Nov 2011 03:16:14 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.6 (--)

P=E1draig Brady <P@HIDDEN> writes:

> Hmm, one of your points above got me thinking.
> Might fallocate() fail to allocate an extent with ENOSPC,
> but there could be fragmented space available to write()?
> That would scupper benefit (2) above :(
> I'll ask linux-fsdevel@HIDDEN
>
> cheers,
> P=E1draig.

fallocate() in no way garanties that the space is continious. Any
filesystem that is halfway smart will use a continious chunk if
possible. But it should only fail if there is really no space, no matter
how fragmented, left.

MfG
        Goswin

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Goswin von Brederlow <goswin-v-b@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Sat, 26 Nov 2011 08:17:02 +0000
Resent-Message-ID: <handler.9500.B9500.13222953777058 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Padraig Brady <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13222953777058
          (code B ref 9500); Sat, 26 Nov 2011 08:17:02 +0000
Received: (at 9500) by debbugs.gnu.org; 26 Nov 2011 08:16:17 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RUDQm-0001pi-SU
	for submit <at> debbugs.gnu.org; Sat, 26 Nov 2011 03:16:17 -0500
Received: from fmmailgate04.web.de ([217.72.192.242])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <goswin-v-b@HIDDEN>) id 1RU9Ni-0004Hv-Qm
	for 9500 <at> debbugs.gnu.org; Fri, 25 Nov 2011 22:56:51 -0500
Received: from moweb002.kundenserver.de (moweb002.kundenserver.de
	[172.19.20.108])
	by fmmailgate04.web.de (Postfix) with ESMTP id 864FD6F1729E
	for <9500 <at> debbugs.gnu.org>; Sat, 26 Nov 2011 04:55:09 +0100 (CET)
Received: from frosties.localnet ([95.208.118.96]) by smtp.web.de (mrweb001)
	with ESMTPA (Nemesis) id 0LopMx-1Qtjnj35Ap-00fxBP;
	Sat, 26 Nov 2011 04:55:08 +0100
Received: from mrvn by frosties.localnet with local (Exim 4.76)
	(envelope-from <goswin-v-b@HIDDEN>)
	id 1RU9M4-0002Qo-2q; Sat, 26 Nov 2011 04:55:08 +0100
From: Goswin von Brederlow <goswin-v-b@HIDDEN>
In-Reply-To: <4ECF74FD.1010609@HIDDEN> ("Padraig Brady"'s message of
	"Fri, 25 Nov 2011 10:59:09 +0000")
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN> <4ECF74FD.1010609@HIDDEN>
User-Agent: Gnus/5.110009 (No Gnus v0.9) XEmacs/21.4.22 (linux, no MULE)
Date: Sat, 26 Nov 2011 04:55:08 +0100
Message-ID: <87d3cfmtb7.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
X-Provags-ID: V02:K0:d2kEpqJqWCeHuJ4n09xt37xg7YTBninvwztFVk+WNps
	2yDquWmnf/YTcOFXylJSH2TgxM4L3zFoJ7xlR1ZOnIVMC27/QS
	k2VTORC9lTQcZkc7YNN12AZBN/kFUh3ptNRHIzShrN+4L0Gle6
	Y5TkpUtoycLyhzeSw9Z7DI5ElsOLRaAlaY5KUAHfpBUU4Jz5Dz
	KrPcPLM40PK5U7kmoheTg==
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.6 (--)
X-Mailman-Approved-At: Sat, 26 Nov 2011 03:16:14 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.6 (--)

P=E1draig Brady <P@HIDDEN> writes:

> On 11/25/2011 10:13 AM, P=E1draig Brady wrote:
>
>> Hmm, one of your points above got me thinking.
>> Might fallocate() fail to allocate an extent with ENOSPC,
>> but there could be fragmented space available to write()?
>> That would scupper benefit (2) above :(
>> I'll ask linux-fsdevel@HIDDEN
>
> And the response from there is that fallocate() will
> check the free blocks first, and then try to allocate
> as a contiguous extent, but that part is not guaranteed.
> So we still get benefit (2).
>
> cheers,
> P=E1draig.

Don't forget that a sparse file uses less space than its size. So any cp
mode that checks for zero filled blocks and omits them on the
destination might use less blocks than the source uses or than the
fiemap indicates.

So cp can only fail on ENOSPC from fallocate() if it does not skip zero
filled blocks. I.e. when --sparse=3Dnever or --sparse=3Dauto doesn't dete=
ct
a sparse file. But that is quite often the case.

MfG
        Goswin

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Tue, 29 Nov 2011 13:02:01 +0000
Resent-Message-ID: <handler.9500.B9500.132257166518969 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: 9500 <at> debbugs.gnu.org
Cc: Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132257166518969
          (code B ref 9500); Tue, 29 Nov 2011 13:02:01 +0000
Received: (at 9500) by debbugs.gnu.org; 29 Nov 2011 13:01:05 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RVNJ3-0004vs-2q
	for submit <at> debbugs.gnu.org; Tue, 29 Nov 2011 08:01:05 -0500
Received: from mail2.vodafone.ie ([213.233.128.44])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <P@HIDDEN>) id 1RVNJ1-0004vZ-36
	for 9500 <at> debbugs.gnu.org; Tue, 29 Nov 2011 08:01:04 -0500
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApMBAFLW1E5tTdZN/2dsb2JhbAAMN61tAQEBBDIBRhALDQsJFg8JAwIBAgFFEwEFAgEBiAm3FogJgxUEmjmDJ4kF
Received: from unknown (HELO [192.168.1.79]) ([109.77.214.77])
	by mail2.vodafone.ie with ESMTP; 29 Nov 2011 12:59:01 +0000
Message-ID: <4ED4D715.7010803@HIDDEN>
Date: Tue, 29 Nov 2011 12:59:01 +0000
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN> <4ECF74FD.1010609@HIDDEN>
In-Reply-To: <4ECF74FD.1010609@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.6 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.6 (--)

On 11/25/2011 10:59 AM, P�draig Brady wrote:
> On 11/25/2011 10:13 AM, P�draig Brady wrote:
> 
>> Hmm, one of your points above got me thinking.
>> Might fallocate() fail to allocate an extent with ENOSPC,
>> but there could be fragmented space available to write()?
>> That would scupper benefit (2) above :(
>> I'll ask linux-fsdevel@HIDDEN
> 
> And the response from there is that fallocate() will
> check the free blocks first, and then try to allocate
> as a contiguous extent, but that part is not guaranteed.
> So we still get benefit (2).

However that thread has continued, and it
was mentioned that benefit (1) might actually
be detrimental, at least for small files.

http://thread.gmane.org/gmane.linux.file-systems/59092

le sigh,
P�draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Jim Meyering <jim@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Tue, 29 Nov 2011 13:25:02 +0000
Resent-Message-ID: <handler.9500.B9500.132257307720994 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132257307720994
          (code B ref 9500); Tue, 29 Nov 2011 13:25:02 +0000
Received: (at 9500) by debbugs.gnu.org; 29 Nov 2011 13:24:37 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RVNfp-0005SZ-AQ
	for submit <at> debbugs.gnu.org; Tue, 29 Nov 2011 08:24:37 -0500
Received: from mx.meyering.net ([88.168.87.75])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <jim@HIDDEN>) id 1RVNfm-0005SQ-CX
	for 9500 <at> debbugs.gnu.org; Tue, 29 Nov 2011 08:24:35 -0500
Received: from rho.meyering.net (localhost.localdomain [127.0.0.1])
	by rho.meyering.net (Acme Bit-Twister) with ESMTP id 26C8360020;
	Tue, 29 Nov 2011 14:22:33 +0100 (CET)
From: Jim Meyering <jim@HIDDEN>
In-Reply-To: <4ED4D715.7010803@HIDDEN> ("=?UTF-8?Q?P=C3=A1draig?= Brady"'s message of
	"Tue, 29 Nov 2011 12:59:01 +0000")
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN> <4ECF74FD.1010609@HIDDEN>
	<4ED4D715.7010803@HIDDEN>
Date: Tue, 29 Nov 2011 14:22:33 +0100
Message-ID: <87y5uz12sm.fsf@HIDDEN>
Lines: 23
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.7 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.7 (--)

P=E1draig Brady wrote:
> On 11/25/2011 10:59 AM, P=E1draig Brady wrote:
>> On 11/25/2011 10:13 AM, P=E1draig Brady wrote:
>>
>>> Hmm, one of your points above got me thinking.
>>> Might fallocate() fail to allocate an extent with ENOSPC,
>>> but there could be fragmented space available to write()?
>>> That would scupper benefit (2) above :(
>>> I'll ask linux-fsdevel@HIDDEN
>>
>> And the response from there is that fallocate() will
>> check the free blocks first, and then try to allocate
>> as a contiguous extent, but that part is not guaranteed.
>> So we still get benefit (2).
>
> However that thread has continued, and it
> was mentioned that benefit (1) might actually
> be detrimental, at least for small files.
>
> http://thread.gmane.org/gmane.linux.file-systems/59092

Sigh, indeed.
Do you think it's even worth an option, now?

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Tue, 29 Nov 2011 14:16:02 +0000
Resent-Message-ID: <handler.9500.B9500.132257613226151 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Jim Meyering <jim@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132257613226151
          (code B ref 9500); Tue, 29 Nov 2011 14:16:02 +0000
Received: (at 9500) by debbugs.gnu.org; 29 Nov 2011 14:15:32 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RVOT6-0006nk-6o
	for submit <at> debbugs.gnu.org; Tue, 29 Nov 2011 09:15:32 -0500
Received: from mail2.vodafone.ie ([213.233.128.44])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <P@HIDDEN>) id 1RVOT3-0006nb-GA
	for 9500 <at> debbugs.gnu.org; Tue, 29 Nov 2011 09:15:30 -0500
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApMBANvn1E5tTdZN/2dsb2JhbAAMN61wAQEBBDIBRhALDQsJFg8JAwIBAgFFBg0BBQIBAYgJtzGLHgSaOYMniQU
Received: from unknown (HELO [192.168.1.79]) ([109.77.214.77])
	by mail2.vodafone.ie with ESMTP; 29 Nov 2011 14:13:28 +0000
Message-ID: <4ED4E887.4050702@HIDDEN>
Date: Tue, 29 Nov 2011 14:13:27 +0000
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN> <4ECF74FD.1010609@HIDDEN>
	<4ED4D715.7010803@HIDDEN> <87y5uz12sm.fsf@HIDDEN>
In-Reply-To: <87y5uz12sm.fsf@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.6 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.6 (--)

On 11/29/2011 01:22 PM, Jim Meyering wrote:
> P�draig Brady wrote:
>> On 11/25/2011 10:59 AM, P�draig Brady wrote:
>>> On 11/25/2011 10:13 AM, P�draig Brady wrote:
>>>
>>>> Hmm, one of your points above got me thinking.
>>>> Might fallocate() fail to allocate an extent with ENOSPC,
>>>> but there could be fragmented space available to write()?
>>>> That would scupper benefit (2) above :(
>>>> I'll ask linux-fsdevel@HIDDEN
>>>
>>> And the response from there is that fallocate() will
>>> check the free blocks first, and then try to allocate
>>> as a contiguous extent, but that part is not guaranteed.
>>> So we still get benefit (2).
>>
>> However that thread has continued, and it
>> was mentioned that benefit (1) might actually
>> be detrimental, at least for small files.
>>
>> http://thread.gmane.org/gmane.linux.file-systems/59092
> 
> Sigh, indeed.
> Do you think it's even worth an option, now?

Having an option to enable is much less interesting.
One might auto-enable it for files > 16MB or something.
I'm reserving judgement until I understand the issues better.

cheers,
P�draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Paul Eggert <eggert@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Tue, 29 Nov 2011 16:10:02 +0000
Resent-Message-ID: <handler.9500.B9500.13225829437113 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Jim Meyering <jim@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13225829437113
          (code B ref 9500); Tue, 29 Nov 2011 16:10:02 +0000
Received: (at 9500) by debbugs.gnu.org; 29 Nov 2011 16:09:03 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RVQEw-0001qe-NT
	for submit <at> debbugs.gnu.org; Tue, 29 Nov 2011 11:09:03 -0500
Received: from smtp.cs.ucla.edu ([131.179.128.62])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <eggert@HIDDEN>) id 1RVQEt-0001qC-UB
	for 9500 <at> debbugs.gnu.org; Tue, 29 Nov 2011 11:09:01 -0500
Received: from localhost (localhost.localdomain [127.0.0.1])
	by smtp.cs.ucla.edu (Postfix) with ESMTP id 73C66A60005;
	Tue, 29 Nov 2011 08:06:58 -0800 (PST)
X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu
Received: from smtp.cs.ucla.edu ([127.0.0.1])
	by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id g6JmvC2ZJMn0; Tue, 29 Nov 2011 08:06:58 -0800 (PST)
Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200])
	by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 12CF3A60001;
	Tue, 29 Nov 2011 08:06:58 -0800 (PST)
Message-ID: <4ED50321.8030204@HIDDEN>
Date: Tue, 29 Nov 2011 08:06:57 -0800
From: Paul Eggert <eggert@HIDDEN>
Organization: UCLA Computer Science Department
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:8.0) Gecko/20111115 Thunderbird/8.0
MIME-Version: 1.0
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN> <4ECF74FD.1010609@HIDDEN>
	<4ED4D715.7010803@HIDDEN> <87y5uz12sm.fsf@HIDDEN>
	<4ED4E887.4050702@HIDDEN>
In-Reply-To: <4ED4E887.4050702@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Spam-Score: -3.0 (---)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.0 (---)

My read of the situation is that the filesystem guys have
spent a lot of time optimizing ordinary write but they
haven't gotten around to optimizing fallocate because it's
so rarely used -- which means that if one uses fallocate
one gets lousy performance.

It's a chicken and egg problem.

If coreutils started using fallocate now, one can be pretty
sure they'd tune their filesystems over the next few years,
to make fallocate compatible with delayed-write optimizations.
On the other hand if nobody uses fallocate, there will be little
incentive on their part to make it go fast.

It's a question of whether we want to inflict temporary pain
on users for a long-term benefit (early warning of file system
full, which is something I'd dearly love to have).

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [PATCH]: use posix_fallocate where supported
Resent-From: Goswin von Brederlow <goswin-v-b@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Wed, 30 Nov 2011 17:55:01 +0000
Resent-Message-ID: <handler.9500.B9500.132267566822123 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Paul Eggert <eggert@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org, Kelly Anderson <kelly@HIDDEN>, Padraig Brady <P@HIDDEN>, Jim Meyering <jim@HIDDEN>, Goswin von Brederlow <goswin-v-b@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.132267566822123
          (code B ref 9500); Wed, 30 Nov 2011 17:55:01 +0000
Received: (at 9500) by debbugs.gnu.org; 30 Nov 2011 17:54:28 +0000
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1RVoMW-0005kl-8M
	for submit <at> debbugs.gnu.org; Wed, 30 Nov 2011 12:54:28 -0500
Received: from fmmailgate04.web.de ([217.72.192.242])
	by debbugs.gnu.org with esmtp (Exim 4.69)
	(envelope-from <goswin-v-b@HIDDEN>) id 1RVoMT-0005ke-O4
	for 9500 <at> debbugs.gnu.org; Wed, 30 Nov 2011 12:54:27 -0500
Received: from moweb001.kundenserver.de (moweb001.kundenserver.de
	[172.19.20.114])
	by fmmailgate04.web.de (Postfix) with ESMTP id 8C0806F1EE17
	for <9500 <at> debbugs.gnu.org>; Wed, 30 Nov 2011 18:54:20 +0100 (CET)
Received: from frosties.localnet ([95.208.118.96]) by smtp.web.de (mrweb001)
	with ESMTPA (Nemesis) id 0MJTHn-1RXjEY1zH1-002CuR;
	Wed, 30 Nov 2011 18:54:18 +0100
Received: from mrvn by frosties.localnet with local (Exim 4.77)
	(envelope-from <goswin-v-b@HIDDEN>)
	id 1RVoMJ-0003OE-QX; Wed, 30 Nov 2011 18:54:16 +0100
From: Goswin von Brederlow <goswin-v-b@HIDDEN>
In-Reply-To: <4ED50321.8030204@HIDDEN> (Paul Eggert's message of "Tue, 29
	Nov 2011 08:06:57 -0800")
References: <4E7041CE.4090305@HIDDEN>
	<4E70B4F9.6080303@HIDDEN> <4E70BE57.6030800@HIDDEN>
	<4ECC4307.9040600@HIDDEN> <4ECF66AD.9050204@HIDDEN>
	<4ECF6A53.3060104@HIDDEN> <4ECF74FD.1010609@HIDDEN>
	<4ED4D715.7010803@HIDDEN> <87y5uz12sm.fsf@HIDDEN>
	<4ED4E887.4050702@HIDDEN> <4ED50321.8030204@HIDDEN>
User-Agent: Gnus/5.110009 (No Gnus v0.9) XEmacs/21.4.22 (linux, no MULE)
Date: Wed, 30 Nov 2011 18:54:15 +0100
Message-ID: <87ty5liji0.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Provags-ID: V02:K0:2z7WkGKBL4Sx+wgmpW38qXK5+bLnDd4hg3uM336Qm3c
	7eP5S9tI7NTtQihIxC54x7qE4jmKWkMYkoVIbbWsDVcHS2ZhtQ
	0WNUGjDbMiZn5byE1McNDnyCxuFDXsWlD7/iEYkgaYExUwK/yC
	+1sdFNypH1pa9XoKjG1RVZsDFOgwVOuA3WNa4PwkRJ1aJy7f+R
	fW2idRAV7/Da5+tx7uG3A==
X-Spam-Score: -2.6 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.6 (--)

Paul Eggert <eggert@HIDDEN> writes:

> My read of the situation is that the filesystem guys have
> spent a lot of time optimizing ordinary write but they
> haven't gotten around to optimizing fallocate because it's
> so rarely used -- which means that if one uses fallocate
> one gets lousy performance.
>
> It's a chicken and egg problem.
>
> If coreutils started using fallocate now, one can be pretty
> sure they'd tune their filesystems over the next few years,
> to make fallocate compatible with delayed-write optimizations.
> On the other hand if nobody uses fallocate, there will be little
> incentive on their part to make it go fast.
>
> It's a question of whether we want to inflict temporary pain
> on users for a long-term benefit (early warning of file system
> full, which is something I'd dearly love to have).

I totaly agree.

I also don't buy Daves analysis that fallocate() will hurt the
filesystem.

Sure, it will place blocks on the disk disjunct from the write
pattern. So if you have a lot of files being written in parallel the
fallocate() will make the disk seek more. But the data for each file
will end up sequentially on disk. Without fallocate() it will be layed
out in order of the write pattern, i.e. 4MB of this file, 4MB of that
file, 4MB of the next file, 4MB of the first file and so on. Lots of
fragments of a size the systems cache and delayed allocation allowed.

So fallocate() might hurt write speed with many parallel writes but it
will keep the fragmentation down and speed up future reads. A one time
penalty for many times advantages in the future.

As for filesystem aligning all fallocate() chunks and creating
fragmentation in their free space: Too bad for them. FIX THE FILESYSTEM.
If I tell the FS that I'm only going to write 32k then it should not
force alignment to a 1MB chunk of free space. Instead it should find
some nice little 32k fragment of free space left over somewhere else.
Put all the 32K files together to fill up the 1MB stripe of an raid.

As for fallocate() causing more IOPS I don't buy that either. Either the
data is too big for the cache so that it is forced out with fallocate()
or delayed allocation or it is so small that it remains in cache in both
cases and the elevator code should write it out sequentially. I mean we
are not talking about opening 1000 files, fallocat()ing them to 100GB
each and then writing 4k chunks to each in a round-robin way. Cp is
writing ONE sequential stream as fast as possible.

As for his assertion that three major Linux filesystems (XFS, BTRFS and
ext4) don't need fallocate() because they use delayed allocation that is
plain not true for large files.

As a compromise cp could start with using fallocate() only for largish
files, say 16MB and above. Anything smaller can probably be handled by
delayed allocation or goes out so fast it stays in one chunk anyway.

MfG
        Goswin

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 11 May 2012 15:56:02 +0000
Resent-Message-ID: <handler.9500.B9500.13367517598316 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: 9500 <at> debbugs.gnu.org, Mark <markk@HIDDEN>
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.13367517598316
          (code B ref 9500); Fri, 11 May 2012 15:56:02 +0000
Received: (at 9500) by debbugs.gnu.org; 11 May 2012 15:55:59 +0000
Received: from localhost ([127.0.0.1]:49304 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1SSsCE-0002A4-3a
	for submit <at> debbugs.gnu.org; Fri, 11 May 2012 11:55:58 -0400
Received: from mail2.vodafone.ie ([213.233.128.44]:64692)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <P@HIDDEN>) id 1SSsCA-00029o-Sm
	for 9500 <at> debbugs.gnu.org; Fri, 11 May 2012 11:55:56 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApMBAC82rU9tTAC//2dsb2JhbAANN7dBAQEBAwEyAUsLCw0LCRYPCQMCAQIBRQYBDAgBAYgFuxCLF4YdBJcOhQaFKYdq
Received: from unknown (HELO [192.168.1.79]) ([109.76.0.191])
	by mail2.vodafone.ie with ESMTP; 11 May 2012 16:55:36 +0100
Message-ID: <4FAD3677.2080906@HIDDEN>
Date: Fri, 11 May 2012 16:55:35 +0100
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <90cdbf76a2ca009ab20d719e55385197.squirrel@HIDDEN>
In-Reply-To: <90cdbf76a2ca009ab20d719e55385197.squirrel@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -1.9 (-)

On 05/11/2012 04:03 PM, Mark wrote:
> Hi,
>
> Here's a patch for cp which adds a new --preallocate option. When
> specified, cp allocates disk space for the destination file before writing
> data. It uses fallocate() with FALLOC_FL_KEEP_SIZE on Linux, falling back
> to posix_fallocate() if that fails.

Thanks for taking the time to do this.
This feature is already under consideration.
See the comments at: http://bugs.gnu.org/9500

> Benefits of preallocation:
>  - Disk fragmentation can be greatly reduced. That means faster file
> access and less filesystem overhead (fewer extents).
>  - Recovering data after filesystem corruption should be more successful,
> since files are more likely to be contiguous.
>  - If you're e.g. copying a virtual machine disk image file, the
> destination should be (almost) contiguous, meaning that running a disk
> optimiser/defragmenter in the guest OS would work as it should (i.e.
> improve performance).
>
> This is a very preliminary patch for testing. Hopefully someone will find
> it useful. And hopefully someone who (a) has a clue when it comes to C
> programming, and (b) is familiar with the coreutils source (I'm neither)
> can work from this to produce something which could be included in a
> future release.
>
> Note that posix_fallocate() sets the destination file size. If your system
> doesn't support fallocate() with FALLOC_FL_KEEP_SIZE, you can't e.g. do
> "ls -l destfilename" to monitor the progress of a large file copy; the
> length shown will always be the final length.
>
> Pre-allocating space can defeat the object of --sparse=always (or the
> default sparse-checking heuristic). If copying files with large holes you
> probably won't want to use --preallocate. If you do, regions in the
> destination corresponding to holes in the source will be allocated but
> unwritten. You'll lose the disk-space-saving benefit, but keep the
> fast-reading-of-holes benefit. On the other hand, that feature could be
> useful sometimes.
>
> In the general case of copying non-sparse files, it should be beneficial
> to use --preallocate. However on some systems, when the destination
> filesystem does not support pre-allocation (e.g. FAT32), the
> implementation of posix_fallocate() might try to fill the region to be
> pre-allocated with zeros. That would double copy time for no benefit.
>
> To-do list:
>  - Add --preallocate option to mv as well
>  - Should the option name be changed to --pre-allocate?
>  - Maybe have an option to tell cp to pre-allocate space for all
> destination files in one go, rather than pre-allocating space for each
> individual file before copying?

I don't think there should be an option at all.
cp should have enough info to do the right thing.
Why would you even not want to preallocate?
In saying that, using fallocate with XFS triggers
alignment behavior that causes fragmentation.
But this might change, and the user can't be expected to know this.
BTW I'm thinking of adding a new FALLOC_FL_ALIGN flag
to the kernel, that XFS can use in its tools to enable that
separate functionality.

>  - Check the error code that fallocate() returns. If it says the
> filesystem does not support fallocate(), don't call it again for every
> other file being copied.
>  - Better handling of sparse files, e.g. don't call fallocate() if source
> file is sparse and --sparse=always is given.

That's an important consideration.

>  - If pre-allocation fails due to insufficient disk space, cp prints a
> message and continues. So typically it will fill up the disk then abort
> with an out-of-disk-space error. It would be nice to be able to tell cp
> to abort when a pre-allocation fails, so it can exit without wasting
> time.

Yes it should exit immediately on ENOSPC

cheers,
P�draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 11 May 2012 19:20:02 +0000
Resent-Message-ID: <handler.9500.B9500.133676397826448 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Mark <markk@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.133676397826448
          (code B ref 9500); Fri, 11 May 2012 19:20:02 +0000
Received: (at 9500) by debbugs.gnu.org; 11 May 2012 19:19:38 +0000
Received: from localhost ([127.0.0.1]:49433 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1SSvNJ-0006sW-Ey
	for submit <at> debbugs.gnu.org; Fri, 11 May 2012 15:19:38 -0400
Received: from mail2.vodafone.ie ([213.233.128.44]:41392)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <P@HIDDEN>) id 1SSvN5-0006s5-5Q
	for 9500 <at> debbugs.gnu.org; Fri, 11 May 2012 15:19:36 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApMBAGZlrU9tTAC//2dsb2JhbAANN7Z+AQEBAwEyAUYFCwsNCwkWDwkDAgECAUUGDQEFAgEBiAW7PYsXhh0EnBSNE4FVGQ
Received: from unknown (HELO [192.168.1.79]) ([109.76.0.191])
	by mail2.vodafone.ie with ESMTP; 11 May 2012 20:19:10 +0100
Message-ID: <4FAD662D.5030306@HIDDEN>
Date: Fri, 11 May 2012 20:19:09 +0100
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <90cdbf76a2ca009ab20d719e55385197.squirrel@HIDDEN>
	<4FAD342A.3050503@HIDDEN>
	<d7a3877bd5e2895596533d7b10a3c9b1.squirrel@HIDDEN>
In-Reply-To: <d7a3877bd5e2895596533d7b10a3c9b1.squirrel@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -1.9 (-)

On 05/11/2012 06:40 PM, Mark wrote:
> Hi,
> 
> On Fri, May 11, 2012 16:45, P�draig Brady wrote:
>> On 05/11/2012 04:03 PM, Mark wrote:
>>> Here's a patch for cp which adds a new --preallocate option. When
>>> specified, cp allocates disk space for the destination file before
>>> ...
>>>
>>> To-do list:
>>>  - Add --preallocate option to mv as well
>>>  - Should the option name be changed to --pre-allocate?
>>>  - Maybe have an option to tell cp to pre-allocate space for all
>>> destination files in one go, rather than pre-allocating space for each
>>> individual file before copying?
>>
>> I don't think there should be an option at all.
>> cp should have enough info to do the right thing.
>> Why would you even not want to preallocate?
> 
> Apart from the case I mentioned where posix_fallocate() might write out
> zeros on some systems, I can't think of many cases where pre-allocating
> wouldn't be a good idea. So perhaps at some point in the future it would
> be the default, with a --no-preallocate option to disable.
> 
> 
>>>  - Better handling of sparse files, e.g. don't call fallocate() if
>>> source file is sparse and --sparse=always is given.
>>
>> That's an important consideration.
> 
> Even for sparse files, if cp can determine where the holes are beforehand
> (using SEEK_HOLE/SEEK_DATA or fiemap), it could pre-allocate the non-hole
> regions. So with more work, pre-allocation could be used with sparse
> files. Implementing that is beyond my abilities though...
> 
> It would still be helpful to allow pre-allocation to be disabled. (E.g.
> source file has many written/allocated all-zero regions, which the user
> wants to turn into holes in the destination file.)
> 
> Still another option: the ability to use FALLOC_FL_PUNCH_HOLE to punch
> holes in (pre-allocated) destination files. Then any all-zero non-hole
> regions in the source could be turned into holes in the destination in
> conjunction with --sparse=always. By pre-allocating, cp would at least
> guarantee the copy won't fail due to insufficient disk space, which may be
> useful in some cases.
> 
> 
>>>  - If pre-allocation fails due to insufficient disk space, cp prints a
>>> message and continues. So typically it will fill up the disk then abort
>>> with an out-of-disk-space error. It would be nice to be able to tell cp
>>> to abort when a pre-allocation fails, so it can exit without wasting
>>> time.
>>
>> Yes it should exit immediately on ENOSPC
> 
> Aborting by default would make sense. Ideally that would be configurable.
> In some cases the user might prefer to get a warning, so they can Ctrl-Z
> suspend cp and delete/move other files to make space, instead of having cp
> abort.
> 
> 
> An interesting aside: I tried using cp to pre-allocate space for a very
> large file on an ext4 partition, much larger than the amount of free
> space. IMHO it would be best for the filesystem to fail immediately in
> that case. ext4 does a lot of work (there was a lot of disk activity and
> it took a long time to fail). ext4 pre-allocates as much of the requested
> region as possible, rather than succeeding or failing all-or-nothing. So
> you get a disk-full condition. (Of course that's no worse than what
> happens when you run cp normally. But it would happen much more quickly
> with pre-allocation.)

Well that's bad as you get a delay in addition to the normal copy.
However, I don't see that behavior with 2.6.40.4-5.fc15.x86_64 at least?

$ df -Th .
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sdb1     ext4     97G   85G  7.1G  93% /home

$ for s in 1 10 100; do time fallocate -l ${s}G testfile; rm testfile; done
sys	0m0.121s
fallocate: testfile: fallocate failed: No space left on device
sys	0m0.345s
fallocate: testfile: fallocate failed: No space left on device
sys	0m0.001s

> You can try that by doing something like this on an ext4 partition:
> truncate -s 9999999999999 testfile   # (create a sparse file larger than
> free space)
> cp --preallocate testfile testfile.copy

Note I'mm CC'ing the bug address rather than the coreutils@HIDDEN address
to keep all of this thread associated with the bug.

cheers,
P�draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files
Resent-From: "Mark" <markk@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 11 May 2012 20:22:02 +0000
Resent-Message-ID: <handler.9500.B9500.133676769431805 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: "=?UTF-8?Q?P=C3=A1draig?= Brady" <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.133676769431805
          (code B ref 9500); Fri, 11 May 2012 20:22:02 +0000
Received: (at 9500) by debbugs.gnu.org; 11 May 2012 20:21:34 +0000
Received: from localhost ([127.0.0.1]:49462 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1SSwKx-0008Gd-NG
	for submit <at> debbugs.gnu.org; Fri, 11 May 2012 16:21:34 -0400
Received: from claranet-outbound-smtp06.uk.clara.net ([195.8.89.39]:41604)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <markk@HIDDEN>) id 1SSvmc-0007SR-LW
	for 9500 <at> debbugs.gnu.org; Fri, 11 May 2012 15:45:48 -0400
Received: from webmail04.mail.eu.clara.net ([213.253.3.104]:47570
	helo=webmail.clara.net)
	by relay06.mail.eu.clara.net (relay.clara.net [213.253.3.46]:1025)
	with esmtpa (authdaemon_plain:markk@HIDDEN) id 1SSvmO-0008RH-Ja 
	(return-path <markk@HIDDEN>); Fri, 11 May 2012 19:45:32 +0000
Received: from 2.27.247.8 (SquirrelMail authenticated user markk@HIDDEN)
	by ssl-webmail-vh.clara.net with HTTP;
	Fri, 11 May 2012 20:45:32 +0100
Message-ID: <2ce007600b32104cc3d7b64158ac7ba8.squirrel@HIDDEN>
In-Reply-To: <4FAD662D.5030306@HIDDEN>
References: <90cdbf76a2ca009ab20d719e55385197.squirrel@HIDDEN>
	<4FAD342A.3050503@HIDDEN>
	<d7a3877bd5e2895596533d7b10a3c9b1.squirrel@HIDDEN>
	<4FAD662D.5030306@HIDDEN>
Date: Fri, 11 May 2012 20:45:32 +0100
From: "Mark" <markk@HIDDEN>
User-Agent: SquirrelMail/1.4.21
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Score: -1.9 (-)
X-Mailman-Approved-At: Fri, 11 May 2012 16:21:14 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -1.9 (-)

On Fri, May 11, 2012 20:19, P�draig Brady wrote:
> On 05/11/2012 06:40 PM, Mark wrote:
>> ...
>> An interesting aside: I tried using cp to pre-allocate space for a very
>> large file on an ext4 partition, much larger than the amount of free
>> space. IMHO it would be best for the filesystem to fail immediately in
>> that case. ext4 does a lot of work (there was a lot of disk activity and
>> it took a long time to fail). ext4 pre-allocates as much of the
>> requested
>> region as possible, rather than succeeding or failing all-or-nothing. So
>> you get a disk-full condition. (Of course that's no worse than what
>> happens when you run cp normally. But it would happen much more quickly
>> with pre-allocation.)
>
> Well that's bad as you get a delay in addition to the normal copy.
> However, I don't see that behavior with 2.6.40.4-5.fc15.x86_64 at least?

I'm using kernel 3.0.0-19-generic #32-Ubuntu here. But probably more
relevant, the partition I tested on was ~1.7TB ext4 on an external USB 2.0
drive which was almost full and probably *very* fragmented, i.e. free
space spread all over the disk in thousands of small chunks. ext4 seems to
be pretty slow at allocating space in that case.

If I were designing a filesystem, I'd have it immediately return failure
if fallocate() is specified with additional size larger than the amount of
free space. Though for the filesystem, determining how much extra space a
fallocate() call would need can be quite involved in some cases and
require a significant amount of disk access...

Imagine a huge sparse file with many thousands of holes, and the requested
region for fallocate() serving to "fill in" many of the holes. But any
non-hole parts within the fallocate() region would reduce the amount of
additional space required for fallocate() to succeed. So it's not as
simple as comparing length of fallocate() region with amount of free
space...

Unless you're creating a new file, which is what cp does most of the time.
So maybe a workaround could be added to cp. If --preallocate is specified,
cp could check the amount of free space before writing to the destination
file and abort without even needing to call fallocate() if there isn't
enough. (In fact, cp could do that anyway in most cases I think?)


-- Mark

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 11 May 2012 20:38:02 +0000
Resent-Message-ID: <handler.9500.B9500.1336768625757 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: Mark <markk@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.1336768625757
          (code B ref 9500); Fri, 11 May 2012 20:38:02 +0000
Received: (at 9500) by debbugs.gnu.org; 11 May 2012 20:37:05 +0000
Received: from localhost ([127.0.0.1]:49471 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1SSwaG-0000CA-JC
	for submit <at> debbugs.gnu.org; Fri, 11 May 2012 16:37:04 -0400
Received: from mail2.vodafone.ie ([213.233.128.44]:29413)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <P@HIDDEN>) id 1SSwaE-0000Bf-HL
	for 9500 <at> debbugs.gnu.org; Fri, 11 May 2012 16:37:03 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApoBAAh3rU9tTAC//2dsb2JhbAANN7AWhmgBAQEDATIBRgULCw0LCRYPCQMCAQIBRQYNAQcBAYgFuyWLFxQFhgQEnBSNE4FWCA
Received: from unknown (HELO [192.168.1.79]) ([109.76.0.191])
	by mail2.vodafone.ie with ESMTP; 11 May 2012 21:36:49 +0100
Message-ID: <4FAD7860.7090008@HIDDEN>
Date: Fri, 11 May 2012 21:36:48 +0100
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:6.0) Gecko/20110816 Thunderbird/6.0
MIME-Version: 1.0
References: <90cdbf76a2ca009ab20d719e55385197.squirrel@HIDDEN>
	<4FAD342A.3050503@HIDDEN>
	<d7a3877bd5e2895596533d7b10a3c9b1.squirrel@HIDDEN>
	<4FAD662D.5030306@HIDDEN>
	<2ce007600b32104cc3d7b64158ac7ba8.squirrel@HIDDEN>
In-Reply-To: <2ce007600b32104cc3d7b64158ac7ba8.squirrel@HIDDEN>
X-Enigmail-Version: 1.3.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -1.9 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -1.9 (-)

On 05/11/2012 08:45 PM, Mark wrote:
> On Fri, May 11, 2012 20:19, P�draig Brady wrote:
>> On 05/11/2012 06:40 PM, Mark wrote:
>>> ...
>>> An interesting aside: I tried using cp to pre-allocate space for a very
>>> large file on an ext4 partition, much larger than the amount of free
>>> space. IMHO it would be best for the filesystem to fail immediately in
>>> that case. ext4 does a lot of work (there was a lot of disk activity and
>>> it took a long time to fail). ext4 pre-allocates as much of the
>>> requested
>>> region as possible, rather than succeeding or failing all-or-nothing. So
>>> you get a disk-full condition. (Of course that's no worse than what
>>> happens when you run cp normally. But it would happen much more quickly
>>> with pre-allocation.)
>>
>> Well that's bad as you get a delay in addition to the normal copy.
>> However, I don't see that behavior with 2.6.40.4-5.fc15.x86_64 at least?
> 
> I'm using kernel 3.0.0-19-generic #32-Ubuntu here. But probably more
> relevant, the partition I tested on was ~1.7TB ext4 on an external USB 2.0
> drive which was almost full and probably *very* fragmented, i.e. free
> space spread all over the disk in thousands of small chunks. ext4 seems to
> be pretty slow at allocating space in that case.

But you asked to fallocate(10GB).
That should have failed immediately,
because it's bigger than the file system

Could you run the fallocate loop from my previous mail,
across an appropriate range of sizes, just to confirm,
and maybe prepare input to a kernel bug report?

> If I were designing a filesystem, I'd have it immediately return failure
> if fallocate() is specified with additional size larger than the amount of
> free space. Though for the filesystem, determining how much extra space a
> fallocate() call would need can be quite involved in some cases and
> require a significant amount of disk access...
> 
> Imagine a huge sparse file with many thousands of holes, and the requested
> region for fallocate() serving to "fill in" many of the holes. But any
> non-hole parts within the fallocate() region would reduce the amount of
> additional space required for fallocate() to succeed. So it's not as
> simple as comparing length of fallocate() region with amount of free
> space...
> 
> Unless you're creating a new file, which is what cp does most of the time.
> So maybe a workaround could be added to cp. If --preallocate is specified,
> cp could check the amount of free space before writing to the destination
> file and abort without even needing to call fallocate() if there isn't
> enough. (In fact, cp could do that anyway in most cases I think?)

Good analysis. Still though for a new file the file system
should be able to to do the simple short calculation
of fallocate_request - free_space > 0.

I can understand inefficiencies in fallocating
around the free space limit, but otherwise
this seems like a bug in ext4.
(maybe a regression since I don't see it on my ext4 system).

cheers,
P�draig.

Message sent to bug-coreutils@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files
References: <4E7041CE.4090305@HIDDEN>
In-Reply-To: <4E7041CE.4090305@HIDDEN>
Resent-From: "Mark" <markk@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Tue, 15 May 2012 11:46:01 +0000
Resent-Message-ID: <handler.9500.B9500.133708231232556 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 9500
X-GNU-PR-Package: coreutils
X-GNU-PR-Keywords: patch
To: "=?UTF-8?Q?P=C3=A1draig?= Brady" <P@HIDDEN>
Cc: 9500 <at> debbugs.gnu.org
Received: via spool by 9500-submit <at> debbugs.gnu.org id=B9500.133708231232556
          (code B ref 9500); Tue, 15 May 2012 11:46:01 +0000
Received: (at 9500) by debbugs.gnu.org; 15 May 2012 11:45:12 +0000
Received: from localhost ([127.0.0.1]:55491 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1SUGBj-0008T2-6C
	for submit <at> debbugs.gnu.org; Tue, 15 May 2012 07:45:11 -0400
Received: from claranet-outbound-smtp03.uk.clara.net ([195.8.89.36]:35834)
	by debbugs.gnu.org with esmtp (Exim 4.72)
	(envelope-from <markk@HIDDEN>) id 1SUGBg-0008So-1w
	for 9500 <at> debbugs.gnu.org; Tue, 15 May 2012 07:45:09 -0400
Received: from webmail04.mail.eu.clara.net ([213.253.3.104]:34254
	helo=webmail.clara.net)
	by relay03.mail.eu.clara.net (relay.clara.net [213.253.3.43]:1025)
	with esmtpa (authdaemon_plain:markk@HIDDEN) id 1SUGBS-000355-Ax 
	(return-path <markk@HIDDEN>); Tue, 15 May 2012 11:44:54 +0000
Received: from 2.26.247.91 (SquirrelMail authenticated user markk@HIDDEN)
	by ssl-webmail-vh.clara.net with HTTP;
	Tue, 15 May 2012 12:44:54 +0100
Message-ID: <2391da8b6dc54895745706cd97c8963c.squirrel@HIDDEN>
Date: Tue, 15 May 2012 12:44:54 +0100
From: "Mark" <markk@HIDDEN>
User-Agent: SquirrelMail/1.4.21
MIME-Version: 1.0
Content-Type: multipart/mixed;boundary="----=_20120515124454_68119"
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Score: -1.9 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
	<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -1.9 (-)

------=_20120515124454_68119
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit

Hi,

[Resending message cc'ed to 9500 <at> debbugs.gnu.org as requested.]

On Fri, May 11, 2012 21:36, P�draig Brady wrote:
> On 05/11/2012 08:45 PM, Mark wrote:
>> ...
>> I'm using kernel 3.0.0-19-generic #32-Ubuntu here. But probably more
relevant, the partition I tested on was ~1.7TB ext4 on an external USB
2.0
>> drive which was almost full and probably *very* fragmented, i.e. free
space spread all over the disk in thousands of small chunks. ext4 seems
to be pretty slow at allocating space in that case.
> But you asked to fallocate(10GB).
> That should have failed immediately,
> because it's bigger than the file system
> Could you run the fallocate loop from my previous mail,
> across an appropriate range of sizes, just to confirm,
> and maybe prepare input to a kernel bug report?

Unfortunately I have freed up a lot of space on the partition in question,
but I did some testing with a new small empty test partition, see below.

>> If I were designing a filesystem, I'd have it immediately return
failure
>> if fallocate() is specified with additional size larger than the amount of
>> free space. Though for the filesystem, determining how much extra space a
>> fallocate() call would need can be quite involved in some cases and
require a significant amount of disk access...
>> Imagine a huge sparse file with many thousands of holes, and the requested
>> region for fallocate() serving to "fill in" many of the holes. But any
non-hole parts within the fallocate() region would reduce the amount of
additional space required for fallocate() to succeed. So it's not as
simple as comparing length of fallocate() region with amount of free
space...
>> Unless you're creating a new file, which is what cp does most of the time.
>> So maybe a workaround could be added to cp. If --preallocate is specified,
>> cp could check the amount of free space before writing to the
>> destination
>> file and abort without even needing to call fallocate() if there isn't
enough. (In fact, cp could do that anyway in most cases I think?)
>
> Good analysis. Still though for a new file the file system
> should be able to to do the simple short calculation
> of fallocate_request - free_space > 0.

Yes.

From the filesystem's perspective it could easily immediately fail
fallocate() in some cases.

When fallocate() is called, the approximate space needed for success is
  (length of region passed to fallocate) - (amount already allocated to
file which overlaps the fallocate region).

The worst case is that the entire overlap between the fallocate region and
the file is a hole. Then
  space needed = fallocate region size.

However the filesystem could narrow it down a little. Consider the
difference between a file's apparent size and its on-disk size. Roughly
  size of all holes = apparent size - on-disk size

The space needed will not be more than:
  (size of fallocate region past end of file) + min((size of fallocate
region overlapping file), size of all holes)

So the fallocate() call could be failed without doing any work if there is
less free space than that.

> I can understand inefficiencies in fallocating
> around the free space limit, but otherwise
> this seems like a bug in ext4.
> (maybe a regression since I don't see it on my ext4 system).

I also saw slowness (i.e. a lot of disk I/O) when writing a file to that
ext4 partition without pre-allocation. It seemed like once the file got
above a certain size, there was a *lot* of disk I/O for several seconds.
Or maybe all the disk I/O happened when the kernel writeback kicked in.
Perhaps that's related to the fallocate() slowness, if the filesystem had
to seek all over the disk again and again to find free space.

As I mentioned though, that filesystem was/is *very* fragmented, with a
relatively small proportion of free space. I might not be able to
reproduce the issue now, because I shifted about 250GB of data off that
partition the other day.

Also worth noting, most ext4 partitions have a certain percentage reserved
for the root user. Maybe testing should be done either on a partition with
no reserved space, or as root.

I posted to the ext4 list a while ago about fallocate() behaviour. Not
specifically about this issue, but the fact that it's not atomic. If it
fails due to lack of space, it allocates all space on the partition with
no way to easily undo that.

For example, suppose the user has a very large sparse file with large
holes, maybe a virtual machine hard disk image or similar. User wants to
make the file non-sparse so calls fallocate() to allocate all the holes.
But it turns out there wasn't enough disk space for that to succeed. Or
maybe some other program allocated a lot of space in the mean time.
fallocate() fails after allocating all remaining space on the partition.
Other than deleting the file, the user would need to roll their own
hole-punching program to reclaim space.

You can read the thread about that at
  http://comments.gmane.org/gmane.comp.file-systems.ext4/29942

Back to the brief test I mentioned above. See the attached file. That
doesn't demonstrate any slowness (since the test was done on a very small
empty partition), but does demonstrate ext4 fallocate() not returning an
error on attempting to allocate more than the partition size. Instead it
allocates all remaining space then fails, leaving the user to manually fix
things afterwards.

Regards,
-- Mark
------=_20120515124454_68119
Content-Type: text/plain; name="ext4_fallocate_test.txt"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment; filename="ext4_fallocate_test.txt"

Demonstration that using fallocate() to attempt to allocate more space than the size of the partition does not immediately fail, but instead allocates all free space then fails.

$ ddpt if=/dev/zero of=testimage.ext4 bs=1048576 count=128
128+0 records in
128+0 records out
time to transfer data: 1.885648 secs at 71.18 MB/sec

$ mkfs.ext4 testimage.ext4 
mke2fs 1.41.14 (22-Dec-2010)
testimage.ext4 is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
32768 inodes, 131072 blocks
6553 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
16 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks: 
	8193, 24577, 40961, 57345, 73729

Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

$ sudo mount -o noatime,loop -t ext4 testimage.ext4 /media/test

$ cd /media/test
ls -l
total 12
drwx------ 2 root root 12288 2012-05-14 12:52 lost+found

$ df | grep test
/dev/loop0              126931      5646    114732   5% /media/test

$ time fallocate -o0 -l999999999 testfile
fallocate: testfile: fallocate failed: No space left on device

real	0m0.002s
user	0m0.000s
sys	0m0.000s

$ df | grep test
/dev/loop0              126931    120378         0 100% /media/test

$ du -B 1 testfile 
117485568	testfile

$ ls -l
total 114744
drwx------ 2 root root     12288 2012-05-14 12:52 lost+found
-rw-r--r-- 1 mark mark 117484544 2012-05-14 12:54 testfile
------=_20120515124454_68119--

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 30 Oct 2018 04:20:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Oct 30 00:20:09 2018
Received: from localhost ([127.0.0.1]:52979 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gHLVh-0006Ya-DL
	for submit <at> debbugs.gnu.org; Tue, 30 Oct 2018 00:20:09 -0400
Received: from mail-pf1-f172.google.com ([209.85.210.172]:35791)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <assafgordon@HIDDEN>) id 1gHLVe-0006Xv-Jy
 for control <at> debbugs.gnu.org; Tue, 30 Oct 2018 00:20:06 -0400
Received: by mail-pf1-f172.google.com with SMTP id z2-v6so2542235pfe.2
 for <control <at> debbugs.gnu.org>; Mon, 29 Oct 2018 21:20:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=to:from:message-id:date:user-agent:mime-version:content-language
 :content-transfer-encoding;
 bh=46VpvEFntM+L6kLVZ6AMhWlyGueDjFlelsFQXHLchBw=;
 b=jI+E+YOtaCCXinN0/J550ZkVJJxAzVqxIDELmnwo+WhKj495buxXV31rOF36VJdCqs
 gcePXHQv+2a2BmQ7HHEhIi7f3Td/xjeRcLg0fMxxqudMpRYKVfIEceQsDH1JPnsH9bdK
 7/BZJmzeyu41qlKFrvikLKyo4TMPtEIMghfqiRF9BfaNaPMtnPJ7RuUpp7MXafHZJCsV
 KSuJgyr8QBGfqBQdrfS5bTwhN0wM2p5m9c4k2xGQlRJEXZnG1IucaFdk4L5N/+4vrRnG
 ewyt/iJHy+7fGjlR6hHIEyMZA8kecAuXSzXTL6XcW97Fo6w4LAaLmbSrl5KBte+Nc/HJ
 JHjA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:to:from:message-id:date:user-agent:mime-version
 :content-language:content-transfer-encoding;
 bh=46VpvEFntM+L6kLVZ6AMhWlyGueDjFlelsFQXHLchBw=;
 b=Gv/S2rYVmVGha3Ou9agUGxlsbSE6d/dSau1qXlYe4HNykNQoHwG6icxZ7RZ7pvksPk
 5YmveoWGi/0XpO+V8GPwRblNGT2VwSj8qdXmFliq4FzTq3dI4krWqzcFVVnPJBJ871X+
 ++LGWKvXp4xc7bApzd3rMPNwQeZsToa/SwYalXghzR9hwpfL3WBharuZbDLRazxvAxfg
 pxlSsEyN67E78Y5u2oLWWbEd52kKLUwZTrdV07bQChWihrBqZ1P1fTgjJc2ozvT+Vwod
 3RUVEF3++lYxpfr2XLD9PZE/EtEJs+DLEJ53z6T5LiHVsPXuQHLJrITQoWXxjlzCVwUu
 9NBA==
X-Gm-Message-State: AGRZ1gLSSVVF2zh8R/SCLA3Itc/F2hubW9i2Q+4BKyonfWg6X9d2b8PT
 LNqwEVH06mekJPIuwFxklMTI6lAhtvo=
X-Google-Smtp-Source: AJdET5dRiCjH/1g3m6Wu0ZGni8gg0kFV5WqMAGrCDIT4F7B3zVyfmabDjBK8EkXK33sc7zkxxYt4RA==
X-Received: by 2002:a62:90db:: with SMTP id
 q88-v6mr1303036pfk.98.1540873199992; 
 Mon, 29 Oct 2018 21:19:59 -0700 (PDT)
Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38])
 by smtp.googlemail.com with ESMTPSA id
 a64-v6sm26691657pfe.32.2018.10.29.21.19.58
 for <control <at> debbugs.gnu.org>
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Mon, 29 Oct 2018 21:19:58 -0700 (PDT)
To: control <at> debbugs.gnu.org
From: Assaf Gordon <assafgordon@HIDDEN>
Message-ID: <eb037728-13f6-2284-118a-3ffd8646de0c@HIDDEN>
Date: Mon, 29 Oct 2018 22:19:57 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 2.0 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: severity 9500 wishlist retitle 9500 cp: use posix_fallocate
 where supported [...] 
 Content analysis details:   (2.0 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, no
 trust [209.85.210.172 listed in list.dnswl.org]
 -0.0 SPF_PASS               SPF: sender matches SPF record
 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
 (assafgordon[at]gmail.com)
 1.8 MISSING_SUBJECT        Missing Subject: header
 0.2 NO_SUBJECT             Extra score for no subject
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

severity 9500 wishlist
retitle 9500 cp: use posix_fallocate where supported

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 30 Oct 2018 04:20:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Oct 30 00:20:09 2018
Received: from localhost ([127.0.0.1]:52979 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gHLVh-0006Ya-DL
	for submit <at> debbugs.gnu.org; Tue, 30 Oct 2018 00:20:09 -0400
Received: from mail-pf1-f172.google.com ([209.85.210.172]:35791)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <assafgordon@HIDDEN>) id 1gHLVe-0006Xv-Jy
 for control <at> debbugs.gnu.org; Tue, 30 Oct 2018 00:20:06 -0400
Received: by mail-pf1-f172.google.com with SMTP id z2-v6so2542235pfe.2
 for <control <at> debbugs.gnu.org>; Mon, 29 Oct 2018 21:20:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=to:from:message-id:date:user-agent:mime-version:content-language
 :content-transfer-encoding;
 bh=46VpvEFntM+L6kLVZ6AMhWlyGueDjFlelsFQXHLchBw=;
 b=jI+E+YOtaCCXinN0/J550ZkVJJxAzVqxIDELmnwo+WhKj495buxXV31rOF36VJdCqs
 gcePXHQv+2a2BmQ7HHEhIi7f3Td/xjeRcLg0fMxxqudMpRYKVfIEceQsDH1JPnsH9bdK
 7/BZJmzeyu41qlKFrvikLKyo4TMPtEIMghfqiRF9BfaNaPMtnPJ7RuUpp7MXafHZJCsV
 KSuJgyr8QBGfqBQdrfS5bTwhN0wM2p5m9c4k2xGQlRJEXZnG1IucaFdk4L5N/+4vrRnG
 ewyt/iJHy+7fGjlR6hHIEyMZA8kecAuXSzXTL6XcW97Fo6w4LAaLmbSrl5KBte+Nc/HJ
 JHjA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:to:from:message-id:date:user-agent:mime-version
 :content-language:content-transfer-encoding;
 bh=46VpvEFntM+L6kLVZ6AMhWlyGueDjFlelsFQXHLchBw=;
 b=Gv/S2rYVmVGha3Ou9agUGxlsbSE6d/dSau1qXlYe4HNykNQoHwG6icxZ7RZ7pvksPk
 5YmveoWGi/0XpO+V8GPwRblNGT2VwSj8qdXmFliq4FzTq3dI4krWqzcFVVnPJBJ871X+
 ++LGWKvXp4xc7bApzd3rMPNwQeZsToa/SwYalXghzR9hwpfL3WBharuZbDLRazxvAxfg
 pxlSsEyN67E78Y5u2oLWWbEd52kKLUwZTrdV07bQChWihrBqZ1P1fTgjJc2ozvT+Vwod
 3RUVEF3++lYxpfr2XLD9PZE/EtEJs+DLEJ53z6T5LiHVsPXuQHLJrITQoWXxjlzCVwUu
 9NBA==
X-Gm-Message-State: AGRZ1gLSSVVF2zh8R/SCLA3Itc/F2hubW9i2Q+4BKyonfWg6X9d2b8PT
 LNqwEVH06mekJPIuwFxklMTI6lAhtvo=
X-Google-Smtp-Source: AJdET5dRiCjH/1g3m6Wu0ZGni8gg0kFV5WqMAGrCDIT4F7B3zVyfmabDjBK8EkXK33sc7zkxxYt4RA==
X-Received: by 2002:a62:90db:: with SMTP id
 q88-v6mr1303036pfk.98.1540873199992; 
 Mon, 29 Oct 2018 21:19:59 -0700 (PDT)
Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38])
 by smtp.googlemail.com with ESMTPSA id
 a64-v6sm26691657pfe.32.2018.10.29.21.19.58
 for <control <at> debbugs.gnu.org>
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Mon, 29 Oct 2018 21:19:58 -0700 (PDT)
To: control <at> debbugs.gnu.org
From: Assaf Gordon <assafgordon@HIDDEN>
Message-ID: <eb037728-13f6-2284-118a-3ffd8646de0c@HIDDEN>
Date: Mon, 29 Oct 2018 22:19:57 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 2.0 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: severity 9500 wishlist retitle 9500 cp: use posix_fallocate
 where supported [...] 
 Content analysis details:   (2.0 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, no
 trust [209.85.210.172 listed in list.dnswl.org]
 -0.0 SPF_PASS               SPF: sender matches SPF record
 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
 (assafgordon[at]gmail.com)
 1.8 MISSING_SUBJECT        Missing Subject: header
 0.2 NO_SUBJECT             Extra score for no subject
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

severity 9500 wishlist
retitle 9500 cp: use posix_fallocate where supported

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.