GNU bug report logs - #28528
Split command problems.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28528 in the body.
You can then email your comments to 28528 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Nick Farrow <farrow.nick <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: Split command problems.
Date: Wed, 20 Sep 2017 02:04:36 -0500

[Message part 1 (text/plain, inline)]

When I use the coreutils split command on a file. I get the parts with no
problem. But when trying the same exact command on the same exact file on a
different server the hashes checks of the parts vs the other server don’t
match. Is there a way to process a file the exact same way despite the OS?


If there is a way to split the files exactly the same then multisource
file-transfer could be as easy as using the scp command.


split --version

split (GNU coreutils) 8.23



-- 
Thanks,
Time is Under your Feet!
-Nick Farrow

[Message part 2 (text/html, inline)]

[server2.txt (text/plain, attachment)]

[server1.txt (text/plain, attachment)]

Message #8 received at 28528 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Nick Farrow <farrow.nick <at> gmail.com>, 28528 <at> debbugs.gnu.org
Subject: Re: bug#28528: Split command problems.
Date: Wed, 20 Sep 2017 10:42:47 -0600

Hello,

On 2017-09-20 01:04 AM, Nick Farrow wrote:
> When I use the coreutils split command on a file. I get the parts with no
> problem. But when trying the same exact command on the same exact file on a
> different server the hashes checks of the parts vs the other server don’t
> match. Is there a way to process a file the exact same way despite the OS?

To understand the issue better,

can you share the exact command you have used on both systems,
and the OS/system details (e.g. running 'uname -a' if using Unix type
machines) ?

Is it the same version of 'split' (8.23) on both machines?

If one of the OSes is windows, please tell us the OS version,
and what is the origin of the coreutils programs (e.g. cygwin, GNuWin32,
etc.).

regards,
 - assaf

Message #11 received at 28528 <at> debbugs.gnu.org (full text, mbox):

From: Nick Farrow <farrow.nick <at> gmail.com>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 28528 <at> debbugs.gnu.org
Subject: Re: bug#28528: Split command problems.
Date: Fri, 29 Sep 2017 05:11:06 -0500

[Message part 1 (text/plain, inline)]

Assaf,
Sorry for the late reply.  I have tried using a few of the spit command
still with the same hash differences.
split -l 82 -d largefile largefile.
split -b 3276800 -d largefile largefile.

ServerA(pi3)
raspberrypi:~ $ uname -a
Linux raspberrypi 4.4.50-v7+ #970 SMP Mon Feb 20 19:18:29 GMT 2017 armv7l
GNU/Linux

split --version
split (GNU coreutils) 8.23




ServerB (pi2)
:~$ uname -a
Linux osmc-pi2 4.4.27-9-osmc #1 SMP PREEMPT Tue Mar 14 20:54:19 UTC 2017
armv7l GNU/Linux

split --version
split (GNU coreutils) 8.23





On Wed, Sep 20, 2017 at 11:42 AM, Assaf Gordon <assafgordon <at> gmail.com>
wrote:

> Hello,
>
> On 2017-09-20 01:04 AM, Nick Farrow wrote:
> > When I use the coreutils split command on a file. I get the parts with no
> > problem. But when trying the same exact command on the same exact file
> on a
> > different server the hashes checks of the parts vs the other server don’t
> > match. Is there a way to process a file the exact same way despite the
> OS?
>
> To understand the issue better,
>
> can you share the exact command you have used on both systems,
> and the OS/system details (e.g. running 'uname -a' if using Unix type
> machines) ?
>
> Is it the same version of 'split' (8.23) on both machines?
>
> If one of the OSes is windows, please tell us the OS version,
> and what is the origin of the coreutils programs (e.g. cygwin, GNuWin32,
> etc.).
>
> regards,
>  - assaf
>
>


-- 

Time is Under your Feet!

-Nick Farrow
<https://www.facebook.com/VeggieVampire360/>
<http://www.twitter.com/farrownick>
<https://plus.google.com/u/0/+NickFarrow/posts/p/pub>
<http://instagram.com/veggievampire>  <https://twitter.com/farrownick>
<https://www.twitch.tv/veggievampire360>  <http://n-i-c-k.com>
<https://github.com/veggievampire>  <http://www.thingiverse.com/Nfarrow>
<https://app.box.com/s/sk0hevygmq21yv8cmawskyb7x2o4aabg>
<http://www.instructables.com/member/nfarrow>
<https://twitter.com/farrownick>

[Message part 2 (text/html, inline)]

Message #14 received at 28528 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Nick Farrow <farrow.nick <at> gmail.com>
Cc: 28528 <at> debbugs.gnu.org
Subject: Re: bug#28528: Split command problems.
Date: Sat, 30 Sep 2017 14:30:56 -0600

Hello,

On 2017-09-29 04:11 AM, Nick Farrow wrote:
> still with the same hash differences. 
>
> split -b 3276800 -d largefile largefile.

I'm unable to reproduce this issue.

I've tried the above command on two different machines,
one of which is an RPi3 similar to yours (armv7l with split version
8.23), the other an amd64 machine with split version 8.25 .

For completeness, these are the commands I've used:

    $ dd if=/dev/urandom of=largefile bs=1M count=100
    $ sha1sum largefile
    087fd7665a93875e69af2b9b2fd04aed6681b944  largefile

    $ split -b 3276800 -d largefile largefile.
    $ sha1sum largefile.* > hash-amd64

    [ copy 'largefile' to the RPi3 machine, snf run there: ]

    $ sha1sum largefile
    087fd7665a93875e69af2b9b2fd04aed6681b944  largefile

    $ split -b 3276800 -d largefile largefile.
    $ sha1sum largefile.* > hash-rpi3

The resulting two hash files (hash-amd64 and hash-rpi3) are identical in
my case - indicating the generated split files are identical on both
machines with using "split -b".

As an additional verification, concatenating the split files should
result in the same sha1 checksum as the original file, on both machines
(this should be true even if the split used differnt "-b X" value)

    $ cat largefile.* | sha1sum
    087fd7665a93875e69af2b9b2fd04aed6681b944  -


If you observe different result, please try the above commands
and let us know which ones differ (e.g. is the checksum of the input
file the same on both machines, is the checksum of the concatenated
files the same on both machines).


regards,
 - assaf

Message #17 received at 28528 <at> debbugs.gnu.org (full text, mbox):

From: Nick Farrow <farrow.nick <at> gmail.com>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 28528 <at> debbugs.gnu.org
Subject: Re: bug#28528: Split command problems.
Date: Sun, 1 Oct 2017 19:09:35 -0500

[Message part 1 (text/plain, inline)]

Assaf,

Thank you so much. when I was doing your steps I changed a few things on my
end to get the sha1sum of each part and I ended up removing the tar command
to speed it up. And from what it looks like the compression was the
problem. Each server was compressing differently compared to the other.



Again thanks for the help,
Nick Farrow

PS: Do you want my script to have 2:1 ratio for scp transfers? It 1/2s your
download time if you have the bandwidth. It's kind like p2p but private and
perfect for local transfers.





ServerA
>sha1sum largefile
8545610c1425ea5f3ceab133632d49201b79236c  largefile
>split -b 3276800 -d largefile largefile.

largefile.00    2c426533291c350c7856cccdcbfb2088ea42a46a
largefile.01    1adee73a813c1ff5f75630de428d9d45916a1c13
largefile.02    cfff15d4199116d83fe888ce5cbc21c69adbd32f
largefile.03    861f94e889305d8d44a842c57003b4a0fceb2124
largefile.04    b08107007cb606a18bacece69b6ffb92b8ce71ab
largefile.05    395d7f027509b6ee262794a002b7d05cb301a287
largefile.06    2d9caae1eebbf8f80b7086f823a8efc50d904b98
largefile.07    5348b67281358d4742b4158606558d6b720186ea
largefile.08    8c2a4b32689779d195f93a39450506eaeab5dafc
largefile.09    b7e93d5d66f3bbf6981182f5fac2973c764e7152
largefile.10    e860f777ed90ccc12105aae81209aede5dda77e2
largefile.11    b1e15b1f1e4c0bf4ed800c6b68dfef5c2edf106a
largefile.12    7cded7436ccaeae421b50ef71e631e5961a514b5
largefile.13    38be5ae048d4e4b27f45ea48244e658de3715278
largefile.14    b2c1cc1bd10940e68c8f3714a57ebbd46e65278d
largefile.15    60fe06eb42529ec3b9c7c91e33b5bc0c3afd720c
largefile.16    dfe1fa527c4569f18dc71ea519f50a207344d44e
largefile.17    d45ee86978f62625029d65d10ba18c5bd054b43e
largefile.18    62ddfad8e778bde1a238ae27d84ef15d75a575d4
largefile.19    e97a07e22bd17a595d02a304ccb0e90fe8bdb0bb
largefile.20    27046e32db4804fbc614bc7863bfaf9edeb4eee8
largefile.21    5f84d619eef796df02d985df14fd7878cb8ec424
largefile.22    97fae277f962aa3a78d0f0e7a81c3ed0501e3b99
largefile.23    d2a547e2e40a7d671712600ba41970e72d9b1bc4
largefile.24    b5e70678a1956f7320cba196df4a7dbe24f2310b
largefile.25    89bc004ebf93a6060fe4ec67318b26073e910ec7
largefile.26    29da870b4a6ecb00be4d7576b7d1973951eba29d
largefile.27    311f7b5621bfa4a916df16a2a84b43ddee5a945c
largefile.28    cee7e1157d53899c25ac268d890ce06f0a56fa3c
largefile.29    9a48931c73cf7d32643050ea3301215ec3157a97
largefile.30    4ca76628bdbb902169444008295e086e0ded7d86
largefile.31    d1c99a0dd4562a726caa48014416f6759ff8cde9
largefile.32    e4234b62b771dd4c0eaf49bd234f184b337752ec
largefile.33    80fb13f5c8972dad4b6178df1ac7c7e26606c822
largefile.34    707d8e6231e3585cff9c8b5e9cfaf18ef5b892f8
largefile.35    45249553f59bc9f5dc61c5be18c3e1ccc16c1834
largefile.36    a65131beee7af720a519ec4bbae9403bf4ead2b0
largefile.37    0506c2300b48d60b0c73fa3d83904cdfdbb7d60a
largefile.38    2e69f1fd7ec7abdabab25fc0a8e0876ce34d8363
largefile.39    c4bb4925feceab729e5dee5ffeb23a692ffe52f5
largefile.40    83fb19e14aacbfd479cb2c39f04b1f38fe54d9a2
largefile.41    a77ac2dd69b9eeb60f464339621fef30d4b040bf
largefile.42    2a4820a2d32a1f2f8ebc9f53f2f1c599ce7db473
largefile.43    5f3dbeeae98264f31219113d71446322894efcfb
largefile.44    918f4aef3a551f03eb790ba16c7ce089bd1feef7
largefile.45    80de63dba780a80f89bc48a6460b3583bd5aa459
largefile.46    3f228d01cf4bfee914a9eb3ebe21410ffb5ca20f
largefile.47    697b305ffdf4608a64db60cfc6b900bf81cb21ee
largefile.48    578794ab84dda28742fedea1ccad48dd93dd9b39
largefile.49    910261f8616a77681adabf123875a04a94dc4805
largefile.50    f9f51501e90f973b2d280a07f43340b6c0389944
largefile.51    02aa6dbc601d9f1df5963b2dabcf9bb72f635388
largefile.52    a726e4af8eda331269805e0fc8b6538cd4d5f4e2
largefile.53    7c17502abc5534620bbd4239307c39b3893cf13e
largefile.54    2038e75452233a29badc4d4ae1134feaf7b159df
largefile.55    b73bc8f70076bf23556c92556f6d3e3cb17e2a45
largefile.56    c3e71980ef9727986df53cd182c95ff55c5a20ac
largefile.57    d3351ab71f06c8af1d7ac6d187e5988776a35c37
largefile.58    f18929942d24e436d25833c4ae959298f9bffa58
largefile.59    644be61bba313094088ef519c1d336dfb8d4fe0f
largefile.60    2abe93596de4d62fc4056ed9531c98ad4cf601ad
largefile.61    cc4f91b521f16028fb83ea9ebecf1b4c8f70b737
largefile.62    a38497e9c318382f82ff98fca1c785d56b0add48
largefile.63    591179342ed5c9a3e5dfab5d0c3ecb2744869a44
largefile.64    6ebbfc5e6e7d5db7e6d4bea38d0aa824300b0e17
largefile.65    26b4e38911c35d529ac551b4e01e3e9fc6a67d09
largefile.66    b616df3e578c69ed1723e82778248a9f39aeb517
largefile.67    a6bff2b4e4625b77ed4ab146bb492ca417905874



ServerB
>sha1sum largefile
8545610c1425ea5f3ceab133632d49201b79236c  largefile
>split -b 3276800 -d largefile largefile.

largefile.00    2c426533291c350c7856cccdcbfb2088ea42a46a
largefile.01    1adee73a813c1ff5f75630de428d9d45916a1c13
largefile.02    cfff15d4199116d83fe888ce5cbc21c69adbd32f
largefile.03    861f94e889305d8d44a842c57003b4a0fceb2124
largefile.04    b08107007cb606a18bacece69b6ffb92b8ce71ab
largefile.05    395d7f027509b6ee262794a002b7d05cb301a287
largefile.06    2d9caae1eebbf8f80b7086f823a8efc50d904b98
largefile.07    5348b67281358d4742b4158606558d6b720186ea
largefile.08    8c2a4b32689779d195f93a39450506eaeab5dafc
largefile.09    b7e93d5d66f3bbf6981182f5fac2973c764e7152
largefile.10    e860f777ed90ccc12105aae81209aede5dda77e2
largefile.11    b1e15b1f1e4c0bf4ed800c6b68dfef5c2edf106a
largefile.12    7cded7436ccaeae421b50ef71e631e5961a514b5
largefile.13    38be5ae048d4e4b27f45ea48244e658de3715278
largefile.14    b2c1cc1bd10940e68c8f3714a57ebbd46e65278d
largefile.15    60fe06eb42529ec3b9c7c91e33b5bc0c3afd720c
largefile.16    dfe1fa527c4569f18dc71ea519f50a207344d44e
largefile.17    d45ee86978f62625029d65d10ba18c5bd054b43e
largefile.18    62ddfad8e778bde1a238ae27d84ef15d75a575d4
largefile.19    e97a07e22bd17a595d02a304ccb0e90fe8bdb0bb
largefile.20    27046e32db4804fbc614bc7863bfaf9edeb4eee8
largefile.21    5f84d619eef796df02d985df14fd7878cb8ec424
largefile.22    97fae277f962aa3a78d0f0e7a81c3ed0501e3b99
largefile.23    d2a547e2e40a7d671712600ba41970e72d9b1bc4
largefile.24    b5e70678a1956f7320cba196df4a7dbe24f2310b
largefile.25    89bc004ebf93a6060fe4ec67318b26073e910ec7
largefile.26    29da870b4a6ecb00be4d7576b7d1973951eba29d
largefile.27    311f7b5621bfa4a916df16a2a84b43ddee5a945c
largefile.28    cee7e1157d53899c25ac268d890ce06f0a56fa3c
largefile.29    9a48931c73cf7d32643050ea3301215ec3157a97
largefile.30    4ca76628bdbb902169444008295e086e0ded7d86
largefile.31    d1c99a0dd4562a726caa48014416f6759ff8cde9
largefile.32    e4234b62b771dd4c0eaf49bd234f184b337752ec
largefile.33    80fb13f5c8972dad4b6178df1ac7c7e26606c822
largefile.34    707d8e6231e3585cff9c8b5e9cfaf18ef5b892f8
largefile.35    45249553f59bc9f5dc61c5be18c3e1ccc16c1834
largefile.36    a65131beee7af720a519ec4bbae9403bf4ead2b0
largefile.37    0506c2300b48d60b0c73fa3d83904cdfdbb7d60a
largefile.38    2e69f1fd7ec7abdabab25fc0a8e0876ce34d8363
largefile.39    c4bb4925feceab729e5dee5ffeb23a692ffe52f5
largefile.40    83fb19e14aacbfd479cb2c39f04b1f38fe54d9a2
largefile.41    a77ac2dd69b9eeb60f464339621fef30d4b040bf
largefile.42    2a4820a2d32a1f2f8ebc9f53f2f1c599ce7db473
largefile.43    5f3dbeeae98264f31219113d71446322894efcfb
largefile.44    918f4aef3a551f03eb790ba16c7ce089bd1feef7
largefile.45    80de63dba780a80f89bc48a6460b3583bd5aa459
largefile.46    3f228d01cf4bfee914a9eb3ebe21410ffb5ca20f
largefile.47    697b305ffdf4608a64db60cfc6b900bf81cb21ee
largefile.48    578794ab84dda28742fedea1ccad48dd93dd9b39
largefile.49    910261f8616a77681adabf123875a04a94dc4805
largefile.50    f9f51501e90f973b2d280a07f43340b6c0389944
largefile.51    02aa6dbc601d9f1df5963b2dabcf9bb72f635388
largefile.52    a726e4af8eda331269805e0fc8b6538cd4d5f4e2
largefile.53    7c17502abc5534620bbd4239307c39b3893cf13e
largefile.54    2038e75452233a29badc4d4ae1134feaf7b159df
largefile.55    b73bc8f70076bf23556c92556f6d3e3cb17e2a45
largefile.56    c3e71980ef9727986df53cd182c95ff55c5a20ac
largefile.57    d3351ab71f06c8af1d7ac6d187e5988776a35c37
largefile.58    f18929942d24e436d25833c4ae959298f9bffa58
largefile.59    644be61bba313094088ef519c1d336dfb8d4fe0f
largefile.60    2abe93596de4d62fc4056ed9531c98ad4cf601ad
largefile.61    cc4f91b521f16028fb83ea9ebecf1b4c8f70b737
largefile.62    a38497e9c318382f82ff98fca1c785d56b0add48
largefile.63    591179342ed5c9a3e5dfab5d0c3ecb2744869a44
largefile.64    6ebbfc5e6e7d5db7e6d4bea38d0aa824300b0e17
largefile.65    26b4e38911c35d529ac551b4e01e3e9fc6a67d09
largefile.66    b616df3e578c69ed1723e82778248a9f39aeb517
largefile.67    a6bff2b4e4625b77ed4ab146bb492ca417905874

On Sat, Sep 30, 2017 at 3:30 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:

> Hello,
>
> On 2017-09-29 04:11 AM, Nick Farrow wrote:
> > still with the same hash differences.
> >
> > split -b 3276800 -d largefile largefile.
>
> I'm unable to reproduce this issue.
>
> I've tried the above command on two different machines,
> one of which is an RPi3 similar to yours (armv7l with split version
> 8.23), the other an amd64 machine with split version 8.25 .
>
> For completeness, these are the commands I've used:
>
>     $ dd if=/dev/urandom of=largefile bs=1M count=100
>     $ sha1sum largefile
>     087fd7665a93875e69af2b9b2fd04aed6681b944  largefile
>
>     $ split -b 3276800 -d largefile largefile.
>     $ sha1sum largefile.* > hash-amd64
>
>     [ copy 'largefile' to the RPi3 machine, snf run there: ]
>
>     $ sha1sum largefile
>     087fd7665a93875e69af2b9b2fd04aed6681b944  largefile
>
>     $ split -b 3276800 -d largefile largefile.
>     $ sha1sum largefile.* > hash-rpi3
>
> The resulting two hash files (hash-amd64 and hash-rpi3) are identical in
> my case - indicating the generated split files are identical on both
> machines with using "split -b".
>
> As an additional verification, concatenating the split files should
> result in the same sha1 checksum as the original file, on both machines
> (this should be true even if the split used differnt "-b X" value)
>
>     $ cat largefile.* | sha1sum
>     087fd7665a93875e69af2b9b2fd04aed6681b944  -
>
>
> If you observe different result, please try the above commands
> and let us know which ones differ (e.g. is the checksum of the input
> file the same on both machines, is the checksum of the concatenated
> files the same on both machines).
>
>
> regards,
>  - assaf
>



-- 

Time is Under your Feet!

-Nick Farrow
<https://www.facebook.com/VeggieVampire360/>
<http://www.twitter.com/farrownick>
<https://plus.google.com/u/0/+NickFarrow/posts/p/pub>
<http://instagram.com/veggievampire>  <https://twitter.com/farrownick>
<https://www.twitch.tv/veggievampire360>  <http://n-i-c-k.com>
<https://github.com/veggievampire>  <http://www.thingiverse.com/Nfarrow>
<https://app.box.com/s/sk0hevygmq21yv8cmawskyb7x2o4aabg>
<http://www.instructables.com/member/nfarrow>
<https://twitter.com/farrownick>

[Message part 2 (text/html, inline)]

Message #20 received at 28528 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 28528 <at> debbugs.gnu.org
Subject: Re: bug#28528: Split command problems.
Date: Mon, 29 Oct 2018 19:25:53 -0600

tags 28528 notabug
close 28528
stop

(triaging old bugs)

On 2017-10-01 6:09 p.m., Nick Farrow wrote:
> [...] And from what it looks like the compression was 
> the problem. Each server was compressing differently compared to the other.

Given the above, I'm closing this bug.

-assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.