3

In order to verify the integrity of some files I have stored on an external disk (EXT4) connected to my Pi Model B using Rasbian Jessie (latest), I use the following:

openssl sha1 file.tar

Which generates a result such as:

SHA1(file.tar)= 1391314ca210b8034342330faac51298fad24a24

This works successfully for Raspbian Stretch only on files that are less than 2GB in size. On files larger than 2GB in size I receive the following error: 

Value too large for defined data type

After doing some research, I found this information on the error.

This appears to be an issue with all Debian Stretch versions for both variants of Raspbian OS ( both +Desktop and Lite) but is not an issue on Raspbian Jessie. I have tested the following fresh installs on my Rasberry Pi Model B for both +Desktop and Lite distributions:

2017-04-10-raspbian-jessie

2017-07-05-raspbian-jessie => last Jessie version

2017-08-10-raspbian-stretch => first Stretch version

2018-03-13-raspbian-stretch => latest Stretch version

On the two Stretch versions above, I receive the "Value too large ..." error when running the openssl sha1 command on files larger than 2GB. I receive no error when running the openssl sha1 command on the same files under the Jessie versions. It generates the SHA-1 hashes successfully for files larger than 2GB.

Additionally, I've performed the same test on a separate machine with an installation of Debian netinst (https://www.debian.org/CD/netinst/) currently based on Stretch. This contains a minimal install of Debian and that also successfully works using the openssl sha1 command on >2GB files.

This leads me to believe that the GNU utilities such as openssl (and probably others) have not been compiled with large file support since the first release of Raspbian Stretch.

Can anyone think of a reason why this might not have been done? Has anyone experiences similar issues perhaps with other GNU utilities?

Improve this question
4
  • Hash function SHA-1 is insecure and outdated (from 1995). To check file integrity I recommend to use SHA-2 (i.e. sha512sum file.tar > file.tar.shasum). en.wikipedia.org/wiki/SHA-1 | en.wikipedia.org/wiki/Sha512sum
    – Fabian
    Apr 4, 2018 at 12:08
  • @SDsolar I don't think Arduino SE Tour is of any relevance to the OP.
    – Dmitry Grigoryev
    Apr 5, 2018 at 9:04
  • When you say you tested on Debian what architecture of Debian did you test with? i386? amd64? armht? Also note that openssl is not a gnu utility.
    – Peter Green
    Sep 17, 2018 at 17:47
  • It was tested on both i386 and amd64 architecture. Two different laptops with different CPUs, running the same Linux distribution (the distrivutors compile for both), the variant native to each architecture.
    – jimjamz
    Dec 30, 2018 at 15:57

4 Answers 4

Reset to default
3

On 32-bit Linux systems there are two sets of filesystem syscalls, the old 32-bit ones which for compatibility are still used unless a program requests otherwise and the new 64 bit ones. That error is what you get when you try to open a large file with the old open syscall.

What is puzzling me is why this bug seems to happen on arm (both debian armhf and raspbian) but not i386. I have done some grepping of the source tree, but nothing particular has stood out. It also seems to have been fixed in buster, but I have not managed to pin down exactly how/when it was fixed.

I think given this is fixed in buster and given that i suspect most people who have run into it have worked around it by now I don't think it's worth expending huge efforts trying to track it down further. If someone comes back with a targetted fix I'd certainly consider it though.

Improve this answer
2
  • It is likely at link time so if you are scouring source check the Makefiles :)
    – crasic
    Apr 12, 2019 at 3:58
  • nah, large file support is a compile time thing, not a link time thing.
    – Peter Green
    Sep 21, 2019 at 15:05
0

Well, 2GB is the limit of a signed 32-bit int, so you might well be right. Why is it signed? That's a harder question – it could be just defensive programming against the chance that somebody used an int where they should have used in unsigned int. Or, it could be that the math involved needs negative numbers.

As to why it hasn't been compiled with large file support, that doesn't make sense to me...

Improve this answer
0

I can only guess this was done to ensure compatibility with older software which wouldn't work correctly with LFS. A lot of software uses OpenSSL to manipulate small chunks of data (certificates / message digests / e-mail signatures), so it may make sense to prefer compatibility over large file support.

One practical workaround to this limitation is input redirection:

openssl sha1 < file.tar

Another workaround is sha1sum, which supports LFS:

sha1sum file.tar
Improve this answer
0

The problem is even on 64 bit systems, commands like bzip2 asd sha1 have not been compiled with big file support. You could have a million bit operating system, and without that compiler flag having been set when the utility is built, you'll hit an error on large files.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.