MD5 vs SHA-1: Different Results?

TL;DR

Yes, a file can have the same MD5SUM calculated by different tools but produce different SHA-1 hashes. This is usually due to differences in how line endings (carriage returns and newlines) are handled, especially when dealing with files created on different operating systems. It’s rare for critical security applications, but important to be aware of.

Why MD5SUMs Can Match But SHA-1 Hashes Differ

MD5 and SHA-1 are both cryptographic hash functions used to create a ‘fingerprint’ of a file. However, they aren’t perfect clones. They react differently to subtle changes in the input data.

Step-by-step Guide: Investigating Hash Differences

1. Understand Line Endings: The main culprit is often how operating systems represent the end of a line in text files.
   • Windows: Uses Carriage Return + Line Feed (CRLF – rn)
   • macOS/Linux: Uses just Line Feed (LF – n)
2. Create a Test File: Create a simple text file on both Windows and macOS/Linux. For example, create a file named ‘test.txt’ with the content “This is a test.”.
3. Calculate MD5SUMs: Use command-line tools to calculate the MD5SUM of the files.
   • Windows (PowerShell):

     Get-FileHash test.txt -Algorithm MD5

   • macOS/Linux:

     md5sum test.txt

   You’ll likely find the MD5SUMs are identical.

4. Calculate SHA-1 Hashes: Now calculate the SHA-1 hashes.
   • Windows (PowerShell):

     Get-FileHash test.txt -Algorithm SHA1

   • macOS/Linux:

     sha1sum test.txt

   The SHA-1 hashes will almost certainly be different.

5. Convert Line Endings (if needed): If you need to compare files created on different systems, convert the line endings to a consistent format.
   • Windows to macOS/Linux: Use tools like dos2unix.

     dos2unix test.txt

   • macOS/Linux to Windows: Use tools like unix2dos.

     unix2dos test.txt

   After converting, recalculate the SHA-1 hashes; they should now match.

6. Consider Binary Files: This issue is less common with binary files (like images or executables) as line endings aren’t relevant. However, different tools might still have slight variations in how they handle padding or metadata, leading to hash differences in rare cases.

Important Security Note

While MD5SUMs matching with differing SHA-1 hashes isn’t a major security breach on its own, it highlights the importance of using strong and consistent hashing algorithms for cyber security purposes. MD5 is considered cryptographically broken and should not be used for verifying data integrity where security is critical. SHA-1 is also becoming less reliable; SHA-256 or SHA-3 are preferred.