Sign up ×

I have a shell script in which I need to check if two files are the same or not. I do this a for a lot of files, and in my script this diffing seems to be the performance bottleneck.

Here's the line:

diff -q $dst $new > /dev/null

if ($status) then ...

Could there be a faster way to compare the files, maybe a custom algorithm instead of the default diff?


share|improve this question
This is really nitpicking, but you're not asking to see if two files are the same, you're asking if two files have identical content. Same files have identical inodes (and same device). –  Zano Nov 4 '14 at 9:08

4 Answers 4

up vote 56 down vote accepted

I believe cmp will stop at the first byte difference:

cmp --silent $old $new || echo "files are different"
share|improve this answer
How can I add more commands than only one? I want to copy a file and roboot. –  Daniel Brunner Jun 14 '14 at 15:09
@DanielBrunner: You can copy from the standard input to both a file and standard output by using the tee command. –  Anders Rabo Thorbeck Jun 18 '14 at 6:51
Note that on my cmp I didn't have to shortcut it to echo, it will print a message if they differ or stay silent if they don't. –  eresonance May 11 at 17:44
@eresonance Right, the example is simply meant to show how you'd capture the return status in order to script a conditional. –  Alex Howansky May 11 at 17:56

Why don't you get the hash of both files content?

Try this script, call it for example and then run it as follows: file1.txt file2.txt


file1=`md5 $1`
file2=`md5 $2`

if [ "$file1" = "$file2" ]
    echo "Files have the same content"
    echo "Files have NOT the same content"
share|improve this answer
sorry not sure what you're referring to, I haven't been scripting unix too long. –  JDS Oct 15 '12 at 17:09
If two files are the same, then they will have the same hash value. For example if 'file1.txt' contents 'aaa' and 'file2.txt' so, then when you get the md5 hash: md5 file1.txt you will get: 5c9597f3c8245907ea71a89d9d39d08e which will be the same output as md5 file2.txt, if you work out both hashes and they are the same you can assure that they both have the same content :) –  jabaldonedo Oct 15 '12 at 17:18
Oh right like some sort of check sum. I considered this as a possibility; is there a simple unix command for this? –  JDS Oct 15 '12 at 17:32
md5 is the unix command to find the md5 hash of two files. –  hlin117 Jan 9 at 5:10
Also, @jabaldonedo, I think you only need one = sign to check for the equality of two strings in bash. –  hlin117 Jan 9 at 5:17

For files that are not different, any method will require having read both files entirely, even if the read was in the past.

There is no alternative. So creating hashes or checksums at some point in time requires reading the whole file. Big files take time.

File metadata retrieval is much faster than reading a large file.

So, is there any file metadata you can use to establish that the files are different? File size ? or even results of the file command which does just read a small portion of the file?

File size example code fragment:

  ls -l $1 $2 | 
  awk 'NR==1{a=$5} NR==2{b=$5} 
       END{val=(a==b)?0 :1; exit( val) }'

[ $? -eq 0 ] && echo 'same' || echo 'different'  

If the files are the same size then you are stuck with full file reads.

share|improve this answer

I like @Alex Howansky have used 'cmp --silent' for this. But I need both positive and negitive response so I use:

cmp --silent file1 file2 && echo '### SUCCESS: Files Are Identical! ###' || echo '### WARNING: Files Are Different! ###'"

I can then run this with a ssh to check files against a constant file.

share|improve this answer

Your Answer


