wget - options to save all new files

Question

I am attempting to use wget to download all new versions of a single file from a server. The file is updated once an hour at an unspecified and variable time.

I am running a cron script that runs wget every 10 minutes with the -N option to only download the new versions of the file (they all have the same name). However with the -N version the previous file on my RasPi is overwritten so I lose the older versions I need to keep. If I don't use -N I get a new copy every 10 minutes 5 of which are duplicates. Is there an option that will allow wget to download the new versions of the file but not overwrite the old ones ? If not is there an alternative solution ?

Thanks John

Answer 1

It may be possible to use some of the many wget options to accomplish your objective, but as I don't use it everyday, I'd have to experiment to provide that answer. But yes, of course there are alternatives - many of them. I'd use a simple shell script were I in your situation, and not intimately familiar with wget options; here it is:

1. from the command line, start the pico editor, and type the following shell script into it:

   #!/usr/bin/env bash 
   newfn=$(date '+%Y%m%d%H%M%S')
   newfn+="TheNewFileName.txt" 
   mv ~/TheOldFileName.txt ~/$newfn
   

Note that you have some options: mv will rename the existing file. If you'd rather leave the old file in place, use cp in place of mv. If you want to operate on the files in different folders than ~/ (your home dir), substitute the correct folder names.

The script will take an existing file named "TheOldFileName.txt", and rename it by concatenating the current date time stamp with a string "TheNewFileName", yielding, e.g. "20180614223302TheNewFileName.txt" as the new file name.

2. Save this file as "filenamer.sh", and exit pico, and make it executable:

   chmod 775 filenamer.sh
   

3. Add this to your crontab to execute when wget does.

Let us know if you have questions.

Answer 2

After a bit of thinking and searching I came up with the following that seems to work Ok. It is rather specific to the folder structure on my Raspberry Pi and to the files I am trying to download but with these changed should be of more general use.

--- shell script ----

    #!/bin/sh

change to the data folder

cd ~/Camera_Data/Download

-N option is supposed to only get new files - only works if file is still in the destination folder at next run of WGET -N

wget -N -a WGETout.log http://oiswww.eumetsat.org/IPPS/html/latestImages/EUMETSAT_MSG_RGBFog_WesternEurope.jpg

set variable to download filename

latestfile=EUMETSAT_MSG_RGBFog_WesternEurope.jpg

get checksum of downloaded (latest) file

file1_sha=$( md5sum $latestfile | awk '{print $1}')

check only files starting with 20 to prevent comparing file EUMETSAT_MSG_RGBFog_WesternEurope.jpg with itself

Also prevents checking image files which begin with 'IMG*.JPG' in the same folder

for filename in 20*.jpg ; do

this variable has to be set in the loop

testfile=$filename

Get checksums of next file

file2_sha=$( md5sum $testfile | awk '{print $1}') # Compare the checksums if [ "x$file1_sha" = "x$file2_sha" ] then printf "Files %s and %s are the same\n" "$latestfile" "$testfile" # current file already exists exit 0 else printf "Files %s and %s are different\n" "$latestfile" "$testfile" fi done

executes if no similar files found

printf "No similar files found copy to new name\n" cp EUMETSAT_MSG_RGBFog_WesternEurope.jpg "$(date +"%Y%m%d-%H%M.jpg")" exit 0

-- end of script --

The print statements can be removed - they are only there to show it is working.

If you have lots of files in the destination directory it would be better to sort them by latest modification time in the for loop. There are examples of how to do this on the web using 'ls' piped to 'sort' but they did not work for me.

The rejection of files starting with IMG prevents checking the camera image files that get downloaded to the same folder by a python programme. Since there are normally several hundred it provides a performance gain.

I could not get the code block to work so the comments are in in most cases in large bold rather than having the preceding comment #.