0

I am attempting to use wget to download all new versions of a single file from a server. The file is updated once an hour at an unspecified and variable time.

I am running a cron script that runs wget every 10 minutes with the -N option to only download the new versions of the file (they all have the same name). However with the -N version the previous file on my RasPi is overwritten so I lose the older versions I need to keep. If I don't use -N I get a new copy every 10 minutes 5 of which are duplicates. Is there an option that will allow wget to download the new versions of the file but not overwrite the old ones ? If not is there an alternative solution ?

Thanks John

1
  • One obvious alternative is to write a script which removes duplicates. Anyway, it doesn't look like your question is about a Raspberry Pi. Jun 21 '18 at 6:58
2

It may be possible to use some of the many wget options to accomplish your objective, but as I don't use it everyday, I'd have to experiment to provide that answer. But yes, of course there are alternatives - many of them. I'd use a simple shell script were I in your situation, and not intimately familiar with wget options; here it is:

  1. from the command line, start the pico editor, and type the following shell script into it:

    #!/usr/bin/env bash 
    newfn=$(date '+%Y%m%d%H%M%S')
    newfn+="TheNewFileName.txt" 
    mv ~/TheOldFileName.txt ~/$newfn
    

Note that you have some options: mv will rename the existing file. If you'd rather leave the old file in place, use cp in place of mv. If you want to operate on the files in different folders than ~/ (your home dir), substitute the correct folder names.

The script will take an existing file named "TheOldFileName.txt", and rename it by concatenating the current date time stamp with a string "TheNewFileName", yielding, e.g. "20180614223302TheNewFileName.txt" as the new file name.

  1. Save this file as "filenamer.sh", and exit pico, and make it executable:

    chmod 775 filenamer.sh
    
  2. Add this to your crontab to execute when wget does.

Let us know if you have questions.

4
  • Thanks Seamus - that is where I got to as well. The problem is as soon as you rename or move the file WGET -N downloaded it 'forgets' that it has downloaded the file already and downloads it again when the script next runs :-( . I have asked the same question on the WGET-bug list to see if they have an answer. If not I may be forced to use one of the programmes or scripts that delete duplicate files that have different names but the same contents.
    – JohnM
    Jun 15 '18 at 13:52
  • OK - simple enough to cure. We'll simply calculate the secure hash for the existing file before we run the remainder of the script. Kinda' busy now, but will post a revision in a day or two.
    – Seamus
    Jun 15 '18 at 21:50
  • Hi @Seamus, I think I have managed to write a shell script that works. All I need to do is to test it tonight. It seems to run ok manually but the real test is when it runs from a cron script. It should only download new files using WGET-N. My script uses the md5 checksum to check if a new version has been downloaded and if so saves it under a date-time similar to your code above. Is it worth posting the code here for others ? It is crude in that it calculates the md5 checksum for all the files every 10 minutes. I will 'answer' my question when the script has been tested.
    – JohnM
    Jun 18 '18 at 16:42
  • Good for you! If it works manually, I can almost guarantee that we'll get it to work under cron - don't worry about that. And wrt posting your code, I think that's a great idea - test it before you post it.
    – Seamus
    Jun 18 '18 at 17:00
0

After a bit of thinking and searching I came up with the following that seems to work Ok. It is rather specific to the folder structure on my Raspberry Pi and to the files I am trying to download but with these changed should be of more general use.

--- shell script ----

    #!/bin/sh

change to the data folder

cd ~/Camera_Data/Download

-N option is supposed to only get new files - only works if file is still in the destination folder at next run of WGET -N

wget -N -a WGETout.log http://oiswww.eumetsat.org/IPPS/html/latestImages/EUMETSAT_MSG_RGBFog_WesternEurope.jpg

set variable to download filename

latestfile=EUMETSAT_MSG_RGBFog_WesternEurope.jpg

get checksum of downloaded (latest) file

file1_sha=$( md5sum $latestfile | awk '{print $1}')

check only files starting with 20 to prevent comparing file EUMETSAT_MSG_RGBFog_WesternEurope.jpg with itself

Also prevents checking image files which begin with 'IMG*.JPG' in the same folder

for filename in 20*.jpg ; do

this variable has to be set in the loop

testfile=$filename

Get checksums of next file

file2_sha=$( md5sum $testfile | awk '{print $1}') # Compare the checksums if [ "x$file1_sha" = "x$file2_sha" ] then printf "Files %s and %s are the same\n" "$latestfile" "$testfile" # current file already exists exit 0 else printf "Files %s and %s are different\n" "$latestfile" "$testfile" fi done

executes if no similar files found

printf "No similar files found copy to new name\n" cp EUMETSAT_MSG_RGBFog_WesternEurope.jpg "$(date +"%Y%m%d-%H%M.jpg")" exit 0

-- end of script --

The print statements can be removed - they are only there to show it is working.

If you have lots of files in the destination directory it would be better to sort them by latest modification time in the for loop. There are examples of how to do this on the web using 'ls' piped to 'sort' but they did not work for me.

The rejection of files starting with IMG prevents checking the camera image files that get downloaded to the same folder by a python programme. Since there are normally several hundred it provides a performance gain.

I could not get the code block to work so the comments are in in most cases in large bold rather than having the preceding comment #.

4
  • Re the code blocks here: Yes, I've had problems with that also. It seems it's not exactly markdown, but a slight variation on it. While "fencing" doesn't seem to work here, I have found that 8 spaces in front of each line of code seems to get the job done.
    – Seamus
    Jun 20 '18 at 19:01
  • Just one other very minor thing: I noticed that you use _sha in your variable names. That confused me until I read the code as I thought you were doing SHA instead of MD5.
    – Seamus
    Jun 20 '18 at 19:03
  • @Seamus - that is because I stole that bit of the code from another post - I changed the check to MD5 but left the variable names the same !!
    – JohnM
    Jun 20 '18 at 21:32
  • If anyone wants to see a better formated version of the script(s) they are on my website at www.JohnMurrell.org.uk
    – JohnM
    Jun 20 '18 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.