0

I succeded in transfering files via ftp from a remote server to a raspberry,

#!/bin/bash -vx
ftp -in IP_SERVER<<END_SCRIPT
quote USER rem_user
quote PASS rem_pass
bin
prompt:off
cd /path_to_server_files
lcd /path_to_local_files
mget *.mp3
bye
END_SCRIPT

but there was a problem with ftp, which was overwriting the mp3 files everytime it ran a cronjob.

I have a company that provides background music to other companies. My method was leaving a computer in each one playing 24/7 or with other specific cronjobs, depending on the client. And the raspberry is a great way to do that instead a computer. The method i have right know that is working is a cronjob per folder. Each folder has a type of music. So i will be putting different music from time to time in the server and the cronjob will transfer those files once a week. It is set to transfer every mp3 file in that folder to the RPi. The thing is, it will transfer all the files there including the ones that were already there. If i put there, for example, 150 music files, it will take a long time transfering those, not to mention if it is done with all the folders since the RPi ARM is not that powerfull. The solution would be not overwriting the files already there, just the new ones. Then after some time another cronjob will delete all the files that have more than * days old.

I searched but it seems ftp doesnt have an option like this yet. So I found the wget command which allows to transfer without overwriting but i couldn't make it transfer multiple files. I have been trying to convert the script above with the wget command without success. Can someone with experience in this matter help out? it could be http also. Thanks in advance.

EDIT:

Tried to change mget with wget in the script but it returns ?invalid command, tried to get ftp line out, the login credentials, nothing works. My problem is the conversion to wget, I have no idea how to do it. Probably i have to make a totally diferent script, hence this post.

Some solution like this would solve the issue

https://stackoverflow.com/questions/4944295/wget-skip-if-files-exist

if it would be possible to convert the script to it, stopping the overwriting of files.

SOLUTION - Thanks to ifermon I got this script working like a charm. Thanks for your impecable support and all the others who made efforts for this to work

#!/usr/bin/python
import os
from ftplib import FTP

local_path='/path_to_local_files/'
os.chdir(local_path)
ftp = FTP(host='server_name_or_IP',user='username', passwd='password')
ftp.cwd('/path_to_local_files/')
f_list = ftp.nlst()
for f in f_list:
if not f.endswith("mp3"):
  continue
new_f_name = local_path + f
if os.path.exists(new_f_name):
  continue
print("Copying remote file <{0}>to local file <{1}>".format(f,new_f_name))
ftp.retrbinary('RETR '+ f, open(new_f_name,'wb').write)

you may need to install this in order for the script to work:

sudo apt-get install python-dev

closed as off-topic by goldilocks Jul 14 '15 at 17:04

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be specific to the Raspberry Pi within the scope defined in the help center." – goldilocks
If this question can be reworded to fit the rules in the help center, please edit the question.

  • Show us wget commands you've tried so far ? It will give us more idea why it is failing. – dhruvvyas90 Jul 14 '15 at 10:19
  • A few wget examples for getting an entire folder from an ftp. serverfault.com/questions/25199/… – dhruvvyas90 Jul 14 '15 at 10:26
  • @dastaan I edited the question with the replie to your comment. Anything else you need let me know. – angelfmf Jul 14 '15 at 10:58
  • @dastaan I changed the script with this #!/bin/bash -vx wget -r --timestamp ftp://user:pass@ip /path_to_remote_files bye END_SCRIPT Doesnt work, it shows a lot of errors. command not found``bad port number Can´t you give an example of how to put those commands in a script? – angelfmf Jul 14 '15 at 11:06
  • wget is not an ftp command, it's a shell command (vs. mget which stands for multiple get which is an ftp command). – ifermon Jul 14 '15 at 11:08
1

wget is not an ftp command, it's a shell command (vs. mget which stands for multiple get which is an ftp command).

Probably a better way to do this is to use something like python. The below script logs into the server, looks for files ending in ".mp3", checks to see if the same file exists locally. If file does exist, it moves on to the next one. If it does not exist locally it copies it over.

#!/usr/local/bin/python
import os
from ftplib import FTP

local_path='local directory name here'
os.chdir(local_path)
ftp = FTP(host='hostname', user='username', passwd='pwd')
ftp.cwd('remote directory name here')
f_list = ftp.nlst()
for f in f_list:
    if not f.endswith("mp3"):
        continue
    new_f_name = local_path + f
    if os.path.exists(new_f_name):
        continue
    print("Copying remote file <{0}> to local file <{1}>".format(f,
            new_f_name))
    ftp.retrbinary('RETR ' + f, open(new_f_name,'wb').write)

Save the above code to a file (e.g. get_mp3.py). Edit it to change paths to match your needed path. Make the file executable (e.g. chmod +x get_mp3.py). Then you can call it directly from a cron job. To install pysftp do the following:

You might have to install ftp first:

sudo apt-get install ftp 

That's it. You should be good to go.

That should work, but it's not tested.

  • thanks for the reply but i don't understand exactly your answer. I have no experience with python, but i think i understood the main idea from answer. I will clarify what i have done with ftp wich is working. I have a website that runs music from a server using ftp. Each music type has a folder, like christmas theme, pop, smooth jazz, etc, and each one had a folder in which a script would be executed with cron from that type of music in the server to the folder of the same type in raspberry. Thats, why i need to transfer multiple files – angelfmf Jul 14 '15 at 12:03
  • because each folder will be getting lots of music from time to time. I understand that probably i had to use code language to do it but i was hoping to avoid it, since the problem with ftp was just the files being overwriting everytime. It would be best if there was comparison with the files from the server and the local ones and if they had the same name, not copying. – angelfmf Jul 14 '15 at 12:03
  • I added an example script that should get you 90% of the way there. Let me know if you need help with more specifics. Tested the python script and it does work. – ifermon Jul 14 '15 at 12:04
  • Sorry the long wait, i have been trying to execute your solution, but i am stuck with the error: /bin/sh: 1: /home/pi/Desktop/scripts_cron/get_mp3.py: not found but it is there. I thought it could be due to permission, but after changing it from root to user, it stayed the same. I will try your other solution as well – angelfmf Jul 14 '15 at 15:47
  • I just noticed that you were thinking of downloading the mp3 files to diferent directories, but that would fill up the raspberry pi in an instant from diferent folders with the same mp3 music. That was just the issue i had with goldilocks solution. Isn't it possible to just not overwrite the music if the files have the same name? – angelfmf Jul 14 '15 at 15:56
0

First, use a timestamp to create a unique location:

#!/bin/bash

mp3dir=/path_to_local_files/$(date +%m-%d_%H:%M)
mkdir $mp3dir
cd $mp3dir

That date call will return something like 07-14_08:51; there is an explanation of the format in man date, but that's month-day_hour:minute, where hour is 00-23, so unless you do this more than once per minute, it will always be unique.

Since you've now changed directory (cd) into that, you don't have to use lcd inside the ftp script:

ftp -in IP_SERVER<<END_SCRIPT
quote USER rem_user
quote PASS rem_pass
bin
prompt:off
cd /path_to_server_files
mget *.mp3
bye
END_SCRIPT

Done. The mp3 files should all be in the /path_to_local_files/07-14_08:51 (or whatever) directory.

  • Thanks goldilocks, i tried your solution and it works. There is an issue though. I set the cronjob to run every minute, and it creates a new folder every minute in that paste. That will end up filling the raspberry SD card. If i put a folder to copy, for example, every monday at 1 a.m., at the end of the week, i would have several music folders, and the music wouldn't play automatically because the path to the files would be constantly changeing while the path in which the website gets the folders remains the same. Sorry if i wasn't clear enough. Is there to just prevent the overwriting? – angelfmf Jul 14 '15 at 13:56
  • There's always a way, although there is not always an easy way, lol -- but that is subjective. Anyway, it's not totally clear what you are doing; if the files that stand to be overwritten are always exactly the same (as opposed to being new versions of something with the same name), then you want to skip the download completely, right? That means giving more specific instructions to ftp -- I suggest you ask about this on Unix & Linux, but describe in more detail exactly what you are trying to do and why. There may be better tools around for this purpose. – goldilocks Jul 14 '15 at 17:03
  • For example: If you only want mp3's that are new on the server since the last time you checked, there might be a way to filter based on the timestamp of the files. That's why the more relevant details you can provide, the more likely someone is to spot an appropriate solution. – goldilocks Jul 14 '15 at 17:06
  • I have updated the question clarifying what i need. Hope it is clearer now. It´s not that easy though because i am not english native, nor i have experience posting here. Anyway i will make a post at Unix & Linux like you suggested. I´d still be grateful if you could keep helping in this matter. Your solution was very close to what i needed. Like you said on your last comment i only want the new mp3's that are in the server yes. Do have more details about your solution related to the filteron the timestamp? – angelfmf Jul 15 '15 at 14:47
  • I don't use ftp, so it's hard for me to say. What comes to mind is asking it for a list of what's currently on the server, then comparing that to the current download directory and downloading each file that isn't there specifically. That means a bit of coding, in bash or whatever your preference is. If you don't code, then hopefully there's a way to filter in this manner directly with ftp. – goldilocks Jul 15 '15 at 14:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.