ftplib - download all files from directory and the newer one

This is the place for queries that don't fit in any of the other categories.

ftplib - download all files from directory and the newer one

Postby precyoz » Sun Aug 25, 2013 3:47 pm

Does anyone here know any scripts that downloads all the files from an ftp directory? And if every time the ftp server/site updates(adds a newer data/file), and that when you'll run the script the newer file will only be downloaded and not the existing file.

I really don't have any idea and I really need help about this.
Last edited by Mekire on Wed Nov 06, 2013 7:56 am, edited 1 time in total.
Reason: post_title.lower()
precyoz
 
Posts: 18
Joined: Wed Jul 03, 2013 6:23 am

Re: FTPLIB - DOWNLOAD ALL FILES FROM DIRECTORY AND THE NEWER

Postby metulburr » Sun Aug 25, 2013 4:16 pm

if server file's last modified date > local directory file's download date, then download, else file is up to date
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1312
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: FTPLIB - DOWNLOAD ALL FILES FROM DIRECTORY AND THE NEWER

Postby precyoz » Mon Aug 26, 2013 2:58 am

I really didn't get what you mean by that. Can you give me an example script instead?
precyoz
 
Posts: 18
Joined: Wed Jul 03, 2013 6:23 am

Re: FTPLIB - DOWNLOAD ALL FILES FROM DIRECTORY AND THE NEWER

Postby metulburr » Mon Aug 26, 2013 4:50 am

Code: Select all
import os
import time

filename = "test123.txt"

if not os.path.exists(filename):
    open(filename, 'w')


print("file was modified on {}".format(time.ctime(os.stat(filename).st_mtime)))
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1312
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: FTPLIB - DOWNLOAD ALL FILES FROM DIRECTORY AND THE NEWER

Postby precyoz » Mon Aug 26, 2013 6:44 am

Along with that script you gave, what should be the script to add so that I can download all the files from the directory? Thanks in advance!
precyoz
 
Posts: 18
Joined: Wed Jul 03, 2013 6:23 am

Re: FTPLIB - DOWNLOAD ALL FILES FROM DIRECTORY AND THE NEWER

Postby tnknepp » Mon Aug 26, 2013 7:07 pm

From your title I assume you want to download from ftp, correct? Nearly everything I do with ftp involves sftp, but I do have one example of open ftp available (albeit a bit of a kludge). FYI: I was transferring data TO an ftp server for several data sources (hence the dictionaries), but the same general concept should hold.

Code: Select all
import ftplib, os, time, glob

ftp = ftplib.FTP('site','uname','password')
       
# Local/Remote storage directories.
locald = {0:'dir'}

remoted = {0:'dest'}

file_type = {0:'r''}

for d in [0,1]:
    # Transfer level0 data, but only if file is not already on ftp server. 
    transfer = False
    tries = 0

    while transfer == False and tries != 10:
        try:
            ftp.cwd('/root/'+remoted[d]+'/'+'L0') # Change to proper ftp directory. 
            files = glob.glob(locald[d] + '\\*.*')

            for f in files:
               # Here, you should be able to do a modtime = f.sendcmd("MDTM file.name") to get the mod date (I THINK!!)
                if f.split('\\')[-1] not in ftp.nlst():
                    t = open(f,file_type[d])
                    ftp.storlines('STOR ' + f.split('\\')[-1],t)
                    t.close()
        except: transfer = False; tries += 1
        else: transfer = True
    if transfer == True:
        print remoted[d] + ' lev0 files transfered.  '
    else: print remoted[d] + ' lev0 transfer FAILURE.'
Python: 2.7 via Anaconda
Numpy: 1.7
Pandas: 0.11
OS: Windows 7
IDE: Spyder/IPython
User avatar
tnknepp
 
Posts: 119
Joined: Mon Mar 11, 2013 7:41 pm

Re: FTPLIB - DOWNLOAD ALL FILES FROM DIRECTORY AND THE NEWER

Postby precyoz » Wed Nov 06, 2013 7:05 am

I've tried the script but there's an error.
And where should I place my ftp directory and destination folder?


Code: Select all
dest lev0 transfer FAILURE.

Traceback (most recent call last):
  File "C:/Users/turtle/Desktop/Jesse Palmer FTP Downloader/FTP DOWNLOAD/Modified.py", line 32, in <module>
    else: print remoted[d] + ' lev0 transfer FAILURE.'
KeyError: 1
precyoz
 
Posts: 18
Joined: Wed Jul 03, 2013 6:23 am


Return to General Coding Help

Who is online

Users browsing this forum: freddyhard and 2 guests