FTPOSError
_______________________________________________ ftputil mailing list ftputil@... http://codespeak.net/mailman/listinfo/ftputil
_______________________________________________ ftputil mailing list ftputil@... http://codespeak.net/mailman/listinfo/ftputil
Hello, Here's an application of the callback parameter for uploads and downloads which I didn't think of when I added callbacks. Forwarded with consent from Alec. :) Stefan -------- Original Message -------- Subject: Re: throttled upload with ftputil? Date: Sun, 27 Mar 2011 11:28:56 +0200 From: Stefan Schwarzer <sschwarzer@...> To: Alec Bennett <wrybread@...> Hello Alec, Thanks for your feedback! On 2011-03-27 06:53, Alec Bennett wrote: > I was wondering if its possible to throttle the upload speed when uploading > a file through ftputil? > > I'm currently uploading like this, if its a factor: > > ftp = ftputil.FTPHost(hostname, username, password) > ftp.upload(fname, target, mode = "b") ftputil doesn't have a parameter to specifically manipulate the upload or download speed. Usually such limiting is done on the operating system level. That said, I guess you could achieve something useful by using a callback parameter [1] in the `upload` method. (Callbacks were added in ftputil 2.5.) I think you could just use import time import ftputil def throttle_callback(chunk): """Insert some sleep time to decrease the transfer speed.""" # Use whatever interval you find appropriate. time.sleep(1) ftp = ftputil.FTPHost(hostname, username, password) ftp.upload(fname, target, mode='b', callback=throttle_callback) If you need more control, e. g. want to throttle down to a specific transfer rate, you could count the bytes transferred so far (the sum of the lengths of the chunks) and the elapsed time (`time.time()` minus start time of transfer) and insert corresponding sleep intervals if the rate is too high. If you need more concrete help, let me know. > And thanks for the spectacular module, by the way. Thanks! You're welcome. :) Although you wrote me in private, I think the question and the approach to it are interesting. I had never had such an application in mind when I added the callback support. I'd like to forward this e-mail to the ftputil mailing list. Would that be ok for you? [1] http://ftputil.sschwarzer.net/trac/wiki/Documentation#uploading-and-downloading-files Stefan
Hi All,
I am having an issue trying to download files from our remote FTP
server using ftputils-2.2.3.
In local development, we can connect to the remote FTP server and
download the files to our local box, no problem. When we deploy the
codebase to our development webserver and try, the code just hangs,
throwing no exception. Our webhosting company, Webfaction, says it
doesn't look like an issue from their end.
Basically the user selects the required files from the page via
checkboxes ('download' parameter).
Code that downloads the files:
c = Context({})
if request.method == 'POST' and request.is_ajax():
cloud_user = get_object_or_404(User,username=username)
user_profile = UserProfile.objects.get(user=cloud_user)
data = {}
client =
FTPHost
(settings
.CLOUD_SERVER,user_profile.cloud_username,user_profile.cloud_password)
adir = os.path.dirname(user_profile.download_dir)
if not os.path.isabs(adir):
adir = ('%s%s' % ("/",adir))
if os.path.isdir(adir):
downloads = request.POST.getlist('download')
for download in downloads:
filename = os.path.split(download)[1]
localfile = os.path.join(adir,filename)
client.download(download, localfile)
data['outcome'] = True
data['results_text'] = ('%s%s' % ("All files ",
"successfully downloaded."))
client.close()
else:
data['outcome'] = False
data['results_text'] = ('%s%s%s' % ("The download
directory ",user_profile.download_dir, " does not exist on your local
system. No files downloaded."))
return _json_response(data)
return render_to_response('cloud/directory.html', c,
context_instance=RequestContext(request))
Any help much appreciated!
Cheers
Rick
Hello, For several years, the ftputil mailing lists (i. e. the general discussion list and the ticket mailing list) have thankfully been hosted by Codespeak. However, as nowadays there are lots of hosting possibilities for open source projects, Codespeak is going to shut down its hosting. So we/I have to migrate the ftputil mailing lists to a new provider. (I'm not really eager to host the lists myself.) I opened a ticket [1] on this and outlined what's to do. I'd like to know if I forgot something. Also, I'm very interested in suggestions of appropriate mailing list providers. I'd like to finish the migration by the end of April but have nothing against finishing it earlier. :) As usual, I'm looking forward to your feedback.[1] http://ftputil.sschwarzer.net/trac/ticket/57 Stefan
ftputil 2.6 is now available from http://ftputil.sschwarzer.net/download . Changes since version 2.5 ------------------------- - The stat caching has been improved. There's now an "auto-grow" feature for `FTPHost.listdir` which in turn applies to `FTPHost.walk`. Moreover, there are several performance optimizations. - A few bugs were fixed [1-3]. What is ftputil? ---------------- ftputil is a high-level FTP client library for the Python programming language. ftputil implements a virtual file system for accessing FTP servers, that is, it can generate file-like objects for remote files. The library supports many functions similar to those in the os, os.path and shutil modules. ftputil has convenience functions for conditional uploads and downloads, and handles FTP clients and servers in different timezones. Read the documentation at http://ftputil.sschwarzer.net/documentation . License ------- ftputil is Open Source software, released under the revised BSD license (see http://www.opensource.org/licenses/bsd-license.php ). [1] http://ftputil.sschwarzer.net/trac/ticket/53 [2] http://ftputil.sschwarzer.net/trac/ticket/55 [3] http://ftputil.sschwarzer.net/trac/ticket/56 Stefan
ftputil 2.6b is now available from http://ftputil.sschwarzer.net/download . Changes since version 2.5 ------------------------- - The stat caching has been improved. I've added an "auto-grow" feature for `FTPHost.listdir` which in turn applies to `FTPHost.walk`. Moreover, there were several performance optimizations. - A few bugs were fixed [1-3]. What is ftputil? ---------------- ftputil is a high-level FTP client library for the Python programming language. ftputil implements a virtual file system for accessing FTP servers, that is, it can generate file-like objects for remote files. The library supports many functions similar to those in the os, os.path and shutil modules. ftputil has convenience functions for conditional uploads and downloads, and handles FTP clients and servers in different timezones. Read the documentation at http://ftputil.sschwarzer.net/documentation . License ------- ftputil is Open Source software, released under the revised BSD license (see http://www.opensource.org/licenses/bsd-license.php ). [1] http://ftputil.sschwarzer.net/trac/ticket/53 [2] http://ftputil.sschwarzer.net/trac/ticket/55 [3] http://ftputil.sschwarzer.net/trac/ticket/56 Stefan
Here is the steps to reproduce this bug
Python version 2.6.6
ftputil-2.5-py2.6
py2exe-0.6.9.win32-py2.6.exe
ftp_example.py
import re
import ftputil
pattern = re.compile('^.*$', re.I)
print pattern
connection = ftputil.FTPHost('127.0.0.1', 'anonymous', '')
print connection
setup.py
from distutils.core import setup
import py2exe
setup(
console=['ftp_example.py'],
options={"py2exe": { "includes" : "ftputil"}})
Here is the output
D:\PROJECTS\IKISPFU\_DB\IKIS_Queue\ASOPDExchange\src>python setup.py py2exe
C:\DevTools\Python26\lib\site-packages\py2exe\build_exe.py:16:
DeprecationWarning: the sets module is deprecated
import sets
running py2exe
*** searching for required modules ***
Traceback (most recent call last):
File "setup.py", line 6, in <module>
options={"py2exe": { "includes" : "ftputil"}})
File "C:\DevTools\Python26\lib\distutils\core.py", line 152, in setup
dist.run_commands()
File "C:\DevTools\Python26\lib\distutils\dist.py", line 975, in run_commands
self.run_command(cmd)
File "C:\DevTools\Python26\lib\distutils\dist.py", line 995, in run_command
cmd_obj.run()
File "C:\DevTools\Python26\lib\site-packages\py2exe\build_exe.py",
line 243, in run
self._run()
File "C:\DevTools\Python26\lib\site-packages\py2exe\build_exe.py",
line 296, in _run
self.find_needed_modules(mf, required_files, required_modules)
File "C:\DevTools\Python26\lib\site-packages\py2exe\build_exe.py",
line 1297, in find_needed_modules
mf.import_hook(mod)
File "C:\DevTools\Python26\lib\site-packages\py2exe\mf.py", line
719, in import_hook
return Base.import_hook(self,name,caller,fromlist,level)
File "C:\DevTools\Python26\lib\site-packages\py2exe\mf.py", line
136, in import_hook
q, tail = self.find_head_package(parent, name)
File "C:\DevTools\Python26\lib\site-packages\py2exe\mf.py", line
204, in find_head_package
raise ImportError, "No module named " + qname
ImportError: No module named ftputil
--
--
С уважением, Владимир.
_______________________________________________
ftputil mailing list
ftputil <at> codespeak.net
http://codespeak.net/mailman/listinfo/ftputil
ftputil 2.5 is now available from http://ftputil.sschwarzer.net/download . Changes since version 2.4.2 --------------------------- - As announced over a year ago [1], the `xreadlines` method for FTP file objects has been removed, and exceptions can no longer be accessed via the `ftputil` namespace. Only use `ftp_error` to access the exceptions. The distribution contains a small tool `find_deprecated_code.py` to scan a directory tree for the deprecated uses. Invoke the program with the `--help` option to see a description. - Upload and download methods now accept a `callback` argument to do things during a transfer. Modification time comparisons in `upload_if_newer` and `download_if_newer` now consider the timestamp precision of the remote file which may lead to some unnecessary transfers. These can be avoided by waiting at least a minute between calls of `upload_if_newer` (or `download_if_newer`) for the same file. See the documentation for details [2]. - The `FTPHost` class got a `keep_alive` method. It should be used carefully though, not routinely. Please read the description [3] in the documentation. - Several bugs were fixed [4-7]. - The source code was restructured. The tests are now in a `test` subdirectory and are no longer part of the release archive. You can still get them via the source repository. Licensing matters have been moved to a common `LICENSE` file. What is ftputil? ---------------- ftputil is a high-level FTP client library for the Python programming language. ftputil implements a virtual file system for accessing FTP servers, that is, it can generate file-like objects for remote files. The library supports many functions similar to those in the os, os.path and shutil modules. ftputil has convenience functions for conditional uploads and downloads, and handles FTP clients and servers in different timezones. Read the documentation at http://ftputil.sschwarzer.net/documentation . License ------- ftputil is Open Source software, released under the revised BSD license (see http://www.opensource.org/licenses/bsd-license.php ). [1] http://codespeak.net/pipermail/ftputil/2009q1/000256.html [2] http://ftputil.sschwarzer.net/trac/wiki/Documentation#uploading-and-downloading-files [3] http://ftputil.sschwarzer.net/trac/wiki/Documentation#keep-alive [4] http://ftputil.sschwarzer.net/trac/ticket/44 [5] http://ftputil.sschwarzer.net/trac/ticket/46 [6] http://ftputil.sschwarzer.net/trac/ticket/47 [7] http://ftputil.sschwarzer.net/trac/ticket/51 Stefan
Hi everyone, I discovered ftputil because I wanted to index ftp servers. So I have to list directories with walk. But the current code make too many requests (1 per file). In a directory with thousands of files it is unusable.
To this is a patch which use retrlines("LIST"). It makes only one request per dir.
def walk(self, top, topdown=True, onerror=None):
"""
Iterate over directory tree and return a tuple (dirpath,
dirnames, filenames) on each iteration, like the `os.walk`
function (see http://docs.python.org/lib/os-file-dir.html ).
"""
# Hack with retrlines("LIST").
# The following code is copied from `os.walk` in Python 2.4
# and adapted to ftputil.
dirs, nondirs = [], []
def callback(line):
#example of a line :
#drwxrwxrwx 2 ftp nogroup 159744 Oct 22 04:48 Upload
#their neither is . and ..
name = line[55:] #Ugly hack, must check that is correct. It work ith proftpd.
if line.startswith("d"):
dirs.append(name)
elif not line.startswith("l"): #No links
nondirs.append(name)
try:
raw = self._session.retrlines("LIST "+ top, callback)
except ftp_error.FTPOSError:
if onerror is not None:
onerror(err)
return
if topdown:
yield top, dirs, nondirs
for name in dirs:
path = self.path.join(top, name)
for item in self.walk(path, topdown, onerror):
yield item
if not topdown:
yield top, dirs, nondirs
Matthieu Bizien
_______________________________________________ ftputil mailing list ftputil@... http://codespeak.net/mailman/listinfo/ftputil
Hello Stefan and all the silent folks,
I run into a strange problem:
* If the current dir is the root of my FTP account
* and I start ftputil's walk (host.walk or host.path.walk)
* with a root of '.',
my script hangs _or_ breaks at the point where it encounters files or
directories whose names contain non-ASCII chars.
I still don't understand, when I'll get a traceback and when not.
The traceback I get with ftputil 2.4.2 is:
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/site-packages/ftputil/ftp_stat.py", line 549, in
__call_with_parser_retry
result = method(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/site-packages/ftputil/ftp_stat.py", line 421, in _real_listdir
loop_path = self._path.join(path, stat_result._st_name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/posixpath.py", line 70, in join
path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position
41: ordinal not in range(128)
In ftputil 2.5b it is:
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/site-packages/ftputil/ftp_stat.py", line 565, in
__call_with_parser_retry
result = method(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/site-packages/ftputil/ftp_stat.py", line 437, in _real_listdir
loop_path = self._path.join(path, stat_result._st_name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/posixpath.py", line 70, in join
path += '/' + b
(In this case some file names contain 'ß'.)
With a root of '/', './' or every other absolute or relative path I
tried, it works (I get everything as byte strings, regardless what
their encoding might once have been, and that's expected FTP behaviour).
Since the real problem seems to be in posixpath, I don't know if it's
possible to work around this in ftputil.
Somewhat confused greetlings from Lake Constance!
Hraban
---
http://www.fiee.net
https://www.cacert.org (I'm an assurer)
ftputil 2.5b is now available from http://ftputil.sschwarzer.net/download . Changes since version 2.4.2 --------------------------- - As announced over a year ago [1], the `xreadlines` method for FTP file objects has been removed, and exceptions can no longer be accessed via the `ftputil` namespace. Only use `ftp_error` to access the exceptions. The distribution contains a small tool `find_deprecated_code.py` to scan a directory tree for the deprecated uses. Invoke the program with the `--help` option to see a description. - Upload and download methods now accept a `callback` argument to do things during a transfer. Modification time comparisons in `upload_if_newer` and `download_if_newer` now consider the timestamp precision of the remote file which may lead to some unnecessary transfers. These can be avoided by waiting at least a minute between calls of `upload_if_newer` (or `download_if_newer`) for the same file. See the documentation for details [2]. - The `FTPHost` class got a `keep_alive` method. It should be used carefully though, not routinely. Please read the description [3] in the documentation. - Several bugs were fixed [4-7]. - The source code was restructured. The tests are now in a `test` subdirectory and are no longer part of the release archive. You can still get them via the source repository. Licensing matters have been moved to a common `LICENSE` file. What is ftputil? ---------------- ftputil is a high-level FTP client library for the Python programming language. ftputil implements a virtual file system for accessing FTP servers, that is, it can generate file-like objects for remote files. The library supports many functions similar to those in the os, os.path and shutil modules. ftputil has convenience functions for conditional uploads and downloads, and handles FTP clients and servers in different timezones. Read the documentation at http://ftputil.sschwarzer.net/documentation . License ------- ftputil is Open Source software, released under the revised BSD license (see http://www.opensource.org/licenses/bsd-license.php ). [1] http://codespeak.net/pipermail/ftputil/2009q1/000256.html [2] http://ftputil.sschwarzer.net/trac/wiki/Documentation#uploading-and-downloading-files [3] http://ftputil.sschwarzer.net/trac/wiki/Documentation#keep-alive [4] http://ftputil.sschwarzer.net/trac/ticket/44 [5] http://ftputil.sschwarzer.net/trac/ticket/46 [6] http://ftputil.sschwarzer.net/trac/ticket/47 [7] http://ftputil.sschwarzer.net/trac/ticket/51 Stefan
| Mon | Tue | Wed | Thu | Fri | Sat | Sun |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| 13 | 14 | 15 | 16 | 17 | 18 | 19 |
| 20 | 21 | 22 | 23 | 24 | 25 | 26 |
| 27 | 28 | 29 |
RSS Feed2 | |
|---|---|
3 | |
2 | |
3 | |
10 | |
5 | |
8 | |
2 | |
20 | |
1 | |
4 | |
1 | |
1 | |
2 | |
11 | |
17 | |
1 | |
11 | |
1 | |
5 | |
4 | |
5 | |
6 | |
9 | |
3 | |
4 | |
3 | |
2 | |
1 | |
1 | |
6 | |
2 | |
54 | |
20 | |
3 | |
12 | |
5 | |
8 | |
27 | |
9 | |
2 | |
2 |