多线程下载文件代码,Python代码库,德仔网

代码语言

代码分类

【Python】多线程下载文件

作者:speedmancs / 发布于2011/10/9/ 1088

import httplib
import urllib2
import time
from threading import Thread
from Queue import Queue
from time import sleep</div>
<div>proxy = 'your proxy';
opener = urllib2.build_opener( urllib2.ProxyHandler({'http':proxy}) )
urllib2.install_opener( opener )</div>
<div>ids = {};
for i in range(1,110):
try:
listUrl = "<a href="http://www.someweb.net/sort/list_8_%d.shtml">http://www.someweb.net/sort/list_8_%d.shtml</a>" % (i);
print listUrl;
page = urllib2.urlopen(listUrl).read();
speUrl = "<a href="http://www.someweb.net/soft/">http://www.someweb.net/soft/</a>";
speUrlLen = len(speUrl);
idx = page.find(speUrl,0);
while idx!=-1:
dotIdx = page.find(".",idx + speUrlLen);
if dotIdx != -1:
id = page[idx + speUrlLen:dotIdx];
ids[id] = 1;
idx = page.find("<a href="http://www.someweb.net/soft/",idx">http://www.someweb.net/soft/",idx</a> + speUrlLen);
except:
pass;</div>
<div>
q = Queue()
NUM = 5
failedId = [];</div>
<div>def do_somthing_using(id):
try:
url = "<a href="http://www.someweb.net/download.php?softid=%s&type=dx">http://www.someweb.net/download.php?softid=%s&type=dx</a>" % (id);
h2 = httplib.HTTPConnection("your proxy", "you port");
h2.request("HEAD", url);
resp = h2.getresponse();
header = resp.getheaders();
location = header[3][1];
sContent = urllib2.urlopen(location).read();
savePath = "C:\\someweb\\%s.rar" % (id);
file=open(savePath,'wb');
file.write(sContent);
file.close();
print savePath + " saved";
except:
pass;</div>
<div>def working():
 while True:
 arguments = q.get()
 do_somthing_using(arguments)
 sleep(1)
 q.task_done()</div>
<div>for i in range(NUM):
 t = Thread(target=working)
 t.setDaemon(True)
 t.start()
for id in ids:
 q.put(id)</div>
<div>q.join()
</div>

试试其它关键字

　多线程下载文件　

同语言下

可能有用的

speedmancs贡献的其它代码(2)