1、另起线程
核心就是threading.Thread
:
from threading import Thread
def square(x):
print(x * x)
def mul(x, y):
print(x * y)
if __name__ == "__main__":
thread1 = Thread(target=square, args=(10, ))
thread2 = Thread(target=mul, args=(10, 10))
thread1.start()
thread2.start()
thread1.join()
thread2.join()
注意:
- 必须调用
start
,线程才会被启动。 join
会将线程合并到当前线程,当前线程被阻塞,直到合并的线程执行完毕。若不合并,主线程退出后,新起的线程将自动退出。
2、多进程并行任务
核心就是multiprocessing.Pool.map
和multiprocessing.Pool.starmap
。
2.1、单参数函数
Pool.map
用起来简单顺手,两行代码就可以并行执行多任务:
from multiprocessing import Pool
def square(x):
return x * x
if __name__ == "__main__":
with Pool() as pool:
res = pool.map(square, range(10))
print(res)
2.2、绑定多参数函数的多余参数
如果函数有多个参数,然后只遍历其中一个参数,一个直观的想法是:
from multiprocessing import Pool
def mul(x, y):
return x * y
if __name__ == "__main__":
double = lambda x: mul(x, 2)
with Pool() as pool:
print(pool.map(double, range(10)))
可惜这个方法行不通,会报pickle
序列化错误:
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f9d3e447f70>: attribute lookup <lambda> on __main__ failed
一个解决办法是使用functools.partial
,也就多一行代码:
from multiprocessing import Pool
from functools import partial
def mul(x, y):
return x * y
if __name__ == "__main__":
double = partial(mul, y=2)
with Pool() as pool:
print(pool.map(double, range(10)))
2.3、多参数函数
并行执行多参数函数需要联合使用starmap
和zip
:
from multiprocessing import Pool
def mul(x, y):
return x * y
if __name__ == "__main__":
with Pool() as pool:
print(pool.starmap(mul, zip(range(10), range(10)))
2.4、并行计算任意任务
上面三种都是并行执行同样的函数(作用于多个参数)。如果要并行执行任意任务,参考下面例子,也就几行代码:
from multiprocessing import Pool
def mul(x, y):
return x * y
if __name__ == "__main__":
with Pool() as pool:
res = []
for i in range(10):
res.append(pool.async_apply(mul, (i, i)))
print([r.get() for r in res])
Q. E. D.