pmap is as simple as
>>> import pytraj as pt
>>> pt.pmap(pt.radgyr, traj, n_cores=4)
pytraj.parallel.multiprocessing_.pmap(func=None, traj=None, *args, **kwd)¶use python’s multiprocessing to accelerate calculation. Limited calculations.
| Parameters: | func : a pytraj’s methods or a list of string or simply as a cpptraj’ text traj : pytraj.TrajectoryIterator n_cores : int, number of cores to be used, default 2. Specify n_cores=-1 to use all available cores iter_options : dict, default {}
frame_indices : {None, array-like}, default None, optional
*args, **kwd: additional keywords |
|---|---|
| Returns: | out : OrderedDict |
See also
pytraj.pmap_mpi
Notes
Rule of thumbs: start with small number of frames (saying 10 frames), varying n_cores=1, 2, 3, 4 to see if the data makes sense or not.
There are two modes in this method, use pytraj’s methods (pytraj.rmsd, pytraj.radgyr, ...) or use cpptraj’s command text syntax (‘autoimage’, ‘rms’, ...)
If using cpptraj syntax:
pytraj only supports limited cpptraj's Actions (not Analysis, checm Amber15 manual
about Action and Analysis), say no to 'matrix', 'atomicfluct', ... or any action
that results output depending on the number of frames.
This method only benifits you if your calculation is quite long (saying few minutes to few hours). For calculation that takes less than 1 minutes, you won’t see the significant speed up (or even slower) since pytraj need to warm up and need to gather data when the calculation done.
The parallel cacluation is very simple, trajectory will be split (almost equal) to different chunk (n_chunks = n_cores), pytraj/cpptraj perform calculation for each chunk in each core and then send data back to master. Note that we are using Python’s built-in multiprocessing module, so you can use this method interactively in Ipython and ipython/jupyter notebook. This behavior is different from using MPI, in which you need to write a script, escaping ipython ession and type something like:
mpirun -n 4 python my_script.py
vs:
In [1]: pt.pmap(pt.radgyr, traj, n_cores=4)
Out[1]:
OrderedDict([('RoG_00000',
array([ 18.91114428, 18.93654996, 18.84969884, 18.90449256,
18.8568644 , 18.88917208, 18.9430491 , 18.88878079,
18.91669565, 18.87069722]))])
This is experimental method, you should expect its syntax, default output will be changed.
When sending Topology to different cores, pytraj will reload Topology from
traj.top.filename, so if you need to update Topology (in the fly), save it to disk and
reload before using pytraj.pmap
Examples
>>> import numpy as np
>>> import pytraj as pt
>>> traj = pt.load_sample_data('tz2')
>>> # use iter_options
>>> iter_options = {'autoimage': True, 'rmsfit': (0, '@CA')}
>>> data = pt.pmap(pt.mean_structure, traj, iter_options=iter_options)
>>> # cpptraj command style
>>> data = pt.pmap(['distance :3 :7', 'vector mask :3 :12'], traj, n_cores=4)
>>> # use reference. Need to explicitly use 'refindex', which is index of reflist
>>> data = pt.pmap(['rms @CA refindex 0'], traj, ref=[traj[3],], n_cores=3)
>>> data
OrderedDict([('RMSD_00001', array([ 2.68820312e-01, 3.11804885e-01, 2.58835452e-01,
9.10475988e-08, 2.93310737e-01, 4.10197322e-01,
3.96226694e-01, 3.66059215e-01, 3.90890362e-01,
4.89180497e-01]))])
>>> # use reference: if not want to use 'refindex', can use 'reference'
>>> # the advantage is you can not specify a list of reference
>>> data = pt.pmap(['rms @CA reference'], traj, ref=[traj[3],], n_cores=3)
>>> data
OrderedDict([('RMSD_00001', array([ 2.68820312e-01, 3.11804885e-01, 2.58835452e-01,
9.10475988e-08, 2.93310737e-01, 4.10197322e-01,
3.96226694e-01, 3.66059215e-01, 3.90890362e-01,
4.89180497e-01]))])
>>> # use different references. Need to explicitly use 'refindex', which is index of reflist
>>> # create a list of references
>>> reflist = traj[3], traj[4]
>>> # make sure to specify `refindex`
>>> # `refindex 0` is equal to `reflist[0]`
>>> # `refindex 1` is equal to `reflist[1]`
>>> data = pt.pmap(['rms @CA refindex 0', 'rms !@H= refindex 1'], traj, ref=reflist, n_cores=2)
>>> # convert to ndarray
>>> data_arr = pt.tools.dict_to_ndarray(data)
>>> # perform parallel calculation with given frame_indices
>>> traj = pt.datafiles.load_tz2()
>>> data = pt.pmap(pt.radgyr, traj, '@CA', frame_indices=range(10, 50), n_cores=4)
>>> # serial version
>>> data = pt.radgyr(traj, '@CA', frame_indices=range(10, 50))
pytraj.parallel.mpi.pmap_mpi(func, traj, *args, **kwd)¶parallel with MPI (mpi4py)
| Parameters: | func : a function traj : pytraj.TrajectoryIterator *args, **kwd: additional arguments |
|---|
Examples
$ # create test_radgyr.py file
$ cat > test_radgyr.py <<EOF
import pytraj as pt
from mpi4py import MPI
comm = MPI.COMM_WORLD
traj = pt.iterload('tz2.nc', 'tz2.parm7')
result_arr = pt.pmap_mpi(pt.radgyr, traj, "@CA")
if comm.rank == 0:
# save data to disk to read later by pytraj.read_pickle
# pt.to_pickle(result_arr, 'output.pk')
print(result_arr)
EOF
$ # run in parallel
$ mpirun -n 4 python ./test_radgyr.py
[array([ 8.10916061, 7.7643485 , 8.09693108, ..., 9.70825678,
9.3161563 , 8.86720964]), array([ 8.82037273, 8.89008289, 9.48540176, ..., 9.29585981,
9.53138062, 9.19155977]), array([ 9.13735723, 8.94651001, 8.97810478, ..., 7.68751186,
8.31361647, 7.83763754]), array([ 7.37423766, 7.05637263, 6.52135566, ..., 6.38061648,
6.24139008, 6.48994552])]