pytraj.parallel

pmap is as simple as

>>> import pytraj as pt
>>> pt.pmap(pt.radgyr, traj, n_cores=4)
pytraj.parallel.multiprocessing_.pmap(func=None, traj=None, *args, **kwd)

use python’s multiprocessing to accelerate calculation. Limited calculations.

Parameters:

func : a pytraj’s methods or a list of string or simply as a cpptraj’ text

traj : pytraj.TrajectoryIterator

n_cores : int, number of cores to be used, default 2. Specify n_cores=-1 to use all available cores

iter_options : dict, default {}

Specify trajectory iterating option. This will be done before calling func.

frame_indices : {None, array-like}, default None, optional

if provided, pytraj will split this frame_indices into different chunks and let cpptraj perform calculation for specific indices. frame_indices must be pickable so is can be sent to different cores.

*args, **kwd: additional keywords

Returns:

out : OrderedDict

See also

pytraj.pmap_mpi

Notes

  • If you not sure about parallel’s results, you should compare the output to serial run.
  • This is absolutely experimental. The syntax might be changed in future.

Rule of thumbs: start with small number of frames (saying 10 frames), varying n_cores=1, 2, 3, 4 to see if the data makes sense or not.

There are two modes in this method, use pytraj’s methods (pytraj.rmsd, pytraj.radgyr, ...) or use cpptraj’s command text syntax (‘autoimage’, ‘rms’, ...)

If using cpptraj syntax:

pytraj only supports limited cpptraj's Actions (not Analysis, checm Amber15 manual
about Action and Analysis), say no  to 'matrix', 'atomicfluct', ... or any action
that results output depending on the number of frames.

This method only benifits you if your calculation is quite long (saying few minutes to few hours). For calculation that takes less than 1 minutes, you won’t see the significant speed up (or even slower) since pytraj need to warm up and need to gather data when the calculation done.

The parallel cacluation is very simple, trajectory will be split (almost equal) to different chunk (n_chunks = n_cores), pytraj/cpptraj perform calculation for each chunk in each core and then send data back to master. Note that we are using Python’s built-in multiprocessing module, so you can use this method interactively in Ipython and ipython/jupyter notebook. This behavior is different from using MPI, in which you need to write a script, escaping ipython ession and type something like:

mpirun -n 4 python my_script.py

vs:

In [1]: pt.pmap(pt.radgyr, traj, n_cores=4)
Out[1]:
OrderedDict([('RoG_00000',
              array([ 18.91114428,  18.93654996,  18.84969884,  18.90449256,
                      18.8568644 ,  18.88917208,  18.9430491 ,  18.88878079,
                      18.91669565,  18.87069722]))])

This is experimental method, you should expect its syntax, default output will be changed.

When sending Topology to different cores, pytraj will reload Topology from traj.top.filename, so if you need to update Topology (in the fly), save it to disk and reload before using pytraj.pmap

Examples

>>> import numpy as np
>>> import pytraj as pt
>>> traj = pt.load_sample_data('tz2')
>>> # use iter_options
>>> iter_options = {'autoimage': True, 'rmsfit': (0, '@CA')}
>>> data = pt.pmap(pt.mean_structure, traj, iter_options=iter_options)
>>> # cpptraj command style
>>> data = pt.pmap(['distance :3 :7', 'vector mask :3 :12'], traj, n_cores=4)
>>> # use reference. Need to explicitly use 'refindex', which is index of reflist
>>> data = pt.pmap(['rms @CA refindex 0'], traj, ref=[traj[3],], n_cores=3)
>>> data
OrderedDict([('RMSD_00001', array([  2.68820312e-01,   3.11804885e-01,   2.58835452e-01,
         9.10475988e-08,   2.93310737e-01,   4.10197322e-01,
         3.96226694e-01,   3.66059215e-01,   3.90890362e-01,
         4.89180497e-01]))])
>>> # use reference: if not want to use 'refindex', can use 'reference'
>>> # the advantage is you can not specify a list of reference
>>> data = pt.pmap(['rms @CA reference'], traj, ref=[traj[3],], n_cores=3)
>>> data
OrderedDict([('RMSD_00001', array([  2.68820312e-01,   3.11804885e-01,   2.58835452e-01,
         9.10475988e-08,   2.93310737e-01,   4.10197322e-01,
         3.96226694e-01,   3.66059215e-01,   3.90890362e-01,
         4.89180497e-01]))])
>>> # use different references. Need to explicitly use 'refindex', which is index of reflist
>>> # create a list of references
>>> reflist = traj[3], traj[4]
>>> # make sure to specify `refindex`
>>> # `refindex 0` is equal to `reflist[0]`
>>> # `refindex 1` is equal to `reflist[1]`
>>> data = pt.pmap(['rms @CA refindex 0', 'rms !@H= refindex 1'], traj, ref=reflist, n_cores=2)
>>> # convert to ndarray
>>> data_arr = pt.tools.dict_to_ndarray(data)
>>> # perform parallel calculation with given frame_indices
>>> traj = pt.datafiles.load_tz2()
>>> data = pt.pmap(pt.radgyr, traj, '@CA', frame_indices=range(10, 50), n_cores=4)
>>> # serial version
>>> data = pt.radgyr(traj, '@CA', frame_indices=range(10, 50))
pytraj.parallel.mpi.pmap_mpi(func, traj, *args, **kwd)

parallel with MPI (mpi4py)

Parameters:

func : a function

traj : pytraj.TrajectoryIterator

*args, **kwd: additional arguments

Examples

$ # create test_radgyr.py file
$ cat > test_radgyr.py <<EOF
import pytraj as pt
from mpi4py import MPI
comm = MPI.COMM_WORLD

traj = pt.iterload('tz2.nc', 'tz2.parm7')

result_arr = pt.pmap_mpi(pt.radgyr, traj, "@CA")

if comm.rank == 0:
    # save data to disk to read later by pytraj.read_pickle
    # pt.to_pickle(result_arr, 'output.pk')
    print(result_arr)
EOF

$ # run in parallel
$ mpirun -n 4 python ./test_radgyr.py
[array([ 8.10916061,  7.7643485 ,  8.09693108, ...,  9.70825678,
        9.3161563 ,  8.86720964]), array([ 8.82037273,  8.89008289,  9.48540176, ...,  9.29585981,
        9.53138062,  9.19155977]), array([ 9.13735723,  8.94651001,  8.97810478, ...,  7.68751186,
        8.31361647,  7.83763754]), array([ 7.37423766,  7.05637263,  6.52135566, ...,  6.38061648,
        6.24139008,  6.48994552])]