Firefly version 8.0 and above supports two new parallel modes of execution, namely the standard eXtreme Parallel (XP) and extended XP modes. The idea behind these modes is to efficiently utilize the multi-level parallelism inherent to many typical QC calculations by splitting the entire job into quasi-independent pieces which can be at high level processed in parallel.
For instance, in runs involving numerical gradients, the energy at each displaced geometry used in finite differencing can be calculated independently. If the underlying theoretical method is not programmed to run in parallel, it is then natural to use the standard XP mode to run multiple serial energy calculations at once and in parallel. If it is, any of three parallel modes (standard parallel, standard XP, and extended XP) could be used, with extended XP being the most scalable and most highly-performing alternative.
For example, if a typical single point energy computation for a particular combination of QC method and model system is scalable to e.g. 64 cores, and the numerical gradient code requires ca. 100 reevaluations of energy, the calculations of numerical gradients in extended XP modes would be scalable up to 64*100=6400 cores and would be extremely fast.
The current implementation of XP modes in Firefly supports two levels of parallelism. This will be changed in the future to support the arbitrary number of parallelism levels provided there will be the need in this. Presently, any job involving numerical gradients can be run in either standard XP or extended XP modes, or both. Other jobs types are not currently allowed to run in XP modes. This limitation will be relaxed in the future by allowing semi-numerical Hessian computations, semi-analytic Raman activities computations, surface scans and some other types of jobs to run in XP modes as well.
The XP or extended XP modes are invoked by running Firefly in parallel using dedicated command line options. There is no way to force XP modes using information in the input file as preparations for these modes of operation need to be done at the very early stages of Firefly's initialization. Namely, to launch Firefly in standard XP mode, use the -xp command line option. Similarly, to launch Firefly in extended XP mode, the -xp:N (or, alternatively, -xp=N) command line options can be used. Here, N is the number of processes belonging to each process group. For instance, launching Firefly on 32 cores using -xp=4 (or -xp:4) command line option will create eight groups of processes, with each group consisting of four separate processes.
A note on P2P interface and dynamic load balancing.
Firefly's P2P interface and dynamic load balancing over P2P can be used in any of three parallel modes of execution. Working in the extended XP mode, Firefly supports two levels of P2P communications and dynamic load balancing. The high-level one is global for the entire parallel Firefly process, and serves primarily for the communications between the local masters of separate groups. For instance, high-level dynamic load balancing is used to distribute the high-level jobs between individual process groups. The global P2P interface is controlled by the usual $p2p control group of the input file. Each separate group of processes can be interconnected via its own local P2P interface which is virtually identical to the global one but is limited in its scope to the members of its group only. The behavior of the second-level, local P2P interfaces is controlled using the new $lp2p control group of the input file. The $lp2p input group has exactly the same keywords as the standard $p2p input group.
Last updated: July 30, 2012