Developer Guide
Directory hierarchy, or “Where are all the files?”
The main directory with the code is the cg
subproject; the header files are in cg/include/cg
, whereas the source files are in cg/src
. The directory hierarchy in both of these is, for the most part, replicated. For example, one can find function declarations in cg/include/cg/pid/eval_forces.h
and function definitions in cg/src/cg/pid/eval_forces.cpp
. We will henceforth skip the cg/include/cg
and cg/src/cg
prefixes and just refer to files like pid/eval_forces.h
and pid/eval_forces.cpp
with the understanding that the .h
files go into the include/
directory, and that .cpp
files go into the src/
directory (for the most part, anyways).
(Rough) Control flow of the program, Notes
Most of the “action” happens in the
simul/
directories:program.h
,program.cpp
- Program instance, invoked in thesrc/main.cpp
file, creates a state instance and the execution system and runs it;state.h
,state_io.cpp
- State instance.cg::simul::state
contains the entire state of the program at a given point (so, things like positions of the monomers, velocities, amino acid types, Verlet list etc.)state.cpp
contains setup functions, which are invoked during various phases of the simulation.thread.h
,thread_setup.cpp
,thread_main.cpp
- Thread system. The (OpenMP) thread instances modify thecg::simul::state
instance, but themselves are stateless (for example, one can* copy the thread instances) - this separation makes the code (hopefully) more modifiable, as new features can be added to thestate
and at best one needs to then add appropriate code to thethread
. Another major advantage (compared to just havingstate
instance with various functions) is that the threads can maintain ephemeral buffers, like thread-private force buffers, which avoids costly atomic operations. Next, it also makes it less dependent on using OpenMP, if one were to ditch it in the future.thread_main.cpp
contains most of the procedures. In particular, there is the finite state machine for the management of the phases of the simulation, as well as the coreadvance_step
function implementation which does most of the heavy lifting, computation-wise.thread_setup.cpp
binds the state variables to the kernels used by the threads to perform operations.
The kernels (for example
pid::eval_forces
ortether::eval_forces
ornl::legacy_update
) are function-like objects which take in the state variables (or rather the pointers thereto) and perform a given operation. To answer “Why not just the functions?”: (a) It allows for different modes of execution - for example, one can implement evaluation of forces sequentially (operator()
) or via OpenMP asynchronous procedure (omp_async()
) or just a slice of the given task (for_slice(from, to)
), (b) One doesn’t need to pass all the parameters all the time, just need to bind them during the instantiation of the kernel. (NOTE: One needs to take care to properly initialize said kernels, as the bugs may arise if some of the variables and parameters are uninitialized)The runtime system (
simul/runtime
) is responsible for splitting the main work of the program, i.e. evaluation of forces, into appropriate slices, and assigning said slices deterministically to the appropriate threads, to balance the workload. One potential issue with using#pragma for schedule(static) nowait
is that it’s unclear how the work is distributed if there are many such for-loops one after another, as is the case in thevoid thread::eval_forces()
- the runtime system makes it explicit. It also allows for a more elegant implementation of vectorized procedures (seepid::eval_forces::vect_iter
and the source filepid/eval_forces.cpp
for details);Most of the rest of the code is either kernels, input system (
input
), utilities (utils
) or file format stuff (files
);One unorthodox/custom feature is the use of a custom-built library for heterogeneous vectors and computations. This, from the developer’s perspective, mostly appears through some strange-looking declarations of data, like in
pid/bundle.h
. The documentation for this part is unfinished, but if one wants to create new datatypes, one can roughly follow how it’s done in all these various already-written examples and it should work out-of-the-box.