# Developer Guide

## Directory hierarchy, or "Where are all the files?"

The main directory with the code is the `cg` subproject; the header files are in `cg/include/cg`, whereas the source files are in `cg/src`. The directory hierarchy in both of these is, for the most part, replicated. For example, one can find function declarations in `cg/include/cg/pid/eval_forces.h` and function definitions in `cg/src/cg/pid/eval_forces.cpp`. We will henceforth skip the `cg/include/cg` and `cg/src/cg` prefixes and just refer to files like `pid/eval_forces.h` and `pid/eval_forces.cpp` with the understanding that the `.h` files go into the `include/` directory, and that `.cpp` files go into the `src/` directory (for the most part, anyways).

## (Rough) Control flow of the program, Notes

- Most of the "action" happens in the `simul/` directories:
  - `program.h`, `program.cpp` - Program instance, invoked in the `src/main.cpp` file, creates a state instance and the execution system and runs it;
  - `state.h`, `state_io.cpp` - State instance. `cg::simul::state` contains the *entire* state of the program at a given point (so, things like positions of the monomers, velocities, amino acid types, Verlet list etc.) `state.cpp` contains setup functions, which are invoked during various phases of the simulation.
  - `thread.h`, `thread_setup.cpp`, `thread_main.cpp` - Thread system. The (OpenMP) thread instances modify the `cg::simul::state` instance, but themselves are *stateless* (for example, one can\* copy the thread instances) - this separation makes the code (hopefully) more modifiable, as new features can be added to the `state` and at best one needs to then add appropriate code to the `thread`. Another major advantage (compared to just having `state` instance with various functions) is that the threads can maintain ephemeral buffers, like thread-private force buffers, which avoids costly atomic operations. Next, it also makes it less dependent on using OpenMP, if one were to ditch it in the future.
    - `thread_main.cpp` contains most of the procedures. In particular, there is the finite state machine for the management of the phases of the simulation, as well as the core `advance_step` function implementation which does most of the heavy lifting, computation-wise.
    - `thread_setup.cpp` binds the state variables to the *kernels* used by the threads to perform operations.
  - The kernels (for example `pid::eval_forces` or `tether::eval_forces` or `nl::legacy_update`) are function-like objects which take in the state variables (or rather the pointers thereto) and perform a given operation. To answer "Why not just the functions?": (a) It allows for different modes of execution - for example, one can implement evaluation of forces sequentially (`operator()`) or via OpenMP asynchronous procedure (`omp_async()`) or just a slice of the given task (`for_slice(from, to)`), (b) One doesn't need to pass all the parameters all the time, just need to bind them during the instantiation of the kernel. (NOTE: One needs to take care to properly initialize said kernels, as the bugs may arise if some of the variables and parameters are uninitialized)
  - The runtime system (`simul/runtime`) is responsible for splitting the main work of the program, i.e. evaluation of forces, into appropriate *slices*, and assigning said slices deterministically to the appropriate threads, to balance the workload. One potential issue with using `#pragma for schedule(static) nowait` is that it's unclear how the work is distributed if there are many such for-loops one after another, as is the case in the `void thread::eval_forces()` - the runtime system makes it explicit. It also allows for a more elegant implementation of vectorized procedures (see `pid::eval_forces::vect_iter` and the source file `pid/eval_forces.cpp` for details);
  - Most of the rest of the code is either kernels, input system (`input`), utilities (`utils`) or file format stuff (`files`);
  - One unorthodox/custom feature is the use of a custom-built library for heterogeneous vectors and computations. This, from the developer's perspective, mostly appears through some strange-looking declarations of data, like in `pid/bundle.h`. The documentation for this part is unfinished, but if one wants to create new datatypes, one can roughly follow how it's done in all these various already-written examples and it should work out-of-the-box.