# Developer Notes ## Style Guide Please check your code for style issues by running `make format` which uses `clang-format`. All style fixes provided by `make format` should be committed. In addition to those automatically enforced style rules, Ratel tends to follow the following code style conventions: - Variable names: `snake_case` - Struct members: `snake_case` - Function names: `PascalCase` - Type names: `PascalCase` - Constant names: `CAPS_SNAKE_CASE` In general, variable and function names should avoid abbreviations and err on the side of verbosity to improve readability. Prefer the use of PETSc functions, e.g. `PetscSNPrintf`, over their standard library versions, e.g. `snprintf`. Also, documentation files should have one sentence per line to help make git diffs clearer and less disruptive. ## Function Conventions ### Naming All functions in the Ratel library should be prefixed by `Ratel` and take a `Ratel` object (or a struct containing a `Ratel` object) as its first argument. If a function takes, for example, a `RatelMaterial` as its first argument, then it should be prefixed with `RatelMaterial`. ### Style Functions should adhere mostly to the PETSc function style, specifically: 1. All local variables of a particular type (for example, PetscInt) should be listed on the same line if possible; otherwise, they should be listed on adjacent lines. For example, ```c // Correct PetscInt a, b, c; PetscInt *d, *e; PetscInt **f; // Incorrect PetscInt a, b, c, *d, *e, **f; ``` 2. Local variables should be initialized in their declaration when possible. 3. Nearly all functions should have a return type of `PetscErrorCode` to allow for error checking. 4. Nearly all functions must start with a single blank line between the local variable declarations followed by `RatelLogFunctionBegin(ratel);`. This requirement differs from the PETSc guidelines, as `RatelLogFunctionBegin(ratel);` includes `PetscFunctionBeginUser;` macro and automatic logging. There are a few exceptions: - In some cases, a function may need to make function calls or do some other logic before safely accessing the Ratel context. In those cases, the function should start with `PetscFunctionBeginUser;` and then use the `RatelLogFunctionBeginLate(ratel);` macro once the Ratel context is available. - If a function is frequently called in a loop or would otherwise be unhelpful to log, it is acceptable to begin with `PetscFunctionBeginUser;` and end with `PetscFunctionReturn(PETSC_SUCCESS)`. - Any `*View` function that may be used with logging should begin with `PetscFunctionBeginUser;` and end with `PetscFunctionReturn(PETSC_SUCCESS)`. 5. Ratel functions which begin with `RatelLogFunctionBegin(ratel);` or use `RatelLogFunctionBeginLate(ratel);` must end with `RatelLogFunctionReturn(ratel, PETSC_SUCCESS);`. 6. All PETSc or Ratel function calls must have their return value checked for errors using the `PetscCall()` macro. This should be wrapped around the function in question. 7. libCEED function calls must have their return value checked for errors using the `RatelCallCeed(ratel, ...)` macro. This should be wrapped around the function in question. 8. In Ratel functions, variables must be declared at the beginning of the code block (C90 style), never mixed in with code. However, when variables are only used in a limited scope, it is encouraged to declare them in that scope. 9. Do not put a blank line immediately after `RatelLogFunctionBegin(ratel);` or a blank line immediately before `RatelLogFunctionReturn(ratel, PETSC_SUCCESS);`. 10. All Ratel functions must use Doxygen comment blocks before their *definition* (not declaration). The block should begin with `/**` and end with `**/`, each on their own line. The block should be indented by two spaces and should contain an `@brief` tag and description, a newline, a line stating whether the function is collective, a newline, `@param` tags for each parameter, a newline, and a `@return` line formatted exactly as in the example below. All parameter lines in the Doxygen block should be formatted such that parameter names and descriptions are aligned. There should be a exactly two spaces between `@param[dir]` (where `dir` is `in`, `out`, or `in,out`) and the parameter name for the closest pair, as well as between the parameter name and description. For example: ```c /** @brief Brief description of function. [C|Not c]ollective across MPI processes. @param[in] ratel `Ratel` context @param[in] my_int_param An integer that is input only @param[in,out] my_mutable_int_param An integer which is input/output @param[out] my_output_array An array that will be filled with values @return An error code: 0 - success, otherwise - failure **/ PetscErrorCode RatelFunctionThatDoesSomething(Ratel ratel, PetscInt my_int_param, PetscInt *my_mutable_int_param, PetscScalar my_output_array[]) { ``` 11. Function declarations should include parameter names, which must exactly match those in the function definition. 12. External functions, i.e. those used in tests or examples, must have their *declarations* prefixed with `RATEL_EXTERN`. All other functions should have their *declarations* prefixed with `RATEL_INTERN`. Function *definitions* should have neither. ## Logging Ratel supports logging at four levels: trace, debug, info, and warn. Each level has the following usage: - **trace**: Log detailed information at a sub-function granularity. Most libCEED objects should be logged at this level, as well as most log statement which occur inside of functions. Any information necessary to understand the internal data structures and operations being performed on it should be logged at this level. - **debug**: Log information useful for debugging programs. Ratel function logging (`RatelFunctionName...` and `RatelFunctionName Success!`) is performed at this level. Any information necessary to understand the control flow of the program should be logged at this level. If the environment variables `RATEL_DEBUG`, `DEBUG`, or `DBG` are set to `1`, the default log level is set to debug. - **info**: Log general information about what is being done in the program. This is rarely used currently. Any information which would be useful for an end user that is not viewable though other command-line arguments should be logged at this level. For example, short convergence summaries for MPM swarm-to-mesh projection solves are logged at this level. - **warn**: Log loud warnings about unsupported behavior or poor configuration. Any information that an end user should know must be logged at this level. This includes warning about the use of experimental or comparison-only features, as well as unsupported, but technically allowable, configuration choices. This is the default logging level, unless the `-quiet` flag is used to suppress all logging. There are corresponding logging macros for each level: `RatelLogTrace`, `RatelLogDebug`, `RatelLogInfo`, and `RatelLogWarn`. These macros each take the `Ratel` context followed by the arguments to `printf` as arguments. Previously, Ratel used macros `RatelDebug` and `RatelDebug256` for plain and colored debug output. These macros are now deprecated. ### Logging Complex Data Ratel also supports logging data which cannot be natively passed to `printf`. The preferred method for such logging is to define a callback function for the data to a viewer. The signature for such a function is: ```c PetscErrorCode MyViewerFunction(..., PetscViewer viewer); ``` where `...` can be any number of arguments needed for the log function. The callback function should use `PetscViewerASCIIPrintf()` calls to log whatever data is required. The logging is then performed using one of the macros: `RatelLogGenericTrace`, `RatelLogGenericDebug`, `RatelLogGenericInfo`, and `RatelLogGenericWarn`. For example, ```c PetscErrorCode LogArray(const PetscScalar array[], PetscInt length, PetscViewer viewer) { PetscFunctionBeginUser; PetscCall(PetscViewerASCIIPrintf(viewer, "[")); PetscCall(PetscViewerASCIIUseTabs(viewer, PETSC_FALSE)); for (PetscInt i = 0; i < length; i++) PetscCall(PetscViewerASCIIPrintf(viewer, "%g%s", array[i], i == length - 1 ? "]\n" : ", ")); PetscCall(PetscViewerASCIIUseTabs(viewer, PETSC_TRUE)); PetscFunctionReturn(PETSC_SUCCESS); } // later in code PetscInt array_len = 10; PetscScalar my_array[10] = {0} PetscCall(RatelLogGenericDebug(ratel, LogArray, my_array, array_len)); ``` This interface *should not* be used for PETSc objects, as they require special handling due to their MPI communicators. Instead, use the dedicated `RatelLogPetsc` interface: `RatelLogPetscTrace`, `RatelLogPetscDebug`, `RatelLogPetscInfo`, and `RatelLogPetscWarn`. For example, ```c KSP ksp; // Set up KSP... PetscCall(RatelLogPetscTrace(ratel, (PetscObject)ksp)); ``` Note, the PETSc data structure should be cast with `(PetscObject)`. Similarly, helper logging macros are functions for printing libCEED objects are provided as `RatelLogCeed` (takes `level` as argument), `RatelLogCeedTrace`, `RatelLogCeedDebug`, `RatelLogCeedInfo`, and `RatelLogCeedWarn`. For example, ```c CeedOperator op; // Set up operator PetscCall(RatelLogCeedTrace(ratel, op)); ``` ## Clang-tidy Please check your code for common issues by running `make tidy` which uses the `clang-tidy` utility included in recent releases of Clang. This tool is much slower than actual compilation (`make -j` parallelism helps). All issues reported by `make tidy` should be fixed. ## Header Files Header inclusion for source files should follow the principal of 'include what you use' rather than relying upon transitive `#include` to define all symbols. Every symbol that is used in the source file `foo.c` should be defined in `foo.c`, `foo.h`, or in a header file ``#include``d in one of these two locations. Please check your code by running the tool `include-what-you-use` to see recommendations for changes to your source. Most issues reported by `include-what-you-use` should be fixed; however this rule is flexible to account for differences in header file organization in external libraries. Header files should be listed in alphabetical order, with installed headers preceding local headers. ```c #include #include #include #include #include "include/ratel.h" ``` Header guard macros should be done using `#pragma once`. This must be the very first non-comment line of the file. ## `restrict` Semantics QFunction arguments can be assumed to have `restrict` semantics. That is, each input and output array must reside in distinct memory without overlap. ## Continuous Integration (CI) and Testing Whenever a merge request is created, the Ratel test suite is run against a variety of hardware and software configurations. If one of the tests in the CI pipeline fails, you should look at the error in the CI job log. The log will provide the command which failed and relevant details (such as `diff` output) to help debug the issue. You can also see the CGNS or CSV output of a failed run, if applicable, by selecting "Download" on the Job Artifacts (to the right of the log). Artifacts from failed runs will be contained in the `test_failure_artifacts` folder inside the artifacts archive. ## Adding New Boundary Conditions Each boundary condition (e.g. pressure, slip, traction, flux, etc.) gets its own folder `include/ratel/boundary/[bc-name]` containing: - `params.h`: Necessary data structures, must be JiT-safe - `qf.h`: QFunctions for setup (optional) and action(s) of the boundary condition, (obviously) must be JiT-safe - `[bc-name].h`: Header for functions specific to the boundary condition, which will in the future be standardized into a uniform interface. This header should include `./params.h`, but NOT include `./qf.h`. An example for `[bc-name].h` for non-Dirichlet BCs is `include/ratel/boundary/traction/traction.h`: ```c /// @file /// Ratel Traction Boundary Conditions Functions #pragma once #include #include "params.h" RATEL_INTERN PetscErrorCode RatelBoundaryTractionDataFromOptions(Ratel ratel, PetscInt i, RatelBCTractionParams *params_traction); RATEL_INTERN PetscErrorCode RatelBoundaryTractionSetupEnergySuboperators(Ratel ratel, DM dm_energy, CeedOperator op_external_energy); RATEL_INTERN PetscErrorCode RatelBoundaryTractionSetupSuboperators(Ratel ratel, DM dm, CeedOperator op_residual); ``` For Dirichlet BCs, see `include/ratel/boundary/clamp/clamp.h`: ```c /// @file /// Ratel Clamp Boundary Conditions Functions #pragma once #include #include "params.h" RATEL_INTERN PetscErrorCode RatelBoundaryClampDataFromOptions(Ratel ratel, PetscInt i, PetscInt j, RatelBCClampParams *params_clamp); RATEL_INTERN PetscErrorCode RatelBoundaryClampSetupDirichletSuboperators(Ratel ratel, DM dm, PetscBool incremental, CeedOperator op_dirichlet); ``` The functions that should exist in the header are a subset of the following, along with any needed helper functions: - `RatelBoundary[bc-name]DataFromOptions`, reads command-line options into the relevant data structure(s) defined in `params.h`, may also take integers for `face_id` and/or `field_index` - `RatelBoundary[bc-name]SetupDirichletSuboperators`, for Dirichlet boundaries - `RatelBoundary[bc-name]SetupSuboperators`, for adding suboperators to any of `op_residual_u`, `op_residual_ut`, `op_residual_utt`, `op_jacobian` - `RatelBoundary[bc-name]SetupMaterialSuboperators`, like `*SetupSuboperators`, but for BCs with explicit material dependence (right now only Nitsche contact). The first argument should be `RatelMaterial` instead of `Ratel`. - `RatelBoundary[bc-name]SetupEnergySuboperators`, for adding suboperators to the external energy operator, if applicable These will eventually be changed to a unified interface under a `RatelBoundary` data structure. The implementations for these headers go in the corresponding `src/boundary/[bc-name].c` file. The existing header `ratel-boundary.h` now contains utility functions for setting up boundary conditions. Most critical of these are: - `RatelBoundarySetupSuboperators`: all BCs which define a `RatelBoundary[bc-name]SetupSuboperators` should have those called here. If a new suboperator may be added to `op_jacobian`, special care must be taken to set up the multigrid level (see below). - `RatelBoundarySetupMaterialSuboperators`: all BCs which define a `RatelBoundary[bc-name]SetupMaterialSuboperators` should be called here. See below for details on `op_jacobian` suboperators. - `RatelBoundarySetupDirichletSuboperators`: all BCs which define a `RatelBoundary[bc-name]SetupDirichletSuboperators` should be called here - `RatelBoundarySetupEnergySuboperators`: all BCs which define a `RatelBoundary[bc-name]SetupEnergySuboperators` should be called here ### BCs with Jacobian suboperators There is now a simple process for setting up multigrid levels for BCs with Jacobian suboperators. Now, both penalty contact and pressure BCs use `RatelMaterialSetBoundaryJacobianMultigridInfo` and the new `RatelBoundarySetupMultigridLevel_Standard` and `RatelBoundarySetupMultigridLevel_CellToFace` helpers to ensure that MG levels are set up for all Jacobian suboperators. As an implementation detail, the Jacobian indices for these boundaries are considered to live on the last material, but this choice is arbitrary and not relevant to the process of adding new boundary conditions. For non-material dependent BCs, the call to `RatelBoundary[bc-name]SetupSuboperators` should be wrapped like: ```c const RatelMaterial last_material = ratel->materials[ratel->num_materials - 1]; CeedInt old_num_jacobian_sub_operators, new_num_jacobian_sub_operators; RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &old_num_jacobian_sub_operators)); PetscCall(RatelBoundary[bc-name]SetupSuboperators(ratel, dm, op_residual_u, op_jacobian)); RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &new_num_jacobian_sub_operators)); // -- Set Jacobian info on last material PetscCall(RatelMaterialSetBoundaryJacobianMultigridInfo(last_material, op_jacobian, old_num_jacobian_sub_operators, new_num_jacobian_sub_operators, &RatelBoundarySetupMultigridLevel_Standard)); ``` For material-dependent BCs, the call to `RatelBoundary[bc-name]SetupMaterialSuboperators` should be wrapped like: ```c RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &old_num_jacobian_sub_operators)); PetscCall(RatelBoundary[bc-name]SetupMaterialSuboperators(material, u_dot, op_residual_u, op_jacobian)); RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &new_num_jacobian_sub_operators)); // Set Jacobian info for material PetscCall(RatelMaterialSetBoundaryJacobianMultigridInfo(material, op_jacobian, old_num_jacobian_sub_operators, new_num_jacobian_sub_operators, &RatelBoundarySetupMultigridLevel_CellToFace)); ``` ## Adding New Initial Conditions Ratel supports non-zero initial condition for linear poromechanincs MMS and as a constant value for other cases. If user needs to add non-zero initial condition as function of coordinate, a function with the same structure as `RatelSetupInitialConditionMMS` needs to be added with its own Qfucntion expressing the initial condition of each active field.