Developer Notes¶

Style Guide¶

Please check your code for style issues by running

make format

which uses clang-format. All style fixes provided by make format should be committed.

In addition to those automatically enforced style rules, Ratel tends to follow the following code style conventions:

Variable names: snake_case
Struct members: snake_case
Function names: PascalCase
Type names: PascalCase
Constant names: CAPS_SNAKE_CASE

In general, variable and function names should avoid abbreviations and err on the side of verbosity to improve readability.

Prefer the use of PETSc functions, e.g. PetscSNPrintf, over their standard library versions, e.g. snprintf.

Also, documentation files should have one sentence per line to help make git diffs clearer and less disruptive.

Function Conventions¶

Naming¶

All functions in the Ratel library should be prefixed by Ratel and take a Ratel object (or a struct containing a Ratel object) as its first argument. If a function takes, for example, a RatelMaterial as its first argument, then it should be prefixed with RatelMaterial.

Style¶

Functions should adhere mostly to the PETSc function style, specifically:

All local variables of a particular type (for example, PetscInt) should be listed on the same line if possible; otherwise, they should be listed on adjacent lines. For example,

// Correct
PetscInt   a, b, c;
PetscInt  *d, *e;
PetscInt **f;

// Incorrect
PetscInt a, b, c, *d, *e, **f;

Local variables should be initialized in their declaration when possible.
Nearly all functions should have a return type of PetscErrorCode to allow for error checking.
Nearly all functions must start with a single blank line between the local variable declarations followed by RatelLogFunctionBegin(ratel);. This requirement differs from the PETSc guidelines, as RatelLogFunctionBegin(ratel); includes PetscFunctionBeginUser; macro and automatic logging. There are a few exceptions:
- In some cases, a function may need to make function calls or do some other logic before safely accessing the Ratel context. In those cases, the function should start with PetscFunctionBeginUser; and then use the RatelLogFunctionBeginLate(ratel); macro once the Ratel context is available.
- If a function is frequently called in a loop or would otherwise be unhelpful to log, it is acceptable to begin with PetscFunctionBeginUser; and end with PetscFunctionReturn(PETSC_SUCCESS).
- Any *View function that may be used with logging should begin with PetscFunctionBeginUser; and end with PetscFunctionReturn(PETSC_SUCCESS).
Ratel functions which begin with RatelLogFunctionBegin(ratel); or use RatelLogFunctionBeginLate(ratel); must end with RatelLogFunctionReturn(ratel, PETSC_SUCCESS);.
All PETSc or Ratel function calls must have their return value checked for errors using the PetscCall() macro. This should be wrapped around the function in question.
libCEED function calls must have their return value checked for errors using the RatelCallCeed(ratel, ...) macro. This should be wrapped around the function in question.
In Ratel functions, variables must be declared at the beginning of the code block (C90 style), never mixed in with code. However, when variables are only used in a limited scope, it is encouraged to declare them in that scope.
Do not put a blank line immediately after RatelLogFunctionBegin(ratel); or a blank line immediately before RatelLogFunctionReturn(ratel, PETSC_SUCCESS);.
All Ratel functions must use Doxygen comment blocks before their definition (not declaration). The block should begin with /** and end with **/, each on their own line. The block should be indented by two spaces and should contain an @brief tag and description, a newline, a line stating whether the function is collective, a newline, @param tags for each parameter, a newline, and a @return line formatted exactly as in the example below. All parameter lines in the Doxygen block should be formatted such that parameter names and descriptions are aligned. There should be a exactly two spaces between @param[dir] (where dir is in, out, or in,out) and the parameter name for the closest pair, as well as between the parameter name and description. For example:

/**
  @brief Brief description of function.

  [C|Not c]ollective across MPI processes.

  @param[in]      ratel                 `Ratel` context
  @param[in]      my_int_param          An integer that is input only
  @param[in,out]  my_mutable_int_param  An integer which is input/output
  @param[out]     my_output_array       An array that will be filled with values

  @return An error code: 0 - success, otherwise - failure
**/
PetscErrorCode RatelFunctionThatDoesSomething(Ratel ratel, PetscInt my_int_param, PetscInt *my_mutable_int_param, PetscScalar my_output_array[]) {

Function declarations should include parameter names, which must exactly match those in the function definition.
External functions, i.e. those used in tests or examples, must have their declarations prefixed with RATEL_EXTERN. All other functions should have their declarations prefixed with RATEL_INTERN. Function definitions should have neither.

Logging¶

Ratel supports logging at four levels: trace, debug, info, and warn. Each level has the following usage:

trace: Log detailed information at a sub-function granularity. Most libCEED objects should be logged at this level, as well as most log statement which occur inside of functions. Any information necessary to understand the internal data structures and operations being performed on it should be logged at this level.
debug: Log information useful for debugging programs. Ratel function logging (RatelFunctionName... and RatelFunctionName Success!) is performed at this level. Any information necessary to understand the control flow of the program should be logged at this level. If the environment variables RATEL_DEBUG, DEBUG, or DBG are set to 1, the default log level is set to debug.
info: Log general information about what is being done in the program. This is rarely used currently. Any information which would be useful for an end user that is not viewable though other command-line arguments should be logged at this level. For example, short convergence summaries for MPM swarm-to-mesh projection solves are logged at this level.
warn: Log loud warnings about unsupported behavior or poor configuration. Any information that an end user should know must be logged at this level. This includes warning about the use of experimental or comparison-only features, as well as unsupported, but technically allowable, configuration choices. This is the default logging level, unless the -quiet flag is used to suppress all logging.

There are corresponding logging macros for each level: RatelLogTrace, RatelLogDebug, RatelLogInfo, and RatelLogWarn. These macros each take the Ratel context followed by the arguments to printf as arguments.

Previously, Ratel used macros RatelDebug and RatelDebug256 for plain and colored debug output. These macros are now deprecated.

Logging Complex Data¶

Ratel also supports logging data which cannot be natively passed to printf. The preferred method for such logging is to define a callback function for the data to a viewer. The signature for such a function is:

PetscErrorCode MyViewerFunction(..., PetscViewer viewer);

where ... can be any number of arguments needed for the log function. The callback function should use PetscViewerASCIIPrintf() calls to log whatever data is required.

The logging is then performed using one of the macros: RatelLogGenericTrace, RatelLogGenericDebug, RatelLogGenericInfo, and RatelLogGenericWarn. For example,

PetscErrorCode LogArray(const PetscScalar array[], PetscInt length, PetscViewer viewer) {
  PetscFunctionBeginUser;
  PetscCall(PetscViewerASCIIPrintf(viewer, "["));
  PetscCall(PetscViewerASCIIUseTabs(viewer, PETSC_FALSE));
  for (PetscInt i = 0; i < length; i++) PetscCall(PetscViewerASCIIPrintf(viewer, "%g%s", array[i], i == length - 1 ? "]\n" : ", "));
  PetscCall(PetscViewerASCIIUseTabs(viewer, PETSC_TRUE));
  PetscFunctionReturn(PETSC_SUCCESS);
}

// later in code
PetscInt    array_len    = 10;
PetscScalar my_array[10] = {0}

PetscCall(RatelLogGenericDebug(ratel, LogArray, my_array, array_len));

Helper functions for printing libCEED objects are provided as RatelLogFnCeed[Type], where [Type] is Operator, QFunction, QFunctionContext, Basis, or ElemRestriction. For example,

CeedOperator op;

// Set up operator
PetscCall(RatelLogGenericTrace(ratel, RatelLogFnCeedOperator, op));

This interface should not be used for PETSc objects, as they require special handling due to their MPI communicators. Instead, use the dedicated RatelLogPetsc interface: RatelLogPetscTrace, RatelLogPetscDebug, RatelLogPetscInfo, and RatelLogPetscWarn. For example,

KSP ksp;

// Set up KSP...
PetscCall(RatelLogPetscTrace(ratel, (PetscObject)ksp));

Note, the PETSc data structure should be cast with (PetscObject).

Clang-tidy¶

Please check your code for common issues by running

make tidy

which uses the clang-tidy utility included in recent releases of Clang. This tool is much slower than actual compilation (make -j parallelism helps). All issues reported by make tidy should be fixed.

Header Files¶

Header inclusion for source files should follow the principal of ‘include what you use’ rather than relying upon transitive #include to define all symbols.

Every symbol that is used in the source file foo.c should be defined in foo.c, foo.h, or in a header file #included in one of these two locations. Please check your code by running the tool include-what-you-use to see recommendations for changes to your source. Most issues reported by include-what-you-use should be fixed; however this rule is flexible to account for differences in header file organization in external libraries.

Header files should be listed in alphabetical order, with installed headers preceding local headers.

#include <ceed.h>
#include <petsc.h>
#include <stdbool.h>
#include <string.h>

#include "include/ratel.h"

Header guard macros should be done using #pragma once. This must be the very first non-comment line of the file.

`restrict` Semantics¶

QFunction arguments can be assumed to have restrict semantics. That is, each input and output array must reside in distinct memory without overlap.

Continuous Integration (CI) and Testing¶

Whenever a merge request is created, the Ratel test suite is run against a variety of hardware and software configurations. If one of the tests in the CI pipeline fails, you should look at the error in the CI job log. The log will provide the command which failed and relevant details (such as diff output) to help debug the issue. You can also see the CGNS or CSV output of a failed run, if applicable, by selecting “Download” on the Job Artifacts (to the right of the log). Artifacts from failed runs will be contained in the test_failure_artifacts folder inside the artifacts archive.

Adding New Boundary Conditions¶

Each boundary condition (e.g. pressure, slip, traction, flux, etc.) gets its own folder include/ratel/boundary/[bc-name] containing:

params.h: Necessary data structures, must be JiT-safe
qf.h: QFunctions for setup (optional) and action(s) of the boundary condition, (obviously) must be JiT-safe
[bc-name].h: Header for functions specific to the boundary condition, which will in the future be standardized into a uniform interface. This header should include ./params.h, but NOT include ./qf.h.

An example for [bc-name].h for non-Dirichlet BCs is include/ratel/boundary/traction/traction.h:

/// @file
/// Ratel Traction Boundary Conditions Functions
#pragma once
#include <ratel-impl.h>

#include "params.h"

RATEL_INTERN PetscErrorCode RatelBoundaryTractionDataFromOptions(Ratel ratel, PetscInt i, RatelBCTractionParams *params_traction);
RATEL_INTERN PetscErrorCode RatelBoundaryTractionSetupEnergySuboperators(Ratel ratel, DM dm_energy, CeedOperator op_external_energy);
RATEL_INTERN PetscErrorCode RatelBoundaryTractionSetupSuboperators(Ratel ratel, DM dm, CeedOperator op_residual);

For Dirichlet BCs, see include/ratel/boundary/clamp/clamp.h:

/// @file
/// Ratel Clamp Boundary Conditions Functions
#pragma once
#include <ratel-impl.h>

#include "params.h"

RATEL_INTERN PetscErrorCode RatelBoundaryClampDataFromOptions(Ratel ratel, PetscInt i, PetscInt j, RatelBCClampParams *params_clamp);
RATEL_INTERN PetscErrorCode RatelBoundaryClampSetupDirichletSuboperators(Ratel ratel, DM dm, PetscBool incremental, CeedOperator op_dirichlet);

The functions that should exist in the header are a subset of the following, along with any needed helper functions:

RatelBoundary[bc-name]DataFromOptions, reads command-line options into the relevant data structure(s) defined in params.h, may also take integers for face_id and/or field_index
RatelBoundary[bc-name]SetupDirichletSuboperators, for Dirichlet boundaries
RatelBoundary[bc-name]SetupSuboperators, for adding suboperators to any of op_residual_u, op_residual_ut, op_residual_utt, op_jacobian
RatelBoundary[bc-name]SetupMaterialSuboperators, like *SetupSuboperators, but for BCs with explicit material dependence (right now only Nitsche contact). The first argument should be RatelMaterial instead of Ratel.
RatelBoundary[bc-name]SetupEnergySuboperators, for adding suboperators to the external energy operator, if applicable

These will eventually be changed to a unified interface under a RatelBoundary data structure.

The implementations for these headers go in the corresponding src/boundary/[bc-name].c file.

The existing header ratel-boundary.h now contains utility functions for setting up boundary conditions. Most critical of these are:

RatelBoundarySetupSuboperators: all BCs which define a RatelBoundary[bc-name]SetupSuboperators should have those called here. If a new suboperator may be added to op_jacobian, special care must be taken to set up the multigrid level (see below).
RatelBoundarySetupMaterialSuboperators: all BCs which define a RatelBoundary[bc-name]SetupMaterialSuboperators should be called here. See below for details on op_jacobian suboperators.
RatelBoundarySetupDirichletSuboperators: all BCs which define a RatelBoundary[bc-name]SetupDirichletSuboperators should be called here
RatelBoundarySetupEnergySuboperators: all BCs which define a RatelBoundary[bc-name]SetupEnergySuboperators should be called here

BCs with Jacobian suboperators¶

There is now a simple process for setting up multigrid levels for BCs with Jacobian suboperators. Now, both penalty contact and pressure BCs use RatelMaterialSetBoundaryJacobianMultigridInfo and the new RatelBoundarySetupMultigridLevel_Standard and RatelBoundarySetupMultigridLevel_CellToFace helpers to ensure that MG levels are set up for all Jacobian suboperators. As an implementation detail, the Jacobian indices for these boundaries are considered to live on the last material, but this choice is arbitrary and not relevant to the process of adding new boundary conditions.

For non-material dependent BCs, the call to RatelBoundary[bc-name]SetupSuboperators should be wrapped like:

const RatelMaterial last_material = ratel->materials[ratel->num_materials - 1];
CeedInt             old_num_jacobian_sub_operators, new_num_jacobian_sub_operators;

RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &old_num_jacobian_sub_operators));
PetscCall(RatelBoundary[bc-name]SetupSuboperators(ratel, dm, op_residual_u, op_jacobian));
RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &new_num_jacobian_sub_operators));
// -- Set Jacobian info on last material
PetscCall(RatelMaterialSetBoundaryJacobianMultigridInfo(last_material, op_jacobian, old_num_jacobian_sub_operators, new_num_jacobian_sub_operators,
                                                        &RatelBoundarySetupMultigridLevel_Standard));

For material-dependent BCs, the call to RatelBoundary[bc-name]SetupMaterialSuboperators should be wrapped like:

RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &old_num_jacobian_sub_operators));
PetscCall(RatelBoundary[bc-name]SetupMaterialSuboperators(material, u_dot, op_residual_u, op_jacobian));
RatelCallCeed(ratel, CeedOperatorCompositeGetNumSub(op_jacobian, &new_num_jacobian_sub_operators));
// Set Jacobian info for material
PetscCall(RatelMaterialSetBoundaryJacobianMultigridInfo(material, op_jacobian, old_num_jacobian_sub_operators, new_num_jacobian_sub_operators,
                                                        &RatelBoundarySetupMultigridLevel_CellToFace));

Adding New Initial Conditions¶

Ratel supports non-zero initial condition for linear poromechanincs MMS and as a constant value for other cases. If user needs to add non-zero initial condition as function of coordinate, a function with the same structure as RatelSetupInitialConditionMMS needs to be added with its own Qfucntion expressing the initial condition of each active field.