- Introduced ELPA_ONE_STAGE (see #4683).
- QR factorization (unsafe is now internal).
- Removed keyword ELPA_QR_UNSAFE.
- Updated to ELPA 2025.06.002.
- Disabled IEEE exceptions.
- Test coverage.
- For the OpenCL based GPU code-path, or by choice, ELPA may not be built with GPU support.
- Introduced __NO_OFFLOAD_ELPA to allow CPU-only ELPA (related to #4617).
- Allow compile-time selection of ELPA solver-kind (1-stage vs 2-stage).
- For ELPA/SYCL, use 1-stage ELPA solver by default for the time being.
- ELPA/SYCL (MKL in general?) has an index-fix currently only in a branch.
Index/ILP issue (undefined behavior):
- ELPA eventually mismatches index-kinds (LP64 vs ILP64 when linking BLAS/MKL).
- For static linkage it usually does not not matter (at least CPU-only).
- CP2K links the correct BLAS flavor (SO and GPU-build may be affected?).
- ELPA 2-stage may be even affected beyond MKL (or for GPU in general).
- ELPA 1-stage is apparently not affected by index issue.
- Introduced offload_mempool_stats_print to Fortran API.
- Print memory statistics independent of DBM (and grid).
- Avoid circular dependency (string_utilities/strlcpy_c2f).
- Use up-leveled MPI/C wrapper.
* MPI/C wrapper (dbm_mpi_ to cp_mpi_)
- Adjusted license of C-MPI wrapper (consequence of mpiwrap).
- Moved wrapper for C languages in to mpiwrap directory.
- Avoid name clash (same basename as for Fortran).
- Renamed prefix of all MPI wrapper functions.
* Other (DBM):
- Improved handling errors (OFFLOAD_CHECK before launch).
- Permit/Adjusted preprocessor symbol (precommit hook).
- Avoid multi-line control-flow w/o curly braces.
- Double-check #4428 (DBM_MULTIPLY_COMM_MEMPOOL).
* TODO
- Apply offload/print_func pattern to GRID's print_func.
- Leverage offload_mempool for grid backend.
- Split backend_upload_packs into two phases on GPU.
- Initial phase is chosen to have not triggered upload.
- Offload: offloadEventQuery to check if GPU is busy.
- Removed dbm_multiply_gpu_download_results
- Transfers are launched as early as possible.
- Improved overlap between transfer/compute.
- Idea: proceed on CPU if GPU is busy.
Other changes:
- Documented way to perform validation (ctx=NULL).
- Permit DBM_ prefix for preprocessor symbols.
- Removed C-shard pointer from GPU-context.
- Per-shard event stored/provided by shard.
- Migrated CPU-options into context.
- Miniapp: allow to omit validation.
- Use preincrement (atomic).
- Experimented with specialized vs DBM's universal kernel.
- Performance is equal likey due to data mix/sort (TODO).
- DBCSR/unrelated: Likely introduce DBM-kernel (cleaner).
- Rely on OPENCL_LIBSMM_PFORMAT.
Mempool:
- Revised default to skip mempool for host-allocs if offload
is not compiled-in or if offload-allocs on host are disabled.
- Renamed and split DBM_ALLOCATION_FACTOR.
- Separate DBM_OVERCOMMIT_DEVICE/HOST.
- More rigid control of mempool config
- Conditionally define mempool lists.
Other:
- Passing batch/task data in const-correct fashion.
- Revised verbose output.
- Apparently, MPI_Alloc_mem/free are not fully thread-safe.
- Debug shows issue similar to issues reported elsewhere.
- Note: MPI_Alloc_mem was taken out of critical section.
- Internally, use separate pools for device/host memory.
- Let hyperparams enable/disable pooled memory
- DBM_OFFLOAD_ALLOC
- DBM_MEMPOOL_DEVICE
- DBM_MEMPOOL_HOST
- Let hyperparam control offload memory (host).
- DBM_MULTIPLY_SM to use SLM (instead of regs).
- Adjusted permitted preprocessor symbols.
- Control GEN depending on other kernel parameters.
- Avoid bank-conflicts (POT).
- Makefile: miniapp
- Allow to select GNU Compiler by the means of "make GNU=1".
- Find <acc_opencl.h> after PR #3994.
- Offload/OCL
- Implemented offloadGetErrorName and offloadGetLastError.
- Backward-compatible with DBCSR (OpenCL backend).
- Print issues with in categories: error, warn, info messages.
- Trace additional events and sources of deadlocks and other issues.
- Allow compilation with GNU even if "omp-tools.h" is not present.
- Allow singular translation units under BSD-3 license.
- Print more accurate location of warning.
CP2K_OMP_TRACE=1: assert errors at the place/location of the issue (nice debug experience)
CP2K_OMP_TRACE=2: print error issues (printing the code location needs debug symbols)
CP2K_OMP_TRACE=3: print warnings or info messages (when CP2K terminates)
Printing instead of asserting issues (errors, warnings, info) benefits from debug symbols. A negative value of CP2K_OMP_TRACE represents maximum verbosity.
* There are three main implementations
1. DBM_MULTIPLY_SPLIT=1: default kernel with smallest LOC-number uses private/register-backed buffer. This implementation is simple and performs best even on different vendor's GPUs. There is moderate unrolling/code-bloat leveraging macros. Though, there is no "old school GPU" like copy-into SLM; if there is re-use, it leverage existing caches between registers and global memory.
2. DBM_MULTIPLY_SPLIT=2..N with DBM_MULTIPLY_SPLIT=8 matching the CUDA implementation using shared/local memory. This implementation matches the existing CUDA implementation and perform equal and reasonable across GPUs. There is moderate unrolling/code-bloat leveraging macros; heuristics are revised/different from CUDA implementation (less cases).
3. DBM_MULTIPLY_GEN=1: uses https://github.com/intel/tiny-tensor-compiler and IR describing DBM's input format. The IR-file (https://github.com/hfp/cp2k/blob/master/src/dbm/dbm_multiply_opencl.ir) is not part of PR. The generated kernel-code is not yet competitive.
* The amount of code in the OpenCL-kernel and and C based glue-code can be *significantly* reduced (fraction of LOC) when removing for instance implementation 2 and 3 (above mentioned). Reducing the number of LOC can be part of a future cleanup (and once code is settled).
* check_file_properties.py: support C-comments, and omit checking OpenCL source code (it is usually compiled at runtime, i.e., machine-driven rather than user-controlled).
* This work "short-cuts" supporting OpenCL by leveraging DBCSR's OpenCL backend to implement CP2K's Offload interface.
- Future work (may be soon) can decide/settle where the actual glue code is hosted making the "OpenCL backend".
- OpenCL is very close to CUDA (streams/queues, events, etc), however, an OpenCL backend accounts for differences.
- Pointer arithmetic on the host using device pointers is not supported with OpenCL without additional effort.
- OpenCL abstracts device memory being "cl_mem" structure and does not expose actual device pointers.
* Some functions on CP2K's Offload interface (once derived from DBCSR's ACC interface) remain "TODO".
* For the time being offload_runtime.h "disables" support for Grid and PW/FFTs
- Defines for __NO_OFFLOAD_GRID and __NO_OFFLOAD_PW must be still given for the Fortran code.
* Some changes remove CUDA and HIP specific code-paths by relying on Offload runtime (e.g., offload_create_buffer).
* Some changes also address problematic code format (multi-line control-flow w/o curly braces/block, etc).
* Further updating INSTALL.md and white-listing/removing certain items are part of follow-up changes.
* Cleaned FLAG_EXCEPTIONS (tools/precommit/check_file_properties.py).
