- Experimented with specialized vs DBM's universal kernel.
- Performance is equal likey due to data mix/sort (TODO).
- DBCSR/unrelated: Likely introduce DBM-kernel (cleaner).
- Rely on OPENCL_LIBSMM_PFORMAT.
Mempool:
- Revised default to skip mempool for host-allocs if offload
is not compiled-in or if offload-allocs on host are disabled.
- Renamed and split DBM_ALLOCATION_FACTOR.
- Separate DBM_OVERCOMMIT_DEVICE/HOST.
- More rigid control of mempool config
- Conditionally define mempool lists.
Other:
- Passing batch/task data in const-correct fashion.
- Revised verbose output.
- Apparently, MPI_Alloc_mem/free are not fully thread-safe.
- Debug shows issue similar to issues reported elsewhere.
- Note: MPI_Alloc_mem was taken out of critical section.
- Internally, use separate pools for device/host memory.
- Let hyperparams enable/disable pooled memory
- DBM_OFFLOAD_ALLOC
- DBM_MEMPOOL_DEVICE
- DBM_MEMPOOL_HOST
- Let hyperparam control offload memory (host).
- DBM_MULTIPLY_SM to use SLM (instead of regs).
- Adjusted permitted preprocessor symbols.
- Control GEN depending on other kernel parameters.
- Avoid bank-conflicts (POT).
- Makefile: miniapp
- Allow to select GNU Compiler by the means of "make GNU=1".
- Find <acc_opencl.h> after PR #3994.
- Offload/OCL
- Implemented offloadGetErrorName and offloadGetLastError.
- Backward-compatible with DBCSR (OpenCL backend).
- Print issues with in categories: error, warn, info messages.
- Trace additional events and sources of deadlocks and other issues.
- Allow compilation with GNU even if "omp-tools.h" is not present.
- Allow singular translation units under BSD-3 license.
- Print more accurate location of warning.
CP2K_OMP_TRACE=1: assert errors at the place/location of the issue (nice debug experience)
CP2K_OMP_TRACE=2: print error issues (printing the code location needs debug symbols)
CP2K_OMP_TRACE=3: print warnings or info messages (when CP2K terminates)
Printing instead of asserting issues (errors, warnings, info) benefits from debug symbols. A negative value of CP2K_OMP_TRACE represents maximum verbosity.
* There are three main implementations
1. DBM_MULTIPLY_SPLIT=1: default kernel with smallest LOC-number uses private/register-backed buffer. This implementation is simple and performs best even on different vendor's GPUs. There is moderate unrolling/code-bloat leveraging macros. Though, there is no "old school GPU" like copy-into SLM; if there is re-use, it leverage existing caches between registers and global memory.
2. DBM_MULTIPLY_SPLIT=2..N with DBM_MULTIPLY_SPLIT=8 matching the CUDA implementation using shared/local memory. This implementation matches the existing CUDA implementation and perform equal and reasonable across GPUs. There is moderate unrolling/code-bloat leveraging macros; heuristics are revised/different from CUDA implementation (less cases).
3. DBM_MULTIPLY_GEN=1: uses https://github.com/intel/tiny-tensor-compiler and IR describing DBM's input format. The IR-file (https://github.com/hfp/cp2k/blob/master/src/dbm/dbm_multiply_opencl.ir) is not part of PR. The generated kernel-code is not yet competitive.
* The amount of code in the OpenCL-kernel and and C based glue-code can be *significantly* reduced (fraction of LOC) when removing for instance implementation 2 and 3 (above mentioned). Reducing the number of LOC can be part of a future cleanup (and once code is settled).
* check_file_properties.py: support C-comments, and omit checking OpenCL source code (it is usually compiled at runtime, i.e., machine-driven rather than user-controlled).
* This work "short-cuts" supporting OpenCL by leveraging DBCSR's OpenCL backend to implement CP2K's Offload interface.
- Future work (may be soon) can decide/settle where the actual glue code is hosted making the "OpenCL backend".
- OpenCL is very close to CUDA (streams/queues, events, etc), however, an OpenCL backend accounts for differences.
- Pointer arithmetic on the host using device pointers is not supported with OpenCL without additional effort.
- OpenCL abstracts device memory being "cl_mem" structure and does not expose actual device pointers.
* Some functions on CP2K's Offload interface (once derived from DBCSR's ACC interface) remain "TODO".
* For the time being offload_runtime.h "disables" support for Grid and PW/FFTs
- Defines for __NO_OFFLOAD_GRID and __NO_OFFLOAD_PW must be still given for the Fortran code.
* Some changes remove CUDA and HIP specific code-paths by relying on Offload runtime (e.g., offload_create_buffer).
* Some changes also address problematic code format (multi-line control-flow w/o curly braces/block, etc).
* Further updating INSTALL.md and white-listing/removing certain items are part of follow-up changes.
* Cleaned FLAG_EXCEPTIONS (tools/precommit/check_file_properties.py).
