Docs: add troubleshooting.md, convergence.md, CONTRIBUTING.md symbolic link, and contributions to changelog (#5522)

CONTRIBUTING.md

@@ -0,0 +1 @@
+./docs/development/onboarding.md

docs/changelog.md

@@ -5,6 +5,8 @@
 ### New Features
 
 - DFT+U with k-points for Mulliken methods ([#4855](https://github.com/cp2k/cp2k/pull/4855))
+- Energy Correction Harris functional with k-points
+  ([#5031](https://github.com/cp2k/cp2k/pull/5031))
 - Lowdin population analysis for k-points ([#5045](https://github.com/cp2k/cp2k/pull/5045))
 - Wavefunction extrapolation for k-point calculations
   ([#4884](https://github.com/cp2k/cp2k/pull/4884), [#4943](https://github.com/cp2k/cp2k/pull/4943),
@@ -22,6 +24,23 @@
 - K-point symmetry reduction ([#5123](https://github.com/cp2k/cp2k/pull/5123),
   [#5152](https://github.com/cp2k/cp2k/pull/5152), [#5165](https://github.com/cp2k/cp2k/pull/5165),
   [#5173](https://github.com/cp2k/cp2k/pull/5173))
+- Input keyword for cutoff radius of HFX shortrange potential
+  ([#4945](https://github.com/cp2k/cp2k/pull/4945))
+- Input keyword for L-BFGS optimizer print control ([#5274](https://github.com/cp2k/cp2k/pull/5274))
+- New output of NEB including relative energy plot and final structures
+  ([#5382](https://github.com/cp2k/cp2k/pull/5382))
+- New output of final structure from optimization in CIF and EXTXYZ formats
+  ([#5118](https://github.com/cp2k/cp2k/pull/5118), [#5140](https://github.com/cp2k/cp2k/pull/5140))
+- New output of Hessian before mass weighting from revised vibrational analysis printout
+  ([#5395](https://github.com/cp2k/cp2k/pull/5395))
+- Dimer initialization by reading molden files ([#5312](https://github.com/cp2k/cp2k/pull/5312))
+- Parse cell information from EXTXYZ for both initial geometry and each frame of reftraj run
+  ([#4960](https://github.com/cp2k/cp2k/pull/4960), [#5401](https://github.com/cp2k/cp2k/pull/5401))
+- Wrapping input atomic coordinates for each frame before calculation in a reftraj run
+  ([#5479](https://github.com/cp2k/cp2k/pull/5479))
+- Per-thermal-region function for rescaling temperatures in MD (independent of thermostats)
+  ([#5002](https://github.com/cp2k/cp2k/pull/5002))
+- Cell optimization with fixed volume ([#5086](https://github.com/cp2k/cp2k/pull/5086))
 - **TODO**
 
 ### New Libraries
@@ -43,6 +62,11 @@
 - Drop support for GCC 8 ([#5290](https://github.com/cp2k/cp2k/pull/5290))
 - Refactor DOS/PDOS input section ([#5326](https://github.com/cp2k/cp2k/pull/5326))
 - Remove obsolete Cython Python bindings ([#5541](https://github.com/cp2k/cp2k/pull/5541))
+- Remove EVAL_ENERGY_FORCES and EVAL_FORCES keywords in favor of EVAL under &MOTION/&MD/&REFTRAJ
+  ([#5401](https://github.com/cp2k/cp2k/pull/5401))
+- An implementation of the FFTW3 interface may be turned into a hard dependency in a later release.
+  Please consider compiling and CP2K with FFTW3, MKL, AOCL or any other library implementing this
+  interface if you have not used it until now. ([#5454](https://github.com/cp2k/cp2k/pull/5454))
 - **TODO**
 
 ### Fixes

docs/development/onboarding.md

@@ -78,11 +78,11 @@ usually takes less than one hour:
   - You can check the status of your PR at
     [https://github.com/cp2k/cp2k/pulls](https://github.com/cp2k/cp2k/pulls)
   - We use `git rebase` to keep the
-    [Git history linear](https://github.com/cp2k/cp2k/wiki/Git-Tips-&-Tricks), meaning that in your
+    [Git history linear](https://github.com/cp2k/cp2k/wiki/CP2K-CI#git-history), meaning that in
-    final Pull Request there can't be any merge commits
+    your final Pull Request there can't be any merge commits
 - The PR will trigger the
   [CP2K Continuous Integration (CI) system](https://github.com/cp2k/cp2k/wiki/CP2K-CI) to check
-  conventions and running some checks
+  conventions, compatibility, and correctness
   - In the case of success, the PR will be merged by one of the CP2K administrators
   - In the case of error, please check what's wrong in the CI logs, fix it in your branch, and
     commit/push again. The CI will automatically rerun on the new version of the code (no need to

docs/getting-started/foreword-and-faq.md

@@ -157,17 +157,8 @@ distinct perspectives.
 
 ## Does CP2K support k-points?
 
-As an essential element for solid-state electronic structure, there is of course support for
+As an essential element for solid-state electronic structure, k-point sampling is supported for some
-k-points in a broad sense in the `QUICKSTEP` module of CP2K. A few specialized features may not have
+features in the `QUICKSTEP` module of CP2K, as elaborated on [](../methods/dft/k-points).
-complete, verified program implementations for k-point support, or are based on theories and
-algorithms that do not have an updated k-point version (compared with an isolated, non-periodic
-formalism) to begin with. After all, it is not a far stretch to think that a novel k-point
-generalization to existing methods is worthy of one or more academic publications and takes serious
-collaboration and devoted efforts to investigate.
-
-The development status and user opinions about k-point supports can be found at the dedicated
-[github issue](https://github.com/cp2k/cp2k/issues/4854); any request for new features of this kind
-requires providing a reference implementation of k-point formalism in other softwares.
 
 ## Where can I meet the CP2K community?
 
@@ -198,32 +189,51 @@ Raymond's [How To Ask Questions The Smart Way](http://www.catb.org/~esr/faqs/sma
 (**Disclaimer**: this link does not imply any connection between the original author and the CP2K
 developers, nor does it suggest that the original author may be contacted for assistance.)
 
+In the very first place, please refer to the [](./troubleshooting) page for a catalog of well-known
+warning and error messages with readily available explanations and suggestions. Search through the
+venues mentioned above for similar questions from others, and see if there are recent answers and
+advice applicable to the scenario. For the time being, it is not recommended to seek for unofficial
+CP2K-specific suggestions from generic large language models (LLM); even if they have been trained
+on a refined and verified corpus of CP2K materials one day, they can still hallucinate and generate
+superficially convincing but factually incorrect responses. (Unless willing to take responsibility
+for the correctness of any content produced by artificial intelligence as with human authors of a
+formal academic publication, do not bother mentioning anything from AI in the discussion at all.)
+
 Before submitting a question, please compose it with sufficient details, accuracy, and clarity.
 Approach the process in the same way as making a presentation to general audience, or even writing
 the "Methods" section in a formal academic publication; this includes giving explanations to
 uncommon acronyms (say, the abbreviated name of a specific class of materials, or anything that is
 not on the [Acronyms](../acronyms) page) and traceable citations (with publication title, date, and
-DOI, instead of merely showing a screenshot or a paragraph of copy-pasted text). The release date or
+DOI link, instead of merely showing a screenshot or a paragraph of copy-pasted text).
-git version of CP2K, and custom revisions if any, has to be mentioned in the first place.
+
+The release date or git version of CP2K, and custom revisions if any, has to be mentioned at the
+beginning. It is encouraged to try out the latest development version from the master branch of the
+github repository whenever situation permits, as this is likely containing the resolution patches
+already, and if not, works on which will benefit the next release version. Be aware that there are
+distinctive sets of manuals, with [](../CP2K_INPUT) for the latest development version and
+[](../versions) for the past releases; check twice if a page matches the program actually used prior
+to reading it.
 
 For problems related to installation and/or performance, the hardware specification and the
 configuration for linked libraries should be explained. The distribution source and means of
 preparation of dependencies, like with package managers, environment-controlling modules, or just a
-build from source, need clarifying.
+build from source, need clarifying. Faulty libraries are unfortunately very common that problems may
+be localized to a machine X or with a dependency Y, or even in a period of time Z with certain
+external concurrent processes or other users intervening; try ruling out these factors first.
 
 For error terminations and wrong results, it is imperative to provide a complete input deck and the
 output files. The "input deck" encompasses not only the main input file with keyword settings, but
 also all of the external files referenced inside unless they are available under the official `data`
-directory, so that the job can be actually run and tested. Instead of the original intended chemical
+directory, so that the job can be actually run and tested on the developers' side. Suspected wrong
-structure and composition, it is better to use a simplified system that triggers the malfunction
+results should have the precise location in the output and the reference expectation pointed out.
-reliably; this prevents confidential research information to be disclosed and reduces the demand on
-computational resources to ease the load of computers on the developer side.
 
-Please refrain from talking about CP2K-specific suggestions from generic large language model (LLM)
+```{note}
-or other types of artificial intelligence (AI). Even if the AIs have been trained on a refined and
+The input file does not have to use the intended chemical structure and composition in the original
-verified corpus of CP2K-oriented information one day, they can still hallucinate and generate
+encounter. For the [minimal reproducer](https://en.wikipedia.org/wiki/Minimal_reproducible_example),
-superficially convincing but scientifically incorrect responses. As with academic publications, the
+any simplified system is fine and the accuracy-controlling parameters can be tuned down, as long as
-human author is responsible for the correctness of any content produced by AIs.
+the input can reliably trigger the problem. Not only would this reduce the demand on computational
+resources while reproducing, but also confidential research information would not be disclosed.
+```
 
 Lastly, please kindly understand that, despite the CP2K developers having knowledge about the
 algorithm infrastructures and program implementations, they may not be suitable for answering all of
@@ -235,18 +245,16 @@ reproduce reported findings, the original authors. This is not denying any perso
 teach oneself at no cost, but rather hinting the necessity of communicating with the right
 professional people which does not have substitutes.
 
-## May I join in development and send patches?
+## What can I do for the community?
 
-Certainly! CP2K welcomes all sorts of contributions, from a small typo fix to modular code
+Potential forms of contribution, apart from engaging in the discussions, include:
-refactoring, to interfaces with other packages, to novel implementation of cutting-edge
-technology... Sharing kindness is an easy feat, and patches makes it more complete, that is the
-essence of open-source programming.
 
-The CP2K project uses `git` as the version control tool, and the official code repository is on
+- Participating the project development as instructed on [](../development/onboarding.md);
-github as [cp2k](https://github.com/cp2k/cp2k). For detailed instructions see the page
+- Enriching the [cp2k-examples](https://github.com/cp2k/cp2k-examples) repository with example
-[Starting development](https://www.cp2k.org/dev:starting).
+  inputs, outputs, pre- and post-analysis scripts. Interpretation and discussion of the results from
+  the program to complete the workflow would be nice to have.
 
-Another form of contribution is to enrich the [cp2k-examples](https://github.com/cp2k/cp2k-examples)
+It is also strongly advised to share the input files as well as structures as supplementary
-repository with example inputs and outputs, complete with post-analysis workflow down to straight
+materials in a publication. This will not only help other curious readers see the full potential of
-publishable results and discussions if possible. This will help other curious users see the full
+CP2K in terms of scientific and engineering applications, but also bridge the gap between
-potential of CP2K in terms of scientific and engineering applications.
+theoretical configurations and input setup syntax.

docs/getting-started/troubleshooting.md

@@ -0,0 +1,225 @@
+# Troubleshooting
+
+True to any advanced computational task, encounters to warnings and errors can be frequent and
+inevitable when working with CP2K due to various reasons. Don't panic: this page analyzes a selected
+catalog of possible issues and provides hints on how to address them.
+
+This is a dynamic list attempting to cover more topics of interest; feel free to open requests for
+expansion, but please read first and bear in mind the recommendations about asking questions in the
+[Foreward and FAQ](./foreword-and-faq.md#what-is-the-best-practice-to-ask-questions). Moreover, here
+is a gentle reminder that the normal termination of a computational task does not inherently
+guarantee scientifically meaningful, accurate, rigorous and publishable results.
+
+## The Whereabouts of Input & Output
+
+Before diving into the CP2K problems and solutions, here is a quick recap of basics about
+input/output of computer programming in general and of CP2K in particular. Well-versed or impatient
+readers can skip to [](#problems-and-solutions) below.
+
+The [standard streams](https://en.wikipedia.org/wiki/Standard_streams) of input/output connections
+include **standard input** (`stdin`) for reading data, **standard output** (`stdout`) for writing
+data and **standard error** (`stderr`) for writing error messages. In linux systems, they are
+represented by [file descriptors](https://en.wikipedia.org/wiki/File_descriptor), standardized as
+integer values 0 for `stdin`, 1 for `stdout` and 2 for `stderr`. During program execution, message
+passing is controlled by [redirection](<https://en.wikipedia.org/wiki/Redirection_(computing)>) with
+the `>` or `>>` syntax, and [pipeline](<https://en.wikipedia.org/wiki/Pipeline_(Unix)>) with the `|`
+syntax.
+
+A typical MPI-parallel CP2K run with the command below specifies the file `project.out` as `stdout`
+to which a majority of log data is written, and the command-line interface (CLI) screen as `stderr`
+to which error message is written if any.
+
+```bash
+mpirun -np 2 -x OMP_NUM_THREADS=1 cp2k.psmp -i project.inp -o project.out
+```
+
+With the following command, the file `project.out` is still `stdout` due to `1>` redirection, but at
+the same time it is also `stderr` due to the subsequent `2>&1` redirection.
+
+```bash
+mpirun -np 2 -x OMP_NUM_THREADS=1 cp2k.psmp -i project.inp 1>project.out 2>&1
+```
+
+Replacing `-o project.out` with `| tee project.out` as follows, the `stdout` from CP2K will be piped
+to the linux [`tee`](https://www.gnu.org/software/coreutils/manual/coreutils.html#tee-invocation)
+utility, which prints log data to the CLI screen and the file `project.out` simultaneously. The
+`stderr` is still the CLI screen as it is not redirected or piped.
+
+```bash
+mpirun -np 2 -x OMP_NUM_THREADS=1 cp2k.psmp -i project.inp | tee project.out
+```
+
+If CP2K is launched by job scheduling systems in a queue, the `stdout` and `stderr` may be set up
+with its own configurations; for instance, the Slurm Workload Manager defines `--output=<filename>`
+and `--error=<filename>` options for the [`srun` command](https://slurm.schedmd.com/srun.html).
+
+A CP2K input file usually contains some sections named `PRINT` that dictates whether and where a
+chunk of information will be printed. Take
+[FORCE_EVAL/PRINT/FORCES](#CP2K_INPUT.FORCE_EVAL.PRINT.FORCES) as an example, which prints
+`ATOMIC FORCES` if defined:
+
+```text
+&FORCE_EVAL
+  &PRINT
+    &FORCES ON
+    &END FORCES
+  &END PRINT
+&END FORCE_EVAL
+```
+
+The `FILENAME` keyword inside the `&FORCES` section determines where the `ATOMIC FORCES` output
+should go. As stated in the manual, the default value if not explicitly defined is a string written
+as `__STD_OUT__` for "the screen or standard logger", which is synonymous with the `stdout`
+described above. Setting `FILENAME` in other ways will create a separate file to write the output.
+
+Note that certain types of tasks like vibrational analysis, nudged elastic band, swarm, farming,
+..., have ***multiple*** output logs depending on the replica and parallelization in use; sometimes
+the single primary log (specified by `-o` option above) may not be as informative or primitive as
+the secondary logs (automatically generated per replica under the working directory) and it is
+necessary to locate the exact issue(s) from the latter. This applies to warnings and errors too,
+which in addition are typically issued on the first MPI rank of each replica.
+
+## Problems and Solutions
+
+### Program is stuck or killed for unknown reason
+
+When the program appears to freeze and stop updating any of the logs for outputs, check immediately
+whether it is still running with `ps aux` or `top`/`htop` commands or with the job scheduler.
+
+An out-of-memory (OOM) error can occur if the memory allocation for the program fails to meet the
+actual requirement during execution. This may or may not have a clear-cut message due to uncertainty
+in the timing, but may be confirmed in a re-run with the `free` or `ps aux` commands monitoring the
+memory consumption. If allocating more memory is not viable, reducing the number of MPI parallel
+processes or switching to MPI + OpenMP hybrid parallelism for the `psmp` build can reduce the total
+memory consumption as a compromise.
+
+If CP2K is built with [ELPA](../technologies/eigensolvers/elpa), it would be the default
+diagonalization library. Setting [PREFERRED_DIAG_LIBRARY](#CP2K_INPUT.GLOBAL.PREFERRED_DIAG_LIBRARY)
+to `SCALAPACK` can probably circumvent some bugs like in github issue
+[#4484](https://github.com/cp2k/cp2k/issues/4484).
+
+When running a hybrid DFT calculation, where it is well-known that the evaluation of {term}`ERI` for
+[Hartree-Fock exchange](../methods/dft/hartree-fock/index) and the in-core storage tends to be very
+memory-intensive, the upper bound of memory requirement depends on the number of MPI parallel
+processes and the value of [MAX_MEMORY](#CP2K_INPUT.FORCE_EVAL.DFT.XC.HF.MEMORY.MAX_MEMORY) in MB.
+The first SCF cycle is usually very time-consuming due to the necessary initial evaluation of ERI,
+and so do not terminate the program too early as long as the memory is fine.
+
+There are heavy tasks like optimization and nudged elastic band that have been reported to stuck for
+indeterminate time, which have rarely been reproduced reliably. In case it happens, chances are that
+adjusting the [OPTIMIZER](#CP2K_INPUT.MOTION.CELL_OPT.OPTIMIZER) or
+[OPT_TYPE](#CP2K_INPUT.MOTION.BAND.OPTIMIZE_BAND.OPT_TYPE) can help, and utilizing the latest
+restart file to re-run the job until convergence is recommended.
+
+### Interleaved multiplicated output messages
+
+A normal MPI parallel execution started as follows will be shown as 4 processes in `ps aux` or
+`top`/`htop`, with the output log `project.out` reflecting the parallelization setup.
+
+```bash
+mpirun -np 4 -x OMP_NUM_THREADS=1 cp2k.psmp -i project.inp -o project.out
+```
+
+```text
+ DBCSR| MPI: Number of processes                                               4
+ DBCSR| OMP: Current number of threads                                         1
+```
+
+If instead 4 instances of the same message appear...
+
+```text
+ DBCSR| MPI: Number of processes                                               1
+ DBCSR| OMP: Current number of threads                                         1
+```
+
+... along with many other interleaved and multiplicated lines like `PROGRAM STARTED AT {time}` and
+`PROGRAM PROCESS ID {pid}`, then CP2K is not parallelized as intended, but run as several
+independent processes that are all writing to a common output log in some indeterminate order. This
+means that the environment is not configured properly and the active MPI library (as shown by
+`which mpirun`) is not the one that CP2K has been linked to (as shown by `ldd cp2k.psmp`).
+
+Check everything that determines the environment variables and paths, including but not limited to
+`~/.bashrc` and `/etc/profile` files, modules, and conda envs. If the CP2K is built from source
+[with toolchain](./build-from-source.md#toolchain-based-build), do not forget to source the
+`cp2k_env` to load the dependencies as instructed.
+
+### Asterisks or NaN in the output
+
+This is a general fortran behavior about how a fixed-width field format handles a numeric value that
+cannot be fitted in. For example, the edit descriptor `F12.6` specifies a real float-type field with
+exactly 12 characters, including 6 for the fractional part and 1 for the decimal point (and 1 for
+the minus sign if negative), thus leaving only 5 places for the integer part; if the value to be
+printed is larger than 99999.999999 or smaller than -9999.999999, it would not fit in and a string
+of asterisks `************` would be shown instead. If something goes haywire and the value is not
+even a valid number any more, it would be represented as `         NaN`, where NaN is shorthand for
+"Not a Number". Most of these field formats are designed with output layout, value precision and
+possible range in mind, so the presence of asterisks and NaN in the output should invoke some doubts
+on the reliability of very large or very small numeric results even if the calculation looks fine
+otherwise.
+
+### A certain type object was expected, found something else
+
+```text
+A (integer|floating point|string) type object was expected, found (end of line|<something>),
+File: <filename>, Line: <line>, Column: <col>
+```
+
+This kind of error arises from the parser of input files, and naturally is resolved by preparing or
+editing the file in accordance with the manual. Say, a structure input requiring XYZ format should
+not use CIF, and the XYZ file should have number of atoms on the first line, comments on the second
+line and each of the rest of (number of atoms) lines in a format of
+`element-symbol X-coordinate Y-coordinate Z-coordinate`.
+
+### CPASSERT failed
+
+`CPASSERT` is one of the [](../development/error-handling) mechanisms in CP2K intended for
+conditions that should hold most of the time, like some basic sanity checks for correct data types,
+matching matrix dimensions, or availability of essential elements for a certain routine. Therefore,
+`CPASSERT failed` is a minimal blanket statement for these unlikely events, only indicating the code
+location at the lower right corner of the message box.
+
+When `CPASSERT failed` occur for a valid input file strictly following documentation, report to
+developers and justify the situation where the condition fails. The message may be revised for a
+clearer user-oriented phrasing upon reasonable request.
+
+### GEOMETRY wrong or EMAX_SPLINE too small
+
+This error originates from closely contacting or outright overlapping atoms in the geometry that are
+detected while building a neighbor list. Although the keyword `EMAX_SPLINE` is for a Molecular
+Mechanics (MM) calculation with classical force fields, this error does not necessarily come from a
+MM or QM/MM task because neighbor lists are also widely used in quantum mechanics (QM).
+
+Always remember to inspect the input structure in a visualization program as said in the section
+[](../methods/optimization/geometry_and_cell_opt.md#starting-structure-and-cell). Among the many
+possibilities of modelling errors, there are three frequently relevant pitfalls:
+
+- The file conversion procedure during structure preparation involve formats that do not record any
+  information about periodicity or cell definition, or whose record is not yet universally
+  recognized such as the extended XYZ specification;
+- The redefinition of cell vectors (most commonly by means of linear transformation) and the
+  construction of surface slabs or supercells violate the original periodicity, or fail to
+  deduplicate atoms sent to the same coordinate dictated by symmetry-equivalent positions;
+- The structure contains crystallographic disorder where one site has more than one possible type of
+  atom, and the fractional occupancy is not handled well when creating the model, as discussed in
+  [a FAQ](./foreword-and-faq.md#how-do-i-create-the-atomistic-model-for-cp2k-input).
+
+### SCF run NOT converged
+
+Refer to [](../methods/dft/convergence).
+
+### Messages mentioning LSD
+
+`LSD` is an alias for [UKS](#CP2K_INPUT.FORCE_EVAL.DFT.UKS) in some error messages such as
+`Use the LSD option for an odd number of electrons` and `LSD: try to use a different multiplicity`.
+[CHARGE](#CP2K_INPUT.FORCE_EVAL.DFT.CHARGE) and
+[MULTIPLICITY](#CP2K_INPUT.FORCE_EVAL.DFT.MULTIPLICITY) options should be set correctly based on the
+chemistry to be modelled. If the system is intended to be closed-shell, broken geometry like missing
+or duplicated hydrogen atoms may give rise to the errors.
+
+### Index to radix array not found
+
+This error arises from a combination of high [CUTOFF](#CP2K_INPUT.FORCE_EVAL.DFT.MGRID.CUTOFF) for
+the real-space grid and large cell size in some direction, which leads to a high internal plane-wave
+FFT length that the external FFT library may not support. Besides reducing cutoff and/or cell size,
+try enabling [EXTENDED_FFT_LENGTHS](#CP2K_INPUT.GLOBAL.EXTENDED_FFT_LENGTHS) or switching
+[PREFERRED_FFT_LIBRARY](#CP2K_INPUT.GLOBAL.PREFERRED_FFT_LIBRARY).

docs/index.md

@@ -19,6 +19,7 @@ getting-started/build-from-source
 getting-started/build-with-spack
 getting-started/distributions
 getting-started/first-calculation
+getting-started/troubleshooting
 ```
 
 ```{toctree}

docs/methods/dft/convergence.md

@@ -0,0 +1,141 @@
+# How to make a SCF run converge
+
+```text
+ *******************************************************************************
+ *   ___                                                                       *
+ *  /   \                                                                      *
+ * [ABORT]                                                                     *
+ *  \___/     SCF run NOT converged. To continue the calculation regardless,   *
+ *    |             please set the keyword IGNORE_CONVERGENCE_FAILURE.         *
+ *  O/|                                                                        *
+ * /| |                                                                        *
+ * / \                                                            qs_scf.F:702 *
+ *******************************************************************************
+```
+
+Since CP2K 2024.1 version, a failure in SCF convergence in a Quickstep calculation aborts the
+program by default. This means that after reaching [MAX_SCF](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.MAX_SCF)
+cycles (50 by default), the value printed under the `Convergence` column does not meet
+[EPS_SCF](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.EPS_SCF). A variety of measures are available for
+converging the SCF to a reasonable result, which this page discusses, assuming that the reader has
+read [](../../getting-started/foreword-and-faq) and the other documentations under [](./index)
+first.
+
+```{note}
+At the moment, the convergence criterion does not take the absolute change in energy into account,
+and the convergence of the diagonalization algorithm differs from that of the OT algorithm.
+```
+
+```{danger}
+**Take your own risk and responsibility for ignoring convergence failure !!**
+
+Setting [IGNORE_CONVERGENCE_FAILURE](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.IGNORE_CONVERGENCE_FAILURE)
+will instead emit a warning ` *** WARNING in qs_scf.F:700 :: SCF run NOT converged ***` and proceed
+to other calculation. Unfortunately, few do realize the necessity to scrutinize subsequent outcomes
+for accuracy, let alone precision. It can lead to qualitatively and quantitatively incorrect results
+including but not limited to strange electronic occupation and band structure, unphysical response
+properties, and wild atomic motion and out-of-control temperature. These are not credible and useful.
+
+Therefore, `IGNORE_CONVERGENCE_FAILURE` should only be considered as a **last resort** out of
+desperation, rather than a universal remedy used on a regular basis or even as the default.
+Please try achieving SCF convergence on the starting structure in a single-point energy calculation
+in the first place; any other tasks not preceded by it is like putting the cart before the ponies.
+```
+
+## General considerations
+
+The very first ingredient of SCF convergence is a sensible input structure, applicable to every type
+of computation task including geometry and cell optimization, as elaborated on
+[](../../methods/optimization/geometry_and_cell_opt.md#starting-structure-and-cell).
+
+On top of that, there is also the net charge and spin multiplicity as specified by keywords
+[CHARGE](#CP2K_INPUT.FORCE_EVAL.DFT.CHARGE) and
+[MULTIPLICITY](#CP2K_INPUT.FORCE_EVAL.DFT.MULTIPLICITY) respectively that should be set to represent
+the realistic electronic state. Use the [UKS](#CP2K_INPUT.FORCE_EVAL.DFT.UKS) keyword (or
+equivalently written as `LSD`) to request for an unrestricted, spin-polarized calculation of
+open-shell systems.
+
+If the geometry is reasonable, check whether the [XC](#CP2K_INPUT.FORCE_EVAL.DFT.XC) section or the
+respective section of model Hamiltonian has been set up correctly.
+
+The default of [SCF_GUESS](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.SCF_GUESS) is `ATOMIC`, meaning that the
+initial wavefunction and density matrix is generated from the atomic density of each kind of atom.
+In this case the [MAGNETIZATION](#CP2K_INPUT.FORCE_EVAL.SUBSYS.KIND.MAGNETIZATION) keyword and the
+[BS](#CP2K_INPUT.FORCE_EVAL.SUBSYS.KIND.BS) section can be used to provide the orbital occupation
+pattern of different spin channels and quantum numbers, which is especially crucial for systems with
+certain magnetic order like ferromagnetic, ferrimagnetic and antiferromagnetic materials. Relevant
+literature or materials database entry often give information about the atomic magnetization. As for
+how the keywords and sections are set up, prior discussions can be found at a
+[google group thread](https://groups.google.com/g/cp2k/c/8fTVlCEjSME) and page 34-36 of
+[a 2015 tutorial](https://www.cp2k.org/_media/events:2015_cecam_tutorial:ling_hybrids.pdf).
+
+Setting SCF_GUESS to `RESTART` and specifying a wavefunction restart file for the keyword
+[WFN_RESTART_FILE_NAME](#CP2K_INPUT.FORCE_EVAL.DFT.WFN_RESTART_FILE_NAME) will instead parse the
+file for the initial density matrix. If some preliminary cheap calculation can converge, restarting
+from the wavefunction is highly recommended for going to advanced, expensive ones:
+
+- Having used the 2-zeta DZVP-MOLOPT-SR-GTH basis set in a gamma-only calculation, restart another
+  gamma-only calculation with the 3-zeta TZVP-MOLOPT-SR-GTH basis set (note that both should use the
+  same pseudopotential);
+- Having used the pure GGA functional PBE, restart a calculation with hybrid functional PBE0 (which
+  also makes [SCREEN_ON_INITIAL_P](#CP2K_INPUT.FORCE_EVAL.DFT.XC.HF.SCREENING.SCREEN_ON_INITIAL_P)
+  reasonable);
+- After a plain calculation, restart another with special external environments such as a periodic
+  electric field or an implicit solvation model;
+- After a single-point energy evaluation, restart a geometry or cell optimization task, and then
+  after that restart a vibrational analysis task;
+- After a ground-state calculation, use [WFN_MIX](#CP2K_INPUT.FORCE_EVAL.DFT.PRINT.WFN_MIX) to
+  manipulate MO coefficients and restart an excited-state calculation;
+- ...
+
+```{note}
+The format and suffix of wavefunction restart files differ between a gamma-only formalism and
+a k-point sampling scheme: the former is typically `<project>-RESTART.wfn` while the latter is
+typically `<project>-RESTART.kp`. They cannot be interchanged for the purpose of restarting.
+
+Starting from CP2K version 2026.2, it is possible to produce a `<project>-RESTART.kp` from a
+gamma-only calculation by using the Harris functional for energy correction under section
+[DFT/ENERGY_CORRECTION](#CP2K_INPUT.FORCE_EVAL.DFT.ENERGY_CORRECTION).
+```
+
+Some parameters that control the accuracy for Quickstep calculation, such as
+[EPS_DEFAULT](#CP2K_INPUT.FORCE_EVAL.DFT.QS.EPS_DEFAULT),
+[CUTOFF](#CP2K_INPUT.FORCE_EVAL.DFT.MGRID.CUTOFF) and
+[REL_CUTOFF](#CP2K_INPUT.FORCE_EVAL.DFT.MGRID.REL_CUTOFF), can also help convergence when chosen as
+good as necessary. The overall time cost is not necessarily increased with the parameters leaning on
+the more accurate and expensive side: even if each SCF iteration takes longer, the total number of
+iterations to reach convergence may still be reduced.
+
+Similarly, higher number or density of k-points for the Brillouin-zone sampling may be beneficial,
+in particular if the cell is small and the system is not insulating. A convergence test for k-points
+with respect to the target property does not need to start with what is too low to make SCF
+converge.
+
+## For diagonalization
+
+The standard diagonalization algorithm for the Kohn-Sham matrix is activated by setting the
+[DIAGONALIZATION](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.DIAGONALIZATION) section, with full support for the
+mixing and smearing techniques.
+
+The mixing procedures of the density matrix in [MIXING](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.MIXING)
+supports several methods in the keyword [METHOD](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.MIXING.METHOD). The
+default conservative `DIRECT_P_MIXING` option may be swapped with `BROYDEN_MIXING`, `PULAY_MIXING`,
+`KERKER_MIXING`, etc.
+
+Fractional occupation of molecular orbitals, or smearing, is enabled by the section
+[SMEAR](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.SMEAR). It is very useful for systems with small to none band
+gap and strong static correlation. The possibilities provided by the keyword
+[METHOD](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.SMEAR.METHOD) include Fermi-Dirac smearing at a certain
+[ELECTRONIC_TEMPERATURE](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.SMEAR.ELECTRONIC_TEMPERATURE) and several
+broadening methods with width [SIGMA](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.SMEAR.SIGMA); elevated
+ELECTRONIC_TEMPERATURE or SIGMA can handle difficult systems, but extrapolation to 0 is required to
+obtain results comparable with what without smearing.
+
+## For OT
+
+Alternative to the diagonalization is the orbital transformation ({term}`OT`) method, activated by
+setting the [OT](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.OT) section. The most important settings are
+[ALGORITHM](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.OT.ALGORITHM),
+[LINESEARCH](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.OT.LINESEARCH),
+[MINIMIZER](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.OT.MINIMIZER), and
+[PRECONDITIONER](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.OT.PRECONDITIONER).

docs/methods/dft/index.md

@@ -11,6 +11,7 @@ hartree-fock/index
 basis_sets
 pseudopotentials
 k-points
+convergence
 cutoff
 local_ri
 constrained

docs/methods/dft/k-points.md

@@ -72,6 +72,15 @@ support.
 | Periodic electric field       | **Unsupported.** Requires OT first.                                     |
 | Active-space calculations     | **Unsupported.** Only `SCHEME NONE` and `SCHEME GAMMA` available.       |
 
+The reason why a specialized feature does not have support for k-point sampling is twofold: it is
+possible that code implementation in CP2K is not present, complete, or verified yet, but it is also
+possible that the underlying theories and algorithms do not have an updated k-point version
+(compared with an isolated, non-periodic formalism) to begin with. In the latter case it is not a
+far stretch to think that a novel k-point generalization to existing methods is worthy of academic
+publications and takes serious collaboration and devoted efforts to investigate. It is therefore
+strongly suggested that reference implementation of k-point formalism in other softwares be provided
+whenever making a feature request of this kind.
+
 Note that a successful calculation does not by itself establish that a feature--k-point combination
 is reliable for a particular system or property. For a new workflow, converge the k-point mesh and,
 where appropriate, compare with an equivalent real-space supercell calculation.

docs/methods/optimization/geometry_and_cell_opt.md

@@ -123,10 +123,8 @@ transition-state search machinery; it requires additional method-specific settin
 treated as an ordinary minimization.
 
 [MAX_ITER](#CP2K_INPUT.MOTION.GEO_OPT.MAX_ITER) limits the number of optimization iterations. One
-optimization iteration can require more than one force evaluation, depending on the optimizer, line
+optimization iteration can require more than one force evaluation, depending on the optimizer and
-search, and the selected `CELL_OPT` mode.
+line search mode. An optimization may terminate if one of the following events takes place:
-
-An optimization may terminate if one of the following events takes place:
 
 - The convergence criteria (see below) are satisfied before reaching `MAX_ITER`;
 - The `MAX_ITER` is reached, regardless of whether the convergence criteria are satisfied or not;
@@ -317,6 +315,8 @@ of an `iterate.dat` excerpt is reproduced below.
     9   15     1     0  con    0  1.0D+00  1.2D-02  7.569D-04 -1.152D+00
 ```
 
+### More notes about convergence
+
 Geometry and cell optimization normally try to lower the energy, but convergence is determined by
 the active force, displacement, and pressure criteria rather than by the total energy alone.
 Intermediate steps may sometimes increase the energy. A very small energy increase at the final step
@@ -466,22 +466,17 @@ geometry updates could make extrapolation unavailable and automatically fall bac
 ```{danger}
 **Be responsible, and do not ignore SCF convergence failure blindly.**
 
-By default, a failure in SCF convergence aborts the program with a message like:
+The general issue of SCF convergence is discussed in [](../dft/convergence).
-`SCF run NOT converged. To continue the calculation regardless, please set the keyword
+Bad SCF convergence leads to unreliable energy, force, and stress on the current step,
-IGNORE_CONVERGENCE_FAILURE.` Setting
+introducing error to the updated structure and the extrapolated wavefunction on the next
-[IGNORE_CONVERGENCE_FAILURE](#CP2K_INPUT.FORCE_EVAL.DFT.SCF.IGNORE_CONVERGENCE_FAILURE)
+step. An optimization process with most or all steps failing to converge SCF cycles does
-to `.TRUE.` turns it into a warning `SCF run NOT converged` that allows for optimization
+not yield trustworthy results in the end. Abnormal behaviors such as structure "blowing up"
-to continue. However, bad SCF convergence leads to unreliable energy, force, and stress
+should not come off as surprising if convergence failure is ignored.
-on the current step, introducing error to the updated structure and the extrapolated
-wavefunction on the next step. An optimization process with most or all steps failing
-to converge SCF cycles does not yield trustworthy results in the end.
 
-Therefore, `IGNORE_CONVERGENCE_FAILURE` should only be considered as a **last resort**
+Preparing a nice starting structure in the first place is more reliable than using a crude
-out of desperation, rather than a universal remedy used on a regular basis or even as
+structure and wishing that it becomes easier to converge SCF cycles as optimization goes.
-the default. There are dozens of measures available towards SCF convergence on top of
+A single-point energy calculation of the starting structure can be utilized to experiment
-a realistic, chemically sensible model structure and appropriate net charge, spin
+with options that achieve convergence as well as to assess the time consumption.
-multiplicity, atomic magnetization, and k-point sampling; please try achieving SCF
-convergence on the starting structure in a single-point calculation in the first place.
 ```
 
 ## Output files and restarts