[docs][coroutines] Revamp "Debugging C++ coroutines"

[vogelsgesang]

This commit is a major overhaul of the documentation on debugging C++ coroutines with the following goals:

• Make it more accessible to casual C++ programmers, i.e. non-toolchain developers. Move the low-level details around ABI further down, and instead start with real-life examples and copy-paste-friendly code, first.
• Cover LLDB in addition to GCC. Provide copy-pasteable scripts for LLDB and not only GCC.
• Cover additional topics, such as:
  • single-stepping into a coroutine
  • using __builtin_return_address for tracking suspension points (inspired by Folly's blog series on coroutine debugging)
• Document LLDB's support for devirtualization of std::coroutine_handle, both from an end user perspective as well as its internal implementation

[llvmbot]

@llvm/pr-subscribers-clang

@llvm/pr-subscribers-coroutines

Author: Adrian Vogelsgesang (vogelsgesang)

Changes

This commit is a major overhaul of the documentation on debugging C++ coroutines with the following goals:

• Make it more accessible to casual C++ programmers, i.e. non-toolchain developers. Move the low-level details around ABI further down, and instead start with real-life examples and copy-paste-friendly code, first.
• Cover LLDB in addition to GCC. Provide copy-pasteable scripts for LLDB and not only GCC.
• Cover additional topics, such as:
  • single-stepping into a coroutine
  • using __builtin_return_address for tracking suspension points (inspired by Folly's blog series on coroutine debugging)
• Document LLDB's support for devirtualization of std::coroutine_handle, both from an end user perspective as well as its internal implementation

---

Patch is 64.52 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/142651.diff

4 Files Affected:

• (modified) clang/docs/DebuggingCoroutines.rst (+844-459)
• (added) clang/docs/coro-async-task-continuations.png ()
• (added) clang/docs/coro-generator-suspended.png ()
• (added) clang/docs/coro-generator-variables.png ()

diff --git a/clang/docs/DebuggingCoroutines.rst b/clang/docs/DebuggingCoroutines.rst
index 80df321340724..b33e2cda91a30 100644
--- a/clang/docs/DebuggingCoroutines.rst
+++ b/clang/docs/DebuggingCoroutines.rst
@@ -8,470 +8,966 @@ Debugging C++ Coroutines
 Introduction
 ============
 
-For performance and other architectural reasons, the C++ Coroutines feature in
-the Clang compiler is implemented in two parts of the compiler.  Semantic
-analysis is performed in Clang, and Coroutine construction and optimization
-takes place in the LLVM middle-end.
+Coroutines in C++ were introduced in C++20, and their user experience for
+debugging them can still be challenging. This document guides you how to most
+efficiently debug coroutines and how to navigate existing shortcomings in
+debuggers and compilers.
+
+Coroutines are generally used either as generators or for asynchronous
+programming. In this document, we will discuss both use cases. Even if you are
+using coroutines for asynchronous programming, you should still read the
+generators section, as it will introduce foundational debugging techniques also
+applicable to the debugging of asynchronous programming.
+
+Both compilers (clang, gcc, ...) and debuggers (lldb, gdb, ...) are
+still improving their support for coroutines. As such, we recommend to use the
+latest available version of your toolchain.
+
+This document focuses on clang and lldb. The screenshots show
+[lldb-dap](https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.lldb-dap)
+in combination with VS Code. The same techniques can also be used in other
+IDEs.
+
+Debugging clang-compiled binaries with gdb is possible, but requires more
+scripting. This guide comes with a basic GDB script for coroutine debugging.
+
+This guide will first showcase the more polished, bleeding-edge experience, but
+will also show you how to debug coroutines with older toolchains. In general,
+the older your toolchain, the deeper you will have to dive into the
+implementation details of coroutines (such as their ABI). The further down in
+this document, the more low-level, technical the content will become. If you
+are on an up-to-date toolchain, you will hopefully be able to stop reading
+earlier.
+
+Debugging generators
+====================
+
+The first major use case for coroutines in C++ are generators, i.e. functions
+which can produce values via ``co_yield``. Values are produced lazily,
+on-demand. For that purpose, every time a new value is requested the coroutine
+gets resumed. As soon as it reaches a ``co_yield`` and thereby returns the
+requested value, the coroutine is suspended again.
+
+This logic is encapsulated in a ``generator`` type similar to
 
-However, this design forces us to generate insufficient debugging information.
-Typically, the compiler generates debug information in the Clang frontend, as
-debug information is highly language specific. However, this is not possible
-for Coroutine frames because the frames are constructed in the LLVM middle-end.
-
-To mitigate this problem, the LLVM middle end attempts to generate some debug
-information, which is unfortunately incomplete, since much of the language
-specific information is missing in the middle end.
+.. code-block:: c++
 
-This document describes how to use this debug information to better debug
-coroutines.
+  // generator.hpp
+  #include <coroutine>
 
-Terminology
-===========
+  // `generator` is a stripped down, minimal generator type.
+  template<typename T>
+  struct generator {
+    struct promise_type {
+      T current_value{};
 
-Due to the recent nature of C++20 Coroutines, the terminology used to describe
-the concepts of Coroutines is not settled.  This section defines a common,
-understandable terminology to be used consistently throughout this document.
+      auto get_return_object() {
+        return std::coroutine_handle<promise_type>::from_promise(*this);
+      }
+      auto initial_suspend() { return std::suspend_always(); }
+      auto final_suspend() noexcept { return std::suspend_always(); }
+      auto return_void() { return std::suspend_always(); }
+      void unhandled_exception() { __builtin_unreachable(); }
+      auto yield_value(T v) {
+        current_value = v;
+        return std::suspend_always();
+      }
+    };
 
-coroutine type
---------------
+    generator(std::coroutine_handle<promise_type> h) : hdl(h) { hdl.resume(); }
+    ~generator() { hdl.destroy(); }
 
-A `coroutine function` is any function that contains any of the Coroutine
-Keywords `co_await`, `co_yield`, or `co_return`.  A `coroutine type` is a
-possible return type of one of these `coroutine functions`.  `Task` and
-`Generator` are commonly referred to coroutine types.
+    generator<int>& operator++() { hdl.resume(); return *this; } // resume the coroutine
+    int operator*() const { return hdl.promise().current_value; }
 
-coroutine
----------
+    private:
+    std::coroutine_handle<promise_type> hdl;
+  };
 
-By technical definition, a `coroutine` is a suspendable function. However,
-programmers typically use `coroutine` to refer to an individual instance.
-For example:
+We can then use this ``generator`` class to print the Fibonacci sequence:
 
 .. code-block:: c++
 
-  std::vector<Task> Coros; // Task is a coroutine type.
-  for (int i = 0; i < 3; i++)
-    Coros.push_back(CoroTask()); // CoroTask is a coroutine function, which
-                                 // would return a coroutine type 'Task'.
+  #include "generator.hpp"
+  #include <iostream>
 
-In practice, we typically say "`Coros` contains 3 coroutines" in the above
-example, though this is not strictly correct.  More technically, this should
-say "`Coros` contains 3 coroutine instances" or "Coros contains 3 coroutine
-objects."
+  generator<int> fibonacci() {
+    co_yield 0;
+    int prev = 0;
+    co_yield 1;
+    int current = 1;
+    while (true) {
+      int next = current + prev;
+      co_yield next;
+      prev = current;
+      current = next;
+    }
+  }
 
-In this document, we follow the common practice of using `coroutine` to refer
-to an individual `coroutine instance`, since the terms `coroutine instance` and
-`coroutine object` aren't sufficiently defined in this case.
+  template<typename T>
+  void print10Elements(generator<T>& gen) {
+    for (unsigned i = 0; i < 10; ++i) {
+      std::cerr << *gen << "\n";
+      ++gen;
+    }
+  }
 
-coroutine frame
----------------
+  int main() {
+    std::cerr << "Fibonacci sequence - here we go\n";
+    generator<int> fib = fibonacci();
+    for (unsigned i = 0; i < 5; ++i) {
+      ++fib;
+    }
+    print10Elements(fib);
+  }
 
-The C++ Standard uses `coroutine state` to describe the allocated storage. In
-the compiler, we use `coroutine frame` to describe the generated data structure
-that contains the necessary information.
+To compile this code, use ``clang++ --std=c++23 generator-example.cpp -g``.
 
-The structure of coroutine frames
-=================================
+Breakpoints inside the generators
+---------------------------------
 
-The structure of coroutine frames is defined as:
+We can set breakpoints inside coroutines just as we set them in regular
+functions. For VS Code, that means clicking next the line number in the editor.
+In the ``lldb`` CLI or in ``gdb``, you can use ``b`` to set a breakpoint.
 
-.. code-block:: c++
+Inspecting variables in a coroutine
+-----------------------------------
 
-  struct {
-    void (*__r)(); // function pointer to the `resume` function
-    void (*__d)(); // function pointer to the `destroy` function
-    promise_type; // the corresponding `promise_type`
-    ... // Any other needed information
-  }
+If you hit a breakpoint inside the ``fibonacci`` function, you should be able
+to inspect all local variables (``prev```, ``current```, ``next``) just like in
+a regular function.
 
-In the debugger, the function's name is obtainable from the address of the
-function. And the name of `resume` function is equal to the name of the
-coroutine function. So the name of the coroutine is obtainable once the
-address of the coroutine is known.
+.. image:: ./coro-generator-variables.png
 
-Print promise_type
-==================
+Note the two additional variables ``__promise`` and ``__coro_frame``. Those
+show the internal state of the coroutine. They are not relevant for our
+generator example, but will be relevant for asynchronous programming described
+in the next section.
 
-Every coroutine has a `promise_type`, which defines the behavior
-for the corresponding coroutine. In other words, if two coroutines have the
-same `promise_type`, they should behave in the same way.
-To print a `promise_type` in a debugger when stopped at a breakpoint inside a
-coroutine, printing the `promise_type` can be done by:
+Stepping out of a coroutine
+---------------------------
 
-.. parsed-literal::
+When single-stepping, you will notice that the debugger will leave the
+``fibonacci`` function as soon as you hit a ``co_yield`` statement. You might
+find yourself inside some standard library code. After stepping out of the
+library code, you will be back in the ``main`` function.
 
-  print __promise
+Stepping into a coroutine
+-------------------------
 
-It is also possible to print the `promise_type` of a coroutine from the address
-of the coroutine frame. For example, if the address of a coroutine frame is
-0x416eb0, and the type of the `promise_type` is `task::promise_type`, printing
-the `promise_type` can be done by:
+If you stop at ``++fib`` and try to step into the generator, you will first
+find yourself inside ``operator++``. Stepping into the ``handle.resume()`` will
+not work by default.
 
-.. parsed-literal::
+This is because lldb does not step int functions from the standard library by
+default. To make this work, you first need to run ``settings set
+target.process.thread.step-avoid-regexp ""``. You can do so from the "Debug
+Console" towards the bottom of the screen. With that setting change, you can
+step through ``coroutine_handle::resume`` and into your generator.
 
-  print (task::promise_type)*(0x416eb0+0x10)
+You might find yourself at the top of the coroutine at first, instead of at
+your previous suspension point. In that case, single-step and you will arrive
+at the previously suspended ``co_yield`` statement.
 
-This is possible because the `promise_type` is guaranteed by the ABI to be at a
-16 bit offset from the coroutine frame.
 
-Note that there is also an ABI independent method:
+Inspecting a suspended coroutine
+--------------------------------
 
-.. parsed-literal::
+The ``print10Elements`` function receives an opaque ``generator`` type. Let's
+assume we are suspended at the ``++gen;`` line, and want to inspect the
+generator and its internal state.
 
-  print std::coroutine_handle<task::promise_type>::from_address((void*)0x416eb0).promise()
+To do so, we can simply look into the ``gen.hdl`` variable. LLDB comes with a
+pretty printer for ``std::coroutine_handle`` which will show us the internal
+state of the coroutine. For GDB, you will have to use the ``show-coro-frame``
+command provided by the :ref:`GDB Debugger Script`.
 
-The functions `from_address(void*)` and `promise()` are often small enough to
-be removed during optimization, so this method may not be possible.
+.. image:: ./coro-generator-suspended.png
 
-Print coroutine frames
-======================
+We can see two function pointers ``resume`` and ``destroy``. These pointers
+point to the resume / destroy functions. By inspecting those function pointers,
+we can see that our ``generator`` is actually backed by our ``fibonacci``
+coroutine. When using VS Code + lldb-dap, you can Cmd+Click on the function
+address (``0x555...`` in the screenshot) to directly jump to the function
+definition backing your coroutine handle.
 
-LLVM generates the debug information for the coroutine frame in the LLVM middle
-end, which permits printing of the coroutine frame in the debugger. Much like
-the `promise_type`, when stopped at a breakpoint inside a coroutine we can
-print the coroutine frame by:
+Next, we see the ``promise``. In our case, this reveals the current value of
+our generator.
 
-.. parsed-literal::
+The ``coro_frame`` member represents the internal state of the coroutine. It
+contains our internal coroutine state ``prev``, ``current``, ``next``.
+Furthermore, it contains many internal, compiler-specific members, which are
+named based on their type. These represent temporary values which the compiler
+decided to spill across suspension points, but which were not declared in our
+original source code and hence have no proper user-provided name.
 
-  print __coro_frame
+Tracking the exact suspension point
+-----------------------------------
 
+Among the compiler-generated members, the ``__coro_index`` is particularly
+important. This member identifies the suspension point at which the coroutine
+is currently suspended.
 
-Just as printing the `promise_type` is possible from the coroutine address,
-printing the details of the coroutine frame from an address is also possible:
+However, it is non-trivial to map this number back to a source code location.
+In simple cases, one might correctly guess the source code location. In more
+complex cases, we can modify the C++ code to store additional information in
+the promise type:
 
-::
+.. code-block:: c++
 
-  (gdb) # Get the address of coroutine frame
-  (gdb) print/x *0x418eb0
-  $1 = 0x4019e0
-  (gdb) # Get the linkage name for the coroutine
-  (gdb) x 0x4019e0
-  0x4019e0 <_ZL9coro_taski>:  0xe5894855
-  (gdb) # Turn off the demangler temporarily to avoid the debugger misunderstanding the name.
-  (gdb) set demangle-style none
-  (gdb) # The coroutine frame type is 'linkage_name.coro_frame_ty'
-  (gdb) print  ('_ZL9coro_taski.coro_frame_ty')*(0x418eb0)
-  $2 = {__resume_fn = 0x4019e0 <coro_task(int)>, __destroy_fn = 0x402000 <coro_task(int)>, __promise = {...}, ...}
+  // For all promise_types we need a new `line_number variable`:
+  class promise_type {
+    ...
+    void* _coro_return_address = nullptr;
+  };
 
-The above is possible because:
+  #include <source_location>
 
-(1) The name of the debug type of the coroutine frame is the `linkage_name`,
-plus the `.coro_frame_ty` suffix because each coroutine function shares the
-same coroutine type.
+  // For all the awaiter types we need:
+  class awaiter {
+    ...
+    template <typename Promise>
+    __attribute__((noinline)) auto await_suspend(std::coroutine_handle<Promise> handle) {
+          ...
+          handle.promise()._coro_return_address = __builtin_return_address(0);
+    }
+  };
 
-(2) The coroutine function name is accessible from the address of the coroutine
-frame.
+This stores the return address of ``await_suspend`` within the promise.
+Thereby, we can read it back from the promise of a suspended coroutine, and map
+it to an exact source code location. For a complete example, see the ``task``
+type used below for asynchronous programming.
 
-The above commands can be simplified by placing them in debug scripts.
+Alternatively, we can modify the C++ code to store the line number in the
+promise type. We can use a ``std::source_location`` to get the line number of
+the await and store it inside the ``promise_type``. Since we can get the
+promise of a suspended coroutine, we thereby get access to the line_number.
 
-Examples to print coroutine frames
-----------------------------------
+.. code-block:: c++
+
+  // For all the awaiter types we need:
+  class awaiter {
+    ...
+    template <typename Promise>
+    void await_suspend(std::coroutine_handle<Promise> handle,
+                       std::source_location sl = std::source_location::current()) {
+          ...
+          handle.promise().line_number = sl.line();
+    }
+  };
+
+The downside of both approaches is that they come at the price of additional
+runtime cost. In particular the second approach increases binary size, since it
+requires additional ``std::source_location`` objects, and those source
+locations are not stripped by split-dwarf. Whether the first approach is worth
+the additional runtime cost is a trade-off you need to make yourself.
+
+Async stack traces
+==================
 
-The print examples below use the following definition:
+Besides generators, the second common use case for coroutines in C++ is
+asynchronous programming, usually involving libraries such as stdexec, folly,
+cppcoro, boost::asio or similar libraries. Some of those libraries already
+provide custom debugging support, so in addition to this guide, you might want
+to check out their documentation.
+
+When using coroutines for asynchronous programming, your library usually
+provides you some ``task`` type. This type usually looks similar to this:
 
 .. code-block:: c++
 
+  // async-task-library.hpp
   #include <coroutine>
-  #include <iostream>
+  #include <utility>
 
-  struct task{
+  struct task {
     struct promise_type {
       task get_return_object() { return std::coroutine_handle<promise_type>::from_promise(*this); }
-      std::suspend_always initial_suspend() { return {}; }
-      std::suspend_always final_suspend() noexcept { return {}; }
-      void return_void() noexcept {}
+      auto initial_suspend() { return std::suspend_always{}; }
+
       void unhandled_exception() noexcept {}
 
-      int count = 0;
-    };
+      auto final_suspend() noexcept {
+        struct FinalSuspend {
+          std::coroutine_handle<> continuation;
+          auto await_ready() noexcept { return false; }
+          auto await_suspend(std::coroutine_handle<> handle) noexcept {
+            return continuation;
+          }
+          void await_resume() noexcept {}
+        };
+        return FinalSuspend{continuation};
+      }
 
-    void resume() noexcept {
-      handle.resume();
-    }
+      void return_value(int res) { result = res; }
 
-    task(std::coroutine_handle<promise_type> hdl) : handle(hdl) {}
+      std::coroutine_handle<> continuation = std::noop_coroutine();
+      int result = 0;
+      #ifndef NDEBUG
+      void* _coro_suspension_point_addr = nullptr;
+      #endif
+    };
+
+    task(std::coroutine_handle<promise_type> handle) : handle(handle) {}
     ~task() {
       if (handle)
         handle.destroy();
     }
 
-    std::coroutine_handle<> handle;
-  };
-
-  class await_counter : public std::suspend_always {
-    public:
-      template<class PromiseType>
-      void await_suspend(std::coroutine_handle<PromiseType> handle) noexcept {
-          handle.promise().count++;
+    struct Awaiter {
+      std::coroutine_handle<promise_type> handle;
+      auto await_ready() { return false; }
+
+      template <typename P>
+      #ifndef NDEBUG
+      __attribute__((noinline))
+      #endif
+      auto await_suspend(std::coroutine_handle<P> continuation) {
+        handle.promise().continuation = continuation;
+        #ifndef NDEBUG
+        continuation.promise()._coro_suspension_point_addr = __builtin_return_address(0);
+        #endif
+        return handle;
       }
+      int await_resume() {
+        return handle.promise().result;
+      }
+    };
+
+    auto operator co_await() {
+      return Awaiter{handle};
+    }
+
+    int syncStart() {
+      handle.resume();
+      return handle.promise().result;
+    }
+
+  private:
+    std::coroutine_handle<promise_type> handle;
   };
 
-  static task coro_task(int v) {
-    int a = v;
-    co_await await_counter{};
-    a++;
-    std::cout << a << "\n";
-    a++;
-    std::cout << a << "\n";
-    a++;
-    std::cout << a << "\n";
-    co_await await_counter{};
-    a++;
-    std::cout << a << "\n";
-    a++;
-    std::cout << a << "\n";
+Note how the ``task::promise_type`` has a member variable
+``std::coroutine_handle<> continuation``. This is the handle of the coroutine
+that will be resumed when the current coroutine is finished executing (see
+``final_suspend``). In a sense, this is the "return address" of the coroutine.
+It is as soon as the caller coroutine ``co_await`` on the called coroutine in
+``operator co_await``.
+
+The result value is returned via the ``int result`` member. It is written in
+``return_value`` and read by ``Awaiter::await_resume``. Usually, the result
+type of a task is a template argument. For simplicity's sake, we hard-coded the
+``int`` type in this example.
+
+Stack traces of in-fli...
[truncated]