Rider backend crash. MacOS Arm64. Collect Managed Call Stack for the hanged or crashed Rider.Backend process

Background/Situation

After MacOS Sonoma release, something got broken in the process of collecting a crash dump of Rider.Backend process on M-based macOS using lldb debugger. (How to collect a dump on a specific crashed or hanged process | Unix-based Operating systems | How to collect a crash dump)

Different errors like Error: Fail to initialize CoreCLR 80070008 , SOS_HOSTING: Fail to initialize hosting runtime '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.29/libcoreclr.dylib' 80004005 , error: this file does not represent a loadable dylib, and unexpected behaviors like lldb session termination during attempts to invoke setsymbolserver complicate the process of taking the dump. One of the known solutions is to build lldb locally and use its binaries, however, it takes time.

Question

How to avoid these errors and take that clrstack -all dump using lldb without building llvm locally?

Workaround

• On macOS Developer mode has to be enabled, as well as user should be added to the admin or developer group:

  sudo DevToolsSecurity --enable
  sudo dseditgroup -o edit -a $USER -t user _developer
  

• Apply the below workaround for lldb (taken from https://github.com/dotnet/diagnostics/blob/main/documentation/FAQ.md)

  $ sudo cp /Applications/Xcode.app/Contents/Developer/usr/bin/lldb /usr/local/bin
  $ sudo install_name_tool -add_rpath /Applications/Xcode.app/Contents/SharedFrameworks /usr/local/bin/lldb
  $ sudo codesign --force --sign - /usr/local/bin/lldb
  

  Ensure to adjust the path to xcode in the above commands (if you have several xcode installations or have a custom path installation, for ex, with Xcode 16.2 installed as /Applications/Xcode16.2.app/, add "16.2" to the paths in commands);

• Comment out any customizations made to ~/.lldbinit (add # => # yourEarlierAddedCommands ). At least comment out any sos plugin loading;

• Get the PID of the Rider.Backend process ps -eo pid,command | grep Rider.Backend | grep -v grep

• Get a path to Rider-bundled sos plugin:

  • You have the path to Rider installation printed after the above command execution. Take the part ending with macos-arm64 (including);
  • Append it with /dotnet/sos/libsosplugin.dylib ;
  • So the sample path is /Users/Dmitry.Kazantsev/Applications/Rider.app/Contents/lib/ReSharperHost/macos-arm64/dotnet/sos/libsosplugin.dylib

• Attach LLDB to a process by PID /usr/local/bin/lldb -p 52083 ;

• Invoke commands one by one:

  process handle -s false SIGUSR1
  plugin load /your/path/to/macos-arm64/libsosplugin.dylib
  setsymbolserver https://resources.jetbrains.com/pdb
  setsymbolserver https://msdl.microsoft.com/download/symbols
  loadsymbols
  

  • Note: Loading symbols might take a while (1-3 minutes), wait while (lldb) appears on the next line;
  • If still invoking plugin load fails, exit from session and launch lldb from xcode installation: /Applications/Xcode.app/Contents/Developer/usr/bin/lldb

• Once it is finished, invoke c to resume the backend process;

• Reproduce the issue in Rider;

• When Rider backend turns into a broken state:

  • crashes -> switch to lldb console;
  • hangs -> switch to lldb console and invoke process interrupt;

• Adjust the file path in the below code snippet using any text editor and copy it to the clipboard:

  with open('/Users/Dmitry.Kazantsev/Area/clrstack_out.txt', 'w') as f:
      res = lldb.SBCommandReturnObject()
      lldb.debugger.GetCommandInterpreter().HandleCommand('clrstack -all', res)
      if res.Succeeded():
          f.write(res.GetOutput())
      else:   
          f.write(res.GetError())
  
  

• Switch back to the lldb terminal and invoke script ;

• Paste the snippet from the clipboard (CMD+V) and press Enter twice;

• Generating a call stack and writing it to a file might take ~2 minutes;

• Once it is finished, leave the interactive python session via Ctrl+D;

• (End), optionally c for resume, exit for detaching debugger;

Find the file written by the debugger (for ex, /Users/Dmitry.Kazantsev/Area/clrstack_out.txt) and share where requested.