IDE crashes when Debug button is clicked (cause: MSI Afterburner)

Problem description

When the debug button is clicked, the debugger UI doesn't open. The application crashes and closes immediately.

Reinstalling the IDE doesn't help, it still crashes every time. Disabling/uninstalling Antivirus software doesn't help either.

The crash happens during loading of native libraries (KERNEL32.DLL) after the debugger is launched:

# A fatal error has been detected by the Java Runtime Environment: 
# 
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffc050d8d80, pid=17992, tid=14088 
# 
# JRE version: OpenJDK Runtime Environment JBR-21.0.5+8-631.28-jcef (21.0.5+8) (build 21.0.5+8-b631.28) 
# Java VM: OpenJDK 64-Bit Server VM JBR-21.0.5+8-631.28-jcef (21.0.5+8-b631.28, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64) 
# Problematic frame: 
# C  [KERNEL32.DLL+0x18d80] 
# 
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows 
# 
# If you would like to submit a bug report, please visit: 
#   https://youtrack.jetbrains.com/issues/JBR 
# The crash happened outside the Java Virtual Machine in native code. 
# See problematic frame for where to report the bug. 
# 

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) 
C  [KERNEL32.DLL+0x18d80] 

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) 
j  jdk.internal.loader.NativeLibraries.load(...)Z+0 java.base
j  ...
j  java.lang.System.loadLibrary(Ljava/lang/String;)V+7 java.base
j  com.sun.tools.jdi.SharedMemoryTransportService.<init>()V+11 jdk.jdi
j  ...
j  com.jetbrains.jdi.SharedMemoryAttachingConnector.createSharedMemoryTransportService()
j  com.jetbrains.jdi.SunCommandLineLauncher.<init>()V
j  com.jetbrains.jdi.VirtualMachineManagerImpl.<init>()V
j  com.intellij.debugger.engine.DebugProcessImpl.findConnector(...)

How to identify this crash

In the crash log (java_error_in_idea_*.log), look for:

• RSI register pointing to RTSSHooks64.dll
• RTSSHooks64.dll in the loaded Dynamic libraries section (path: C:\Program Files (x86)\RivaTuner Statistics Server\RTSSHooks64.dll)
• The return address on the stack (stack at sp + 0 slots) pointing into RTSSHooks64.dll

Root cause

The crash log refers to RTSSHooks64.dll. This library is from the tool called "RivaTuner" that comes with MSI Afterburner (graphics card overclocking utility). RTSS hooks Windows API calls (including LoadLibrary) to inject its GPU monitoring overlay. When the IDE's debugger tries to load the shared memory transport native library (dt_shmem.dll), RTSS's hook intercepts the call and causes a native crash.

Solution

Option 1 (recommended): Add the IDE to the RTSS exclusion list

This allows the user to keep MSI Afterburner and RTSS running for GPU monitoring while preventing the crash:

1. Open RivaTuner Statistics Server (usually in the system tray)
2. Click the green Add button at the bottom of the window
3. Browse to the IDE executable (e.g. C:\Program Files\JetBrains\IntelliJ IDEA\bin\idea64.exe)
4. Set Application detection level to None
5. Restart the IDE

Option 2: Disable MSI Afterburner

Close MSI Afterburner (which also stops RivaTuner) before launching the IDE. Check the system tray to make sure RTSS is not running separately.