How to collect performance metrics in tests

We use Opentelemetry to collect performance metrics in integration tests.

Since it is almost always not possible to collect performance metrics in IDE functionality itself (IDE starts some actions and does not know when they will be finished), we collect them in Performance plugins' commands. We call the functionality of IDE in a command and find a way in the command how we could be sure that the called action has finished. It makes it possible to collect performance metrics since we know exactly when we called a functionality and when it is finished.

Since tests are under continuous development, collected metrics are subject to change (the metric's name can be changed, and how we collect it can also be changed)

We can provide here an overview of the main metrics for some old and important functionality that we measure in our integration tests.

Tests' metrics

Indexing

For now, the main metrics in this functionality that should be tracked are indexingTimeWithoutPauses and scanningTimeWithoutPauses. It shows the indexing and scanning times without interruptions (pauses). There are also indexSize and processingSpeed and processingTimemetrics, which display the overall index size, indexing speed and time of different file types. They, as well as the metric numberOfIndexedFiles, can be used in the investigations of performance issues.

Import

Different metrics related to the import/ reimport of Gradle and Maven projects. For more detailed information, please take a look at Gradle Import and Maven Import dashboards.

Completion

The main metric that should be tracked here is completion. The metric is a sum of all individual completion times executed in the test. So if you invoke completion ten times, the completion metric will show how long they take together. The metric measures the execution time from calling completion till all possible elements have been added to the completion list.

Note: The metric relies on a com.intellij.codeInsight.completion.CompletionPhaseListener#completionPhaseChanged listener that can be triggered after typing so feel free to ask @maxim.kolmakov if you find the result strange.

Show Intentions

The main metric here is the showQuickFixes metric. It shows how long it took to show the quick fixes after calling Alt + Enter. You can also track the following two metrics which show maximum and average AWT delay time: test#average_awt_delay and test#max_awt_delay

Local inspections

The main metric here is called localInspections. It displays the execution time of the restarted DaemonAnalyzer (the file is highlighted and all local inspections are executed). firstCodeAnalysis is also very useful and shows how long analysis took the first time a file was opened.

Inspections

Metric globalInspections displays the execution time of all inspections defined in the default inspection profile (the time of displaying results is included).

FindUsages

Metrics findUsages and findUsages#number are the main metrics here. findUsages is the execution time which takes from calling find usages on an element till all find usages are found (not displayed). findUsages#number displays the number of found usages for a given element.

File Structure

File structure can be called on the different types of files. The main metric is FileStructurePopup. The time is needed to display and fill a popup with information about the structure of a given file.

Search everywhere

Can be called with different Modes: Action, File, Class. Depending on the called functionality, should be tracked one of these three main metrics: searchEverywhere_action, searchEverywhere_class, or searchEverywhere_file.

Typing

We measure the typing execution time (with a given tempo) - metric typing. Typing places characters with a specified delay in the current TypingTarget (usually, this is the Editor). Then the editor inserts it and reports when it’s ready to accept new input. These two metrics, which measure AWT delay time can also be interesting here: test#average_awt_delay and test#max_awt_delay There are many metrics which measure responsiveness time in the typing tests: test#max_awt_delay metric shows valid delays on AWT as well any long WA during typing typing#latency#max metric shows delays during typing (the time between an event type till the text is shown in the editor). It will not show delays if we type and another process steals AWT thread

System metrics

CPU | Load |Total % 95th pctl - 95th percentile of system CPU load Memory | IDE | RESIDENT SIZE (MB) 95th pctl - 95 percentile of system memory that was occupied by IDE and all its children processes

Garbage Collector's metrics

fullGCPause - the time that full GC was active (IDE is fully paused) gcPause - the time spent in GC (including minor collections without pausing) gcPauseCount - the number of minor GC's pauses freedMemoryByGC - the amount of memory that was freed by the GC

Pages' metrics

Page Table is a data structure used by the virtual memory system to store the mapping between logical addresses and physical addresses.

The CPU generates logical addresses for the pages of the processes therefore they are generally used by the processes.

Physical addresses are the actual frame address of the memory. They are generally used by the hardware or more specifically by RAM subsystems.

pageHit - When the CPU attempts to obtain a needed page from main memory and the page exists in main memory (RAM), it is referred to as a PAGE HIT. This metric displays the number of successful Pages' obtainments. pageLoad - is the number of regular Pages' loads. pageMiss - If the needed page has not existed in the main memory (RAM), it is known as PAGE MISS. The metric displays the number of unsuccessful Pages' obtainments.

How do we measure AWT delay?

We place new empty actions in the EventQueue and see how long it takes to get the action from the queue. The time in the queue is called responsiveness or delay. It’s a rough but pretty good approximation of delays that customers experience plus it’s a rough approximation of how long it takes for a character to appear on the screen.

What are all other metrics for?

Please ask in #ij-integration-performance-tests-help, Platform QA Automation team will provide an answer and update this document if needed.