Indexing diagnostics 101

This article describes new diagnostics, introduced in 2023.2 and covered by Indexing diagnostic: rework to describe separate scanning & indexing stages

• IntelliJ-based IDEs rely heavily on indexes, which are precomputed information about which files in project contain which data: text, functions with certain names, etc.
• Developers edit files in their projects, so indexes should be updated. Updating contains of two parts: scanning, when we determine which files changed, and indexing, when new values of indexes are computed and saved for them. Each opening of a project results in one full scanning with telling type "full on project open".
• Currently no two activities of scannings and indexings ever intersect in time. After each scanning that found at least one file that should be reindexed, happens an indexing that does it. However it's possible that after two scannings comes single indexing that handles all collected files. More about it in Perform "Scanning files to index..." in smart mode.
• Many actions can't be done correctly when indexes are outdated. So they are disabled in so called dumb mode. Indexing always enables dumb mode while it runs. Scanning doesn't start it until some preconfigured number of changed files is found.
• Log directory ('Help | Show Log in ...') contains indexing-diagnostic directory which for every opened project contains a directory named as its name plus hash. Indexing diagnostics files for that project lay there. By default they are limited by total size, and oldest ones are removed on excess.

Summary of activities in report.html

File report.html contains a table with list of latest indexings and scannings of project. They are ordered by beginning time (Time | Started). Scannings have blue background.

• Scanning Type may be full, partial or refresh: whole project was checked, some structural subset of it, for example, new library was added and indexes should be computed for its files, or virtual file system reported a set of changed files. Also scanning may be "forced", which means user deliberately called it with Repair IDE... action.

• Scanning ID(s) Each scanning has unique id. For indexings this column contains ids of scannings which provided some of the files they are handling.

• Time

  • For every activity there are Start time, Started dumb mode and Finished times. For indexings Started and Started dumb mode times are always the same. Scannings may not start dumb mode at all if they don't find enough changed files.
  • Total time is time between Start and Finished moments.
  • Both indexings and scannings may be put on hold either by user clicking on pause button on progress indicator, or automatically during heavy world-changing operations like import from Maven. Time spent on pause is written in Time spent on pause. It may be important for analysis as part of total time that can't be removed by optimizing indexing subsystem.
  • Full dumb mode time is the time dumb mode was enabled including pauses. May be interesting to evaluate time a user was waiting for IDE to become fully functional. Dumb mode time w/o pauses tells us how much of that time is actually on indexing subsystem.

• Files

  • Scanned - is available for scannings only. Number of files that were checked if indexes should be recalculated for them.
  • When shared indexes are used, it may occur during scanning that index values for a changed file are already provided by shared indexes, and needn't be calculated during indexing. Number of such files is provided in Shared indexes (w/o content loading). Not available for indexings.
  • Number of scanned files for which index values should be recalculated are provided in Scheduled for indexing. Not available for indexings.
  • Number of files which were indexed during indexing using shared indexes is provided in Shared indexes (content loaded). Not available for scannings.
  • Total number of indexed files during an indexing is provided in Total indexed (shared indexes included)
  • Collector of changed files is checked during every indexing. Retrieved files are also indexed, and included in numbers in report.html: (incl. *** changed changed in VFS) in Total indexed (shared indexes included).

  Detailed reports of activities

  Every detailed report of activity begins with Application info and Runtime tables.

  Detailed report of a scanning

  • Overview table contains all data provided in report.html plus some additional values.
  • After certain preparations platform starts iterating files on multiple threads checking if their values in indexes should be reevaluated. Related times are provided in Concurrent part of scanning table. Time there is a sum of CPU times on multiple threads with pauses unless stated otherwise.
    • Files to be checked during scanning are defined via so-called providers which give them to platform one after another. Each provider is processed on a separate thread.
    • For each provider first it's iterated to get all of its files, and then scanners are applied to them. See Time of iterating VFS and applying scanners
    • Next step is checking each of provided files if it should be indexed. See Time of checking files.
    • Processing a file may include checking if its index values are already provided by shared indexes. Sum of durations of all such checks on all threads, including pauses, if they happened at that time, is provided in Time of running shared indexes (without loading content, part of checking files).
    • Other parts of file checking are mentioned in Scanning per provider table.
  • Information on every provider's handling is provided in Scanning per provider table. Rows are ordered by total time.
    • All times in the table include pauses. They can't be longer than Wall time with pauses of concurrent part, but can exceed Wall time w/o pauses
    • Only longest to handle providers get into the table by default. By unchecking Hide minor data one can get the full list of providers there.
    • Stages of provider's handling are mentioned in description of Scanning per provider table.
    • Every file handling is basically pushing and getting status. Pushing is writing into file's attributes various important for indexing parameters like language level or dialect. Status is should file be indexed or not. Time of getting files' statuses is provided, and pushing time is the rest of Total time.
    • Getting file status includes processing up-to-date files, updating content-less indexes, indexing by shared indexes.
      • Shared indexes may require evaluating index values for files that seem up-to-date. Sum of durations of such checks is provided in Time processing up-to-date files
      • During file checking values of content-less indexes like FilenameIndex may be evaluated. Sum of durations of such evaluations is provided in Time updating content-less indexes
      • Some files may not need to be indexed because values of their indexes are already provided via shared indexes. Sum of durations of such checks is provided in Time indexing by shared indexes without content, and number of such files is provided in Number of files fully indexed by shared indexes

Detailed report of an indexing

• Overview table contains all data provided in report.html plus some additional values.
• All provided durations include pauses unless specified otherwise.
• File system performance is important for indexing, so two times are provided: Content loading time and Index writing time.
• Content loading is a part of code which is run in parallel, so time here should be regarded as visible time of loading if all files' content were loaded on indexing threads simultaneously. In fact relation of CPU content loading tile to total CPU processing time is multiplied by wall time of processing.
• Index writing time is calculated the same way.
• After indexing files that came from scannings, platform's changed files are requested and also indexed. Their number and time of their retrieval are provided in Number of files changed in VFS and Time of retrieving list of the files changed in VFS.
• Durations in tables Statistics per file type, Statistics per indexer, Indexing are wall time including pauses.
• Only rows with longest durations are provided in tables by default. By unchecking Hide minor data one can get the full data there.

Important system properties & settings

With default values. To check the code see IndexDiagnosticDumper.

• -Dintellij.indexes.diagnostics.should.dump.for.interrupted.index.updaters=false
• -Dintellij.indexes.diagnostics.limit.of.files=300
• -Dintellij.indexes.diagnostics.size.limit.of.files.MiB.per.project=10
• -Dintellij.indexes.diagnostics.should.dump.paths.of.indexed.files=false
• -Dintellij.indexes.diagnostics.should.dump.provider.root.paths=false
• -Dintellij.indexes.diagnostics.should.provide.visible.and.all.threads.time.info=false
• com.intellij.util.indexing.diagnostic.IndexDiagnosticDumper.Companion#getShouldDumpInUnitTestMode
• -Dintellij.indexes.diagnostics.should.dump.paths.indexed.by.infrastructure.extensions = ApplicationManagerEx.isInIntegrationTest()