the Chromium logo

The Chromium Projects
For Developers >

Crash Reports

The crash reporting system for Google Chrome is called Crashpad. It is integrated into the Chromium source in //components/crash. Crashpad is enabled by default for both Chromium and Chrome, though for Chromium crash reports are only stored locally. For Chrome, crash reports can be uploaded to Google's crash reporting infrastructure, but isn't until the user has opted in. Uploaded crash reports are only available to Google employees.

Differences on Chrome OS

Crash reporting on Chrome OS works differently than on other platforms. For more information, see the Chrome OS crash reporting FAQ.

Generating Crash Dumps Locally

Crash reports will be generated by default, with details available by visiting chrome://crashes.

To test this, a renderer process can be triggered to crash by visiting chrome://crash, or for real fun, a browser process crash can be triggered by visiting chrome://inducebrowsercrashforrealz. The crash dump file will be stored in a directory relative to the User Data Directory.

For example, for Chromium running on macOS:

$ ls -l ~/Library/Application\ Support/Chromium/Crashpad/completed
total 448
-rw-------@ 1 <owner> <group> 229248 Mar 13 2015 4bfca440-039f-4bc6-bbd4-6933cef5efd4.dmp

To get crash reports for content_shell and layout tests, see this page.

Life of a Crash Report

When a Chrome process crashes, Crashpad springs into action by gathering information about the exception state, callstacks, stack memory, and loaded modules. It takes all of this information and puts it into a minidump file. This minidump is then HTTP POST uploaded to Google, along with some metadata. The metadata contains information like Chrome version, OS name and version, and crash keys.

Once the crash report and metadata are received, a server begins processing it. The first step is to symbolize the instruction pointers to get function names associated with the callstack. The processing servers have platform-neutral Breakpad symbol files for Chrome executables and operating system shared libraries, which allow symbolication of both Chrome functions and OS functions. After symbolizing the crash, the processor generates signatures to help group/bucket crash reports by their cause, implicated by shared stack frames.

The tools used by the processor to symbolize the stack are effectively the same as the minidump_stackwalk and minidump_dump tools available in Breakpad.

Once a crash report is processed, it is made available for view in the crash dashboard. The dashboard allows developers to drill down into crash reports, grouped by metadata. More documentation on this is available internally, linked to from the dashboard. The original minidump file can be downloaded from this frontend, as well.

Crash Signatures

There are three types of signatures generated by the crash processor:

Stack Signature

The stack signature is the topmost symbol name in the crashed thread, suffixed with a small hash of the callstack. This signature provides a high-level grouping and easy identification (as it is human-readable), but it can collide across versions for different complete stack traces. Variance in the crashing thread's callstack can also produce several signatures for the same root cause.

Stack Signature 2

This signature is merely a hash of the crashed stack. This makes it accurate for grouping crashes precisely, but it is not human readable and slight variance in a crashed stack defeat this signature's bucketing properties.

Magic Signature

The magic signature is heuristic-based, using application-specific knowledge of the Chromium codebase to find the most germane symbol to produce a signature. The magic signature creator has knowledge of CHECK/DCHECK, hangs caught by the ThreadWatcher watchdog, out-of-memory conditions, memory corruption signs, and other heuristics. This signature is highly simplified, which gives it good bucketing properties and protection against callstack variance, but the signature can often be generated for disjoint crashes from different versions. More information can be found in the processor's design document and this explanation.

Working with Minidumps

The minidump format originated on Windows, and can be natively loaded into a debugger on that platform. Doing this gives you access to local variables, callstacks, and integrates with the source viewer in Visual Studio.

On POSIX systems, however, a minidump is an opaque binary blob. It is possible to examine local variable state, but it requires manual work to do so. To work with minidumps, Breakpad provides two tools (more information here):

minidump_stackwalk prints the stack trace for all of the threads in the minidump. Note that without Breakpad symbol files, placed in a special directory structure, this will not symbolize the stack. It will merely print the %EIP, %EBP, and %ESP (or the x64 equivalent) for each frame and the code module in which the frame resides. minidump_dump outputs the stack memory for each thread as a hexadecimal string.

Extracting Local Variables on POSIX

This is one workflow for manually analyzing a minidump:

  1. Run minidump_dump and minidump_stackwalk on the target minidump file, piping output to text files.
  2. Find the target thread in the stackwalk (via the threadid).
  3. Find the corresponding target thread in the stackdump.
  4. Take the raw stack data for that thread, and remove the 0x prefix.
  5. Split the stack data on word-size boundaries (8 hexadecimal characters on a 32-bit platform).
  6. Using the information from the stackwalk, annotate EIP, EBP, and ESP in the stackdump to delineate stack frames. Note that the byte order/endianness in the stackdump may not match host order, i.e. the pointer badbeef may be represented in the minidump as efbedbba.
  7. Download the build products (executable and symbol file) for the version of Chrome you're working with. Disassemble the function you're interested in.
  8. Step through the assembly and see how data is being pushed and popped off the stack, and try to correlate it to the data in the stackdump.

Unfortunately, this process is neither straightforward nor quick. But there are alternatives for saving local variables to be visible in a crash report. See below for more information.

However, it may be possible to get most of the same information using WinDBG on Windows. As long as the CPU architecture is supported by Windows, the debugger will be able to understand the exception record and disassemble the code around the exception, show call stacks (without symbols though), and let you inspect the contents of the stack and registers. That may be enough to get you going on a good percentage of crashes, especially since the symbolized stack may be available on the crash server.

See also https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/crash-reporting/debugging-a-minidump

Symbolizing Minidumps

To symbolize a minidump that was not uploaded, you can use the crsym tool (internal instance of https://github.com/chromium/crsym/):

  1. Build minidump_stackwalk from inside the Chromium or Breakpad trees.
  2. Run minidump_stackwalk -m /path/to/minidump.dmp > /tmp/stack.txt
  3. Go to crsym and set the input type to be a Minidump Stackwalk.
  4. Paste the contents of /tmp/stack.txt into the input field.
  5. Press the Symbolize button and wait for the output.

To symbolize a minidump for a local build, follow the steps in https://code.google.com/p/chromium/issues/detail?id=304846#c14 for Mac, and see https://www.chromium.org/developers/decoding-crash-dumps for linux.

Production Debugging

If a crash is not reproducible, the only tools at one's disposal for investigation are what can be gleaned from a crash report. These are some tips for debugging builds running in production (i.e. on users' machines).

Aliasing Against Optimizations

Production builds are highly optimized to make the binary executable fast and small. As part of this, when the compiler is optimizing, it will reuse space on the stack for different variables. To defeat this optimization, you can use base::debug::Alias() on the variable. This will attempt to force the compiler to keep dedicated space around for the variable, so that its value is inspectable in a minidump stackdump.

Crash Keys

Another way to make variable values accessible in a crash report is to report it as metadata, rather than keeping it in the minidump. The metadata is viewable in the crash dashboard and is not contained in the minidump file. Learn more about debugging with crash keys here.

Checking Out a Release Branch

Production builds are generally not cut from the tip of tree/trunk, but are branched and stabilized via the dev, beta, and stable release channels. Follow these instructions for how to check out a release branch, which will give you the code for the version of the crash you're investigating.

Crash Reports in Chrome

Metadata about crash reports generated by Chrome can be found by visiting chrome://crashes in the Omnibox. Each crash report entry on the page includes the date and time that the crash occurred, along with a status indicating whether the crash report has been uploaded to Google's crash report collection system. For reports that have been uploaded, an "Uploaded Crash Report ID" is shown, providing a unique identifier for the crash.

For Googlers, the crash report ID is particularly useful. It can be used to access the processed crash report by visiting the short link go/crash/<crash report id>.

View Uploaded Crash Report IDs via the Command Line

In some cases, such as persistent crashes upon browser startup, it may not be possible to use chrome://crashes to get the crash report ID. In these situations, the uploaded crash report IDs can typically be obtained via the command line.

For developers with a Chromium checkout, build the crashpad_database_util target (ex: autoninja -C out/Default crashpad_database_util) and then use the resulting executable to show contents similar to what's in chrome://crashes. This will require determining the directory containing the crash reports, which can be done by finding the User Data Directory and then appending the following to the directory path, depending on the platform:

Example output, from macOS:

$ out/Default/crashpad_database_util --database="${HOME}/Library/Application Support/Google/Chrome/Crashpad" --show-pending-reports --show-completed-reports --show-all-report-info
...
Report 8b1a1c49-fb17-461b-8462-2e6b5d653fa5:
  Path: /Users/<username>/Library/Application Support/Google/Chrome/Crashpad/completed/8b1a1c49-fb17-461b-8462-2e6b5d653fa5.dmp
  Remote ID: acbf91ecd2dabaeb
  Creation time: 2025-12-17 14:53:41 EST
  Uploaded: true
  Last upload attempt time: 2025-12-17 14:58:29 EST
  Upload attempts: 1
...

For additional information on crashpad_database_util, see the corresponding man page.