the Chromium logo

The Chromium Projects
For Developers >

Crash Reports

The crash reporting system for Google Chrome is called Breakpad, although it is being replaced by Crashpad, which is already in use on Windows and Mac. Both are integrated into the Chromium source in //components/crash. Breakpad is disabled by default in Chromium, and crash reports are only available to Google employees, though they are frequently copied into the bug tracker for tracking and fixing.

Enabling Breakpad

Breakpad is disabled by default. The cross-platform way to turn on crash reporting is to perform an official, branded build of Chromium by setting GN arguments is_chrome_branded=true and is_official_build=true, which will require a src-internal checkout.

To enable Breakpad without doing a branded build, each platform has a unique way of enabling breakpad:

In Chromium

Windows

Windows uses crashpad by default. It's compiled in and enabled by default.

Linux

Breakpad is compiled in, but is not enabled by default. See IsCrashReportingEnabled() in chrome_browser_main_linux.cc for details on when it is enabled.

Set the gyp variable (GYP_DEFINES) linux_dump_symbols=1 to dump the debugging symbols for the chrome binary in the Breakpad symbol format as part of a build.

Chrome OS

Breakpad is compiled in, similar to Linux above. See the Chrome OS crash reporting FAQ for differences.

Mac

Mac uses Crashpad as its crash reporter instead of Breakpad. Crashpad is always compiled in and enabled, though reports are only stored locally and not uploaded by default.

Generating Crash Dumps Locally

After Breakpad is enabled, you can have crash report minidumps generated locally.

To get crash reports for content_shell and layout tests, see this page.

On Windows

Run chrome.exe --noerrdialogs to disable the OS-level error reporter. Any crashes will then cause a minidump file (.dmp) to be generated in %HOME%\AppData\Local\Chromium\User Data\Crashpad\reports.

On Linux

To make chromium write a .dmp file, run it like

CHROME_HEADLESS=1 out/Release/chrome

On Mac

To make Chromium write crash dumps, no additional work is needed. To test it, a renderer process can be triggered to crash by visiting “about:crash”, or for real fun, a browser process crash can be triggered by visiting “about:inducebrowsercrashforrealz”. The crash dump file is stored in ~/Library/Application Support/Chromium/Crashpad/completed.

$ ls -l ~/Library/Application\ Support/Chromium/Crashpad/completed

total 448

-rw-------@ 1 mark staff 229248 Mar 13 2015 4bfca440-039f-4bc6-bbd4-6933cef5efd4.dmp

Life of a Crash Report

When a Chrome process crashes, Breakpad or Crashpad springs into action by gathering information about the exception state, callstacks, stack memory, and loaded modules. It takes all of this information and puts it into a minidump file. This minidump is then HTTP POST uploaded to Google, along with some metadata. The metadata contains information like Chrome version, OS name and version, and crash keys.

Once the crash report and metadata are received, a server begins processing it. The first step is to symbolize the instruction pointers to get function names associated with the callstack. The processing servers have platform-neutral Breakpad symbol files for Chrome executables and operating system shared libraries, which allow symbolication of both Chrome functions and OS functions. After symbolizing the crash, the processor generates signatures to help group/bucket crash reports by their cause, implicated by shared stack frames.

The tools used by the processor to symbolize the stack are effectively the same as the minidump_stackwalk and minidump_dump tools available in Breakpad.

Once a crash report is processed, it is made available for view in the crash dashboard. The dashboard allows developers to drill down into crash reports, grouped by metadata. More documentation on this is available internally, linked to from the dashboard. The original minidump file can be downloaded from this frontend, as well.

Crash Signatures

There are three types of signatures generated by the crash processor:

Stack Signature

The stack signature is the topmost symbol name in the crashed thread, suffixed with a small hash of the callstack. This signature provides a high-level grouping and easy identification (as it is human-readable), but it can collide across versions for different complete stack traces. Variance in the crashing thread's callstack can also produce several signatures for the same root cause.

Stack Signature 2

This signature is merely a hash of the crashed stack. This makes it accurate for grouping crashes precisely, but it is not human readable and slight variance in a crashed stack defeat this signature's bucketing properties.

Magic Signature

The magic signature is heuristic-based, using application-specific knowledge of the Chromium codebase to find the most germane symbol to produce a signature. The magic signature creator has knowledge of CHECK/DCHECK, hangs caught by the ThreadWatcher watchdog, out-of-memory conditions, memory corruption signs, and other heuristics. This signature is highly simplified, which gives it good bucketing properties and protection against callstack variance, but the signature can often be generated for disjoint crashes from different versions. More information can be found in the processor's design document and this explanation.

Working with Minidumps

The minidump format originated on Windows, and can be natively loaded into a debugger on that platform. Doing this gives you access to local variables, callstacks, and integrates with the source viewer in Visual Studio.

On POSIX systems, however, a minidump is an opaque binary blob. It is possible to examine local variable state, but it requires manual work to do so. To work with minidumps, Breakpad provides two tools (more information here):

minidump_stackwalk prints the stack trace for all of the threads in the minidump. Note that without Breakpad symbol files, placed in a special directory structure, this will not symbolize the stack. It will merely print the %EIP, %EBP, and %ESP (or the x64 equivalent) for each frame and the code module in which the frame resides. minidump_dump outputs the stack memory for each thread as a hexadecimal string.

Extracting Local Variables on POSIX

This is one workflow for manually analyzing a minidump:

  1. Run minidump_dump and minidump_stackwalk on the target minidump file, piping output to text files.
  2. Find the target thread in the stackwalk (via the threadid).
  3. Find the corresponding target thread in the stackdump.
  4. Take the raw stack data for that thread, and remove the 0x prefix.
  5. Split the stack data on word-size boundaries (8 hexadecimal characters on a 32-bit platform).
  6. Using the information from the stackwalk, annotate EIP, EBP, and ESP in the stackdump to delineate stack frames. Note that the byte order/endianness in the stackdump may not match host order, i.e. the pointer badbeef may be represented in the minidump as efbedbba.
  7. Download the build products (executable and symbol file) for the version of Chrome you're working with. Disassemble the function you're interested in.
  8. Step through the assembly and see how data is being pushed and popped off the stack, and try to correlate it to the data in the stackdump.

Unfortunately, this process is neither straightforward nor quick. But there are alternatives for saving local variables to be visible in a crash report. See below for more information.

However, it may be possible to get most of the same information using WinDBG on Windows. As long as the CPU architecture is supported by Windows, the debugger will be able to understand the exception record and disassemble the code around the exception, show call stacks (without symbols though), and let you inspect the contents of the stack and registers. That may be enough to get you going on a good percentage of crashes, especially since the symbolized stack may be available on the crash server.

See also http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/crash-reporting/debugging-a-minidump

Symbolizing Minidumps

To symbolize a minidump that was not uploaded, you can use the crsym tool (internal instance of https://github.com/chromium/crsym/):

  1. Build minidump_stackwalk from inside the Chromium or Breakpad trees.
  2. Run minidump_stackwalk -m /path/to/minidump.dmp > /tmp/stack.txt
  3. Go to crsym and set the input type to be a Minidump Stackwalk.
  4. Paste the contents of /tmp/stack.txt into the input field.
  5. Press the Symbolize button and wait for the output.

To symbolize a minidump for a local build, follow the steps in https://code.google.com/p/chromium/issues/detail?id=304846#c14 for Mac, and see http://www.chromium.org/developers/decoding-crash-dumps for linux.

Production Debugging

If a crash is not reproducible, the only tools at one's disposal for investigation are what can be gleaned from a crash report. These are some tips for debugging builds running in production (i.e. on users' machines).

Aliasing Against Optimizations

Production builds are highly optimized to make the binary executable fast and small. As part of this, when the compiler is optimizing, it will reuse space on the stack for different variables. To defeat this optimization, you can use base::debug::Alias() on the variable. This will attempt to force the compiler to keep dedicated space around for the variable, so that its value is inspectable in a minidump stackdump.

Crash Keys

Another way to make variable values accessible in a crash report is to report it as metadata, rather than keeping it in the minidump. The metadata is viewable in the crash dashboard and is not contained in the minidump file. Learn more about debugging with crash keys here.

Checking Out a Release Branch

Production builds are generally not cut from the tip of tree/trunk, but are branched and stabilized via the dev, beta, and stable release channels. Follow these instructions for how to check out a release branch, which will give you the code for the version of the crash you're investigating.