the Chromium logo

The Chromium Projects
For Developers > How-Tos >

Using Dr. Memory

Note: Dr.Memory bot has retired and the documentation here is obsolete (added on Jul. 27, 2017)

Dr. Memory is a new open-source tool for finding memory errors - similar to Memcheck (Valgrind) but with Windows support, including detection of Windows handle leaks and GDI usage errors.

Dr. Memory is running on the MFYI waterfall MFYI waterfall where it finds bugs in Chromium on Windows.

If you want to try Dr. Memory on your machine, please do the following:

If for some reason you cannot reproduce a bot failure on your Windows machine, look for the exact command that started the tests and run that. For example, here is one variation of the test command that catches some memory bugs that the above commands may not catch:

src\tools\valgrind\chrome_tests.bat --brave-new-test-launcher --test-launcher-bot-mode --test <test> --tool drmemory_full --target Release --build-dir src\out

OK, I've found a real bug

Great!

Please file a crbug and add the Stability-Memory-DrMemory label.

drmemory_light vs drmemory_full

Light: searches for unaddressable accesses like OOB or use-after-frees, handle leaks, and graphical library usage errors, incurring a moderate execution slowdown.

Full: additionally searches for uninitialized reads but adds a large slowdown.

More on the "full" mode:

If you're familiar with the Valgrind guts you probably know that tools like Valgrind and Dr. Memory need to intercept and handle all system calls to know precisely what data is uninitialized, what is read and so on.

In contrast to the open-source Linux kernel with just a hundred system calls or so, on Windows we have an open-ended problem of handling undocumented system calls. We're trying hard to do that but we haven't finished yet. We have some heuristics that work pretty well to ignore errors stemming from system libraries making system calls as opposed to Chromium code, but if you do see an error report that you believe is a false positive, please create a short reproducer and file it into our issue tracker.

Suppressing error reports from the bots

TODO(rnk): When the DrMemory bots have moved to the chromium.memory.fyi waterfall, this information will be moved into the memory sheriff page.

timurrrr: most of this info sounds DrMemory-specific, do we really want to put it onto the "common" page?

The DrMemory suppression file is at tools/valgrind/drmemory/suppressions[_full].txt. Generally speaking, suppressions in these files are maintained in exactly the same way as those for all of the other memory tools we use for Chromium, except that DrMemory uses a slightly different suppression format.

When the bot generates a report, follow the link to the bot's stdio and search for "hash=#" to find the report quickly. You should see something similar to:

UNADDRESSABLE ACCESS: reading 0x00000000-0x00000004 4 byte(s) # 0 TestingProfile::FinishInit [chrome\test\base\testing_profile.cc:211] # 1 TestingProfile::TestingProfile [chrome\test\base\testing_profile.cc:164] # 2 BrowserAboutHandlerTest_WillHandleBrowserAboutURL_Test::TestBody [chrome\browser\browser_about_handler_unittest.cc:110] # 3 testing::Test::Run [testing\gtest\src\gtest.cc:2162]Note: @0:00:08.656 in thread 1900 Note: instruction: mov (%ebx) -> %esi Suppression (error hash=#30459DB8320B1FA3#): { UNADDRESSABLE ACCESS name=<insert_a_suppression_name_here> *!TestingProfile::FinishInit *!TestingProfile::TestingProfile *!BrowserAboutHandlerTest_WillHandleBrowserAboutURL_Test::TestBody *!testing::Test::Run }

As a starting point, you can copy the suppression (the portion between the curly brackets not including the brackets) into the suppressions.txt file and then widen it so it covers any similar reports on other bots. The curly brackets are not part of the suppression as in all the other memory tools! DrMemory suppressions are separated by blank lines instead, so make sure your suppression does not run into an existing suppression.

Each frame from the stack trace is of the form "<module>!<function>". For functions from Chrome, we generally link the code statically into many different binaries, so we always use a wildcard for the module. For third party code, it is generally helpful to identify which DLL the frame came from, especially since we often do not have debug info for windows system libraries.

As in the other tool suppression formats, '*' is a single frame variable-width wildcard, '?' is a single-character wildcard, and '...' will match any number of frames.

DrMemory has support for a handful of other pattern matching mechanisms. In particular, if you paste the instruction from the report on a line starting with "instruction=", you can match the exact instruction producing the report. The instruction= qualifier is mostly useful when suppressing bit-level false positive uninitialized reads in third party libraries where we don't have an accurate callstack and want to make a tight suppression.

Once you have your suppression, you can test that it suppresses the existing reports using tools/valgrind/waterfall.sh as you would with the other memory tools.

Running Chromium under Dr. Memory on a given set of URLs

First, put the list of URLs into tools\valgrind\reliability\url_list.txt, one URL per line without the http:// prefix. You can also put in the local URLs like file://c:\mypath\repro.htm

Then, run

tools\valgrind\chrome_tests.bat -t reliability --tool drmemory --tool_flags="-no_check_uninitialized -no_count_leaks"

If you're searching for some stability/security bugs, you may consider running Chromium in the single process mode to increase the probability of the report by applying this patch:

Index: tools/valgrind/chrome_tests.py

--- tools/valgrind/chrome_tests.py (revision 96002) +++ tools/valgrind/chrome_tests.py (working copy) @@ -271,7 +271,8 @@

Valgrind timeouts are in seconds.

UI_VALGRIND_ARGS = ["--timeout=7200", "--trace_children", "--indirect"]

UI test timeouts are in milliseconds.

Running your custom command line under Dr. Memory

Sometimes you may need to run your own program or your own custom command line under Dr. Memory. For example, if you want to prefix your own "chrome.exe --my-flags" command with Dr. Memory, do the following: tools\valgrind\drmemory.bat chrome.exe -- --my-flags # don't forget to increase the timeouts If the command line runs Chrome, you may consider increasing the timeouts by appending

--ui-test-action-timeout=120000 --ui-test-action-max-timeout=280000 --ui-test-terminate-timeout=120000

to the commandline.

Random notes for the curious ones

You can find the Dr. Memory suppression file for Chrome at tools\valgrind\drmemory\suppressions[_full].txt

The report files are parsed and re-formatted to be more familiar for Valgrind users (see tools\valgrind\drmemory_analyze.py).

Excluded tests are listed in tools\valgrind\gtest_exclude - see Using Valgrind more for information.

Some debugging tips while using Dr. Memory can be found here.

For those still wanting to use Valgrind, be aware Valgrind requires Chrome be built using the system memory allocator. To do this, add use_allocator = "none" to your gn args.

Feedback?

Drop drmemory-team@ a message