the Chromium logo

The Chromium Projects
ChromiumOS > Guides >

Firmware test manual

This is a Firmware test manual targeted for any tester needing to work with AP, EC, or GSC firmware.

Prerequisites

Bug Filing Procedure

Generate and attach Logs

Run generate_logs on the device, scp the generated .tar.gz file to your desktop and attach it to the bug.

Get PD/EC Console Log while running Tests

First, you must locate the target PTY (pseudo-TTY) to use for console communication.

Once the desired PTY is known, multiple different tools can be used to connect to that interface.

Device Precondition

Unless otherwise specified, device (sometime refer to as DUT - Device Under Test) are assumed to be setup with ChromeOS test image and updated with the target test firmware.

Setup DUT for Case Closed Debugging (CCD)

If your DUT is running a production cr50 image, see the Case Closed Debugging documentation.

Device Mode Setup

Check Device Mode (Developer, Normal or Recovery)

Change Device Mode from Developer to Normal

Groot UI

  1. Check device mode is set to developer.

  2. Reboot the device (via power button).

  3. You should see the screen with:

    You are in developer mode
To return to the recommended secure mode,
select Return to secure mode below.

  4. Select Return to secure mode option.

  5. You should see the screen with:

    Confirm returning to secure mode
This option will disable developer mode and restore your device to its
original state.
Your user data will be wiped in the process.

  6. Select Confirm option.

  7. System will reboot itself and switch to normal mode.

  8. Check to confirm the device is in normal mode.

Pre-Groot UI

  1. Check device mode is set to developer.

  2. Reboot the device (via power button).

  3. You should see the screen with:

    OS verification is OFF
Press SPACE to re-enable

  4. Press space to enable OS verification, you should see:

    OS verification is OFF
Press ENTER to confirm you wish to turn OS verification on.
Your system will reboot and local data will be cleared.
To go back press ESC.

  5. Press enter key to confirm, screen should display: OS verification is ON Your system will reboot and all data will be cleared.

  6. System will reboot itself and switch to normal mode.

  7. Check to confirm the device is in normal mode.

Recovery Mode

Press the following key sequence to put the device into recovery mode.

Change Device Mode from Normal to Developer

Groot UI

  1. Power on the device, if the screen shows You are in developer mode, the device is already in developer mode.

  2. Check device mode is set to normal.

  3. Press Esc + F3/Refresh + Power to enter recovery mode. The screen should display:

    Let’s step you through the recovery process.
Select how you’d like to recover. You can recover using
external storage such as a USB drive.

  4. Press Ctrl-D you should see:

    You are attempting to enable developer mode.
This involves erasing all data from your device, and will
make your device insecure.
Select Cancel to remain protected.

  5. If hitting Ctrl-D does nothing, check Stuck at Recovery Screen.

  6. Select Confirm and press ENTER, a screen would display the message:

    You are in developer mode
To return to recommended secure mode, select “Return to
secure mode” below. After timeout, the device will
automatically boot from the selection below.

  7. The System will reboot itself in 30 seconds (to bypass the wait time, hit Ctrl-D).

  8. Check to confirm the device is in developer mode.

Pre-Groot UI

  1. Power on the device, if the screen shows OS verification is OFF, the device is already in developer mode. On older chromebox, flip the switch on the back to go to the developer mode.

  2. Check device mode is set to normal.

  3. Press Esc + F3/Refresh + Power to enter recovery mode. The screen should display: ChromeOS is missing or damaged. Please insert a recovery USB stick or SD card.

  4. Press Ctrl-D you should see:

    To turn OS verification OFF, press ENTER.
Your system will reboot and local data will be cleared.
To go back, press ESC.

  5. If hitting Ctrl-D does nothing, check Stuck at Recovery Screen.

  6. Press enter, screen should display:

    OS verification is OFF
Press SPACE to re-enable.

  7. System will reboot itself in 30 seconds (to bypass the wait time hit Ctrl-D).

  8. Check to confirm the device is in developer mode.

Developer Mode Screen Sequences

  1. Press Esc+F3+Power at the login screen.
  2. Press Ctrl+D at ChromeOS is missing or damaged..
  3. Press ENTER at To turn OS verification OFF, press ENTER..
  4. Press Ctrl+D at OS verification is OFF.
  5. Screen changes to Your system is transitioning to Developer Mode..
  6. Screen will change to Preparing system for Developer Mode. after 30 seconds.
  7. After few minutes, the system will return to OS login screen.

Firmware Shellball (chromeos-firmwareupdate)

Firmware Shellball is a shell archive or shar. Located on device as /usr/sbin/chromeos-firmwareupdate.

Firmware Update Modes

Refer to the Firmware Updater guide for chromeos-firmwareupdate mode options.

Extract and repack the Firmware Shellball

Refer to the Firmware Updater Package guide for more information.

Follow the below instructions to repack the shellball

Make rootFS Writable

Firmware Versions

Check Firmware Version

crossystem fwid gets the running AP firmware's version from /sys/devices/platform/chromeos_acpi/FWID or /sys/devices/platform/GGL0001:00/FWID or /sys/devices/platform/GOOG0016:00/FWID on x86 devices and /proc/device-tree/firmware/chromeos/firmware-version on ARM devices.

chromeos-firmwareupdate is the installer for both the AP Firmware and the EC firmware. When updating, "Last Boot Version:" in the output is populated via crossystem. "Firmware Updater:" (i.e. the version contained within the updater that it will attempt to install when you execute the updater) is populated via the manifest that is part of chromeos-firmwareupdate.

How to Check cr50 version with chrome://system

Flashing Firmware via Futility

If you are logged-in to the DUT:

futility update -i PATH_TO/image.bin

If the DUT is connected via a servo to a workstation (and servod is running on your workstation):

sudo futility update --servo -i PATH_TO/image.bin

Re-signing the Firmware

In the event that you need to resign a test firmware, refer to futility's inline help:

futility help sign

How to determine Firmware Version from the OS Image

Intel's Management Engine

Write Protect

For details about write protect, and how to enable and disable the feature, please read Write Protection.

Power Delivery

ChromeOS Image

To create a USB stick with a ChromeOS image, follow the instructions in the developer guide at Put your image on a USB disk.

If using a prebuilt image instead of one that is locally built, do the following:

To boot from a USB disk, follow the instructions at Boot from your USB disk in the ChromiumOS Developer guide.

Firmware Update

Firmware Screen Names

Here is a list of the various SCREENs referenced in some test steps:

Legacy UI (devices launched until 2019)

New revised recovery screen for most device created post 2016.

New screen for diagnostic tools device created post 2020.

Diagram

General Procedure

Basic

Virtual Terminals

There are 4 types of terminals.

Update GBB Flags (AKA extend recovery screen timeout)

GBB flags are located in a read-only region of firmware, and control various firmware features (developer mode, EC software sync, etc). They can be manipulated for ease of developer and testing experience:

Stuck at Recovery Screen

Reset EC

Reset TPM Values

The version of the keys used to sign the firmware and kernel are stored in the trusted platform module (TPM) and are used for rollback protection, i.e. in normal mode you cannot boot a firmware or kernel whose key version is less than what is stored in the TPM. The versions of the keys are reported from crossystem as tpm_fwver and tpm_kernver.

In most testing scenarios, you will be using dev-keyed firmware builds. For dev-keyed firmware builds the key versions stored in the firmware will map to a value of 0x10001 for tpm_fwver. You might need to reset the values in the TPM if you were running another firmware build that had higher key versions and you now want to run a dev-keyed build. You can reset the stored values as follows:

Power State

Run power_supply_info from VT2 to report power related stat like details on battery state, battery charge percentage, if AC power is plugged in.

Get device IP

Hardware ID Mismatch

LED Color Charge State

Refer to the "LED behavior" section of the latest requirements for the relevant form factor. See the Latest Chromebook requirements for clamshell devices; that page has links to requirements for other form factors.

Mount USB stick on ChromeOS Device

Storage

Before putting away the device for long term storage, open VT2 and run ectool batterycutoff.

ChromeOS Firmware Troubleshooting

Checking Failure Status

In order for the firmware to help, we will need to know the following details:

Getting Recovery Reason

On recent firmware (both x86 and arm), a new feature is added to help diagnosing boot (recovery screen) failure. When you see the BIOS screen (whether a blue/pink developer screen or recovery screen), try to press the TAB key. Some debug message should appear on left-top corner of the screen, including HWID string and a code for the reason why your device enters recovery mode.

Find the message "Recovery Reason: 0x??", where ?? is a hex number; then find the corresponding entry in this document (or the source header).

Frequently Encountered Recovery Reasons

Here is the list of well-known recovery reasons and possible solutions.

Debugging "OS missing" failures (0x43, 0x5b)

To boot into system, the verified firmware must complete following:

  1. Find internal storage (eMMC, SSD, or NVMe).
  2. Read and parse a valid GPT (Partition table).
  3. Find a partition with "ChromeOS kernel" type and is marked as "good and ready for booting" (by GPT attributes).
  4. Load the kernel contents into memory and verify its digital signature.
  5. Jump into loaded kernel.

The verified boot firmware has changed the recovery code several times. For old devices, failure in step 1~4 may result in error code 0x48, 0x5a, 0x42, or other values. For most devices today the error codes for each steps listed above are:

  1. Failed finding storage: 0x5a VB2_RECOVERY_RW_NO_DISK
  2. Failed getting GPT: 0x5b VB2_RECOVERY_RW_NO_KERNEL
  3. Failed finding kernel entry: 0x5b VB2_RECOVERY_RW_NO_KERNEL
  4. Failed verifying kernel: 0x43 VB2_RECOVERY_RW_INVALID_OS

The difference between 0x5b "No bootable kernel found on disk" and 0x43 "OS kernel or rootfs failed signature check" is if the firmware can find at least one partition table entry with "ChromeOS kernel" type and proper boot priority attributes. However, the two errors can be particularly frustrating to debug, as there are many potential causes:

Collect information

When on the error screen, press [tab] to get debug information and collect.

Since there are three potential reasons to the failure, to debug we need to read back the firmware and disk contents of the device. Connect servo (or enable CCD) and then:

Failure in Finalization

If the device goes to the recovery page with "OS missing" errors immediately after running factory finalization, that may be caused by "unexpected reboot or failure during finalization", since in the process we will de-active test image partitions (2) and enable release partition (4). Check the factory logs for this.

As a reference, we have seen few finalization issues that failed after GBB was cleared (and before the new release partition was activated):

Failure during run-in

If the DUT goes to 0x43 during run-in tests, it's usually caused by DMVERROR or other storage driver issues. The following sections will help you to get more details.

Find active partition

Read gpt.txt and look for partition 2 and 4, which should look like


          69       65536       2  Label: "KERN-A"
                                  Type: ChromeOS kernel
                                  UUID: 8AF3A85D-AC80-A44F-AA0A-CA8C73E66811
                                  Attr: priority=1 tries=0 successful=1
...
       65605       65536       4  Label: "KERN-B"
                                  Type: ChromeOS kernel
                                  UUID: 0BC160CA-E154-1749-9F79-D5838FAAD064
                                  Attr: priority=0 tries=15 successful=0

There must be at least one partition with priority > 0 and (tries > 0 or successful = 1). Use that partition for the following tests.

Note for devices failed immediately after factory finalization, they should have active = 4. For devices failed recovery, active = 2. For failure after OOBE or Auto Update, both partitions may be active, so choose the one with higher priority.

The examples below will assume your active one is 4 (kern.4.gz).

Check kernel integrity

Run the command in chroot to check:

gunzip kern.4.gz
futility verify kern.4

If the kernel looks good you should see some information like:

Keyblock:
  Signature:           ignored
  Size:                0x4b8
  Flags:               7  !DEV DEV !REC
  Data key algorithm:  4 RSA2048 SHA256
  Data key version:    1
  Data key sha1sum:    d6170aa480136f1f29cf339a5ab1b960585fa444
Preamble:
  Size:                0xfb48
  Header version:      2.2
  Kernel version:      1
  Flags:               0
  Body load address:   0x100000
  Body size:           0x7d7000
  Bootloader address:  0x8d6000
  Bootloader size:     0x1000
Body verification succeeded.
Config:
console= loglevel=7 init=/sbin/init cros_secure root=/dev/dm-0 ......

Otherwise, the kernel is either corrupted with unknown reason or by DMVERROR.

Check DMVERROR

Print the first 8 bytes of kernel blob to check.

dd if=kern.4 bs=1 count=8 2>/dev/null

A real ChromeOS kernel should have CHROMEOS.

For devices failed due to DM verity error, that will be DMVERROR. You will want to run a memory stress test (stressapptest) to see if this is caused by memory issues. Otherwise, it may be failure by storage driver, or even hardware failure inside storage.

If you see anything else (or nothing), it is probably fully corrupted or an empty partition.

OS image key mismatch

In normal mode, firmware and storage partitions must be signed with matching keys for boot to succeed. If the keys mismatch, you may encounter this error.

The debug screen contains key information in the following fields:

To change the keys on a device, disable hardware write-protect and flash a recovery image with the desired key. See the Firmware Write Protection document for details.

If you can't access chromeos-hwid or are not sure which keys were installed on the DUT and just want to know if the boot failure is caused by keys mismatch, follow the steps:

futility verify image.bin
futility dump_fmap image.bin -x VBLOCK_A:vb.a VBLOCK_B:vb.b
futility verify kern.4 --publickey vb.a
futility verify kern.4 --publickey vb.b

At least one of vb.a or vb.b should decode kernel partition properly with something like:

Keyblock:
  Signature:           valid
  Size:                0x4b8
  Flags:               7  !DEV DEV !REC
  Data key algorithm:  4 RSA2048 SHA256
  Data key version:    1
  Data key sha1sum:    d6170aa480136f1f29cf339a5ab1b960585fa444
Preamble:
  Size:                0xfb48
  Header version:      2.2
  Kernel version:      1
  Flags:               0
  Body load address:   0x100000
  Body size:           0x7d7000
  Bootloader address:  0x8d6000
  Bootloader size:     0x1000
Body verification succeeded.
Config:
console= loglevel=7 init=/sbin/init cros_secure root=/dev/dm-0 ...

If you can't verify the active kernel by either firmware (in A or B), then it's probably caused by firmware key mismatch.

Corrupted image on storage

The simplest cause of this error is that there is no valid image on the disk. To confirm if this is the case, you should follow the following steps:

Both should return with the string "CHROMEOS". If not, the partition is corrupted, and that is what is preventing boot.

Consult the ChromeOS Disk format document for more details on expected image format.

eMMC/NVMe/SATA communication errors

Data corruption from storage can cause signature checking to fail. This is a good case to check for if using a new storage part or the hardware design is early in development. This is especially suspicious if the system boots reliably over USB.

To diagnose this potential issue, run the quick storage test: tast run storage.QuickStress*

TODO(b/295578792): Replace this with a different test.

Communication errors with cr50

Bad straps

Bad hardware strap information for H1 can disrupt communication between cr50 and the AP (Application processor). The cr50 console will log strapping information at boot, with a line like this: [0.006304 Valid strap: 0x2 properties: 0x21]

If the strapping configuration is invalid, instead, a line like this will be emitted: [0.006244 Invalid strap pins! Default properties = 0x46]

Pulse width too short for SoC

cr50 sends pulses on the interrupt line that are ~2.6us in length. SoCs should be careful to manage clock gating registers to ensure that pulses that short may be detected, otherwise TPM communication may fail.

ChromeOS Installation Troubleshooting Guide

Introduction

This document describes the process to restore your device when the device no lo nger boots or is stuck at the recovery screen. The general problem is usually a key mismatch between OS and firmware. There are three main types of keys: MP, pre-MP and Dev (test). Special care is needed with key changes. Here is a table on when you need special care and when it is a simple upgrade.

To Dev/Test To PreMP To MP version N To MP version N+M
From Dev/Test: Normal Install Normal Key Change (flowchart) Normal key change (flowchart) N/A
From PreMP: Normal key change (flowchart) Normal Install Normal Key Change N/A
From MP version N: Reset TPM Unlikely to Happen Normal Install Normal upgrade (need confirmation)
From MP version N+M: N/A N/A Normal install (need to clear TPM) N/A

Your device must be in working order in order to switch OS with a different key. Once you are stuck at the recovery screen you will need to perform extra steps to restore the device to working order.

Use Dev image as bridge

The fool-proof method is to install a dev sign OS and firmware before switching to the new OS. Here are the high level steps:

Flow Chart

Troubleshooting Flow Chart

Step 1: Before you start

Determine the key of your DUT image

cat /etc/lsb-release and check the value of CHROMEOS_RELEASE_BOARD to determine your image key. For example, on swanky, CHROMEOS_RELEASE_BOARD is set to swankey-signed-mpkeys which indicates that the device has MP signed image on the system. You will need this information in order to determine which path to take below. If you don’t know there is a way to find out in Step 2.

Select migration path

There are 3 types of keys: MP, PreMP and Unsigned (aka test image). If your source image key and target image key is the same, you will have no issue. If the source and target is different you will need special care. It is rare that you will need to go from MP key to PreMP key, if you must, switch to dev image first.

Step 2: State of your device

The device is fully working

Fully working means you can boot to ChromeOS successfully. If your DUT can boot to ChromeOS, skip to section "Migrate your device" and follow the directions.

The device is stuck at recovery screen

You will need to restore your device to a working order before continue. You will need to reinstall the device with an OS signed with the same key type. Once you have done that, you can go to "Migrate your device" and follow the directions.

Device stuck at recovery screen and don’t know the original OS key type

There are two ways to determine the original image key of your DUT if you can get to the OS:

  1. get gbb.rootkey, locate key in factory firmware and get key type
  2. create USB stick for each type and insert stick at the recovery screen until it boots

To get gbb.rootkey, press tab at the recovery screen.

To find the HWID file that describes the keys for a given device, go to the chromeos-hwid repository and find the desired file for the given device name.

Step 3: Migrate your device OS

Below are list of high level steps.

From MP key image to preMP key image or unsigned image

You will need test firmware, and a USB stick with the target OS image flashed onto it.

  1. Disable hardware write protect
    • remove screw, unplug battery, or disable through CCD (mechanism depends on target device)
    • crossystem wpsw_cur from VT2 should return 0 to indicate you have successfully disabled write protect.
  2. disable software write protect, futility flash --wp-status should return disabled
  3. boot from USB
    • change device to dev mode
    • set crossystem dev_boot_usb=1
    • reboot
    • ctrl-U to boot from USB
  4. install OS and firmware from VT2
    • chromeos-firmwareupdate --mode=factory
    • chromeos-install
    • reboot
  5. set GBB value to 0
  6. reset TPM if either tpm_fwver or tpm_kernver != 0x10001

From pre-MP key image to MP key image or unsigned image

Same instructions as the section above.

Reset DUT with unsigned image

You will need an USB stick with the unsigned ChromeOS image.

  1. remove hardware write protect
  2. boot your device to ChromeOS
  3. go to recovery mode
  4. insert USB with OS image and boot
  5. disable software write-protect futility flash --wp-disable
  6. reset firmware chromeos-firmwareupdate --mode=factory
  7. install image with chromeos-install
  8. shutdown -P 0 to shutdown
  9. press power to start OS
  10. reset GBB
  11. reset TPM if either tpm_fwver or tpm_kernver > 0x10001

From unsigned image to MP key image or premp key image

Same as Reset DUT with unsigned image except you will need to boot your device to ChromeOS with:

Device in dev mode

  1. set crossystem dev_boot_usb=1
  2. reboot
  3. ctrl-U to boot from USB

Possible problems

  1. bad USB
    • get a new USB stick or use a known good one.
  2. corrupt image on USB
    • some test will cause USB stick to be corrupted
    • recreate USB stick with image
  3. USB has the right image
    • when in doubt flash a new image and try again

Kernel log Troubleshoot Guide

Commands to inspect kernel log information

Go to VT2, press the dmesg command to check the messages. Linux kernel can print logs and trace messages, which can by default be stored in a ring buffer. dmesg is a one type of shell command on the kernel , which displays the content of the ring buffer. Another way to display the contents of the kernel log is to look at the content of the file /var/log/messages. /var/log/messages includes all the system messages including from the boot process of the chromeOS along with dmesg output.