Google Research has developed a system that can monitor your heart rate simply by looking at your face through your smartphone’s front camera. The technology works passively in the background during normal phone use, capturing brief video clips after you unlock your device with face recognition.

This breakthrough matters because it could make heart health monitoring accessible to the five billion smartphone users worldwide, particularly in areas where fitness trackers and medical devices are too expensive or unavailable. More importantly, Google’s system is the first to achieve medical-grade accuracy across all skin tones – addressing a critical bias problem that has plagued pulse oximeters and other health monitoring devices.

How the technology works

The Passive Heart Rate Monitoring (PHRM) system uses a technique called photoplethysmography, which detects tiny changes in how light reflects off your skin as blood flows through it. When you unlock your phone with face recognition, the system captures an 8-second video clip and uses AI to analyze subtle color changes in your face that correspond to your heartbeat.

The system then processes these measurements throughout the day to calculate your resting heart rate – a key indicator of cardiovascular health. Higher resting heart rates are linked to increased risk of heart disease and death from all causes.

Solving the skin tone accuracy problem

Previous heart rate monitoring technologies have struggled with darker skin tones because melanin makes it harder for cameras and sensors to detect the subtle changes needed for accurate readings. This has created serious equity issues in health tech – pulse oximeters, for example, have been shown to provide less accurate readings for Black patients.

Google addressed this by training their system on over 350,000 video clips from nearly 700 diverse participants. They deliberately ensured that:

• At least 25% of participants had light skin (Monk scale 1-4)
• At least 25% had medium skin (Monk scale 5-7)
• At least 33% had dark skin (Monk scale 8-10)

The company also set a strict requirement that the system’s accuracy must not vary by more than 5 percentage points between different skin tone groups.

Real-world testing results

Google tested the system in both controlled laboratory settings and real-world conditions. In the lab study with 365 participants, PHRM achieved less than 10% error compared to medical-grade electrocardiogram readings across all skin tone groups.

The real-world study was particularly ambitious. Google had 231 people install a special app on their personal phones and use them normally for eight days while wearing chest-strap heart monitors and Fitbit devices for comparison. The app captured an average of 231 video clips per day after each face unlock.

Results showed the system achieved:

• 5.04% error for light skin tones
• 5.12% error for medium skin tones
• 7.84% error for dark skin tones

All groups met medical industry accuracy standards of less than 10% error. For daily resting heart rate estimates, the system matched the accuracy of fitness trackers like the Fitbit Charge 6, with less than 5 beats per minute difference on average.

Privacy and practical challenges

The system faces some limitations that Google acknowledges. Success rates were lower for people with darker skin, though accuracy remained high when measurements were successful. The technology also struggles when people are talking or moving their heads significantly during the brief recording window.

Privacy represents another consideration. While Google’s research version required participants to manually approve video uploads after reviewing them, any commercial implementation would need robust on-device processing to protect user data.

Making research resources available

Google is releasing its dataset and a simplified version of the AI model to qualified researchers with institutional approval. This represents the largest and most diverse collection of smartphone-based heart rate monitoring data ever made available for research.

The dataset is restricted to non-commercial research use, and researchers are prohibited from trying to identify individuals or publicly displaying the video content.

This research could eventually make heart health monitoring as simple as unlocking your phone, potentially helping millions of people track cardiovascular health who couldn’t otherwise afford or access fitness trackers or medical monitoring devices.