Comparing 10 Sleep Trackers (2017)
How well do they track your sleep? A 9-day minute-by-minute comparison.
by Jina Yoon — jina@brown.edu
I'm an undergraduate researcher at Brown University working in a research group that spends a lot of time thinking about sleep tracking devices and apps. Three years ago, sleep specialist and physician Dr. Christopher Winter wrote an article titled "Personal Sleep Monitors: Do They Work?". Today, with over 500 options from more than 15 brands available, I was curious to conduct a similar study three years later on how the data from personal sleep tracking devices compare with each other.
I wore 10 of some of the most popular sleep trackers (including one research-standard actigraph as a control) across 10 nights and then graphed the data against each other. If you're only interested in the results, feel free to skip to the end, but here's a rundown of the devices used, and what data I was looking at. I’m also including links to guides and source code if you’d like to make a similar chart yourself!
Devices and Apps
Sleep happens in five phases, but these are often categorized by consumer-level devices into two to three stages: light sleep, deep sleep, and REM sleep. Detecting REM sleep with a worn device or app is virtually impossible without polysomnography, a test that records your brain waves, blood oxygen levels, heart rate, breathing, as well as eye and leg movements during the study. It requires the placement of electrodes on your head and is usually performed in a clinic for one night.
Because polysomnography is expensive and inconvenient, personal activity trackers offer simpler, more affordable, and longer-term options for people like you and me. They are typically placed on the bed or worn on your wrist, and use accelerometer sensors (to detect your movement), sometimes along with a microphone to detect noise, but they are not nearly as accurate as polysomnography used in clinical sleep studies.
Here's a rundown of each of the devices I used in my experiment. *I wanted to include the BASIS Peak that "won" Dr. Winter's original experiment, but that device was recently recalled due to safety concerns.
| Device | Price | Type | Sensors | Sleep Stages | Notes |
|---|---|---|---|---|---|
| AMI MotionLogger | $795, without software | Wristband | Accelerometer, Temperature, Heart Rate Monitor | Awake, Asleep | Research-standard actigraph. Our control, i.e. "gold standard". We chose to compute sleep using the most commonly known Sadeh algorithm. |
| Fitbit Alta | $130 | Wristband | Accelerometer | Awake, Restless, Asleep | One of the top-selling activity tracker brands on Amazon. |
| Jawbone UP (discontinued) | $34 | Wristband | Accelerometer | Awake, Light Sleep, Deep Sleep | Another top-selling brand. |
| Microsoft Band (discontinued) | $125 | Wristband | Accelerometer, Heart Rate Monitor, Skin Temperature | Awake, Light Sleep, Deep Sleep | We are not sure which sensors are used for sleep stage computation because their algorithms have not been publicly released. |
| Microsoft Band 2 | $224 | Wristband | Accelerometer, Heart Rate Monitor, Skin Temperature | Awake, Light Sleep, Deep Sleep | An update of the original Microsoft Band, though now it is reportedly discontinued as well. |
| SleepCycle iOS | Free ($9.99/yr Premium) | iPhone App | Microphone | Awake, Sleep, Deep Sleep | SleepCycle does not clarify what thresholds are used for classifying sleep stages, so our chart presents our best estimates based on the given images from the app. |
| Sleep as Android | Free ($3.99 Unlock) | Android App | Acclerometer, Microphone | Awake, Light Sleep, Deep Sleep, REMS | The most downloaded sleep app in the Google Play Store. Claims to be able to detect REM sleep. |
| SleepCoacher Android + iOS | Free | iPhone and Android Apps | Accelerometer | Awake, Asleep | A specially developed app by our research group that offers personalized sleep recommendations based on your habits and experiments. |
| Hello Sense (without Voice) | $149 | Tabletop Sense + Clip-on Pill | Accelerometer, Light Sensor, Microphone | Awake, Light, Deep | A system that tracks not only your motion, but also your sleep environment during the night. The exact values they show are unclear, so we do a best estimate. |
Experiment
Gathering the data was simple in theory: I just had to use all ten devices simultaneously for ten nights. So, I wore all five wristbands on the same arm (Fitbit Alta, Jawbone UP, Microsoft Band, Microsoft Band 2, AMI MotionLogger), three phone apps placed side-by-side on my bed (Sleep as Android, SleepCoacher iOS, SleepCoacher Android), one phone app placed on my nightstand (Sleep Cycle iOS), one device clipped to my pillow (Hello Sense Sleep Pill) and one product that required two sensors clipped onto my pillow and placed on my nightstand (Hello Sense). It was, as expected, not very comfortable -- which might be why my sleep patterns seem quite restless over the 10 nights!
Extracting the data from the devices proved to be a bit of a challenge. If you’re interested in a detailed explanation of how I did this, you can check out my data extraction guide. It'll walk you through how to unlock your minute-by-minute (or stage-by-stage) data from your device. This isn't always as straightforward as you would expect -- in fact, we were of the first to unlock data from the Hello Sense (thanks to my advisor)!
Results
Below is the visualization of the data I retrieved, where each chart is one night of sleep and each row is a different sleep tracking device or app. There are a few things to note that you can find in the limitations section later. You can hover over sections of each night to see timestamps, and click through the links to different nights.
As for the data visualization, I used this d3 chart. I tweaked the margins and the legend just a tiny bit for this project, so if you'd like to check out my version, you can find it here (they are virtually the same).
Conclusion
While the data I gathered was better than random, the results from my experiment should still be taken with a grain of salt. My experience with these 10 devices may differ from yours, and each device has its strengths and weaknesses. If we consider only the results from the AMI MotionLogger as the “gold standard” for our accuracy study, the Fitbit Alta seems to be the most accurate among the other 9 in terms of sleep versus awake data. However, each device varies in price, convenience, design, popularity, and plenty of other factors more that you might consider differently based on your lifestyle.
In the end, we found that most devices were similar to each other if they were based on the same type of sensor (accelerometer vs. noise) and the device type (phone vs. wristband vs. other). Ultimately, sleep is best tracked through polysomnography, but these devices offer much more accessible and convenient options for casual users to track their activity and learn more about their habits during both the night and day. These devices offer some powerful data tracking tools, but it’s best to let the experts analyze your sleep if you have any chronic conditions. Our findings tell that these consumer-level sleep reports should be taken with a grain of salt, but regardless we’re happy to see more and more people investing in improving their sleep -- after all, there’s nothing better than a good night’s rest!
If you're interested in trying SleepCoacher, you can download the SleepCoacher app for Andoid. You can read more about how it works, on our Medium article. The iOS app is still under development by researchers in my research group at Brown University, but you can sign up for updates on our sleep website here! By signing up, you'll receive an invitation to be part of the first group to receive personalized recommendations through our app. SleepCoacher works by guiding you through small experiments to find the optimal sleep environment and schedule for you. The more you use it, the better the recommendations get.
*Limitations
- The chart is missing Fitbit Alta data on December 6-7 because my battery died.
- The Band 2 lost contact with my wrist when I was asleep sometimes leading to gaps in the data on quite a few of the nights.
- It was impossible to wear them on the same exact part of my arm every night, and so the different distances from my body may have biased the data.
- I could not physically turn on all the devices on/off at the same exact time, so the start and end times for each device are slightly different.
- I was unable to try a real polysomnograph for this study, so the AMI MotionLogger in this study is the closest approximation of actual sleep/wake.
- 5 of the 10 devices used in this study were all wristbands that used accelerometer sensors to compute sleep data, and they were closer to each other (and my body) than they were to the phone apps. They were also all worn on the same arm as the AMI MotionLogger so seem more similar to it.