Inside Fitbit’s top-secret testing lab: Where sweat is turned into data

About three years ago, a lot of Fitbit users suddenly realised they were being shortchanged. Parents pushing strollers weren’t getting credit for their hard-earned steps, all because of an algorithmic oversight in their fitness trackers. They noticed – and Fitbit got to work: it went out and bought every type of stroller it could find. “Every kind of hand orientation you can imagine,” recalls Tracy Giest, Fitbit’s manager of R&D operations. Once it had amassed a diverse fleet, it filled each enough weight to mimic a baby, and had employees push them around the streets of San Francisco, with a Fitbit to collect the data.

This is the usual – sometimes bizarre – work that happens inside Fitbit’s Human Research lab. Since it was founded in 2007, the company has built a sizeable operation in San Francisco and it's here that sweat is turned into data and ideas evolve into health devices.

See the devices: Our reviews of Fitbit's trackers and smartwatches

At first glance it looks no different to any other gym: rows of running machines and peloton bikes form aisles across the room, with dumbbells and benches in one corner for weightlifting. But look a little closer and you'll see hints at what really happens here: rows of computers, whiteboards scribbled with complex looking equations – and a high-security entrance.

It's here that Fitbit collects data from test subjects to improve the accuracy of its wearables, but it’s also a skunkworks for the company’s future and furthest-reaching ideas. “We get to touch devices and products in all stages of development cycle,” says Giest. ”It’s in some cases very much years out, like two, three years out. It’s [sometimes] just an idea like, hey, maybe can we look at this.”

We built climate chambers to replicate what would happen at very high humidity. That was fun

At any one time the lab can be juggling devices both weeks and years from launch. Certainly, some new ideas are being cooked up right now – perhaps somewhere amid all those equations – that might not surface for years. Some will never materialise: creating a new device is an exercise in compromise between the lab and design team to try different materials and configurations, tweaking prototypes until – if it works – they’re ready to become a product.

Once the device is almost ready to go out the door, employees can volunteer to field test it, often long before anything is announced publicly. In those cases, they're made to wear a black band around the new gadget to keep it from the prying eyes of the public. This stage is essential – it lets Fitbit detect possible edge cases and get a better sense for how its new product is going to perform in the wild.

Fair tests and fake kitchens

The lab's latest batch of participants are being fitted with chest straps and Fitbit devices, ready for their test. The company works with external recruiting firms to find its volunteers, all of whom must complete a physical readiness questionnaire. “It’s a list of questions that if you answer yes to any of them, we say you don’t fit the criteria,” says Giest.

Read this: Puma's wearable future goes beyond smartwatches and self-lacing shoes

In the early tests for a new device, Fitbit may use just a handful of people – as it's doing in our time there – but the size of the group and the exercises they're doing varies a lot.

And those samples have to be diverse, else it can lead to problems. A lot of Fitbit’s own employees tend to be younger, says Giest, so the lab often needs to look further afield. Sample size? “That really depends on what we’re testing, what our hypothesis are, and what the activities are,” says Conor Heneghan, Fitbit’s lead research scientist. “It can be anywhere from five, 10, 15 subjects all the way up to 150 subjects.”

The tests themselves can be as short as 30 minutes, and as long as several hours. Sometimes they need to be wearing the device for days, so Fitbit will kit them out with a Fitbit and send them back to their normal lives.

We'll have people pretend to wash dishes or put away groceries

And sometimes, Fitbit has to go beyond the treadmills to get the right test conditions. It doesn't have its own pool in the building (Heneghan says he'd like one) so it rents out a local one when it needs to test swimming features. And then there are things like temperature and altitude that also have to be considered. “We built climate chambers at one point to replicate what would happen at very high humidity," adds Heneghan. "That was fun.”

While Fitbit does test for more intense training, it admits it's less of a priority. Its optical sensor tech has certainly improved, but its devices still falter during higher-intensity workouts. “Our tech is primarily focused on the sort of 98% of our users so we focus on walking, running at typical speeds for the casual jogger,” says Heneghan. “We do some work with weightlifting but it’s nothing extreme – we’re not doing full CrossFit evaluation. It’s a trade-off.”

But here's one that may surprise you: when it comes to steps, Fitbit’s go-to is still a good old fashioned clicker. Someone will walk and another person will walk alongside them, counting every step with a press of the thumb. And strollers aren’t the only obstacle to Fitbit's algorithms. It creates all sorts of test conditions - including fake kitchens, says Giest. “We will have people pretend or actually wash dishes, or put away groceries or other activities of daily living."

Time for the treadmill test

To get a better sense of how Fitbit does its testing I decided to take part in a short treadmill exercise. For this I'd need to wear a Fitbit (a Versa in this case) and a chest strap to test the optical reading against (Fitbit uses Polar's chest straps to test with).

Fitbit refers to the electrode-based chest strap as its ground truth, a term used to describe the gold standard of accuracy – or as close as is possible to get. Once I have it on, I'm led to a desk where someone points a small handheld gizmo at my arm. It's a Fitzpatrick test, a skin-type analysis often used to measure sensitivity to UV. Here it's used because optical light on wearables has to penetrate through several layers of skin, and Fitbit needs to ensure that it’s equally effective across all skin types.

Then it's time for action, but first I needed to calibrate. I'm told to sit on the edge of a treadmill for two minutes to get my heart pumping at its normal rate. The exercises Fitbit has people do will depend on what it needs data for, and this can vary a lot. For me it was interval training, so I was made to walk for a couple of minutes on the treadmill, followed by a short burst, then walking again, then running – repeated for about 15 minutes.

All the while, one of Fitbit's assistants was talking me through the process, telling me how much time I had left, and checking I was in the right heart rate zone. If my HR was to read too high or too low, it would be a sign that something was off – or it was time for me to step off the treadmill.

And just like that it was over, and the data was fed into the giant Fitbit machine. Meanwhile, on the treadmill next to me, one of Fitbit’s other (read: much fitter) test specimens was just getting started on a VO2 Max test. For the uninitiated, VO2 Max is a measure of how much oxygen your lungs are able to deliver against how much the body is demanding. It’s featured in a lot of today’s fitness wearables; if you’re a Fitbit user it’s referred to as a cardio fitness score in the app – but it’s the same thing.

And by the way, VO2 Max testing is super intense. The test subject is pushed harder and harder until oxygen consumption hits a plateau – signalling a shift from aerobic to anaerobic metabolism – and that's where the VO2 Max score is found.

But it's not all about torching the fat. Guided breathing and meditation have become staples of Fitbit's devices and actually leverage the same heart rate data, but require their own type of testing.

Then there’s sleep – that's an interesting one. Fitbit keeps a bed in one of the rooms here for some testing, but it's difficult to garner large datasets from lab conditions. “Even if we had 50 beds here it would be a pretty artificial environment,” says Heneghan.

Instead Fitbit uses a registered sleep technician who sets up test subjects with an EEG device that they wear in their bed a home, alongside whatever Fitbit device is being tested. “We take maybe 40, 50 people and have them each sleep one night rather than having one person sleep for 40 nights. We’re more interested in diversity.”

Emerging trends

With Fitbit sucking data at a population level – including billions of nights of sleep – fascinating insights have started to surface. “For female menstrual tracking we got some really interesting data on the population of natural heart rate trends, which track the menstrual cycle which people had seen in studies of 200 people," says Tracy Giest.

"We were able to run the data on a million people and see the same trends. People were wondering whether that tracking depended on BMI and it looks like it didn’t. That was a new insight that wouldn’t have been accessible with the current tools.”

That was a new insight that wouldn’t have been accessible with the current tools

Heneghan says Fitbit was also been able to observe how resting heart rate begins to drop after mid-life – confirming previous research done elsewhere, but on a much larger scale.

And in time, more insights will inevitably come to light. The race to build the ultimate health companion has never been more exciting, with wearables graduating from step trackers to heart health monitors – but we're still at the very start of the journey.

In fact, Fitbit’s research VP Shelten Yuen still likes to tell a story about Fitbit's first heart rate sensor. In 2011 Yuen was exploring the possibilities of using new sensors with Fitbit co-founders James Park and Eric Friedman, when someone suggested the idea of using photoplethysmography, the process of shining a light through the skin and seeing how much is scattered by blood flow.

With a few RadioShack parts and a lot of duct tape, they fashioned a wrist-worn optical heart rate tracker. Then Yuen started running on a treadmill and quickly saw it wasn't working. It was giving numbers, but the signal was too messy. Still, it was as much encouragement as they needed, and began the process of prototyping that led to the Fitbit first heart rate tracker, the Surge.

Eight years later, Fitbit is now exploring whether it can track atrial fibrillation and sleep apnea using the same core principle: beaming light through people’s skin.

It thinks it's cracked it.