What it’s like working at Tesla’s Autopilot labeling facilities, where your keystrokes and bathroom breaks are tracked

Much of Tesla’s data annotation is done at its facility in Buffalo, New York.

When should a car begin braking at a stop sign when it’s snowing outside? When should it turn on its blinker? How can you tell the difference between a traffic light and a full moon?

These are just some of the questions Tesla’s Autopilot team deals with on a daily basis.

Tesla’s driver-assist software relies on a small army of data annotators who review thousands of hours of footage from Tesla owners and the company’s in-house test drivers. The annotators gradually teach the company’s AI how to behave like a human driver, one 30-second clip at a time. Tesla employs dozens of annotators who make about $20 per hour in a full-time role at the company.

B-17 spoke with 17 current and former employees on Tesla’s data annotation team to understand what it’s like powering the company’s Autopilot and Full Self-Driving software. The team is split across three different Tesla facilities in Buffalo, New York, Palo Alto, California, and Draper, Utah.

Projects on the teams can take anywhere from months to days and can vary from workflows that require staff to label short clips to trace still images or overlay satellite data.

“Sometimes it can get monotonous,” one former worker said. “You could spend eight hours a day for months on end just labeling lane lines and curbs across thousands of videos.”

An ‘intimate view into someone’s life’

The clips can provide a unique window into Tesla drivers’ everyday lives. At one point, there was a project that required workers to label data taken from inside some owners’ garages via Tesla’s Sentry Mode feature, five workers said.

Another project, called “Selfie,” required some annotators to label data taken from Tesla’s in-cabin cameras, according to two workers who witnessed employees working on the project. Four additional workers said they were aware of the program. The Selfie program was designed to teach Tesla’s system how to identify when a driver wasn’t paying attention to the road while Autopilot was in use, they said.

Tesla has said that the vehicles’ in-cabin camera “shares short cabin camera video clips with Tesla to help us develop future safety enhancements and continuously improve the intelligence of features that rely on the cabin camera,” according to its owner’s manuals. Tesla owners must first opt-in to sharing their data for Tesla’s labelers to be able to access the videos, the company says.

In other cases, workers found themselves labeling routes associated with YouTubers and even Elon Musk himself, B-17 previously reported.

“There is something very strange about having this very intimate view into someone’s life,” a current employee said. “It feels odd to see someone’s daily drive, but it’s also an important part of correcting and refining the program.”

The videos are taken from across the US, as well as some regions in Europe and South America, 15 workers said. Two workers recalled labeling videos they believed had been taken from customer cars in Ukraine during Russia’s invasion.

B-17 reached out to Tesla, Musk, and his legal team for comment but did not receive a response ahead of publication.

Tesla’s driver-assist software can automatically change lanes and stop at traffic lights. 

Workers can run into data from any number of countries in a single workflow, meaning they must constantly be aware of the different road rules for each region. At times, Tesla seemed to take a more relaxed stance on those rules, seven former and current workers said. For example, some workers said they were told to ignore “No Turn on Red” or “No U-Turn” signs, meaning they would not train the system to adhere to those signs.

“It’s a driver-first mentality,” one former worker said. “I think the idea is we want to train it to drive like a human would, not a robot that’s just following the rules.”

Sometimes the role requires workers to label videos from accidents and near-misses. Seven workers recalled labeling videos that included Tesla accidents or those involving nearby vehicles. At one point a worker even distributed a video between employees of an incident that involved a young boy on a bicycle getting hit by a Tesla, four workers said. It was one of many videos and memes workers used to exchange, they said.

Last year, Reuters was the first to report on the bicycle clip and potential privacy issues at the annotation sites. Shortly after the article was published, Tesla began restricting access to clips outside of workers’ designated projects and added watermarks to some of the videos and images so they could easily track down which employees were sharing images, nine workers told B-17.

Tesla’s employee monitoring systems

Tesla has fairly strict employee monitoring systems at its Buffalo site. The location has an array of surveillance cameras at the site that overlooks the workspace, 11 workers told B-17.

Employees are also very closely monitored using two different software systems.

One system, called HuMans, gauges how long they should spend on each clip, four workers said. Annotators who take consistently longer than the allotted time are likely to receive poor performance reviews or be put under a performance improvement plan, or PIP, they said. The software was originally designed to help pilots in the US Air Force and also has the capability of tracking employees’ eye movement and taking audio recordings, according to its website. But it’s unclear if Tesla uses the software to track staff’s eye movement.

The company also uses a measure called “Flide Time” to track annotators’ active time on the labeling software, 17 workers said. It can track keystrokes and how long workers spend with the labeling software open, but it won’t track the time workers spend using other tools on their computer, they said. Depending on their level, workers can be expected to log anywhere from five to seven and a half hours of Flide Time, meaning they must be active on the software for at least that amount of time.

If workers are even five minutes short of hitting their designated Flide Time they could face disciplinary action, six workers said. If they miss Flide Time three times in the span of six months, they could be terminated, the workers added.

Some Tesla workers have attempted to push back on the company’s metrics for staff with little success.

In February 2023, some workers at Tesla’s Buffalo facility attempted to unionize. Union organizers at the Buffalo facility told Bloomberg the company was tracking their keystrokes and said some workers had grown weary of “being treated like robots.”

That same month Tesla laid off dozens of workers at the Buffalo plant. At the time, The National Labor Relations Board (NLRB) filed a complaint alleging that Tesla unlawfully terminated some staff “in retaliation for union activity and to discourage union activity.” But, Tesla denied the allegation, stating that the employees were laid off due to poor performance instead. The NLRB didn’t respond to a request for the current status of the complaint.

When Tesla began building its driver assist program in 2016 the company outsourced data labeling to a California-based company that had offices in Kenya, but Tesla brought the program in-house in 2019, Reuters reported.

More recently, Tesla’s Autopilot team was hit by companywide layoffs in April. Tesla shed nearly 300 of its staff in Buffalo, according to a WARN notice.

Tesla has said its neural network will someday be able to train itself, but for now, it relies on its human workforce.

The work is vital to Musk’s vision for the car company.

Over the years, the Tesla CEO has repeatedly emphasized the importance of Tesla’s efforts toward achieving autonomous driving. In 2022, Musk said Tesla’s self-driving technology is the “difference between Tesla being worth a lot of money or worth basically zero.”

Tesla plans to unveil its autonomous Robotaxi service later this year, which is expected to be built upon that same self-driving software — and, of course, the tedious, clip-by-clip analysis of its labelers.