Apple alerted Amazon about a potential cloud security risk, prompting a change in AWS’s data-deletion process

Apple CEO Tim Cook

Amazon Web Services made major changes to its data-deletion process after Apple alerted the cloud giant about a potential security risk, according to an internal document.

In early 2023, Apple spotted unusual activity around the data and contents associated with its terminated cloud accounts on AWS. By April 2023, Apple escalated concerns to AWS’s security team, and asked the cloud giant to investigate whether the data and contents were deleted from roughly 2,200 of its AWS accounts that had been closed for more than 90 days. AWS customers expect their data to be permanently deleted 90 days after accounts are shut.

An internal AWS investigation found that it had failed to remove almost 2,000 pieces of content or metadata linked to those Apple terminated accounts, according to the document. It’s not clear what specific Apple data was left undeleted. However, the AWS cloud services involved store information such as software, text, audio, video, images, resource identifiers, metadata tags, and permissions.

The oldest content held was from an account terminated in October 2020. Amazon told Apple it would delete all relevant data by the end of June 2023. There was “no unauthorized access” to Apple’s data, the document added.

AWS had “skipped” data deletion for some of these Apple cloud accounts and its internal “data deletion systems or internal processes” were broken for some of the services at the time, according to the internal document.

As a result, the AWS security team in charge recommended establishing clear guidance around “end-to-end data deletion” and identifying a director or VP-level executive responsible for “data deletion correctness at AWS,” the document stated.

“AWS services retained customer content and resources longer than 90 days,” the document said. “Today, there is no centralized monitoring that resources are deleted in a timely manner.”

“Fruitstand” = Apple

The document doesn’t specifically mention Apple. But a separate email obtained refers this case to a customer named “Fruitstand,” which is an internal codename for Apple, according to people familiar with the matter. These people asked not to be identified discussing a sensitive cloud security issue.

The incident gives a rare behind-the-scenes look at how one of AWS’s largest customers prompted security improvements for the cloud giant. It’s also notable because Apple’s marketing focuses so much on privacy and keeping data secure.

The 23-page document from last June is what Amazon internally calls a “Correction of Error” report, an in-depth analysis of an incident that the company wants to prevent happening again. Employees in charge of the report have to explain why the incident took place and how the team plans to remediate it. This particular report, which was marked “Privileged & Confidential,” had more than a dozen Amazon managers and employees involved and was updated at least 30 times.

“No data accessed”

Before the publication, AWS spokesperson Patrick Neighorn said that “this story has a number of inaccuracies.” He also wrote in an email that the internal Amazon document had “some misconceptions.”

“Our deletion processes are comprehensive and have worked nearly 100% of the time,” Neighorn added. “In a very small number of cases in the past, a small amount of data — mostly metadata like configuration data — took longer to be deleted than intended. There was no data accessed, and we quickly fixed these edge use cases as soon as they were identified.”

An Apple spokesperson didn’t respond to requests for comment. Cloud security is becoming more important for Apple. Next week, the iPhone maker is expected to unveil new AI features at its WWDC event, and at least some of these new offerings will have to run in the cloud.

A difficult problem

Cybersecurity experts say cloud vendors in general can have difficulty deleting data associated with terminated accounts.

Customer data is typically scattered around many servers throughout the world and it’s hard for cloud providers to keep track of every piece of data in real-time. Sometimes it’s backed up in multiple data centers, and some applications may not have automatic data-deleting systems in place, these people said.

“It’s a difficult problem,” Justin Cappos, a computer science professor at NYU. said “Cloud providers need to treat this as a serious feature that they provide to customers because the customers should have the right to control their data, where it’s distributed, how long it lives, and more importantly, when it is removed.”

A “nightmare” of “shadow data”

The greater concern is the risk of such data potentially being exposed to a third-party. Fabrice Delhoste, CTO of security startup Mindflow, said deleting data from terminated cloud accounts can be a “nightmare” because of its complexity, and it is the cloud provider’s responsibility to ensure all data is safely removed.
“Abandoned cloud accounts and misconfigured services create unmanaged ‘shadow data,’ which could leave sensitive information exposed. Once abandoned, lack of oversight and updates increase the potential for exploitability over time,” Ken Elefant, managing director of Sorenson Ventures,said.

Security is a top priority for AWS

Amazon Web Services CEO Matt Garm

For AWS, security has always been a top priority. In a recent blog post, Amazon’s new cloud CEO Matt Garman wrote that ensuring security for customers is “job zero.”

As more companies and government agencies rely on cloud infrastructure, major vendors have become a prime target of cyber attacks in recent years.

Most prominently, a hacking group associated with the Chinese government compromised Microsoft’s cloud service last year. A US Homeland Security-backed review published earlier this year blamed Microsoft for allowing a “cascade” of “avoidable errors.”

Microsoft President Brad Smith is scheduled to testify before the House Committee about the cyber intrusion next week.

A “canary-like environment”

According to the Amazon document, AWS employees took Apple’s inquiry seriously and made several recommendations and changes to improve its internal data-deletion process.

The AWS security team suggested taking actions to “meaningfully improve” the quality of the data-deletion process and “define a clear guideline” around it. It also made plans for new tracking and escalation mechanisms, a periodic review plan, and a new data-deletion template for all types of accounts. Up to that point, AWS had depended on a largely manual auditing process.

The team wrote that “post-closure data deletion failures” are raising “customer concerns,” and that the company hadn’t invested enough in the right solutions. Having a “canary-like environment that continuously tests whether service teams are deleting their data” would provide the “highest quality signal” to prevent such issues, it added.

But even with these changes, there were still “open risks” in terms of fully deleting every content or metadata, and AWS would need to invest in “additional systems” that continuously detect data deletion of every AWS service team, the document added.

Neighorn, the AWS spokesperson, said that AWS now continually conducts “automated audits of every AWS service for data deletion compliance,” and any compliance issue receives “immediate attention.” AWS has “always had comprehensive data deletion process guidance,” and it recently made improvements to make it “even easier” for engineers to follow, he added.

Neighorn also said the few AWS customers who asked about data deletion were “satisfied with our resolution.” AWS completed all these fixes and has since implemented “many more enhancements,” like auto-detecting whether resources are present after account closure, he said.

“As with any other AWS service or function, we continually look for ways to make our guidance more helpful,” Neighorn said.