Did This Microsoft Engineer Just Save the Internet From State-Backed Hackers?

The short answer to the question in my headline is probably so. Andre Freund is a software engineer for Microsoft who is located in San Francisco. His normal work is focused on database software called PostgreSQL but recently he stumbled across a backdoor placed in a piece of software which would have eventually spread around the world if he hadn’t spotted it.

The saga began earlier this year, when Mr. Freund was flying back from a visit to his parents in Germany. While reviewing a log of automated tests, he noticed a few error messages he didn’t recognize. He was jet-lagged, and the messages didn’t seem urgent, so he filed them away in his memory.

But a few weeks later, while running some more tests at home, he noticed that an application called SSH, which is used to log into computers remotely, was using more processing power than normal. He traced the issue to a set of data compression tools called xz Utils, and wondered if it was related to the earlier errors he’d seen…

Like other popular open-source software, Linux gets updated all the time, and most bugs are the result of innocent mistakes. But when Mr. Freund looked closely at the source code for xz Utils, he saw clues that it had been intentionally tampered with.

In particular, he found that someone had planted malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine.

Because Linux is used by millions of computers worldwide, the ability to access anyone of those computers and run code would have been a very significant ability for someone. The backdoor could have been used to spread malware or spyware or simply to shut down certain machines. The head of a cybersecurity firm told the NY Times, “This could have been the most widespread and effective backdoor ever planted in any software product.”

So who was behind this? There is a name associated with the person who made the changes to the code: Jia Tan. It appears this person spent years helping out with the xy Utils code until he eventually became a “maintainer” which is sort of the software nerd equivalent of being a moderator on a forum. Once he had this position, he inserted the back door code. That code was part of a recent release but fortunately Andre Freund caught it before it had spread widely.

Who is Jia Tan? Well the name obviously sounds Asian and a quick look at his time spent on the xy Utils code suggests he was working in China’s time zone. However, one analysis suggests that could be a trick as simple as changing the time zone on his computer.

Based on his name, he wanted people to believe he is Asian — specifically Chinese— and the vast majority of his commits (440) appear to have a UTC+08 time stamp. The +0800 is likely CST, the time zone of China (or Indonesia or Philippines or Western Australia), given almost no one lives in Siberia and the Gobi desert.

However, I believe that he is actually from somewhere in the UTC+02 (winter)/UTC+03 (DST) timezone, which includes Eastern Europe (EET), but also Israel (IST), and some others. Forging time zones would be easy — no need to do any math or delay any commits. He likely just changed his system time to Chinese time every time he committed…

…sometimes, he forgot to change his time zone. There are 3 commits and 6 commits, respectively, with UTC+02 and UTC+03. The UTC+02 time zones match perfectly with the winter time (February and November), while the UTC+03 matches with summer (Jun, Jul, and early October). This matches perfectly with the daylight savings time switchover that happens in Eastern Europe…

The same analysis also found that Jia Tan was working during China’s New Year’s week when almost no one in China is working but he did not work on Christmas (which is not a public holiday in China). Other experts who’ve looked at this say it looks like a state-backed effort.

That inhumanly patient approach, along with the technical features and sophistication of the backdoor itself, has led many in the cybersecurity world to believe that Jia Tan must, in fact, be a handle operated by state-sponsored hackers—and very good ones. “This multiyear operation was very cunning, and the implanted backdoor is incredibly deceptive,” says Costin Raiu, who until last year served as the most senior researcher and head of the global research and analysis team at Russian cybersecurity firm Kaspersky. “I’d say this is a nation-state-backed group, one with long-term goals in mind that affords to invest into multiyear infiltration of open source projects.”

As for which nation, Raiu names the usual suspects: China, Russia, and North Korea. He says it’s still too early to know the true culprit. “One thing is for sure clear,” he adds. “This was more cunning than all previous software supply chain attacks I’ve seen.”

…the majority of clues lead back to Russia, and specifically Russia’s APT29 hacking group, argues Dave Aitel, a former NSA hacker and founder of the cybersecurity firm Immunity. Aitel points out that APT29—widely believed to work for Russia’s foreign intelligence agency, known as the SVR—has a reputation for technical care of a kind that few other hacker groups show. APT29 also carried out the Solar Winds compromise, perhaps the most deftly coordinated and effective software supply chain attack in history. That operation matches the style of the XZ Utils backdoor far more than the cruder supply chain attacks of APT41 or Lazarus, by comparison.

“It could very well be someone else,” says Aitel. “But I mean, if you’re looking for the most sophisticated supply chain attacks on the planet, that’s going to be our dear friends at the SVR.”

Whoever was behind this can’t be happy that some random tech worker in San Francisco uncovered the plot they had spent years putting in place. Andre Freund (who was born in Germany and whose name means “friend” in German) declined to have his photo taken for the NY Times story about him so he’s definitely not looking for attention. But someone in the comments suggested the US government ought to honor him with some sort of medal or recognition. It’s a nice idea. It doesn’t have to even be public but why not recognize what was likely a big poke in the eye to our enemies abroad.