PARIS (AP) - On the morning of March 19, 2016, Den Katenberg ran a little test with big stakes.
The previous week, Katenberg's hacking crew had been bombarding the Hillary Clinton campaign's email accounts with fake Google warnings, trying to get her Brooklyn-based staff to panic, enter their passwords and open their digital lives to Russia's intelligence services.
But the going was tough. Even when Clinton staffers clicked the malicious links Katenberg crafted, two-factor authentication - a second, failsafe password test - still kept him out of their accounts.
After a day of testing on March 18, he took a different tack, striking the Clinton's campaign staff at their personal - and generally less secure - Gmail addresses. At 10:30 the next morning he carried out one last experiment, targeting himself at his own Gmail address to make sure his messages weren't being blocked.
An hour later he sent out a barrage of new malicious messages to more than 70 people, including one to Clinton campaign chair John Podesta. By the end of the day, he'd won access to one of the most important inboxes in American politics.
On Friday, the U.S. special counsel said Katenberg was an alias used by Lt. Aleksey Lukashev, an email phishing specialist with Unit 26165 of Russia's Main Intelligence Directorate, often abbreviated GRU.
Katenberg, who did not return multiple messages seeking comment, has been in The Associated Press' sights ever since his email was identified among a massive hacker hit list handed to the news agency by Secureworks last year.
It was that 19,000-line database that allowed the AP to reconstruct Katenberg's digital movements, logging every malicious link he and his colleagues created between March 2015 and May 2016.
The data show that the malicious emails came in waves, some 20 or 30 of them at a time, aimed at diplomats, journalists, defense contractors and other Russian intelligence targets across the world. Between the waves, sometimes only an hour or a few minutes before a major campaign, the hackers sent test emails to their own accounts to make sure they could still dodge Google's spam filters.
Katenberg's GRU hacking group, widely nicknamed "Fancy Bear," was locked in an arms race with the email giant. Every few months, Google would cotton on to the group's tactics and begin blocking its messages. The Secureworks list, along with more than 100 other phishing emails recovered from spying victims, showed how the GRU would respond by firing up a new batch of malicious websites, moving on to a new link shortening service, or trying a new brand of phishing message meant to lure its recipients into giving up their credentials.
"Someone has your password," was one particularly dire-sounding message sent by the GRU to a DNC staffer on March 25, 2016. Some messages played on their targets' fears of being hacked. One offered Gmail users a malicious "Anti-Phishing Guard App" to protect themselves from cybercriminals. Another particularly twisted message warned a Russian journalist that "Government-backed attackers may be trying to steal your password" - before directing him to a booby-trapped link.
But as good as the hackers were at extracting passwords from their victims, they also made mistakes.
For example, the Gmail address the GRU used to test-drive its phishing messages on March 19, 2016, was also used to register a Den Katenberg Twitter account , according to Twitter's "Find friends" feature. The AP also found a Facebook page using the same name and picture, although it's by no means clear that the accounts' black-and-white photograph of a young man in a dark sweater really belongs to Lukashev.
Both social media pages appeared dormant, but Lukashev and his colleagues may not be resting easy. Katenberg's Facebook profile vanished within minutes of the publication of this article. Across the internet, journalists were picking up traces of the once-anonymous hackers' digital trail, like the document posted to the website of a Moscow secondary school that identified Viktor Netyshko as the head of Unit 26165 - just as the U.S. indictment alleged.
For years men like Netyshko and Lukashev are alleged to have hunted America's secrets.
Now the world's media is after theirs.
Jim Heintz in Moscow contributed to this report.
Raphael Satter can be reached at: http://raphaelsatter.com
More AP reporting on Russian hacking: https://www.apnews.com/tag/Russianhacking