#TrollTracker: Facebook’s Midterm Takedown

Analyzing the accounts attributed to Russia’s Internet Research Agency

#TrollTracker: Facebook’s Midterm Takedown

Share this story
THE FOCUS

Analyzing the accounts attributed to Russia’s Internet Research Agency

A troll operation attributed to the Internet Research Agency or “troll factory” in Russia ran over 100 accounts on Instagram and Facebook, and posted divisive content from all political angles, right up to the eve of the U.S. midterm elections, according to traces of this activity left online.

Facebook took the accounts offline on November 5, 2018, the eve of the U.S. midterm elections, after a tip-off from U.S. law enforcement. Initial suspicions were that the accounts were run by the Internet Research Agency or “troll factory” in St. Petersburg, which targeted American social media users throughout 2014 through 2018 with divisive, inflammatory and election-related content. A website which claimed to be affiliated to the troll factory said it ran them, although any such claim should be viewed with caution.

On November 13, Facebook confirmed it had taken down 99 Instagram, 36 Facebook accounts, and six Facebook pages for “coordinated inauthentic behavior.” Around 1.25 million other users followed at least one of them. Facebook said that the inauthentic accounts “may have been connected” to the troll factory, but stopped short of attribution.

Ahead of the November 13 announcement, Facebook shared the Instagram account names with @DFRLab. Enough traces have survived online to illustrate the way this operation worked, posing as online interest groups, building an audience, and progressively inserting more polarizing and political content into the mix.

The overall strategy of promoting divisive political content matched those of earlier Russian operations. The latest operation stood out for its greater focus on celebrity groups, rather than single-issue political ones. This appears likely to have been an attempt to build audience appeal, while allowing the flexibility to switch from one hot political topic to another.

Troll Factory Resemblances

All the accounts which Facebook named were blocked before @DFRLab could review them. This analysis is based on residual posts, either from online caches or from shares. It therefore provides a snapshot of the accounts’ behavior, not a full catalog of their output.

Within that subset of residual posts, some of the suspect accounts shared content directly from the original troll factory. One account which focused on African American communities, @not_your_negro, shared a meme which was watermarked by known Russian troll account @woke_blacks_, and had also been posted by Russian troll account @afrokingdom_ .

Left, post by @not_your_negro_; note the watermark, @woke_blacks_, in the red box. (Source: Instagram / @not_your_negro_, via Google cache of instagimg.com.) Right, post by @afrokingdom_, preserved by @UsHadrons. (Source: Instagram / @afrokingdom_, via Medium / UsHadrons)

Similarly, suspect troll account @4merican.m4de shared a post which was watermarked by original Russian troll account @american.veterans, preserved in the archive created by online researcher UsHadrons.

Left, post by @4merican.m4de, with the “American.Veterans” watermark. Right, post by known IRA troll account american.veterans. (Left, Source: Instagram / 4merican.m4de, via Google cache of insgain.com, October 24, 2018. Right source: Instagram / american.veterans, via Medium / @UsHadrons)

The use of watermarked troll factory content does not, on its own, confirm a link between these suspect accounts and earlier Russian troll operations. It does provide a further data point on the resemblance between the operations, independent of the assessments by the FBI and Facebook, and the claims of the apparent troll factory site.

On another occasion, one account shared content from a Russian state source, without attribution. In October, suspect account @anonymous.speak posted an attack on Facebook, and its decision on October 11, 2018, to take down hundreds of accounts for coordinated inauthentic behavior.

Source: Instagram / @anonymous.speak, via Google cache of insta-stalker.com, October 22, 2018, and oino.site, undated.

The post was unattributed, but its opening sentences were taken from an “exclusive” article on Kremlin outlet Sputnik, whose stated mission is to “secure the national interests of the Russian Federation in the information sphere.” Later content in the Instagram post appeared original, and included the grammatical error, “Those independent activists have a courage to tell people.”

Comparison of the anonymous.speak post and Sputnik article. Yellow indicates identical text. Red highlights the characteristic language error, “have a courage to tell people.” (Left source: Instagram / @anonymous.speak, via Google cache of insta-stalker.com, October 22, 2018, and oino.site, undated. Right source: Sputnik.)

The sentence, “those independent activists have a courage to tell people” is suggestive because it embodies a grammatical error which would be unlikely to come from a native English speaker. Confusion over the use of “the” and “a” is characteristic of speakers of Slavic languages, notably Russian, which do not possess grammatical articles; this weakness was one of the clues in the exposure of earlier Russian troll accounts.

The same blindness to “a” and “the” featured in the central post in this image, from suspected troll account @black.dollar__, “Keep up a great work”.

Posts by suspected troll account @black.dollar__. Note the telltale “Keep up a great work” and, in the right-hand post, “Tiana and other princess of color.” (Source: Instagram / @black.dollar__, via Google cache of complexpic.com on September 23, 2018. )

Linguistic errors also featured in this post from suspected troll account @not_your_negro: “everything else doesn’t matter for the sane people” and “Such asshole like the one who,” a non-English turn of phrase which missed out “an” and replicated the phrasing of the Russian “Такой мудак…”

Post by @not_your_negro; note the language in the red boxes. (Source: Instagram / @not_your_negro_, via Google cache of instagimg.com)

As a further example, this post from Instagram account @anonymous.speak mis-phrased the word order of a question (“why the f**k media never mentioned this?”), in a way that earlier, known Russian troll accounts also did (“why marijuana is illegal?”).

Left, post by suspected troll account @anonymous.speak. Note the word order of the question. Right, post from known Russian troll Instagram account Anonymous_News; note the mangled word order in the question. (Source: Left, Instagram / @anonymous.speak, via Google cache of insta-stalker.com, October 22, 2018. Right, Instagram / Anonymous_News, via Medium / @UsHadrons)

None of these features is enough on its own to provide a firm attribution. They should be understood as adding context to Facebook’s finding that the accounts were coordinated and inauthentic, and the apparent claim by U.S. law enforcement.

Even More Divisive Content

Just like the original troll factory, these accounts posted highly divisive content which targeted both sides of America’s partisan gulf. Some, such as @black.voices__ and @ur.melanated.mind, focused on African American communities. Others, such as @maga.people and @4merican.m4de, posed as conservatives. @feminism_4ever and @lgbt_poc focused on gender and race issues; @american.atheist_ and @proud_muslims focused on religion.

Some of their posts were clearly designed to offend, as in this post, which equated Christianity with child abuse.

Post on Christianity by @american.atheist_; note the attribution in the URL bar. (Source: Instagram / @american.atheist_, from picgrace.com, via Google cache, October 14, 2018)

Religion and politics mixed in this post (right of image), which portrayed a “Republican Jesus” telling a rape victim that “what’s really important” is the rapist’s Senate seat.

Posts by suspected troll account @immi.great. (Source: Instagram / @immi.great, via Google cache of yooying.com on November 2, 2018)

The suspect accounts paid particular attention to race and gender issues, including the hyper-sensitive questions of violence against African Americans, and transgender rights.

Posts by suspected trolls @blknation and @intellectual.dark.web on police violence, transgender rights and women’s employment. (Left source: Instagram / @blknation, via Google cache of yooying.com, October 14, 2018. Right source: Instagram / @intellectual.dark.web, via Google cache of deskgram.net, October 15, 2018.)

Regularly, the troll accounts took both sides on divisive issues, as in these examples on feminism; feminism and gun control; abortion; and President Trump.

Left, repost of meme by suspect troll account @feminism_4ever. Right, repost of meme by suspect troll account @milo.against.feminism. We have anonymized those accounts which reposted the content. (Source: Instagram / @feminism_4ever / @milo.against.feminism, via reposts.)
Upper image, repost of meme by suspected troll account @be.louder.with.crowder on guns as a tool of women’s equality. Lower image, post by suspected troll account @guns.control, on guns as a threat to women. (Source: Instagram / @be.louder.with.crowder / @guns.control)
Left, repost of content by suspected troll account @f.u.libtards. Right, repost of content by suspected troll account @JohnOliverExplains. (Source: Instagram / @f.u.libtards / @JohnOliverExplains)
Posts on Trump by @johnoliverexplains, left, and @intellectual.dark.web, right. (Left source: Instagram / @johnoliverexplains, from repost. Right source: Instagram / @intellectual.dark.web, via Google cache of deskgram.net, October 15, 2018.)

The opposition was particularly stark over the question of gun control, with apparent troll accounts both demanding stricter gun controls, and posting messages that refused to countenance even the possibility of discussion. The latter stance appeared designed to render any attempt at reconciliation useless — an indication of how the trolls’ intent appeared to be to reinforce the most polarizing content.

Post by suspected troll account @guns.control on school shootings. Note the debate the post triggered, on the right. (Source: Instagram / @guns.control, via Google cache of deskgram.net, October 27, 2018.)
Posts by apparent troll account @real_man_cave, showing the rejection of debate. (Source: Instagram / @real_man_cave, via Google webcache of picbon.com, September 20, 2018.)

The trolls were quick to fasten on trending controversies, such as Nike’s decision to make footballer Colin Kaepernick the face of its brand.

Repost of a Nike-related meme from @be.louder.with.crowder. (Source: Instagram / @be.louder.with.crowder)
Post on the Nike / Kaepernick controversy by suspected troll account @blknation. (Source: Instagram / @blknation, via Google cache of yooying.com, October 14, 2018.)
(Source: Instagram / @bring.america.back, via Google cache of bigsta.net on September 21, 2018.)

Kanye West was the subject of particularly polarizing coverage, with a number of suspected troll accounts citing his relationship with Trump as proof that the president and his supporters were not racist, while Democrats and Trump opponents were.

(Source left: Instagram / @bring.america.back. Source right: Instagram / @intellectual.dark.web, via Google cache of picbon.com, October 9, 2018.)

In typical style, other apparent troll accounts took the opposing view.

Share by apparent troll account @bill.maher.time of a post on the Kanye West controversy. (Source: Instagram / @bill.maher.time, via Google cache of picbear.xyz on October 6, 2018.)

This controversy took an unexpected twist on October 30, when West tweeted: “I’ve been used to spread messages I don’t believe in. I am distancing myself from politics and completely focusing on being creative !!!”

https://twitter.com/kanyewest/status/1057382916760707072

The following day, suspected troll account @immi.great shared the tweet, together with a Buzzfeed News article which quoted it. The @immi.great account posed as liberal and anti-Trump.

Posts by suspected troll account @immi.great, the left-hand one focusing on Kanye West’s apparent retraction. (Source: Instagram / @immi.great, via Google cache of yooming.com on November 2, 2018. )

Even apparently apolitical accounts, such as @girlsbeerguns, which largely posted images of girls girls with guns (users were apparently expected to bring their own beer), posted the occasional political content, suggesting that their far higher proportion of innocuous content was meant as an audience building measure, which will be discussed below.

Post by @girlsbeerguns on Twitter’s decision to ban disinformation actor Alex Jones of Infowars, blaming it on “leftist pressure.” We have anonymized the account which reposted it, and blurred out the image to keep @DFRLab safe for work.

Posting divisive content is not, again, sufficient to expose a Russian troll account; indeed, without American trolls, Russian trolls would have nobody to disguise themselves as. However, it is in character with earlier Russian troll operations.

Celebrity Targeting

One striking difference about the latest set of accounts was the way they focused on celebrities or media personalities. This differed from earlier accounts run by the troll factory, which either posed as individuals, or as single-issue activist communities whose issues ranged from LGBT rights and the Black Lives Matter movement to Texan secession and Islamophobia.

Accounts in the latest cluster included @john.oliver.explains, dedicated to comedian John Oliver, and @tomi.lahren.fans, dedicated to conservative pundit Tomi Lahren. The former account had over 95,000 fans, the latter almost 15,000.

Profile images for @johnoliverexplains and @tomi.lahren.fans, from online caches. (Source top: Instagram / @johnoliverexplains, via Google cache of pikview.com, October 27, 2018. Source bottom: Instagram / @tomi.lahren.fans, via Google cache of deskgram.net, September 21, 2018)

Others included @be.louder.with.crowder, dedicated to conservative comedian Steve Crowder, and @bill.maher.time, dedicated to comedian Bill Maher. These had more modest followings, with the “Crowder” account slightly ahead of the “Maher” one.

(Source top: Instagram / @be.louder.with.crowder, via Google cache of picbear.xyz, October 1, 2018. Source bottom: Instagram / @bill.maher.time, via Google cache of insta-stalker.com, September 5, 2018.)

Other accounts were not even remotely political, and focused on celebrities from the world of entertainment and (non-political) drama, including actress Jennifer Lawrence (jenlawrencefanclub) and rapper Kid Rock (@kidrock_fanpage). These had far fewer followers.

Source top: Instagram / @jenlawrencefanclub, via Google cache of picbear.xyz, October 11, 2018. Source bottom: Instagram / @kidrock_fanpage, via Google cache of applam.com, October 28, 2018.

The surviving posts by these accounts were focused on glamor, and did not appear to pass political messaging.

Posts by @jenlawrencefanclub. (Source: Instagram / @jenlawrencefanclub, via Google cache of picbear.xyz, October 11, 2018)
Posts by @kidrock_fanpage. (Source: Instagram / @kidrock_fanpage, via Google cache of applam.com, October 28, 2018.)

This focus on celebrities is likely to have served two functions. First, it allowed the troll operators to tap into a readymade audience, although receptiveness to these efforts clearly differed across channels, judging by the range of follower numbers. Second, it may have allowed them to switch more credibly between different political hot topics, in a way which apparently single-issue accounts would have found harder to do.

Audience Building

Some of the accounts barely posted on political topics, as we have seen with the celebrity examples. Much more of their content was focused on innocuous, positive or clickbait content. In the context of a suspected information operation, this appears likely to have been an audience building maneuver, as earlier generations of Russian troll accounts are known to have used.

Repost of a typical post by @girlsbeerguns, anonymized to protect the user who reposted it. (Source: Instagram / @girlsbeerguns)
Repost of a post by @fem.voice, anonymized to protect the user. (Source: Instagram / @fem.voice)
Posts by @ur.melanated.mind. (Source: Instagram / @ur.melanated.mind, via Google cache of yooying.com, August 30, 2018)
Post by @4merican_m4de.

Once again, the posts were not driven by any apparent set of coherent values, and seemed designed to appeal to as many different groups as possible. For example, one account, @ur.melanated.mind, argued that “nudity is not necessary to show beauty,” while another, @girlsrepub1ic, promised “the most beautiful and luscious Conservative girls,” mostly in bikinis.

Left, post by @ur.melanated.mind. Right, profile for @girlsrepub1ic, screen name “Conservative Babes.” (Source: Instagram / @ur.melanated.mind / @girlsrepub1ic, via Google cache of picdo.net, October 24, 2018)

The prevalence of audience-building posts suggests that these accounts were in the early stage of their operationalization, and had not yet been turned to more aggressive posting on divisive or political issues. Facebook itself said that most of the accounts dated from late 2017 or later, reinforcing that view.

American Voices

One other important feature of these accounts was the way they repeated partisan and divisive comments by Americans. This is likely, again, to have had multiple purposes. It allowed the trolls to appeal to existing audiences, and reduced the likelihood that they would betray themselves through linguistic errors.

It also revealed America’s fundamental vulnerability to such foreign influence operations, as Americans’ own posts provided the ammunition which foreign trolls used to target American users.

This was especially the case on the conservative side. The troll accounts repeatedly used content from conservative group Turning Point America, whose founder, Charlie Kirk, spread misleading or dated claims of election fraud in the buildup to the midterm elections. Turning Point America, Kirk himself, and the group’s director of communications, Candace Owens, all featured in many different suspected Russian trolls’ posts.

Post by @intellectual.dark.web featuring Candace Owens.
Repost of post by @be.louder.with.crowder, featuring Owens. Anonymized to protect the original reposter. (Source: Instagram / @be.louder.with.crowder)
Posts by @4merican.m4de, left, and @against_libtards, right, featuring the same watermarked Turning Point USA meme. (Left, source: Instagram / @4merican.m4de, via Google cache of insgain.com, October 24, 2018. Right, source: Instagram / @against_libtards_, via Google cache, October 23, 2018. )
Share of a quote from Kirk with the Turning Point USA watermark by @Tomi.Lahren.fans. (Source: Instagram / @Tomi.Lahren.Fans, via Google cache, September 21, 2018. )

Lahren herself was another voice the suspected troll accounts amplified, not only through the @tomi.lahren.fans account, but by others, including @maga.people and @bring.america.back.

Repost of a share, by suspected troll account @maga.people, of a Tomi Lahren tweet. (Source: Google)

One account, @info.warriors, was even dedicated to alt-right conspiracy theorist Alex Jones, a major source of U.S.-based political disinformation.

Profile image and title of @info.warriors. (Source: Instagram / @info.warriors)

The apparently liberal or left-wing accounts drew their comments from a wider range of sources, but also repeatedly shared genuine American comments, sometimes with an additional spin of their own, sometimes only with hashtags.

Posts by suspected troll account @bill.maher.time, sharing and commenting on posts by American commentators. (Source: Instagram / @bill.maher.time, via Google cache of picbear.xyx, October 6, 2018.)
Shares of posts by suspected troll account @feminism_4ever, using imagery and quotes from genuine Americans. (Source: Instagram / @feminism_4ever)

On both sides, this use of American voices both gave the apparent trolls the appropriate local language and color, and gave them a further chance of embedding in like-minded American communities which they could then seek to radicalize.

Conclusions

While these accounts have not been definitively attributed, their behavior closely resembled that of earlier Russian operations. They masqueraded as both right- and left-wing users, and posted divisive and polarizing content. If anything, their content was even more divisive than earlier troll operations.

Some of the accounts appeared focused on audience building, with little or no controversial content, suggesting that this was an operation in its early stages. Their follower numbers ranged from a few hundred to over 100,000, suggesting that other accounts had already been operating for some time, or at some intensity.

The apparent troll accounts focused on celebrity fan groups. This appears a new approach, designed to boost audience and make it easier to switch from one theme to another. Some of the “celebrity” accounts numbered tens of thousands of followers, suggesting that the approach paid off.

Finally, the exposure of this network confirmed that inauthentic troll clusters are still operating, and still targeting American divisions. However, the response, which featured close cooperation between U.S. law enforcement and the social platforms, showed that key players on the U.S. side are more aware of the threat, and both willing and able to tackle it, than in 2016.

The challenge remains to pass that understanding on to the communities which are being targeted.


Follow along for more in-depth analysis from our #DigitalSherlocks.