State-affiliated pro-Kremlin channels dominate the “most relevant” search results

As part of our effort to broaden expertise and understanding of information ecosystems around the world, the DFRLab is publishing this external contribution. The views and assessments in this open-source analysis do not necessarily represent those of the DFRLab.

YouTube is the most popular social media platform in Russia. Despite this, little is known about how YouTube’s search engine works for search queries in Russian.

Much of the independent research on YouTube search has been conducted on English-language search material. There were fundamental studies by Safiya Umoja Noble, such as “Missed Connections: What Search Engines Say About Women” and “Algorithms of Oppression: How Search Engines Reinforce Racism,” as well as studies by The Markup, like “Google’s Top Search Result? Surprise! It’s Google.” According to YouTube’s online Creators’ Academy course, the platform’s search engine works in a manner similar to Google’s, surfacing the most relevant results and ranking by various factors:

Like Google’s search engine, search on YouTube strives to surface the most relevant results according to keyword queries. Videos are ranked based on a variety of factors including how well the title, description, and video content match the viewer’s query. Beyond that, we look at which videos have driven the most engagement for a query. Search results are not a list of the most-viewed videos for a given query.

To understand what Russian-language videos are considered most relevant and thus, likely to be shown to Russian-speaking viewers, researchers used the YouTube Data API Service and official YouTube statistics to search for videos related to COVID-19 in Russian, then compared social media engagement metrics.

The study showed that even though Russia has significantly expanded laws and regulations tightening control over internet infrastructure and online content, the videos produced by anti-Kremlin YouTube channels are more popular and have higher audience engagement than pro-Kremlin channels. These findings support earlier observations regarding the potential of YouTube as a platform for protest in Russia.

Audience engagement, however, did not appear to be a determining factor in which Russian-language videos YouTube’s algorithm recommended. Rather, videos produced by large and often state-affiliated media companies, including those overtly promoting a Kremlin agenda, appeared to have a better chance of landing among YouTube’s most relevant videos, and thus were more likely to be recommended for viewing. This indicates that YouTube recommends videos based not purely on popularity and engagement, but on some other unknown factors.

This research contributes to a much broader and increasingly pressing discussion about ethical SEO (search engine optimization). There has been increasing debate about the role search engines play in amplifying racial bias. Although the promotion of racist stereotypes and the promotion of agendas of dictatorial political regimes refer to different phenomena, the authors argue that both of these tendencies should be considered in the framework of increasing demands for greater responsibility of Big Tech.

Analyzing the most relevant videos

Using the Russian keyword “коронавирус” (translated from Russian: “coronavirus”), they collected data about the 50 most relevant videos surfaced by YouTube. The data were retrieved at 9:10 a.m. EEST on June 16, 2020, constituting a snapshot of the videos and statistics available at that point in time. After collecting the information, the researchers used the programming languages Python and Julia to process the data, then compared some elementary statistics.

The aim of the experiment was to reveal how many pro-Kremlin vs. independent Russian-language YouTube videos about COVID-19 on YouTube were among the most relevant, and compare their social media engagement metrics, including the number of views, likes, dislikes, and comments. Thus, the analysis did not focus on the content of the videos themselves.

The titles of the 10 most relevant videos about COVID-19 indicate that news content is mainly related to statistical and health information. The following is a rough translation of the titles: (0) “Coronavirus: first findings and new challenges;” (1) “Coronavirus in Ukraine: Statistics for June 12;” (2) “Coronavirus in Ukraine: Statistics for June 13;” (3) “Coronavirus in Ukraine: Statistics for June 14;” (4) “Latest news on coronavirus in Russia;” (5) “Coronavirus in Russia and the world: main news;” (6) “How France defeated coronavirus;” (7) “Latest news on coronavirus in Russia;” (8) “Symptoms of the coronavirus;” (9) “Coronavirus in Russia and the world: main news.”

Despite the fact that the word “coronavirus” in Ukrainian is spelled differently than in Russian (“коронавірус” instead of “коронавирус”), YouTube surfaced some results from Ukrainian channels as well. Since the purpose of the study was to understand what kind of information about COVID-19 a YouTube user could get by entering the word “коронавирус” into the YouTube search engine, the researchers did not exclude the results in Ukrainian from the analysis. Russian-speaking citizens of Ukraine are also highly likely to regularly read and consume information in Ukrainian.

A total of 12 channels released the 50 most relevant videos for the search term:

РБК (“RBC,” or RosBiznesConsulting)

Алексей Навальный (opposition activist Alexei Navalny)

СЕГОДНЯ (“TODAY,” a program of the television channel NTV)

RT на русском (“Russia Today in Russian”)

Первый канал (“Channel One”)

Радіо Свобода (“Radio Liberty”)

BIHUS info

Радіо Свобода Україна (“Radio Liberty Ukraine”)

24 Канал (“Channel 24”)

Espreso.TV

Россия 1 (“Russia-1”)

Студия Квартал 95 Online (“Kvartal 95 Studio Online”)

The first thing that catches the eye after cleaning the data and processing the statistics on these videos the imbalance among video relevance assigned by YouTube, the numbers of videos per channel, and the video views count.

Findings in political context: pro-Kremlin and anti-Kremlin channels

Among the top five channels that uploaded the most relevant videos, three are openly pro-Kremlin and financially dependent on the Russian government. Fifty-one percent of the shares of Channel One (Первый канал), one of the media outlets included in the study, are held by representatives of the Russian state. RT in Russian (RT на русском) is part of the Russian state-controlled international television network RT, which is funded by the Russian government through the Federal Agency for the Press and Mass Communications. Russia-1 (Россия 1) is also a state-owned television channel.

The most outwardly pro-opposition media on the list is the YouTube channel of the popular Russian opposition politician Alexei Navalny (Алексей Навальный). In terms of relevance, Alexei Navalny’s video about COVID-19 is second to last, despite the fact that, in terms of views, this video is in second place.

As mentioned, YouTube indicates that the most relevant search results are not a list of the most-viewed videos for a given query. But search results also do not depend solely on other social media engagement metrics, such as comments, likes, and dislikes. It can be that they are factored in, but are ultimately not decisive.

While Navalny’s channel has high viewer engagement in terms of likes and dislikes (Navalny’s video is the most-liked by a wide margin), the channel has not climbed up on the list in terms of relevance.

The primary anti-Kremlin channels in the dataset are Ukrainian (Espreso.TV, 24 Канал), or the Ukrainian branch of Radio Free Europe/Radio Liberty (Радіо Свобода, Радіо Свобода Україна). There are two more Ukrainian channels among the ones that released relevant videos, but both of them rarely cover Russian politics. BIHUS info is a YouTube channel of the Ukrainian investigative journalism project Bihus.info, which reports on corruption in Ukraine. Студия Квартал 95 Online is a humor-based YouTube channel of the Ukrainian entertainment production company Kvartal 95 Studio, created in 2003 by the current Ukrainian President Volodymyr Zelensky.

That means that among the relevant channels, the channel of Alexei Navalny is the only one from Russia with an explicit anti-Kremlin agenda. At the same time, there is no reason to suppose that YouTube somehow promotes certain political — in this case, pro-Kremlin — content. The example of СЕГОДНЯ (“Today”) shows that there are no clear patterns in terms of political content or stance. СЕГОДНЯ is the main information program of the powerful, effective, and unscrupulous pro-Kremlin propaganda television channel NTV. Despite the fact that, in terms of views, СЕГОДНЯ is the most successful among pro-Kremlin YouTube channels, the relevance assigned to its videos by YouTube is not high.

Another interesting case is the YouTube channel РБК — a business news television channel in Russia and part of the RBC Group, which a few years ago was considered moderately oppositional to the Kremlin. However, due to pressure from the Russian authorities and subsequent leadership changes in May 2016, the RBC Group, including the РБК TV channel, now avoids highlighting and discussing pressing political issues, focusing almost exclusively on business news. РБК leads both in the number of videos (24 videos of 50) and in the total number of views. However, in terms of relevance, the most relevant РБК video is only in the third position. The leader in relevance is a video of the openly pro-Kremlin Channel One.

What YouTube users watch in Russian

All the statistics and metrics analyzed in this study indicate that the fact that YouTube recommends a video for viewing does not guarantee that people will watch it. So, to understand trends, perhaps, it’s important to reveal not only what priority a video has in the search engine, but also real views of real people. Dividing the channels into three groups (pro-Kremlin, anti-Kremlin, neutral), we get the following picture in terms of views.

According to this division, YouTube users prefer to watch neutral channels. Viewers of pro-Kremlin and anti-Kremlin channels represent numerically almost equal groups, with a slight margin in favor of the latter. Of course, this is specific to this particular case study, and there is more research and comparative analysis needed, using other search queries, to see if this is a wider trend, and to see if it holds for longer periods of time under study.

Conclusion

YouTube’s recommendation engine is opaque and difficult to study. Civic tech organizations like Algorithmic Justice League, Data for Black Lives, and Data Feminism have raised concerns over the social implications of algorithmic bias, and YouTube’s systems are no exception. A growing body of work — primarily focused on English-language content — also suggests that YouTube’s recommendation systems may contribute to online far-right radicalization and more extreme conspiracy thinking.

This small study examined YouTube recommendations in the context of Russian-language content about COVID-19, and found that videos produced by large and often state-affiliated media companies have a better chance of landing among YouTube’s most relevant videos, and thus are more likely to be recommended for viewing. This is true for both Russian and Ukrainian media, despite the fact that the Ukrainian media channels examined often have more anti-Kremlin viewpoints.

Among the 50 most relevant YouTube videos for the keyword “коронавирус,” there is almost parity between anti-Kremlin and pro-Kremlin channels in terms of viewership. A great deal of the anti-Kremlin content, however, comes from Ukraine, and not from Russia. Additional research is needed to confirm this finding, and greater algorithmic transparency will no doubt aid in that effort.

Andrejs Berdnikovs, Ph.D., is a social movement scholar, queer data science and algorithmic justice enthusiast, data analyst, and editor at the European Journalism Observatory (EJO).

Pavel Marozau is a social media activist, data analyst and founder of ARU TV.

Follow along for more in-depth analysis from our #DigitalSherlocks.