Covid-19: far right violent extremism and tech platforms' response

Tech Against Terrorism is a public-private partnership supported by the United Nations Counter-Terrorism Executive Directorate (UN CTED). Tech Against Terrorism was launched in April 2017 at the United Nations Headquarters in New York and has since been implemented by Quantspark Foundation. As a public-private endeavour, Tech Against Terrorism is funded by both the tech industry via the Global Internet Forum to Counter Terrorism (GIFCT) and by governments. To date, the governments of Spain, the Republic of Korea, Switzerland, and Canada have provided financial support to Tech Against Terrorism.

Our core aim at Tech Against Terrorism is to support the tech industry in building its capacity to tackle the use of the internet for terrorist purposes whilst respecting human rights. We work with all types of tech companies, such as social media, pasting*, file-storage, messaging, fintech platforms**, and web infrastructure providers. Our core mission is providing the global tech industry with the tools needed to effectively tackle terrorist activity on their platforms. In doing so, we work across three streams:

1.   Threat analysis and outreach

We carry out extensive open-source intelligence analysis to identify platforms at risk and build constructive working relationships with the tech sector, as well as facilitating public-private cooperation

2.    Knowledge sharing and best practices

We facilitate intra-industry and cross-sector support mechanisms through online tools, guides, and practical datasets to support policy and content moderation decisions. Here we work closely with the GIFCT in organising global workshops and webinars. We also support companies through our membership and mentorship programmes

3.   Tech development and operational support

We provide technical support and resources for tech companies to improve their counterterrorism mechanisms, for example through data science or development support. Examples of past work within this workstream includes our work with Jihadology.net*** and our current work on the Terrorist Content Analytics Platform****.

Tech Against Terrorism consists of an interdisciplinary team of counterterrorism policy experts, open-source intelligence analysts, and developers and data scientists. Our team composition allows us to comprehensively support the global tech industry in a manner that is effective, accurate, and human rights compliant.

Tech platforms and the Internet have, for a relatively long time now, been  a crucial part of our social fabric, but have become an even more central element in the world’s response to the Covid-19 outbreak. As the world has confined itself, we have seen politicians and international organisations share updates with the public through social media and messaging platforms. We have seen separated friends and families further rely on online communications to stay connected. Coordinated via online platforms, we have also seen entire student and work forces being managed remotely. However, we have also seen the evolution and fragmentation of our information environment, including a surge in online content with clear overlaps with far-right violent extremist messages. Thus, the Internet, whilst connecting people at a time of social distancing, became a reflection of societies’ divisions and antagonisms as terrorists and violent extremists used it to propel their violent ideologies.

In a way, terrorists and violent extremists can be seen as manipulators, exploiting pre-existing grievances and stress factors in our societies and amongst individuals, such as gaps in governments’ capabilities or one’s search for answers in life. In light of this, it should come as no surprise that the Covid-19 pandemic, the ensuing lockdown measures and restrictions, as well as the subsequent economic instability and general anxiety were ideal stress factors for terrorists and violent extremists to exploit. The credibility gaps caused by some governments’ hesitant response to the virus and the flow of online misinformation and disinformation further provided a breach in conventional narratives and in government trust for malevolent actors to take advantage of. Islamist terrorist groups with an on-the-ground presence – such as the Islamic State, al-Qaeda, and al-Shabaab¹– are thus attempting to benefit from fragile local governments being further destabilised by the crisis (especially in regard to health and security capacities) to present themselves as a viable solution and expand their control on the ground². Meanwhile, far-right violent extremists in Western countries are exploiting the so-called Covid-19 ‘infodemic’³ to penetrate the mainstream and distil their ideologies to a larger audience.

Given that the exploitation of the Covid-19 crisis by far-right violent extremists has primarily been an online phenomenon, in contrast with Islamist terrorists, we decided to focus on far-right violent extremists’ online manipulation of the pandemic for this insight piece.

Indeed, there has been a prominent rise in far-right violent extremists’ online discussion and propaganda centred around the virus4 expanding into the mainstream as it was bolstered by a crossover between their ideologies and mainstream discussions of the crisis. Far-right violent extremists capitalising on the virus thus represent an interesting example of violent extremists’ distortion of the information landscape and of their use of online platforms. This paper will thus address the following issues:

• First, it will discuss far-right violent extremists’ manipulation of discourse about the pandemic;
• Second, it will assess the implications for future content moderation through the lens of current tech sector responses to the wave of disinformation and conspiracy theories;
• Finally, we included Tech Against Terrorism’s commentary on the recently dimissed French legislation to combat hate on the Internet, Lutte contre la haine sur internet (or Loi Avia), as this proposed law, and its rebuttal by the French Constitutional Council on the 18 June 2020, touched upon some of the most important debates with regards to online regulation globally.

What is far-right violent extremism?

The label of far-right – or extreme right-wing; or ethnically and racially motivated – violent extremism and terrorism regroups a variety of different ideologies and networks; the main ones being divided between: neo-Nazism, white supremacism, white nationalism, and cultural nationalism. They gained the label of “far-right” as they share a number of ideological features including (but not limited to) racism, ultra-nationalism, ethnocentrism, anti-Semitism, opposition to equality and democratic principles, with mainstream far-right politics. However, important differences exist between the mainstream and the violent extremist far-right, with the major contrast being the use of violence.

“[A] key difference between the radical/populist right and the extreme-right is their willingness to use extra-state violence against political opponents and ‘racial’ enemies. The extreme-right embrace violence (whether fantasies of expanded state violence should they come to power, or terroristic, individual violence in the here and now) in order to promote an ideological agenda and this ideological agenda is, typically, a variation of fascism or neo-Nazism.”*

In contrast with Islamist terrorism, the far-right violent extremism landscape represents a relatively less structured, more decentralised and fragmented movement with a multitude of different groups and networks – some of them existing mostly online. Amongst the most influential groups and movements, we find:

• Ouest Casual (France based)
• The Nordic Resistance Movement (Sweden based)
• Feuerkrieg Division (mostly online, no physical central base)
• National Action and its armed branch Sonnenkrieg Division (UK based, both designated as terrorist organisations by the UK government)
• Atomwaffen Division (U.S. based, with European subgroups; now re-branded as the National Socialist Order)**
• Azov Battalion (Ukraine based, training a number of foreign fighters in the conflict in Ukraine)
• Blood & Honour and its armed branch Combat18 (Originally in the UK with a presence in Germany, Canada and the U.S.)

Timeline of recent far-right terrorist and violent extremist attacks:*

• 2011 Oslo and Utøya attacks, Norway, 77 deaths**
• 2018 Pittsburgh synagogue shooting, United States, 11 deaths***
• 2019 Christchurch mosque shooting, New Zealand, 51 deaths****
• 2019 El Paso shooting, United States, 23 deaths*****
• 2020 Hanau shooting, Germany, 11 deaths******

As the Covid-19 outbreak forced the world into lockdown, the online space witnessed a surge in conspiracy theories and mis/disinformation⁵ about the origin and proliferation of the virus, and governments’ responses to it. While conspiracy theories exist most fervently and openly on the fringes of the Internet, Covid-19 related conspiracy theories quickly found their way onto mainstream tech platforms, particularly social media. Amongst the main conspiracy theories related to Covid-19, which are not exclusively found in far-right circles but also in other environments such as the far-left, we notably find the following ideological strains:⁶

• 5G and anti-government. Amongst the most popular, this theory is centred around the idea of China having caused the pandemic through 5G⁷, as well as governments’ lockdown measures and restrictions being aimed at restricting people’s freedom⁸. In France, researchers have observed an outburst of anti-5G discourse on social networks with the appearance of the coronavirus and have pointed out a significant porosity between various conspiracy theories ⁹.
• Social control. The driving idea behind this is the assumption that the “end goal [of the virus] is to impose social restrictions and force vaccinations”¹⁰ in order to be able to monitor and track the population.
• Bioweapon. A popular belief amongst the far-right is that the coronavirus is a human-made virus targeting people based on race, released intentionally or by accident.
• QAnon, the ‘deep state’ Emerging in 2017, QAnon and linked ‘deep state’ theories have been amongst the most prolific during the Covid-19 pandemic¹¹, increasingly gaining ground on online platforms and attracting new audiences. QAnon-related theories on Covid-19 encompass different and shifting narratives on the virus – from a plot to damage President Trump’s standing, to a cover-up to arrest ‘deep state’ agents or a ‘spiritual warfare’ – whilst all relying on anti-government and xenophobic rhetoric¹². Analysis of QAnon’s presence on social media underline the movement’s capacity to exploit the pandemic, in particular in using public social media accounts to reach broader and new audiences. QAnon supporters especially used public groups on Facebook discussing the pandemic to attract new followers to their own websites. In doing so, they managed to gather 80 million interactions on Facebook, between January and April 2020, an alarmingly high volume in comparison with the 6.2 million interactions for posts linked to the World Health Organization over the same time period¹³. In just three months, the number of QAnon hashtags on Twitter increased by 21%¹⁴.
• Chaos theories. These are theories comprising the idea that instabilities created by the pandemic should be furthered to bring about chaos and societal changes; these theories are similar to the idea of accelerationism, popular amongst far-right violent extremists. Accelerationism refers to the accelerated collapse of society as we know it through violent means – such as terrorism and guerrilla warfare – and non-violent means such as political and media manipulation to heighten the system’s existing tensions and further polarise society¹⁵.

The QAnon movement

QAnon emerged in 2017 on 4chan, an image board* popular amongst far-right and violent extremists online networks, and is based on anti-government, anti-Semitic and xenophobic elements. The name QAnon is derived from the pseudonym of the 4chan user at the origin of the movement, “Q Clearance Patriot” (the letter Q standing for a secret defence clearance in the United States), warning of a conspiracy within the government to undermine Donald Trump’s presidency. QAnon supporters have an important mistrust of governments, and believe in a ‘deep state’, a global elite “of power brokers hell bent on enriching themselves and maintaining their Satanic child-murdering sex cult.”**

Whilst QAnon is not a violent extremist-led conspiracy movement, it is based on rhetorical elements commonly found in far-right violent extremist circles, and has led to violent acts in the US***. The surge of violence linked to QAnon even led the FBI to warn about the risks of domestic terrorism act motivated by conspiracy theories.**** QAnon is particularly compelling as a case study of how far-right rhetoric can infiltrate into the mainstream and motivate offline attacks. Namely, the conspiracy and its supporters increasingly made their ways from 4chan, a rather fringe platform, to mainstream social media (including Twitter, Facebook and YouTube), expending their audience reach every time. QAnon is also interesting in demonstrating how conspiracy theories can become influential across national borders. Whether through occasional linked conspiracy theories being spread internationally and in different languages,***** or with the broader QAnon movement gaining grounds outside the U.S., as has notably been the case in Canada recently.******

A number of major online platforms have recently taken action against the spread of QAnon related content on their platforms,******* most often on the ground of harmful disinformation, or threats of physical world violence. Twitter was amongst the first ones to do so, in mid-July, by banning thousands of accounts. In an attempt to respond to “shifting and evolving” challenges in moderating content, YouTube expanded its content policy to prohibit conspiracy theories “that have been used to justify real-world violence”. In doing so, it further expanded its policy on harmful misinformation and conspiracy theories. In accordance with its existing guidelines, the e-commerce platform Etsy has removed merchandise related to QAnon on the ground that such items “promote hate, incite violence, or promote or endorse harmful misinformation.” The platform also removed items related to the white supremacist group Proud Boys on the same ground. TikTok has also taken action against the conspiracy theory movement, removing accounts sharing related content on the ground that QAnon violates its disinformation policy. A few days after, the social media platform further expanded its hate speech policy to target “neighbouring ideologies” of neo-Nazism and white supremacy (including white nationalism and white genocide theory). In doing so, they also aimed at targeting “the spread of coded language and symbols that can normalise hateful speech and behaviour”. Finally, Facebook recently changed its community guidelines to limit the organisational capacity of groups that do not directly call for violence, but whose rhetoric demonstrates support for the use of violence. Beside removing QAnon content and networks that support the use of violence, Facebook has also started to implement a redirect counternarrative program, in partnership with the Global Network on Extremism and Technology (GNET, the research arm of the GIFCT and a core TAT partner). Users searching for terms related to QAnon on Facebook and Instagram are now redirected to credible sources from GNET.

Overall, online platforms’ crackdown on QAnon demonstrates a shift in content moderation policy to cover online content and groups that do not necessarily make direct calls for violence, but whose rhetoric expresses support for the use of violence, or groups that present an important threat of offline violence. In doing so, they are indeed directly targeting groups based on assessment of harms and risk of real world violence.

These conspiracy theories rapidly found their way into mainstream discourse on the virus, spreading virally on social media and messaging applications as individuals were searching for answers on the pandemic. Whilst not created by violent extremists, nor specific to one side or the other of the political spectrum, these conspiracy theories do, however, bear important similarities with far-right violent extremist discourses and share tenets usually endemic to these circles. Indeed, often based on the idea of a “global plot to subjugate  humanity”¹⁶ and anti-government rhetoric, these conspiracy theories are for the most part racist, xenophobic, and anti-Semitic in nature. As “the foreign-created virus” became a recurrent topic of online discussion through conspiracy theories, violent extremists saw an opportunity to further exploit ideas that form the basis of their own rhetoric. Through such ideas present in Covid-19 related conspiracy theories, they found a way to sew their own messages into the mainstream. The convergence of such ideas is all the more compelling for some of the conspiracy theories that are directly linked to the far-right, notably the deep state (QAnon) theories, or those apocalyptic in nature and advocating for social chaos. A post by a far-right violent extremist supporter on Telegram (an instant-messaging application), reported by Angie Gad in an article for the Global Network on Extremism and Technology (GNET), particularly exemplifies how the pandemic and the consequent instability benefits violent extremists: “We don’t need to convince the masses of our ideas. The pandemic has already done that… Whites panic bought guns, food, water, and prepare for social unrest all on their own […].”¹⁷

For violent extremists, it was then simply a matter of blending themselves into mainstream discussion and capitalising on the opportunity offered by the pandemic. Far-right violent extremists have thus taken to online platforms en masse to further spread disinformation and conspiracy theories, as well as their own hateful messages. They did so by not openly branding themselves as violent extremists, but rather by injecting themselves into public discussions with those whom they shared rhetoric elements and by posting links to more fringe websites they administered. A study by GNET found that some of the conspiracy groups on Telegram were ran by “known extremist Telegram users who dress [those groups] up as sources of information for those seeking answers”¹⁸.

As such, far-right violent extremists have tried to expand their reach to new audiences and anchor their ideas into the mainstream. Early on in the lockdowns, governments and experts alike voiced concerns about what an increase in social and economic grievances, paired with more time spent online, could do to contribute to radicalisation – not in the least among young people who are confined at home with outside recreational and cultural activities prohibited.¹⁹ The fact that online interactions can hold an important place in one’s radicalisation process has been acknowledged by the counter-terrorism community for quite some time now. However, the particularity of the Covid-19 pandemic, in this regard, is that it led to an unprecedented surge in online activities, especially amongst young people. A phenomenon summarised by Cynthia Miller-Idriss as:

“This dynamic creates a perfect storm for extremist recruitment and radicalization. Extreme isolation and increased online presence on gaming platforms, social media, and more creates growing possibilities for exposure to extremist content and expands the gateways that can lead to extremist radicalizations.”²⁰

Terrorists and violent extremists thrive on insecurities and grievances, posing as the solution to the difficulties one might encounter in life, for example economic hardship, lack of a sense of belonging, feelings of anxiety or disillusion with society… With the pandemic creating an ideal ground for such anxiety and a need for explanations, combined with more time spent online, violent extremists are stepping up their propaganda efforts to appear as the simple solution to an unprecedented time of instability and insecurity.

Violent extremists’ online disruption efforts – the “zoom-bombing”* trend

College campuses in the U.S. have for a while been a favourite recruitment target for far-right violent extremists, plastering some with their propaganda flyers and disrupting on-campus activities. With colleges closing and the movement of classes from the physical world to the virtual world, violent extremists followed the online migration and began ‘Zoom-bombing’ online classes: breaking into classes held on video-conferencing online platforms to harass and ‘troll’** students and professors. Notably, “[u]sers have reported Zoom-bombers shouting racial slurs, drawing neo-Nazi insignia, sharing explicit or violent videos and ‘doxxing’ – sharing the confidential personal information of – faculty and participants.”*** The phenomenon has spread so widely and rapidly that the FBI issued a warning for the users of these platforms, alerting them of the possibility of these intrusions****.

Violent extremists have long attempted to curate and exploit periods of instability and insecurity to spread their ideologies. One trend that has been increasingly present amongst far-right violent extremists over the last year, and that particularly reflects this attempt to feed on polarisation and instability, is accelerationism: the accelerated collapse of society as we know it through violent means – such as terrorism and guerrilla warfare – and non-violent means such as political and media manipulation to heighten the system’s existing tensions and further polarise society²¹. Based on a targeted monitoring and content analysis of 100 of far-right violent extremists, especially neo- Nazi, channels on Telegram, Tech Against Terrorism has studied how talks about accelerationism merged with discussions about the Covid-19 crisis.

The encrypted messenger app Telegram was chosen because of its ever- increasing importance amongst far-right violent extremist online circles, and especially amongst neo-Nazis²².

When reports emerged about violent extremists using Covid-19 as an impetus for accelerationist talks online, we decided to analyse how this was expressed on one of the most important messenger apps amongst far-right violent extremists circle: Telegram, a platform that Tech Against Terrorism has been monitoring for neo-Nazi content since 2019. In order to analyse how neo-Nazi accelerationist circles discussed the pandemic, we selected 100 channels based on a snowball sampling: starting with one neo-Nazi channel, we mapped out a network of neo-Nazi activity on the platform through the links shared on the original channel and included relevant public channels in our sample. The initial channel was located through a simple keyword search of neo-Nazi terms reflecting an adoration of Hitler, Nazi Germany, or a hatred of Jewish communities; neo-Nazi character of the identified channels was verified by Tech Against Terrorism’s research team. The analysis of the channels sampled and of the data retrieved allowed us to further confirm existing relations between the different neo-Nazi channels.

The first mentions of ‘accelerationism’ on violent extremist Telegram channels date back to 2019, and thus precede the Covid-19 pandemic. The first discussions around a new lung disease and mentions of China on violent extremist channels, for their parts, can be located to January and February 2020, approximately at the same time as when Covid-19 became a major news subject, and channels dedicated to the Covid-19 disease followed suit. Around March 2020, the association between accelerationism talks and the Covid-19 crises emerged on Telegram, shortly after the association of the two began to be referenced in a number of news articles²³.

Since then, mentions of accelerationism on Telegram have been continuously increasing. As the pandemic was spreading around the world, accelerationism was becoming a central topic of discussion on violent extremist Telegram channels (see a visualisation of such a surge on one selected Telegram channel, next page). One reason for these more frequent uses of the term over time   is most likely the intensifying media coverage around the pandemic and the measures it led to. This represented a favourable condition for accelerationists, as Covid-19 became a key subject of public discussion – and especially as the pandemic, the lockdown and other restrictions created more instability and contestation.

To better understand the language used by violent extremists discussing the Covid-19 pandemic on Telegram, we subjected a specific channel to further analysis. This channel was chosen for being mostly dedicated to accelerationist and Covid-19 talk, and thus heavily referencing the accelerationist ideology in light of the ongoing global pandemic. In this respect, we conducted a content analysis and mapped out the mention of the term ‘accel’ over time within this particular Telegram channel to better understand the use of language by far-right extremists. ‘Accel’ was deliberately chosen to account for variations, such as accelerate, accelerationism, accelerationist, or acceleration.

It is important to state that any generalisation of this analysis should be treated carefully due to the small sample size. However, given the nature of this group and its quite large number of members, it does provide relevant insights into the change in language used by these circles over time.

Other researchers have also shown a rapid uptick of accelerationism in these channels, especially purposed towards talk of weaponising the virus²⁴, stirring up social unrest, and accelerating the collapse of society²⁵.

This threat has been recognised by federal law enforcements in the U.S., that have reported on far-right violent extremist groups considering weaponising the virus for terrorist purposes²⁶, and has already manifested itself in real world events as we will see in the following section.

France, far-right violent extremism & accelerationism

From a French perspective, it might be easy to conclude that concepts such as accelerationism are irrelevant in France and view the broader far-right violent extremist phenomenon as something alien and with limited influence in the country. However, one would be mistaken to believe in a French exception in this context. Far-right violent extremism is present in France, including talks about an upcoming civil or racial war and the collapse of our current society – both key elements of accelerationism. A podcast recently published by Arte Radio* is a striking example of the entrenchment of such hateful ideas amongst some members of law enforcement in France. Quite interestingly, audio messages captured in the podcast underline that French supporters of violent extremism evolve on the same online platforms (Bitchute, VK) as their international counterparts. An insightful inquiry by the online investigation platform Bellingcat also sheds light on the international connection of French far-right violent extremists by exposing the presence of French elements in an openly neo-Nazi music festival in Ukraine in December 2019.**

More research on French violent extremist networks is required to fully comprehend the extent of the threat in the country and how these networks interact with their international counterparts. In the meantime, the threat should not be overlooked, as highlighted by the recent arrest of a man plotting an attack on the Jewish community in France.***

All this disinformation and far-right violent extremist talk does not limit itself to online platforms but has found its way into real world activities and events through anti-lockdown protests around the world, 5G infrastructure attacks, telecom worker harassment, and folded violent extremist plots.

Anti-lockdown demonstrations criticising lockdown measures on the basis that they restrict freedom and calling for these measures to end have occurred throughout Europe (notably in Germany, Switzerland, Spain and Italy). However, protests in the U.S. have received greater media attention, most likely as a result of their scale and of the militarisation some of these protests have seen.²⁷ Whilst the anti-lockdown protests represent a wide spectrum of political beliefs, with the majority of participants having no affiliation whatsoever with violent extremism, many violent extremist groups saw these protests as an ideal opportunity for publicity and for spreading their ideologies²⁸. Thus, if some far-right violent extremist groups were amongst the first to organise protests in the U.S., they then started to blend themselves into the mainstream demonstrations.²⁹ The Boogaloo (Bois)³⁰, an American meme31 turned violent extremist movement calling for a second civil war, and the Proud Boys, a U.S. neo-fascist gang,³² were amongst the first far-right violent extremists to be spotted at anti-lockdown protests, openly displaying symbols associated with American far-right violent extremism³³. In time, these demonstrations saw an ever-increasing violent extremist presence, as far-right violent extremists continued to capitalise on the pandemic and lockdown measures, and finding a great medium of communications and publicity in the aforementioned conspiracy theories and protests. If the link between violent extremists and anti-lockdown demonstrations has been the most striking in the U.S., the same combining of conspiracy theories, anti-government sentiments, and violent extremist motives can be observed in Europe. Protests in Germany, for instance, have been supported by the mainstream far-right whilst witnessing a mixture of conspiracy theory supporters. It is particularly important here to remember the role of violent extremists and neo-Nazi elements in spreading conspiracy theories.³⁴ In a way, anti-lockdown protests represent the physical-world manifestation of the crossover between violent extremist ideologies and mainstream discussion on the pandemic.

Covid-19 related conspiracy theories also materialised themselves with acts of physical violence. Conspiracy theories linking the virus to 5G technologies, in particular, led to attacks on 5G infrastructure and telecom workers in the UK and in the Netherlands³⁵. This phenomenon was encouraged by far-right accelerationists who saw this as a chance to create further social unrest³⁶. Beyond these attacks and harassments on telecom workers, far-right violent extremists’ plots have also been thwarted in the U.S. and France since the start of the pandemic. On 26 May 2020, a man was arrested in France for planning an attack on the Jewish community. The man, whose online profile on social media showed his engagement with neo-Nazi material, including a French translation of the Christchurch shooter manifesto³⁷, reportedly became increasingly violent over the weeks leading up to his arrest³⁸ and demonstrated a particular interest for the reopening of places of worship in France once the lockdown was lifted³⁹.

At this stage of the investigation, we can only assume that this man’s increased interest in far-right violent extremism over the recent weeks can be linked to the pandemic acting as a catalyst for action. However, other thwarted plans in the U.S. have, for their part, clearly been linked to Covid-19 and accelerationism. Early on in the pandemic, a white supremacist was notably killed during his arrest by the FBI as he planned an attack on a hospital hosting Covid-19 patients. Whilst the individual had been a supporter of white supremacist ideology and a proponent of “kick start[ing] a revolution” before the pandemic, details of a discussion he had with an undercover law enforcement agent reveal that the Covid-19 crisis accelerated his action. The individual viewed the increased media attention given to the health sector as an opportunity to promote and grab attention on violent extremist views⁴⁰. These foiled attack plans show how violent extremists’ taking advantage of the virus, especially accelerationist talks, pose a real threat to security in Western democracies. This threat should in no case be underestimated as “lone-wolf” far-right attackers have demonstrated their capacity to act on their hateful beliefs in recent years – 2019 being a particularly tragic testimony year in this regard as the world witnessed four major far-right motivated terrorist attacks in New Zeland, the U.S., and Germany; including the tragic Christchurch mosques attack and the El Paso Shooting.

As shown in this analysis, far-right violent extremists have capitalised on the pandemic to exploit and distort the media and information landscape, displaying once again their ability to use online platforms for their own agendas. However, tech companies – especially social media and messaging apps – were quick to react to the spread of Covid-19 disinformation and conspiracy theories shared on their platforms. While they did not necessarily target violent extremists in doing so, the content moderation policy changes introduced would also affect violent extremists’ capacity to benefit from the pandemic. In the following section, we will see what these changes mean for the future of online content moderation. In an attempt to ‘curb the spread’ of Covid-19 related online harms, a number of tech platforms have made changes to their content standards, often broadening their definition of harms to prohibit or disrupt the spread of conspiracy theories and mis/disinformation related to the virus. Prominent examples of this include YouTube’s decision to ban all conspiracy theory videos linking the coronavirus to 5G networks – a particularly welcomed move according to some experts given existing discussions about YouTube’s potential for extremist exploitation in reaching out to broad audiences⁴¹ – Twitter’s removal of any information that contradicts official health guidelines, and WhatsApp’s newest restrictions on the forwarding of messages (“messages that have been identified as “highly forwarded” – sent through a chain of five or more people – can only be forwarded to a single person [instead of five usually]”⁴²). Such swift changes in platforms’ policies raise important questions around content moderation (COMO) strategies and the inherent challenges in tackling terrorist and violent extremist use of the Internet and disinformation in a manner that respects human rights.

What are content standards?

“Content standards” designate the broad set of guidelines available for a tech company to lay out what is accepted, or forbidden, on its platform, including: Terms of Service, Community Guidelines, and Acceptable Use Policies. By being the principal way for a platform to inform its users on what is prohibited or not, and thus on its content moderation policy and enforcement, content standards are essential for a platform’s transparency and accountability. Tech Against Terrorism advises tech companies to clearly lay out what they expect of their users, and how they intend to enforce their moderation policies in case of violation, as well as to explicitly prohibit terrorist use in their content standards. Such prohibition is also a requirement for membership to the Global Internet Forum to Counter Terrorism (GIFCT), an industry-led coalition formed by Facebook, Twitter, YouTube, and Microsoft which collaborates with a wide range of NGOs, academic experts, and governments.*

The rapid response of tech companies to adapt their policies and service offerings to protect users and society as a whole has prompted people to question why such a response is not the norm. More than anything else, however, the question should be: what changes are being made to enable such an expedient response now, and how can we keep track of what ground is being covered by these measures for when this is all over?

Part of the answer is that in 2020 tech companies are able to fully rely on the systems, strategies, and partnerships that they have been carefully putting in place since 2017 and the establishement of the GIFCT. This means that in order to deal with the increased risk of misuse of their platforms, they have been able to capitalise on the investment they have put into needful policy work, effective content moderation solutions, and collaboration with researchers, civil society, and experts on the ground or fact-checkers. Over recent years, tech platforms have indeed increasingly been trying to efficiently respond to terrorist use of the Internet. They did so through increased content moderation efforts, including the development of machine-learning solutions, counter-narratives projects or through GIFCT whose hash-sharing consortium and crisis protocol system particularly demonstrate this commitment. Broadly speaking a “hash” is a unique fingerprint given to an online image or video. Through hash-sharing tech platforms have the possibility to quickly locate, take down, or prevent the uploading of, an image or video that has previously been identified as terrorist content. The Covid-19 crisis has also stressed the importance of well-functioning and robust public-private channels between governments and the tech sector.

However, the pandemic and its consequent rapid changes in policies also present the risks of encroachments on the policies that tech platforms had slowly and transparently built over recent years. This is exacerbated by the fact that human beings may be more out of the moderation loop than usual, notably because many employees have been sent home for social distancing purposes⁴³. Whilst already widely used by major tech platforms to assist with their content moderation efforts, the use of artificial intelligence, specifically machine-learning⁴⁴ to counter terrorist use of the Internet, is still in an initial phase. At the moment, such technology notably lacks the understanding of nuance and context (e.g. difficulties distinguishing between terrorist content and journalistic information). If the development of such technologies can be encouraged, human moderators remain essential, especially to avoid over-censoring and the taking down of legitimate content. Moreover, these technologies require important quantities of data to be perfected, which remains a barrier to entry with regard to using it to counter terrorist and violent extremist use of the Internet – particularly when interested in predictive behaviour technologies.

In light of this, the Center for Democracy and Technology released a joint statement about concerns shared by industry actors and NGOs over tech companies’ increased reliance on automated moderation during this period – including the use of machine – learning based tools to detect and even adjudicate infringing content on their websites, rather than manual, human moderation⁴⁵.

An increased reliance on automated moderation solutions raises the risk of false positives in taking down content that is legal, and raises questions about accountability in removal decisions ⁴⁶. It becomes even more complicated when harmful content originates from users that are not affiliated with violent extremism, as is often the case at the moment, or when the content exists in a “grey area” (due to the lack of designation of far-right terrorists and violent extremists as terrorist entities by governments, these groups often exist in what is commonly called a legal “grey area”). Worse, however, is the risk that content denouncing human rights violations, including journalistic content that can serve as evidence of such violation, could be automatically removed ⁴⁷, more so at a time where constitutional guarantees are weakened in certain countries.

There are multiple reports by human rights groups and advocates that the Covid-19 crisis is being used by some governments around the world to “establish or tighten control over the national media and to step up state censorship⁴⁸.” Concerns around conspiracy theories, disinformation, and the threat of violent actors who wish to take advantage of such a situation can thus become easy pretexts for censorship. In the past, anti-democratic governments have used the security threat of terrorist and violent extremist exploitations of the Internet to try and arm-twist tech companies into silencing dissenting voices or restricting access to information. It is perhaps unsurprising that in the midst of this crisis, various governments have shut down the Internet in parts of their territories or restricted access to hundreds of sites⁴⁹. Others have temporarily revoked constitutional guarantees, including around freedom of expression⁵⁰. This is partly why clear content standards across the Internet are so important: they act as a defence against unlawful or ad hoc suppression. Attempts to criminalise information relating to the pandemic may create distrust in institutional information, delay access to reliable information, and have a chilling effect on freedom of expression⁵¹. That platforms are policed in a manner that follows rule of law precisely because it is a crisis and because the circumstances are exceptional is exactly the point.

In a joint announcement, David Kaye (United Nations Special Rapporteur on the promotion and protection of the right to freedom of opinion and expression), Harlem Désir (Organization for Security and Co-operation in Europe Representative on Freedom of the Media, OSCE), and Edison Lanza (Inter-American Commission on Human Rights Special Rapporteur for Freedom of Expression, IACHR) all highlighted the importance of governments and tech companies working together to provide reliable information in the wake of false information about the pandemic that “could lead to health concerns, panic, and disorder.” However, they also emphasised that “resorting to other measures, such as content take-downs and censorship, may result in limiting access to important information for public health and should only be undertaken where they meet the standards of necessity and proportionality.”⁵² 

Such considerations are even more significant in light of recent developments in the world of online regulation: the EU is still drafting its new online regulation proposal ⁵³, which could lead to the imposition of upload filters – automated programmes scanning data according to certain criteria, and allowing or preventing the upload of content depending on the result of the scan – and thus to increased reliance on automated tools⁵⁴. In parallel, France recently attempted to pass a new “cyberhate law” obliging tech platforms to remove notified terrorist content within one hour, and the U.S. president signed an executive order that, whilst contested for constitutionality reasons, could remove tech platforms’ legal protection from liability for user generated content. As with other online regulation laws passed in Europe, notably the 2017 German Network Enforcement Act (NetzDG), human rights advocates and civil society groups have criticised these laws for potentially incentivising overzealous moderation and leading to the removal of lawful content. Nevertheless, many also regard such laws as necessary steps to ensure that tech companies have accountability and transparency due processes in place. What is clear, however, is that tech companies and governments alike must balance confronting new and evolving harms and threats to society in the online sphere whilst protecting human rights.

On 24 June 2020, a new law was promulgated in France on regulating hateful online content⁵⁵, aiming at countering the spread of hateful, discriminatory, and terrorist content online. All content usual to the playbook of the violent extremists discussed earlier.

A year after its first introduction to Parliament, the French national assembly adopted on 13 May 2020 this new, so-called “cyberhate” law or “Loi Avia”⁵⁶. Bearing several similarities with the German NetzDG, this law would have required tech platforms to remove terrorist and child sexual abuse content within one hour once it had been reported to the platform by the French authorities, whereas other hateful and discriminatory content would have had to be removed within 24 hours once reported by the authorities or by users. Platforms failing to comply were to face a fine ranging from 250,000 to 20 million euros, or up to 4% of the platform’s global annual revenue. However, the core of the “cyberhate” law was censored by the French Constitutional Council on 18 June 2020⁵⁷. This effectively meant that the law was to be reduced to its prevention aspect, and that all requirements for tech platforms to take down notified content within one or 24 hours were deemed unconstitutional.

The fight to counter terrorist and violent extremist use of the Internet should not be ignored, especially at a time of intensified instability and disinformation. However, online regulation should not come at the price of accountability and freedom of expression⁵⁸. We previously provided a commentary on tech companies’ slightly adapted content moderation processes to respond to Covid-19, and below we highlight what we consider to be the main problematic characteristics of the dismissed “cyberhate law”. We believe that despite the censoring from the French Constitutional Council, this commentary can still be relevant in underlining limitations common to a number of recent attempts to counter the spread of terrorist and violent extremist content online. This is especially the case given the important similarities between the dismissed “cyberhate law”, the German NetzDG and the draft EU regulation on online terrorist content⁵⁹. Moreover, some of the criticisms we addressed to the law were also present in the Constitutional Council’s decision to censor it.

• A lack of consideration for smaller tech platforms’ The one-hour delay for removal of terrorist content is unrealistic for micro and small platforms. They have few employees (some are run by just one person) and lack the technical and human resources necessary to be able to respond to notifications of terrorist content within such a short deadline. A one-hour time period, and even a 24-hour one, would require constant monitoring from tech platforms to ensure compliance. It is a difficult, if not impossible, endeavour for most tech platforms. This strain bears the risk of reduced competition in the tech sector if smaller platforms are not able to catch up or are financially afflicted by the fines. Furthermore, research conducted by Tech Against Terrorism has shown that smaller platforms are heavily targeted by terrorist exploitation⁶⁰. Yet, smaller platforms that are willing to tackle this exploitation but lack the scope to do so, and thus require the most assistance in tackling terrorist use of the Internet, may end up being left behind by this new legislation.

The short removal delay of the law was underlined by the French Constitutional Council as being particularly difficult for tech platforms to meet, and for being disproportionate with regard to the risk posed to freedom of expression. The Constitutional Council particularly underlined that the law included regulations that were “impossible to satisfy” for tech platforms, thus breaking the principle of equality with regard to public regulations.

• Risks for freedom of Due to the quick deadline and the broad scope of the law, tech platforms will not have the time to properly adjudicate on a content’s legality. Instead, there is the risk of an overzealous removal of content, with platforms indiscriminately taking down all content notified before properly assessing whether such content would in fact be considered illegal under existing French legislation. Even for the bigger tech platforms, the so-called GAFAM (Google, Amazon, Facebook,  Apple  and  Microsoft) in France, the short deadline to remove signalled content would require an important increase in resources dedicated to content moderation. One easy way to do so for the platforms that have technical resources will be an increased reliance on automated content moderation tools. Whilst automated content moderation has its benefits, current solutions are not fairly nuanced and require human overview to avoid the excessive takedown of content. It notably lacks the capacity to comprehend context; for example, whether a content is journalistic or shared in order to denounce a specific position. Thus, an increase in relying on such methods, instead of human moderation, presents important threats for freedom of expression as automated moderation risks excessively taking down lawful content.

The risks for freedom of expression were also underlined by the French Constitutional Council, which deemed that they were neither necessary nor appropriate or proportionate.

• Leaving tech platforms to adjudicate on The law itself does not create a new set of harms, nor does it create a new range of prohibited content. Everything, from hateful and discriminatory to terrorist and child sexual abuse content, is already illegal under French law (respectively as a limit to freedom of expression and under criminal code). However, the existing laws it relies on are themselves broad and can cover a wide range of content – it is the reason why current limitations to freedom of expression have to be decided upon by an independent judiciary body. The law thus shifts the responsibility of (rapidly) deciding what is hateful or discriminatory content to private tech companies, rather than offering tech platforms clear cut definitions and examples of what content is to be removed. Indeed, such a provision may in fact lead to a development where more influence over the online speech rulebook is given to tech companies.

In this regard, the Constitutional Council’s decision was a strong reminder that adjudicating on the legality of online content, in particular terrorist content, is “subject to the sole discretion of the [French] administration.” Yet, the required removal deadlines are deemed insufficient to obtain the necessary court’s decision to assess the illegality of a specific content. This decision is an important reminder of the significance of the rule of law in countering online terrorist content.

Even though the “cyberhate” law presented important risks for freedom of expression and did not consider the technical and human capacities of smaller tech platforms, the law also includes commendable points. Most of these commendable points touch upon the idea of preventing the spread of online hate and calling for greater cooperation, and have not been censored by the Constitutional Council. Hence, even though the law has been stripped of its core requirements for posing a too important risk to freedom of expression, the Avia law has been passed in France and will be implemented along its more “preventive” lines. Below, we provide a commentary on some of its positive requirements.

• Calls for increased cooperation in the tech The law, commendably, broadly promotes tools for cooperation and information sharing between platforms. However, it does so without specifying what these tools would be. More information in the future on how the French government plans to support said cooperation and how to properly account for smaller platforms in this regard would be beneficial.
• Increased transparency and The new legislation also calls for increased transparency and accountability from tech platforms, as well as from the French government, whose Conseil supérieur de l’audiovisuel – the French audiovisual authority – will oversee the enforcement of the law and will publish an annual report on it. However, this call for transparency does not take into account platforms’ sizes and individual capacities. Yet, these are important factors to be considered to avoid unrealistic expectations as to what metrics should be included in the transparency reports.

Concerns around increased content moderation policy bearing the risk of online censorship should not be ignored now, nor forgotten once the Covid-19 public health crisis is declared to be over. Many of these problems will not be dispelled with the end of the pandemic and will have lasting implications for human rights and democratic functioning. In light of this, we recommend the following:

Tech companies should strive to:

1. Improve accountability mechanisms, such as appeal and redress processes.
2. Increase transparency on decision-making behind policy development and enforcement, especially with regard to content removal.
3. Review changes in content moderation made in response to the Covid-19 crisis once the crisis is over.
4. Produce special transparency reports detailing platform governance and content moderation activity during the Covid-19 pandemic. For small and micro-sized tech companies, taking record, even in a rudimentary manner, of at least some of these statistics will be invariably valuable as well⁶¹.
5. Invest in improved understanding of the threat of far-right violent extremism on tech platforms
   • particularly in giving further attention to how these groups can instrumentalise mainstream media and public discussion for their own purposes;
   • in supporting efforts to promote improved understanding of how terrorist and violent extremists target youth online.
6. Support the development of technical solutions to mitigate terrorist and violent use of the Internet whilst safeguarding fundamental rights and particularly freedom of expression.
7. Continue providing support to small and micro-sized tech platforms who do not have the resources to tackle terrorist and violent extremist use of their services.

Governments should strive to:

• 8. Produce transparency reports on their engagement with tech companies and on how they affect online speech, including data on account information, suspension requests and takedown requests.
• 9. Invest in research to better understand how real-world grievances lead to radicalisation and terrorism, and how this manifests itself in the online space, with a particular focus on how the Covid-19 pandemic may have affected this dynamic.
• 10. Use legal powers to promote the rule of law through more comprehensive terrorism designation lists to help increase definitional clarity around terms such as terrorism, as opposed to outsourcing the onus of adjudication on terrorism and illegality onto tech companies.
• 11. Support the tech sector in countering terrorist and violent extremist use of the Internet, for example through supporting smaller tech platforms, public-private partnership endeavours, and digital literacy programmes.
• 12. Promote proportionate and balanced regulation that takes smaller tech platforms’ capacities to comply into account.
• 13. Communicate government policy clearly, especially in times of crisis, to reduce ambiguity around said policy and feelings of instability within societies, and thus to avoid terrorist and violent extremist exploitation of stress factors and insecrurity.