{'fraction_non_alphanumeric': 0.030075468878427856, 'fraction_numerical': 0.013935589927519988, 'mean_word_length': 5.146268656716418, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Privatised enforcement and the right to freedom of expression in a world confronted with terrorism propaganda online

: The purpose of this paper is to explore the risks of privatised enforcement in the field of terrorism propaganda, stemming from the EU Code of conduct on countering illegal hate speech online. By shedding light on this Code, the author argues that implementation of it may undermine the rule of law and give rise to private censorship. In order to outweigh these risks, IT companies should improve their transparency, especially towards users whose content have been affected. Where automated means are used, the companies should always have in place some form of human intervention in order to contextualise posts. At the EU level, the Commission should provide IT companies with clearer guidelines regarding their liability exemption under the e-Commerce Directive. This would help prevent a race-to-the bottom where intermediaries choose to interpret and apply the most stringent national laws in order to secure at utmost their liability. The paper further articulates on the fine line that exists between ‘terrorist content’ and ‘illegal hate speech’ and the need for more detailed definitions.


INTRODUCTION
Terrorism is not a new issue (Ansart, 2011), but terrorism propaganda online is. As early as 2008 the EU Council officially recognised the internet as a medium used by terrorist recruiters for the dissemination of propaganda material (EU Council Framework Decision 2008/919/JHA). Several studies revealed the important role played by social media platforms, predominantly Twitter, in ISIS' 1 propaganda strategy (Badawy & Ferrara, 2017, p. 2). A 2015 report illustrated that members of ISIS, on average, posted 38 propaganda materials each day, ranging from videos to photographs or articles and on a diversity of platforms, including Facebook, Tumblr, Twitter or Surespot (Winter, 2015, p. 10). Countering this type of speech has challenged traditional law enforcement in many ways. In 2014, the EU Commission recognised that traditional law enforcement is insufficient to deal with evolving trends in radicalisation and that all of society ought to be involved in the countering of terrorism online (COM (2013) 941 final, para. 8).
On 31 May 2016, four IT companies (Facebook, Microsoft, Twitter and Youtube, 2016) adopted the EU Code of conduct against illegal hate speech online (hereinafter, the Code). This instrument places enforcement responsibilities into the hands of private companies and gives rise to the practice of 'privatised enforcement'. The dangers stemming from such practice can be illustrated by Twitter's latest biannual report (2017), in which it indicates that from July 2017 through December 2017, 274,460 accounts were suspended because of terrorism' related activities in violation of the company's terms and services. It also specifies on its webpage concerning removal requests that 'out of the 1,661 reports received from trusted reporters and other EU non-governmental organisations (NGOs), 19% resulted in content removal due to terms of service (TOS) violations and 10% in content being withheld in a particular country based on local law(s)'. In other words, more posts seem to have been removed because of noncompliance with the companies' policies than due to illegality. Consequently, when placing private companies at the frontline of law enforcement online, the risk may arise that our right to freedom of expression is merely guided by their terms of service, which may not always be in accordance with the level of protection guaranteed under human rights instruments, such as under Article 10 of the European Convention on Human Rights (hereinafter, ECHR) or Article 11 of the Charter of Fundamental Rights of the European Union. Moreover, taking into account the primary profit-making nature of platforms, it is questionable in how far delegation of such large-scale public functions, which are fundamental to the proper function of our democracy, may be at odd with their business objectives and thereby result in a conflict of interests. As was pointed out in an article which discussed the liability of Google when faced with removal of defamatory content: 'in order to pursue its profit (emphasis added), Google did not adopt precautionary measures that could have prevented the upload of illegal materials […] Google is profiting from people uploading materials on the internet' (Sarter et al.,p. 372). Taking into account the intermediaries' data-driven business model, placing them at the frontline of law enforcement may be dangerous from a legal point of view but also for democracy in general.
Whereas the privatised enforcement phenomenon has already received considerable academic attention, this paper specifically focuses on the risks stemming from the Code, in the field of illegal hate speech and, in particular, terrorism propaganda. Through identifying such risks and by taking into account subsequently adopted EU instruments, recommendations are made on how to better guarantee respect for fundamental human rights in the online environment. These findings are especially relevant as the EU Commission issued, on 12 September 2018, a proposal Privatised enforcement and the right to freedom of expression in a world confronted with terrorism propaganda online for a Regulation on the prevention of terrorist content online. Besides the proposal's general requirement that hosting service providers should remove or disable access to terrorist content within one hour after receipt of a removal order, it also encourages the use of 'referrals', whose content should be assessed against the companies own terms and conditions. In that respect, it makes no reference to the law.
In order to draw a conclusion and make recommendations, the content of the Code and its relationship with privatised enforcement is first discussed. This section also delineates to what degree terrorism propaganda falls within the scope of the Code. Doing so is necessary, seeing as the Code merely focuses on the removal of 'illegal hate speech' whereas the countering of terrorism propaganda formed one of the main incentives for its adoption. This was made clear by EU Commissioner Vera Jourová who declared, when announcing the Code, that recent terror attacks have strengthened the need for it and that 'social media is unfortunately one of the tools that terrorist groups use to radicalise young people' (European Commission, 2016). In other words, it investigates whether and to what extent terrorist propaganda can be countered through hate speech tools. In the second section, different reasons behind privatised enforcement in the field of terrorism propaganda are presented. This is followed by a discussion on the dangers of such practice from a free speech perspective. In the subsequent section, recommendations to outweigh the identified risks are proposed, by taking into account subsequently adopted EU instruments building upon the Code, namely the communication and recommendation on tackling illegal content. The final section presents important developments that have taken place since the adoption of the Code.

PRIVATISED ENFORCEMENT THROUGH THE EU CODE OF CONDUCT ON COUNTERING ILLEGAL HATE SPEECH ONLINE
The Code is a self-regulatory initiative under which Twitter, Microsoft, YouTube and Facebook made a commitment to put in place a notice-and-take down system for the countering of illegal hate speech, the ambit of which is laid down in Framework Decision 2008/913/JHA. This nonbinding instrument encourages companies to assess the legality of a post within 24 hours after being notified and to remove or block access to it in case of unlawfulness. Importantly, it explicitly stipulates that the notified posts have to be primarily reviewed against the company's rules and community guidelines and only 'where necessary' (emphasis added) against national laws transposing the Framework Decision. Through these means, specifically encouraging the companies to 'take the lead' and initiative in tackling illegal hate speech online, the Code stimulates the occurrence of privatised enforcement. This phenomenon was defined as a practice in which private companies undertake 'non-law based "voluntary" enforcement measures' (Council of Europe, 2014, p. 86). Legal scholars define this practice as: 'instances where private parties (voluntarily) undertake law-enforcement measures' (Angelopoulos et al., 2015, p. 6). These two definitions show that privatised enforcement has three key components: enforcement of the law; by a private party; and imposed voluntarily (in the sense that the enforcement measures flow from self-regulatory initiatives and are thus 'non-law based'). This is sometimes also referred to as 'intermediarization' (Farrand, 2013, p. 405) or 'delegated' enforcement, in the sense that the regulator's role is delegated to companies and private sector actors (ADF International, 2016, p. 1). This practice has already been encouraged in different fields of law such as copyright law (EDRi, 2014, pp. 2-14) or the countering of 'fake news' on social media (OSCE, FOM.GAL/3/17, 2017, section 4(a)).
Whereas terrorism propaganda formed one of the main reasons for adopting the Code, such speech is not explicitly mentioned in it. The companies are merely required to counter 'illegal hate speech'. In the Commission's Communication on 'tackling illegal content online' (COM (2017), 555 final) a clear distinction is made between 'incitement to terrorism' and 'xenophobic and racist speech that publicly incites hatred and violence' (p. 2). The latter refers to the type of hate speech that is criminalised under Framework Decision 2008/913/JHA and which serves as legal basis for content removal under the Code. Concerning incitement to terrorism, the Communication refers to Article 5 of the Terrorism Directive (EU Directive 2017/541), which covers the 'public provocation to commit a terrorist offence'. Bearing this in mind, how can the Code thus contribute to the countering of terrorism propaganda?
An important distinction to be drawn between 'incitement to terrorism' and 'illegal hate speech' is that the former only covers incitement to violence (See Article 3(1), point (a) to (i) of the Terrorism Directive) while the latter also extends to incitement to hatred. The relation between these two was made clear by Vera Jourová who stated, in the context of terrorism propaganda, that 'there is growing evidence that online incitement to hatred leads to violence offline' (European Commission, 2015). In this respect, it is important to highlight that the United Nations General Assembly (2013) has determined that 'the likelihood for harm to occur' is a factor that should be taken into account when assessing whether incitement to hatred is present (para. 29). Although 'incitement' is by definition an inchoate crime, there is thus an implicit assumption that the speech has a reasonable probability to incite the intended actions and thereby cause harm. In the Surek v. Turkey case, this implicit relation between incitement to hatred, on the one hand, and actions, on the other, was made clear by the European Court of Human Rights (hereinafter, ECtHR) which noted that the speech was 'capable of inciting to further violence by instilling a deep-seated and irrational hatred' ( §62).
In the context of terrorism, the Commission claimed, in June 2017, that 'countering illegal hate speech online' serves to counter radicalisation (COM (2017), 354 final, p. 3). The link between radicalisation through hate speech and terrorist acts was also made explicit by Julian King, Commissioner for the Security Union who declared that: 'there is a direct link between recent attacks in Europe and the online material used by terrorist groups like Da'esh to radicalise the vulnerable and to sow fear and division in our communities' (European Commission, 2017).
This overlap between incitement to hatred and incitement to terrorism may be explained by the fact that terrorism relies on extremist ideologies. These were identified by Europol (2013) to include religious, ethno-nationalist and separatist ideologies as well as left-wing and anarchistic ones (pp. 16-30).
However, it is relevant to highlight the Leroy v. France case, which illustrates how the Code, and thereby illegal hate speech, would fall short in countering all types of terrorism propaganda.
In this case, a cartoonist was accused of glorification to terrorism after having published, on the day of the 9/11 terrorist attacks, a drawing representing the American Twin Towers. The drawing was interpreted by the Court ( §42) as a call for violence to and glorification of terrorism but was not perceived as a reflection of the cartoonist's anti-American ideologies. This type of speech, in which the underlying extremist ideologies are implicit within the speech -and therefore 'hidden'-will not easily be caught under the Code. Indeed, for 'illegal hate speech' to be present, some kind of discrimination must be expressed (Article 1(a) Framework Decision 2008/913/JHA). Such a discriminatory element is however not required for 'incitement to terrorism' as defined under the Terrorism Directive.
Privatised enforcement and the right to freedom of expression in a world confronted with terrorism propaganda online

DIFFERENT REASONS FOR PRIVATISED ENFORCEMENT IN THE FIELD OF TERRORISM PROPAGANDA
In 2015, the Commission highlighted, in a proposal for a directive, the importance of internet intermediaries in the fight against terrorism propaganda (COM (2015) When it comes to the countering of illegal content online, the Commission has emphasised the favourable position of internet intermediaries. In its recent proposal for a regulation concerned with the online removal of terrorist content, it indicated their 'central role' in the dissemination of such material as well as their 'technological means and capabilities' justifying their 'particular societal responsibilities' (recital 3). Placing internet intermediaries at the frontline of law enforcement online was thus by no means a coincidence. Indeed, as opposed to public authorities, intermediaries have better technological means at their disposal to swiftly notice illegal content, identify infringing authors and, subsequently, block or remove allegedly illegal material. Moreover, taking into account the speed at which terrorist content is disseminated across online services, it seems primordial to involve the parties that are most prone to react quickly.

FREEDOM OF EXPRESSION RISKS STEMMING FROM PRIVATISED ENFORCEMENT
Whilst it was practical to involve internet intermediaries in the counter-terrorism process, to have them enforce the law online constitutes a real potential danger for their users' right to freedom of expression. Importantly, as was made clear in Jersild v. Denmark ( §30), the right to freedom of expression is twofold in the sense that it does not only protect individuals' right to impart information but also the public's right to receive such information. As repeatedly held by the ECtHR, 'freedom of expression constitutes one of the essential foundations of a democratic society and one of the basic conditions for its progress and for each individual's self-fulfillment' (Hertel v. Switzerland,§ 46;Animal Defenders International v. The United Kingdom,§ 100).
This was also recognised at the international level by the Human Rights Committee (2011, para. 2). Importantly, the right is very broad in scope and also applies to ideas that 'offend, shock or disturb any sector of the population' (Handyside v. UK, § 49). Whereas the right is subject to limitations, such limitations are strict as these must meet different requirements, which will be discussed below, in order to be permissible. Taking into account the broad nature of this right, on the one hand, and the strict limitations, on the other, it is argued that the Code gives rise to the risk that the rule of law is undermined and that private censorship may arise.

A CHALLENGE TO THE RULE OF LAW
As specified by the Code, removal of a post shall be primarily based on the company's terms of service and only secondarily and when necessary, on national law. In an issue paper by the Council of Europe (2014) it was warned that such a practice would give rise to the risk that 'general terms and conditions of private-sector entities are not in accordance with international human rights standards' and therefore that the rule of law is threatened (p. 14 and 87). In that same paper, the rule of law was described as 'a principle of governance by which all persons, Consequently, the danger exists that the law may be downgraded to terms of service.
The risk for IT companies to incorrectly interpret and enforce illegal hate speech is even more emphasised when taking into account that under EU law, different factors such as the intent of the speaker, the likelihood for harm to occur and the context of the speech must be considered (Surek v Turkey,§62;Gokceli v Turkey,§ 38). Although in theory the European Commission (2017) has specified that such factors shall also be taken into account by the IT companies in their assessment of illegal hate speech, no reporting activities have yet taken place in which is demonstrated that such elements play a role in their assessment. The only way through which can be inferred that the companies do take these factors into account is by taking a look at their community guidelines. However, when reading those, Twitter merely seems to take into account the 'context of the larger conversation' and Facebook fails to mention 'the likelihood for harm to occur' or require 'incitement' to be present (Allan, 2017).

PRIVATE CENSORSHIP
Another major risk on our right to freedom of expression is that the 'presupposes authorising the provider to exercise the controls that may prevent its liability, i.e., empowering it to exclude all those contents that may generate liability' (p. 376).
Indeed, according to Article 14 (jo. recital 46) of this Directive, hosting providers may be exempted from liability when they 'expeditiously remove or disable access to illegal content' after having been notified of such content's presence. As was argued by legal scholar Aleksandra Kuczerawy (2015), such a mechanism implies a conflict of interests for the intermediary. To put this in her own words: 'they [the internet intermediaries] have to decide swiftly about removing or blocking content in order to exonerate themselves from possible liability, which basically makes them a judge in their own cause' (p. 48). Consequently, as was pointed out by her, they will have the incentive to be over-protective and to remove or disable access to content regardless of their illegality and, sometimes, even without carrying out a balancing of interest.
This may in turn result in users' right to freedom of expression being impeded as 'any potential controversial information would then likely be prevented from reaching public accessibility' (Sartor et al., 2010, pp. 376-377).
During a public consultation on the e-Commerce Directive, the majority of stakeholders (including internet intermediaries) were of the opinion that over-removal of content is partly due to legal uncertainties surrounding the scope and terms of the liability exemption (European Privatised enforcement and the right to freedom of expression in a world confronted with terrorism propaganda online However, both courts were in these cases confronted with liability issues and the interpretation of Article 14 of the e-Commerce Directive. Taking into account aforementioned risks, it is necessary to find out how these could at most be outweighed.

DIFFERENT WAYS TO BALANCE THE DANGERS OF PRIVATIZED ENFORCEMENT ON THE RIGHT TO FREEDOM OF EXPRESSION
One way to counterbalance the issue of overly broad terms of service through which the rule of law may be threatened would be to provide legal safeguards to end users. In this regard, legal scholars (Angelopoulos et al., 2015) stressed, in a study concerned with privatised enforcement and human rights limitations, the importance of IT companies to be transparent and accountable and to take into account due process principles (p. 57). This idea was also supported Whilst the Code states that it promotes transparency, it only does so by encouraging publication of transparency reports. In the two latest periodical reviews, no attention was paid to the existence of transparency measures towards end users whose post had been notified and/or removed (European Commission, 2017;. The main focus was whether the companies had provided feedback to notifying users. Whilst the Commission did stress, in its communication, the importance of transparency reports, it also stressed the importance of being transparent towards users whose post had been notified and that information shall be provided about Turkey, para. 137). The Council of Europe (2014) has already suggested that 'states have an obligation to ensure that general terms and conditions of private companies that are not in accordance with international human rights standards must be held null and void' (p. 114).
Legal scholars also supported this idea and stressed that 'States may be found to be in breach of their positive obligations for their failure to prevent violations of individuals' fundamental rights as a result of privatized law enforcement by online intermediaries' (Angelopoulos et al., 2015, p. 79). However, as these scholars mentioned, different criteria must be taken into account when establishing whether a breach of a state's positive obligation occurred (p. 79). Whilst analysis of such a breach goes beyond the scope of this paper (since it would require a case-by-case analysis), relying on state's positive obligations could help to foster the rule of law in the online environment. However, as was concluded in their study, discussions should find place in order to 'operationalize relevant positive obligations of States in the context of self-regulatory or in which their content is accessible, the safest way for them to act would be to take a restrictive approach and treat the harshest laws as threshold for content removal. In other words, by 'lowering the standards of free speech on the internet to the lowest common regulatory denominator' (Mills, 2015, p. 19) In 2012, the Commission announced an initiative on 'Notice-and-Action' procedures aimed at harmonising, at EU level, the rules on these procedures (COM ( Another possible way to achieve a higher level of legal certainty would, yet again, be through Privatised enforcement and the right to freedom of expression in a world confronted with terrorism propaganda online positive state obligations. Importantly, the ECtHR established in Dink v. Turkey (para. 137) that one of these obligations consists in ensuring that individuals can express themselves without fear. In light of this, legal scholars have held that such a positive obligation could include the duty to reduce internet intermediaries' fear of being held liable, which would be a 'promotional obligation' (Angelopoulos et al., 2015, pp. 32;42). On the one hand, the EU has issued non-binding instruments such as the communication and the recommendation on tackling illegal content online and, on the other, it is in the process of adopting legislation aimed at tackling terrorist content online.

DEVELOPMENTS SINCE THE ADOPTION OF THE EU CODE OF CONDUCT ON COUNTERING ILLEGAL HATE SPEECH ONLINE
Regarding the communication and subsequently adopted recommendation, several actors have criticised these. Concerning the former, the European Federation of Journalists (2017) pointed out to a lack of guidance to platforms for respect of the right to freedom of expression.
According to Jens-Henrik Jeppesen, representative and director for European Affairs, the communication 'describes a regime of privatised law enforcement that does not attempt to draw a bright line between content that violates platforms' terms of service (TOS) and content that breaks the law' (Jeppesen, 2017). Furthermore, Marietje Schaake, member of the European Parliament, warned that 'the good parts on enhancing transparency and accountability for the removal of illegal content are completely overshadowed by the parts that encourage automated measures by online platforms'  A new Audiovisual Media Services Directive is also being adopted which would make private companies accountable for having hate speech videos or videos inciting to terrorism present on their services (COM(2016), 287 final). This general trend to hold intermediaries accountable for illegal content has emerged in different fields of law, such as in intellectual property law with the proposed Copyright Directive (Com (2016), 593 final, Article 13). This regulatory tendency can also be seen at national level. For example, Germany adopted the so-called 'network enforcement law', which threatens social media companies with fines of up to 50 million in case of non-removal of illegal content within a certain time. 2 In a legal review of this (draft) law, commissioned by the Office of the OSCE Representative on Freedom of the Media, Bernd Holznagel (2017) warned that: 'with the risk of high fines in mind, the networks will probably be more inclined to delete a post than to expose themselves to the risk of a penalty payment' (p. 23). He also noted that such regulations may encourage platforms to circumvent the laws' territorial scope through removal of German-language comments (24).

CONCLUSION
The present paper has aimed to demonstrate how the Code has clear implications on internet users' right to freedom of expression. The privatised enforcement system encouraged under it could result in private censorship as well as undermining of the rule of law.
By taking into account different developments since the adoption of the Code, this paper claims that, from an EU-perspective, a shift from the focus on 'speed' to 'legality' should take place.
Whereas the Code adopted a 24-hour framework for removal of illegal content, the recommendation on tackling illegal content online and the recently proposed regulation Despite the shortcomings in the recently adopted EU instruments, these illustrate that some attention is being paid at the EU level for the protection of human rights in the digital environment. However, having regard to the recent proposal for a regulation on the prevention of terrorist content online, more attention is still needed in order to reconcile the practice of privatised enforcement with respect for individuals' fundamental human rights.