Is generating automatic requests illegal? TL; DR; Is generating automated requests to a government site illegal in some way? There is a government site (EU/Ireland) that provides a WebChat service as a contact option. When the chat is active there is a green button, otherwise, it is grey. It is almost impossible to find the green button (although sometimes it is green) so I decided to test the real availability of this service. My idea was to create a script that accesses the website, checks the status of the button and then logs it. My goal was not to DDoS the server or something similar so I thought a request every minute will suffice for getting some statistics. Could this be considered illegal? I know it might depend on each country's laws but if someone in the field could provide some insight or directions is really appreciated. Even more Details: The website is always up even if the chat is not available, so it is not only testing an HTTP request. What I found is that the HTML is loaded without the &quot;button&quot; (it's not a button tag, it's actually a clickable image), then a javascript script connects to the chat service and depending on the status, it loads the &quot;button&quot; to access the chat or to notify it is unavailable. This is an &lt;img&gt; tag. What I do is read the &quot;alt&quot; property description where the status can be identified. Then I put that in a cron job, log the time and the status. Finally, at the end of the day, I summarise the data into a chart showing the percentage of time the chat was really active. The purpose of this experiment is to prove that the customer service is awful with actual data.
What you are doing is commonly referred to &quot;web scrapers&quot; and they are legal in the EU. What you cannot do is extract personal data. Since the data you are aggregating is non-personal data, whether or not a chat button is available, it should be fine. EDIT As some of the commenters said - it's legal, but many websites detect scrapping. To (try to) avoid being blocked by the server, make it act human. Something like. I check once every 15 minutes with 3 minutes +/- is probably enough. That also is probably what a human would do using the website so it should strengthen your argument the website is unavailable.
These are some thoughts about the state of affairs in the US, I do not know how it works in the UK. In the US it seems to be a legal gray area. Gray enough that I do not think any lawyer could say for certain that the use of the data is legal. The data is stolen. If possessing stolen property is illegal then possessing this data is also likely illegal. Of course experts disagree, Stuart Karle, an adjunct media professor at Columbia University and former general counsel for the Wall Street Journal says: ...the documents have been published by the hackers, they are now public by virtue of being put on the Internet. But Barrett Brown was charged with trafficking in stolen authentication when he forwarded a link to some stolen emails. He signed a plea for acting as an accessory after the fact. He spent more than a year in jail while they sorted it out. In the US there is no law banning the download of hacked documents. In fact Bartnicki v. Vopper 532 US 514 (2001) stands for the rule that journalists can report on illegally obtained information. But contrast that with the Barret Brown prosecution! And decide where a data scientist fits. Also there is the question of whether an illegally recorded conversation is of the same "illegal" nature as hacked personal information.
Can't we get a legal action about AdBlocker as web masters? No. I assume that by "web master" you mean a server admin who contracts with entities which are seeking to advertise their goods and services. An advertiser pays the server to transmit (or send(), in terms of a socket API) certain content to whoever submits a GET or POST request ("the client"). The scope of the contract ends there, and the end consumer or client is not a party to that contract. Any point beyond the server admin's routers, it is perfectly lawful for end users to run some program on the client side where the purpose of that program is to (1) parse any packets received in the client socket, (2) filter out some of it, (3) and forward the rest to a browser. Bringing legal action against AdBlocker would be somewhat akin to suing producers of TV remote controls under the allegation that the mute button functionality blocks advertisement content.
The so called "cookie law" is Article 5(3) of Directive 2002/58/EC: Member States shall ensure that the use of electronic communications networks to store information or to gain access to information stored in the terminal equipment of a subscriber or user is only allowed on condition that the subscriber or user concerned is provided with clear and comprehensive information in accordance with Directive 95/46/EC, inter alia about the purposes of the processing, and is offered the right to refuse such processing by the data controller. This shall not prevent any technical storage or access for the sole purpose of carrying out or facilitating the transmission of a communication over an electronic communications network, or as strictly necessary in order to provide an information society service explicitly requested by the subscriber or user. So this does not only apply to cookies, but to any way to store or retrieve information on/from the equipment of the user. When you use a PHP session, the session ID is stored in a session cookie or URL parameter. If a URL parameter is used, it is also stored on the equipment of the user, so the "cookie law" would also apply to that, but I am not sure about that. The Article 29 Data Protection Working Party (Art. 29 WP) has written Opinion 04/2012 on Cookie Consent Exemption. It considers a session-ID to be exempted from the requirement of informed consent: 3.1 “User-input” cookies The term “user input cookies” can be used as a generic term to describe session cookies that are used to keep track of the user’s input in a series of message exchanges with a service provider in a consistent manner. These would be expected to be first party cookies typically relying on a Session-ID (a random temporary unique number) and expire when the session ends at the latest. First party user input session cookies are typically used to keep track of the user’s input when filling online forms over several pages, or as a shopping cart, to keep track of the items the user has selected by clicking on a button (e.g. “add to my shopping cart”). These cookies are clearly needed to provide an information service explicitly requested by the user. Additionally, they are tied to a user’s action (such as clicking on a button or filling a form). As such these cookies are exempted under CRITERION B. So basically, a PHP session ID cookie does not require consent. However as soon as a session id is used for analytics, it is not exempted: 4.3 First party analytics Analytics are statistical audience measuring tools for websites, which often rely on cookies. These tools are notably used by website owners to estimate the number of unique visitors, to detect the most preeminent search engine keywords that lead to a webpage or to track down website navigation issues. Analytics tools available today use a number of different data collection and analysis models each of which present different data protection risks. A firstparty analytic system based on “first party” cookies clearly presents different risks compared to a third-party analytics system based on “third party” cookies. There are also tools which use “first party” cookies with the analysis performed by another party. This other party will be considered as a joint controller or as a processor depending on whether it uses the data for its own purposes or if it is prohibited to do so through technical or contractual arrangements. While they are often considered as a “strictly necessary” tool for website operators, they are not strictly necessary to provide a functionality explicitly requested by the user (or subscriber). In fact, the user can access all the functionalities provided by the website when such cookies are disabled. As a consequence, these cookies do not fall under the exemption defined in CRITERION A or B. However, the Working Party considers that first party analytics cookies are not likely to create a privacy risk when they are strictly limited to first party aggregated statistical purposes and when they are used by websites that already provide clear information about these cookies in their privacy policy as well as adequate privacy safeguards. Such safeguards are expected to include a user friendly mechanism to opt-out from any data collection and comprehensive anonymization mechanisms that are applied to other collected identifiable information such as IP addresses. In this regard, should article 5.3 of the Directive 2002/58/EC be re-visited in the future, the European legislator might appropriately add a third exemption criterion to consent for cookies that are strictly limited to first party anonymized and aggregated statistical purposes. First party analytics should be clearly distinguished from third party analytics, which use a common third party cookie to collect navigation information related to users across distinct websites, and which pose a substantially greater risk to privacy. So Art. 29 WP does not consider it a privacy risk under the conditions in the last two paragraphs. At least in some EU countries (Like the Netherlands) you would be exempted from the requirement of informed consent because of that. In your question you also mention you use "a soft opt-in on continual use". That is not sufficient to obtain consent.
In general this is protected by the first amendment. It is not in general a problem describing how one can one can do something illegal. But there are special cases to be careful with. You might want to do some research into the limits on free speech. It would be hard to provide an answer that fully covers all your different cases and you would need to be more specific about what illegal activity you want to describe. In describing how to do something illegal, you might accidentally share information that you are not allowed to share. When you post things online, this can be considered as publishing or exporting. Therefore certain export restrictions might apply. Also, It is illegal to publish bomb making manuals, with the knowledge or intent that this information be used to commit a federal crime of violence. See https://www.law.cornell.edu/uscode/text/18/842. There are restrictions on publishing material relating to cryptography without having an export license. Granted, this isn't necessarily related to publishing things that are illegal, but just to give an idea about how publishing/exporting knowledge can causes problems. See https://en.wikipedia.org/wiki/Export_of_cryptography_from_the_United_States ITAR (International Traffic in Arms Regulations) sets restrictions on what you can publish about arms. What you publish can't be “directed to inciting or producing imminent lawless action.” See for example https://en.wikipedia.org/wiki/Brandenburg_v._Ohio. One might imagine that you could get into trouble if someone interprets what you do as inciting or producing a lawless action. It might sound obvious, but you want to make sure that you have the right to share the information that you have. The information that you are providing might be copyrighted in some way.
You must get opt-in affirmative consent to process personal data, including tracking people's use of your site or providing targeted advertising. The banner on StackExchange is likely in violation of the GDPR. Do not copy it. It does not have an explicit opt-in, only an opt out which is onerous (leave the site, then manually go in and delete any cookies they set, which may be hard to identify if they are from 3rd parties). The sites you mention that have a gateway are a more correct implementation. Consent must be acquired before processing of data begins, and it must be explicit.
can it be construed as a violation of 66F clause 2 if we try to circumvent the rate limit even though the information is on public domain? No, unless the element of intent to threaten [...] or strike terror can be proved. I'm assuming that by &quot;clause 2&quot; you mean the excerpt you posted. Although circumventing the limit of API calls sounds in &quot;exceeding authorised access&quot; and might even lead to DOS, it does not imply an intent to threaten the security/sovereignty/etc. or strike terror. The statute is premised on such intent. The fact that the information(packets) is(are) on public domain is irrelevant. For instance, an intent to strike terror and/or actually causing a DOS can involve [targeting] a public domain.
The "Contribution activity" is extracted from repositories hosted on Github. Looking at the privacy statement, Github considers itself as a hosting service for those repositories. See EU Directive 2000/31/EC Article 14 for the exact definition and conditions: Article 14 Hosting Where an information society service is provided that consists of the storage of information provided by a recipient of the service, Member States shall ensure that the service provider is not liable for the information stored at the request of a recipient of the service, on condition that: (a) the provider does not have actual knowledge of illegal activity or information and, as regards claims for damages, is not aware of facts or circumstances from which the illegal activity or information is apparent; or (b) the provider, upon obtaining such knowledge or awareness, acts expeditiously to remove or to disable access to the information. Paragraph 1 shall not apply when the recipient of the service is acting under the authority or the control of the provider. This Article shall not affect the possibility for a court or administrative authority, in accordance with Member States' legal systems, of requiring the service provider to terminate or prevent an infringement, nor does it affect the possibility for Member States of establishing procedures governing the removal or disabling of access to information. Article 2(4) GDPR defines that the GDPR does not apply (to Github) in this case This Regulation shall be without prejudice to the application of Directive 2000/31/EC, in particular of the liability rules of intermediary service providers in Articles 12 to 15 of that Directive. Extracting data from the repositories to create a nicely formatted contribution activity list, would still be considered hosting as defined at Directive 2000/31/EC, because Github does not (manually) redact or modify commits. So if you want to hide your contribution activity, you must delete the contributions. And you should not ask Github to do that, but ask the owners of the repositories. And they might have good reasons to deny your request. If an owner of a repository denies your request without a good reason, you can ask Github to do so. But Github would then probably deny that request, because they explain in their privacy statement that modifying the history is not possible: The email address you have supplied via your Git commit settings will always be associated with your commits in the Git system. If you chose to make your email address private, you should also update your Git commit settings. We are unable to change or delete data in the Git commit history — the Git software is designed to maintain a record — but we do enable you to control what information you put in that record. edit To further clarify why Article 14 of Directive 2000/31/EC applies, see case C-236/08 (Google v. Louis Vuitton) where the European Court of Justice clarifies the meaning of that article (ECLI:EU:C:2010:159): In that regard, it follows from recital 42 in the preamble to Directive 2000/31 that the exemptions from liability established in that directive cover only cases in which the activity of the information society service provider is ‘of a mere technical, automatic and passive nature’, which implies that that service provider ‘has neither knowledge of nor control over the information which is transmitted or stored’. Accordingly, in order to establish whether the liability of a referencing service provider may be limited under Article 14 of Directive 2000/31, it is necessary to examine whether the role played by that service provider is neutral, in the sense that its conduct is merely technical, automatic and passive, pointing to a lack of knowledge or control of the data which it stores. The way Github creates/shows the contribution activity is in my opinion technical, automatic and passive. That means it is not liable, and that implies that the GDPR does not directly apply, unless data is unlawful, and someone notifies Github of that: The restriction on liability set out in Article 14(1) of Directive 2000/31 applies to cases ‘[w]here an information society service is provided that consists of the storage of information provided by a recipient of the service’ and means that the provider of such a service cannot be held liable for the data which it has stored at the request of a recipient of that service unless that service provider, after having become aware, because of information supplied by an injured party or otherwise, of the unlawful nature of those data or of activities of that recipient, fails to act expeditiously to remove or to disable access to those data. Surely the GDPR might be the reason data is unlawful. But you need a reason, in particular based on Article 17 ("right to be forgotten"). For example if personal data of a 12 year old child is processed without permission of it's parent. Without such a reason the legitimate interest of the repository owner will probably prevail (who wants to keep the git history complete).