Publishing research obtained by violating Google&#39;s TC I believe Google's Terms and Conditions prohibit bot scraping: that is, submitting lots of (automated) search queries in a short time. This is enforced by mechanisms such as CAPTCHAs and IP blacklisting. But suppose a researcher, say based in UK, managed to work around them, get a big amount of data from Google searches and use it to publish some research. Could they get, theoretically, into trouble? Would they, practically? Would the above depend on how they circumvented Google's checks (i.e., by using lots of proxies)?
First of all, Google's TOS says we reasonably believe that your conduct causes harm or liability to a user, third party, or Google — for example, by hacking, phishing, harassing, spamming, misleading others, or scraping content that doesn’t belong to you And you ask: But suppose a researcher, say based in UK, managed to work around them, get a big amount of data from Google searches and use it to publish some research. You're confusing methods with results. Someone uses methods to get a result, but if the results are not legal, the methods are usually illegal, too. Would the above depend on how they circumvented Google's checks (i.e., by using lots of proxies)? The words &quot;or scraping content that doesn’t belong to you&quot; means just that; it does not give any wiggle room for the actual method used to scrape. Could they get, theoretically, into trouble? Very much so. The researcher would have at least civil liability, and possibly criminal exposure. Would they, practically? Google can be very not kind to people who break their TOS. And Google has lots of money to spend on lawyers and court fees to enforce their TOS. Google would probably be able to easily prove the data came from their servers, and would probably have server logs to help prove it. And see user6726's answer for specific legal citations in the US and UK.
If the operators of a site post them in such a way that anyone with a browser can access them, with no login or other security precaution required, and not even any notice that the files are confidential, they are implicitly inviting anyone to view them, and thus it is legal to do so. There was a case where a site owner gave specific notice to a user not to access the site, and blocked the user's IP. When the user hired a proxy service to get around the blockage, this was held to be unauthorized access, a crime under the CFAA (Computer Fraud and Abuse act). But in that case the individual notice was considered essential to the applicability of the act. That case was Craigslist, Inc v. 3Taps, Inc et al, 942 F.Supp.2d 962 (N.D. Cal. 2015). See also This Wikipedia article and thisJaxEnter article. As the Wikipedia article put it: Craigslist Inc. v. 3Taps Inc., 942 F.Supp.2d 962 (N.D. Cal. 2013) was a Northern District of California Court case in which the court held that sending a cease-and-desist letter and enacting an IP address block is sufficient notice of online trespassing, which a plaintiff can use to claim a violation of the Computer Fraud and Abuse Act. Note that in this case web scraping was apparently impacting the craigslist site.
This is relatively uncharted legal territory, so until multiple cases establish some sort of precedent, we can only guess. I know of no legal requirement that a Browser or User has to submit cookies or referrer data or other meta-information accurately. In that regard, a user is unlikely to be prosecuted just for submitting HTTP headers. It is likely closely related to Free Speech issues. The DMCA spells out that it is illegal to circumvent copyright protection measures. While this law is typically used to make it illegal to copy DVDs, video-games or streaming movies, it is possible that the "3-free articles" policy could be interpreted as a copyright protection mechanism, and defeating it by changing HTTP headers is a circumvention. A good summary is here. A specific site's TOS (Terms of Service) probably contains language that spells out it is a violation to use the site in a manner other than as it is intended. This is a typical anti-hacking, anti-screen-scraping provision. Altering a browser session to circumvent their services is probably a violation of the license to access the site, and may open a user to a civil lawsuit for damages or even criminal hacking charges (the details of which are different state-to-state)
A web site that is serious on protecting some content behind a paywall will put the protected content, or a version of the page with both protected and unprotected content, on separate page or pages, so arranged that a user will not be able to follow the link until that user has signed in and been accepted as an authorized user. A site that merely uses CSS to hide "protected" content is not really protecting it. CSS is designed to be modified by the ultimate user -- that is part of its function. If the site chooses to send you content, you are entitled to read it. Even if some of the content has a CSS tag attached which suppresses or obscures the display of that content, they know perfectly well that any user can supersede this with local CSS, and so I don't see how they have any legal claim, nor any way of knowing if you have accessed the "hidden" content or not. If you attempt to bypass or hack a login screen, that might be circumvention under the US DMCA, or "Unauthorized computer access" under any of several laws.
First I should point out that the Google question is about a different situation, the "snippet" issue where a tiny part of a web page is redistributed, where the issue of resolved in the US by appeal to the "fair use" defense. The proposed scenario as written here is broader since it would go beyond a couple of lines, and goes up to the limit of copy an entire web page. That is copyright infringement, with or without an associated link. Copyright protection is not just about attribution, it is about control. If you can limit your copying appropriately, you may survive under a fair use analysis; but you need to hire a lawyer with experience in copyright litigation to vet your notions of what is "a small amount" etc.
You are clearly seeking legal advice. Answers on this site come from anonymous people on the internet and are not legal advice. You should not act based on information from this site. I am unaware of any lawsuit where one would be sued for merely storing and reading HTML for personal use. Downloading a webpage is probably not a copyright violation. Most things you create, including HTML source code, are protected by copyright and copyright includes the exclusive right to choose who can read what you created. I couldn't find any actual reference to this but I would hazard a guess that displaying an HTML webpage online is implicitly allowing others to read that code. I believe this guess is correct because all modern web browsers have the capability to view source that nobody considers illegal and browsers also include the capability to save webpages to disk. These browsers are made by companies with large legal departments, I doubt Internet Explorer would include this function if using it was a copyright violation. Here begins speculation: However, your expanded question says that not only you wish to read the HTML code but you also wish to process it, extract information from it and use what you learn this way. This could, I think, be prevented by the copyright holder. Still, what you are describing is commonly done in the world. Services such as Google, Bing or the Wayback Machine go far beyond what you are doing. In theory, I can see this as being a copyright violation but again, the fact that these big companies - without any kind of contract with the website owners - keep doing it is big evidence in favor of legality of storing webpages. You should be careful about how you use the stored data, though. For example, computer programs often have a stipulation in EULA that prevents you from reverse engineering the code. I could see that the use of some websites could be protected in such manner. Further (not authoritative) internet pages on this topic: https://stackoverflow.com/q/22819287/1580088 https://answers.yahoo.com/question/index?qid=20120621055815AAvJPvN
Is that extortion? false advertising? or in any way illegal? Not at all. The owner of the site is simply exercising his right as outlined in the terms and conditions from when the user signed up. And giving users an option for continued use of the site (that is, for him not to exercise a right of which they were always aware) does not constitute extortion.
Providing they meet the basic requirements (see What is a contract and what is required for them to be valid?) then they are binding contracts. Consideration is not an issue: the site provides the content, the user provides eyeballs on it. Consent is the major stumbling block. Online Terms of Service are either presented as browsewrap or clickwrap or sign-up wrap. A browsewrap provides notice of the terms of service but there is no specific user assent to them. A clickwrap requires the user to check a box specifically about agreeing to the terms (with or without user registration). A sign-up wrap presents user registration with a "Sign up" and provides notice to the terms at the point of service but doesn't have a check button specifically for the terms. As an example, Stack Exchange provides two types of wrap. For the casual user, there is a Legal link at the bottom of the page - a browsewrap. If you sign-up, you go to a page that says "By clicking "Sign up", you agree to our terms of service, ..." - a sign-up wrap. Whether a Terms of Service is an enforcable contract depends on whether the user provided notice, whether the user gave consent and whether enforcing the agreement is conscionable. Clickwraps and sign-up wraps have the advantage over browsewraps in the first two of these. Assuming that the contract terms are unremarkable (i.e. they are within the range of "normal" for that type of contract) a clickwrap will normally create an enforcable contract - Forrest v Verizon and Motise v America Online being the relevant case law. Browsewraps are more problematic: Specht v Netscape said no contract but where the browsewrap is shown prominently and repeatedly they can form an enforceable contract - Hubbert v Dell and Cairo v CrossMedia Services. Zaltz v Jdate was a sign-up wrap and did create an enforceable contract. All of these turn on the facts of how the information was was presented to the user. For example, in Meyer v Kalanick the Second Circuit said: Where there is no evidence that the offeree had actual notice of the terms of the agreement, the offeree will still be bound by the agreement if a reasonably prudent user would be on inquiry notice of the terms.[sic] So, in general, Terms of Service are enforcable as a contract if a reasonably prudent user could, on inquiry (e.g. by clicking a link), make themselves aware of the terms. This also explains why skrinkwraps (when software came on actual physical media in a shrink-wrapped box with the terms inside) were not enforcable - a reasonably prudent user could not inform themselves of the terms without unwrapping a product they didn't yet own.