Discrepancy between robots.txt and TOS This question made me think of a situation: Alice asks Bob to crawl web site realestate.example.com and return results of the regular expression &quot;Price:([0-9]*).*Size:([0-9]*)&quot; Bob points a major open source web crawler that implements googles robots.txt reading implementation (and it therefore fully consistent with industry standard practices) at realestate.example.com, greps the result, sends data to Alice. He never visits the site manually, he spent about 2 minutes of human time on the whole job and had no need to. realestate.example.com has a robots.txt that says &quot;Take everything&quot; and a sitemap.xml describing the pages of interest because they care about SEO. It has a human readable TOS that says &quot;Scraping is not allowed&quot;. Has Bob done anything wrong?
Probably not There have not been many cases in this area of the law, and those have mostly dealt with &quot;deep linking&quot;, particularly cases where a person knowingly linked to a page bypassing a login or introductory page, when the site was so designed that ordinarily a visitor could only get to other pages by going through such a log-in or intro page. In cases where this deprives the site owner of income, or harms the site's reputation by bypassing disclaimers, this has been held to be actionable. See Nolo's page on Linking, Framing, and Inlining And the Wikipedia article on Deep linking In Intellectual Reserve, Inc. v. Utah Lighthouse Ministry, Inc, 75 F. Supp. 2d 1290 (D. Utah 1999) deep linking was held to be contributory copyright infringement. See The Wikipedia article on the case In that case, the content being linked to had been posted without the authorization of the copyright holder, and no fair use issue was raised by the defense. In general, courts have found that publishing a page on the web invites others to visit it and link to it. In the Wikipedia article on &quot;Deep linking&quot; (linked above) it is said that: In a February 2006 ruling, the Danish Maritime and Commercial Court (Copenhagen) found systematic crawling, indexing and deep linking by portal site ofir.dk of real estate site Home.dk not to conflict with Danish law or the database directive of the European Union. The Court stated that search engines are desirable for the functioning of the Internet, and that, when publishing information on the Internet, one must assume—and accept—that search engines deep-link to individual pages of one's website. In Perfect 10, Inc. v. Amazon.com, Inc. 508 F.3d 1146 (9th Cir. 2007) a US court held that links to copyrighted images as part of an image search were not copyright infringement. The Ninth Circuit court of Appeals held that Google's display and caching of thumbnails was fair use, mainly because they were &quot;highly transformative.&quot; In Craigslist vs 3Taps (see Jaxenter article) Cragslist objected to repeated scraping by PadMapper, sent a cease and desist order to PadMapper , and blocked its IP addresses. PadMapper used the services of 3Taps to bypass this block with a proxy. Craigslist sued and won. The court held that under the US Computer Fraud and Abuse Act (CFAA), the Cease and Desist order and the IP block were sufficient notice of denial of access, and that further access was unauthorized and a violation of the act. The individual notice was considered essential to this holding. The case of Ryanair vs PR Aviation was brought in the European Court of Justice. There, Ryanair had argued that continued scraping was a violation of its TOS and a copyright infringement. The court held that the owners of publicly available databases were entitled to impose access restrictions. It further held that the applicability of a TOS was a matter for national courts to determine. See also this article on &quot;Essential Legal Issues Associated With Web Scraping&quot;. There it is emphasized that much scraping is legal, except when copyright is infringed, or when specific access restrictions under the US CFAA (or similar laws) were violated. Individual facts, such as home prices and sizes, are not subject to copyright protection, although the selection and organization of such facts may be, and a database consisting of such facts may be protected. Pages posted on the web are being made publicly accessible unless specific steps are taken to make them private, such as password protection, requiring a login, or individual notice not to access. The ROBOTS.TXT file, while not technically enforced, is a widely accepted standard, and a visitor is probably entitled to assume that access in accord with the local robots file is authorized, in the absence of specific notice from the site owner to the contrary. Repeated access that negatively impacts the bandwidth or performance of the site might be a different matter.
You are not supposed to "explain" anything. See this site: http://www.dmlp.org/legal-guide/responding-dmca-takedown-notice-targeting-your-content What you have to do is to state, under penalty of perjury, that you have a good faith belief that your material was wrongly removed. That's it. Nothing more, nothing less. Google (or whoever the host is) doesn't care and shouldn't care about the actual copyright situation, only whether you supplied a counter notice where you state that your material was wrongly removed. Now Google should reinstate your app (however, since nobody can force them to host your app at all, I suspect they can remove it at any time for any reason), and whoever put in the DMCA claim can then go and sue you for copyright infringement. If they do, you can use as a defence that they could have and should have a DMCA notice to you. The whole DMCA is about your host, here: Google, to remove itself from any copyright infringement case. By following the rules for a proper notice and proper counter notice, they achieve that. And then the matter is between the complainant and you.
The software being free and open source has no impact on whether it infringes any patents or violates any copyrights. Copyrights attach to fixed representations of creative work in a tangible medium (e.g., the actual code and graphical elements of the software in question). As long as you aren't copying the copyrighted work of someone else, you should be in the clear. So, if you write your own code from scratch, or rely on code that you're allowed to use (e.g., "free" software with a permissive license that allows it to be used freely), you should be fine. On the other hand, if you copy a chunk of code that you aren't allowed to use, and then change the variable names so that it's superficially different, you're likely violating someone's copyright. Patents are a much more difficult question. To determine whether you would infringe any patents, you would have to read the independent claims of every patent that might be related. If you perform all the steps of any one of those claims, then you are infringing that claim (and therefore, the patent in which it is found). Unfortunately, this is much easier said than done. First, it may be difficult to search for all the potentially relevant patents, and once you've found them, there may be far too many to read. Second, claims are written in a type of language that is specific to patents, and someone without experience in patent law may not understand them correctly. Finally, the terms in the claims may not take on their plain English meaning, but rather may have been defined by the language in the rest of that patent, so it's possible that you might incorrectly think you were in the clear based on a misunderstanding arising from that. All that said, it may be best to go ahead with implementing an idea and then waiting to see what happens. Chances are that the implementation will arguably infringe some patent in some way, no matter what's done. But chances are also high that there will never be any worrisome enforcement action taken against it by a patent owner, simply due to the difficulty and expense associated with enforcing patent rights.
"Doing the same thing" is very common. We even have names for certain categories of websites, such as "web shop" and "blog". Such concepts are not protected in general. Obviously, you can't copy the name of existing webshop, or their logo, but things like a "rectangular layout of products for sale" aren't original. There is of course a grey area here, because there's a continuum of similarity. In general, similarities that follow from technical justifications are acceptable (having an upload button is sort of the point for your website), similarities that are cosmetic only (same color choice) might be interpreted as intentional attempts to cause confusion.
is contract text itself subject to copyright? What are my options? It largely depends on the originality of your contract. C &amp; J Management Corp. v. Anderson, 707 F.Supp.2d 858, 862 (2009) points to multiple references against preclusion of "a copyrightable interest in a contract". But you would need to prove that your competitor copied "original elements" of your contract including "a minimum degree of creativity and originality required to support a valid copyright". See Donald v. Uarco Business Forms, 478 F.2d 764, 766 (1973). Your post provides no information that would help identifying or ruling out this issue in your matter. Without realizing, you might have paid dearly for boilerplate language that your lawyer copied from somewhere else. Indeed, there is so much regurgitation and copy/pasting in the legal "profession" (judges included, as is notorious in judicial opinions they release and in the similarities --verbatim-- among the procedure law of many, many U.S. jurisdictions). That regurgitation is not bad in and of itself, though, since what matters is the expeditious administration of justice and the protection of your rights, rather than obtaining creative expressions authored by some lawyer. You might end up wasting valuable energy and money if you went after the competitor for something like this without first assessing the extent of originality in your contract. Focus instead on the much more detrimental fact that your competitor "plays dirty in general".
Yes. Contracts do not excuse a site from obeying the law, and the law requires sites to obey court orders. "It violates my contract with a third party" is not a valid reason to disobey a court, and disobeying a court order is contempt. On the other hand, contract law (at least in the U.S. and U.K.) says that contracts that violate the law or public policy are invalid, and so you could not win a breach of contract suit if the defendant could not legally comply with the contract. Now, a contract might require the company to attempt all possible legal ways to get the court to revoke its order, or to avoid the order in the first place (for instance, if Google does no business in China, a U.S. court might not excuse them from a contract because they capitulated to the Chinese government). However, when push comes to shove, private contracts are lower priority than court orders.
This would probably fall under the "legitimate interest" basis. However the service provider would need to show that the processing they did was aimed at this objective and in accordance with the data protection principles, particularly of data minimisation and storage. They should consider what data they want to extract from the detailed logs, whether the logs can be anonymised, how long they need to keep them, and whether they can achieve their objectives by storing aggregate data (e.g. percentage of people who clicked this link when viewing this page) rather than individual clicks and key presses. They should also consider whether the logs might reveal special category data (e.g. health, sexuality, religion, politics). This can happen if, for instance, the site contains information about a particular disease; people who read it are likely to have that disease or know someone who does.
The user can always claim that. Whether such a claim will be believed is another matter. A party to a contract with a &quot;wet-ink&quot; signature can claim that it was forged or that the signer was impersonated. But such claims are hard to prove, particularly if they are not true. The answer to the question in the title is that terms are binding if the user freely agrees to them, and if they meet the conditions for a legally binding agreement in the relevant jurisdiction. The question in the body is more about what evidence will establish this. Suppose the code behind a web-site is so designed that no user is allowed to proceed without affirmatively accepting the ToS, and the operator can demonstrate this. That will be persuasive evidence that the user did accept the terms. If the site logs the consent in a database along with a timestamp that would be further evidence. A journaling database provides an audit trail of changes, which would be further evidence. Evidence of other users that the site always worked that way, and of coders that the code had never permitted a user to proceed without accepting the ToS might also help. Backups of the DB stored with a 3rd party including the consent logs would also provide evidence that the timestamps were present and unchanged. A hash of the log can be stored on the blockchain. The site could also send a &quot;registration accepted&quot; email to the user (if the user has provided an email address) which could include a mention that the user had accepted the ToS. Such an email would be reserved in the logs of the senders and receiver's email providers, and could be retrieved as evidence. Ultimately it will be a question of what the finder of fact believes, if the matter goes to court. And as user Amon says, the US standard is preponderance of the evidence (that is, the evidence is at least a little bit stronger on one side. This is sometimes loosely expressed as more than 50% or 51% or better proved, but courts do not in fact measure degrees of proof in numbers), not absolute proof.